{ "cells": [ { "cell_type": "markdown", "id": "42ce64ba", "metadata": { "editable": true }, "source": [ "\n", "" ] }, { "cell_type": "markdown", "id": "c987b86b", "metadata": { "editable": true }, "source": [ "# Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", "\n", "Date: **Oct 26, 2023**\n", "\n", "Copyright 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", "id": "0dc0d6ca", "metadata": { "editable": true }, "source": [ "## Plans for week 43\n", "\n", "**Material for the active learning sessions on Tuesday and Wednesday.**\n", "\n", " * Exercise on writing your own neural network code, application to the OR and XOR gates\n", "\n", " * The exercises this week will be continued next week as well\n", "\n", " * Discussion of project 2\n", "\n", " * [Video of lab session](https://youtu.be/Ia6wwDLxqtM)\n", "\n", " \n", "\n", "**Material for the lecture on Thursday October 26, 2023.**\n", "\n", " * Building our own Feed-forward Neural Network and discussion of project 2, continuation from last week\n", "\n", " * Solving differential equations with Neural Networks and intro to **Tensorflow** with examples.\n", "\n", " * [Video of lecture](https://youtu.be/_-AwbBh4G-8)\n", "\n", " * Readings and Videos:\n", "\n", " * These lecture notes\n", "\n", " * [Aurelien Geron's chapters 10-11](https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf)\n", "\n", " * For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7. \n", "\n", " * [Neural Networks demystified](https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs)\n", "\n", " * [Building Neural Networks from scratch](https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex)\n", "\n", " * [Video on Neural Networks](https://www.youtube.com/watch?v=CqOfi41LfDw)\n", "\n", " * [Video on the back propagation algorithm](https://www.youtube.com/watch?v=Ilg3gGewQ5U)\n", "\n", "I also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at ." ] }, { "cell_type": "markdown", "id": "f4d3253b", "metadata": { "editable": true }, "source": [ "## Using Automatic differentiation\n", "a\n", "In our discussions of ordinary differential equations \n", "we will also study the usage of [Autograd](https://www.youtube.com/watch?v=fRf4l5qaX1M&ab_channel=AlexSmola) in computing gradients for deep learning. For the documentation of Autograd and examples see the lectures slides from [week 39](https://compphysics.github.io/MachineLearning/doc/pub/week39/html/week39.html) and the [Autograd documentation](https://github.com/HIPS/autograd).\n", "t" ] }, { "cell_type": "markdown", "id": "9744596b", "metadata": { "editable": true }, "source": [ "## Back propagation and automatic differentiation\n", "\n", "For more details on the back propagation algorithm and automatic differentiation see\n", "1. \n", "\n", "2. \n", "\n", "3. Slides 12-44 at URL\":http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture4.pdf\"" ] }, { "cell_type": "markdown", "id": "bd189f93", "metadata": { "editable": true }, "source": [ "## Material for exercises week 43 and week 44" ] }, { "cell_type": "markdown", "id": "e901e3d4", "metadata": { "editable": true }, "source": [ "## Writing our first neural network code, testing it for the OR and XOR gates\n", "\n", "During week 41 we discussed three different types of gates, the\n", "so-called XOR, the OR and the AND gates. In order to develop a code\n", "for neural networks, it can be useful to set up a simpler system with\n", "only two inputs and one output. This can make it easier to debug and\n", "study the feed forward pass and the back propagation part. In the\n", "exercise this and next week, we propose to study this system with just\n", "one hidden layer and two hidden nodes. There is only one output node\n", "and we can choose to use either a simple regression case (fitting a\n", "line) or just a binary classification case with the corss-entropy as\n", "cost function.\n", "\n", "Their inputs and outputs can be\n", "summarized using the following tables, first for the OR gate with\n", "inputs $x_1$ and $x_2$ and outputs $y$:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
$x_1$ $x_2$ $y$
0 0 0
0 1 1
1 0 1
1 1 1
" ] }, { "cell_type": "markdown", "id": "fca381f9", "metadata": { "editable": true }, "source": [ "## The AND and XOR Gates\n", "\n", "The AND gate is defined as\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
$x_1$ $x_2$ $y$
0 0 0
0 1 0
1 0 0
1 1 1
\n", "\n", "And finally we have the XOR gate\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
$x_1$ $x_2$ $y$
0 0 0
0 1 1
1 0 1
1 1 0
" ] }, { "cell_type": "markdown", "id": "f0252121", "metadata": { "editable": true }, "source": [ "## Representing the Data Sets\n", "\n", "Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads" ] }, { "cell_type": "markdown", "id": "f59fc5ee", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n", " 0 & 1 \\\\\n", "\t\t 1 & 0 \\\\\n", "\t\t 1 & 1 \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "1d74ec2a", "metadata": { "editable": true }, "source": [ "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n", "\n", "Your tasks here are\n", "\n", "1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n", "\n", "2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n", "\n", "3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n", "\n", "4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n", "\n", "5. Set up the cost function (cross entropy for classification of binary cases).\n", "\n", "6. Calculate the gradients needed for the back propagation part.\n", "\n", "7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n", "\n", "8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n", "\n", "Everything you develop here can be used directly into the code for the project." ] }, { "cell_type": "markdown", "id": "8e6565f3", "metadata": { "editable": true }, "source": [ "## Setting up dimensionalities by hand\n", "\n", "It can be useful to test the dimensionalities for the network. Let us assume we have performed an optimization for XOR gate and found that the weights for the hidden layer are given by" ] }, { "cell_type": "markdown", "id": "7f55beb5", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n", " 1 & 1 \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "ba515e4f", "metadata": { "editable": true }, "source": [ "Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives" ] }, { "cell_type": "markdown", "id": "1007d026", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n", " 1 & 1 \\\\\n", "\t\t 1 & 1 \\\\\n", "\t\t 2 & 2 \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "b1ee686a", "metadata": { "editable": true }, "source": [ "Assume also that the bias vector for the hidden layer is" ] }, { "cell_type": "markdown", "id": "2820cfdd", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n", " -1\\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "5b5f6d64", "metadata": { "editable": true }, "source": [ "Adding it gives us the input to the activation function of the hidden layer" ] }, { "cell_type": "markdown", "id": "0814e6d9", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n", " 1 & 0 \\\\\n", "\t\t 1 & 0 \\\\\n", "\t\t 2 & 1 \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "9da10b99", "metadata": { "editable": true }, "source": [ "Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have" ] }, { "cell_type": "markdown", "id": "c0f3ede6", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n", " 1 & 0 \\\\\n", "\t\t 1 & 0 \\\\\n", "\t\t 2 & 1 \\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "f00bc3ad", "metadata": { "editable": true }, "source": [ "Assume also that the bias of the output layer is zero and that the weights of the output layer are" ] }, { "cell_type": "markdown", "id": "da749419", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n", " -2\\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "2e9135f9", "metadata": { "editable": true }, "source": [ "and multiplying with $\\boldsymbol{a}_h$ gives the output" ] }, { "cell_type": "markdown", "id": "a0d879df", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{a}_o=\\begin{bmatrix} 0 & 0 \\\\\n", " 1 & 0 \\\\\n", "\t\t 1 & 0 \\\\\n", "\t\t 2 & 1 \\end{bmatrix}\\begin{bmatrix} 1 \\\\\n", " -2\\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n", "$$" ] }, { "cell_type": "markdown", "id": "988a9a20", "metadata": { "editable": true }, "source": [ "the wanted result. Pay attention to the dimensionalities as well." ] }, { "cell_type": "markdown", "id": "354ad4af", "metadata": { "editable": true }, "source": [ "## Setting up the Neural Network\n", "\n", "We define first our design matrix and the various output vectors for the different gates." ] }, { "cell_type": "code", "execution_count": 1, "id": "1f04aac4", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[[0.61238907]\n", " [0.61939429]\n", " [0.73482109]\n", " [0.70115106]]\n" ] } ], "source": [ "%matplotlib inline\n", "\n", "\"\"\"\n", "Simple code that tests XOR, OR and AND gates with linear regression\n", "\"\"\"\n", "\n", "# import necessary packages\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn import datasets\n", "\n", "def sigmoid(x):\n", " return 1/(1 + np.exp(-x))\n", "\n", "def feed_forward(X):\n", " # weighted sum of inputs to the hidden layer\n", " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", " # activation in the hidden layer\n", " a_h = sigmoid(z_h)\n", " \n", " # weighted sum of inputs to the output layer\n", " z_o = np.matmul(a_h, output_weights) + output_bias\n", " # softmax output\n", " # axis 0 holds each input and axis 1 the probabilities of each category\n", " probabilities = sigmoid(z_o)\n", " return probabilities\n", "\n", "\n", "# ensure the same random numbers appear every time\n", "np.random.seed(0)\n", "\n", "# Design matrix\n", "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", "\n", "# The XOR gate\n", "yXOR = np.array( [ 0, 1 ,1, 0])\n", "# The OR gate\n", "yOR = np.array( [ 0, 1 ,1, 1])\n", "# The AND gate\n", "yAND = np.array( [ 0, 0 ,0, 1])\n", "\n", "# Defining the neural network\n", "n_inputs, n_features = X.shape\n", "n_hidden_neurons = 2\n", "n_categories = 1\n", "n_features = 2\n", "\n", "# we make the weights normally distributed using numpy.random.randn\n", "\n", "# weights and bias in the hidden layer\n", "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n", "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n", "\n", "# weights and bias in the output layer\n", "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n", "output_bias = np.zeros(n_categories) + 0.01\n", "\n", "probabilities = feed_forward(X)\n", "print(probabilities)" ] }, { "cell_type": "markdown", "id": "6e12253c", "metadata": { "editable": true }, "source": [ "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above." ] }, { "cell_type": "markdown", "id": "4ca80040", "metadata": { "editable": true }, "source": [ "## The Code using Scikit-Learn" ] }, { "cell_type": "code", "execution_count": 2, "id": "1b657a96", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning rate = 1e-05\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1e-05\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1e-05\n", "Lambda = 0.001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1e-05\n", "Lambda = 0.01\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1e-05\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1e-05\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1e-05\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.0001\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.0001\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.0001\n", "Lambda = 0.001\n", "Accuracy score on data set: 0.5\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Learning rate = 0.0001\n", "Lambda = 0.01\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.0001\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.0001\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.0001\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 0.001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 0.01\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.001\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.01\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.25\n", "\n", "Learning rate = 0.01\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.75\n", "\n", "Learning rate = 0.01\n", "Lambda = 0.001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.01\n", "Lambda = 0.01\n", "Accuracy score on data set: 0.75\n", "\n", "Learning rate = 0.01\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.01\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.01\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.1\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.1\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.1\n", "Lambda = 0.001\n", "Accuracy score on data set: 1.0\n", "\n", "Learning rate = 0.1\n", "Lambda = 0.01\n", "Accuracy score on data set: 1.0\n", "\n", "Learning rate = 0.1\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.1\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 0.1\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1.0\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.75\n", "\n", "Learning rate = 1.0\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.75\n", "\n", "Learning rate = 1.0\n", "Lambda = 0.001\n", "Accuracy score on data set: 0.75\n", "\n", "Learning rate = 1.0\n", "Lambda = 0.01\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1.0\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1.0\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 1.0\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 1e-05\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 0.0001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 0.001\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 0.01\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 0.1\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 1.0\n", "Accuracy score on data set: 0.5\n", "\n", "Learning rate = 10.0\n", "Lambda = 10.0\n", "Accuracy score on data set: 0.5\n", "\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n", "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", " warnings.warn(\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABrR0lEQVR4nO3dd3xUddbH8e9QJgMpQISQSOgEYpQSDEVpIqDriuIKPsqKC4tEmrCCCzYUUGwLigKCq1J0kQWlWigi7q4U6ShCCE1aME0gCSE9mecPJLvjBBhg7r2Z5PN+XvN6mN/cO/fk7BA5c8691+Z0Op0CAAAAABNUsDoAAAAAAOUHBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAICBuNcrAACuKlkdAACY6emnn9ayZcsuuU2dOnX0zTffXPOxPv30Ux0+fFhPP/30ZbddvHixnnvuOXXq1EkffPDBNR8bAIDSyubk6zkA5cjx48d1+vTp4uczZ85UXFycZsyYUbxmt9sVFRV1zce6/fbb1bZtW7322muX3bZv377KzMzUoUOH9NVXX6lu3brXfHwAAEojOiAAypV69eqpXr16xc+Dg4Nlt9vVqlUry2I6cuSIdu7cqffee09jxozRJ598oieffNKyeAAAMBLngABACQ4cOKDBgwerdevWat26tYYPH64TJ064bPOPf/xDv/vd79S8eXN16tRJEyZMUGZmpqTz3Y+TJ09q2bJlatasmRISEi56rCVLligwMFC33HKLfve732nJkiXKy8tz227Pnj0aNGiQbr75ZrVv316jRo1SYmJi8eunTp3Ss88+q1tvvVXR0dF6+OGHtWPHjuLXmzVrpunTp7u85/Tp09WsWbPi508//bT69++v8ePHKyYmRn/4wx9UUFCg06dPa+LEieratatuuukmtW3bVsOHD3f7ub788kvdf//9atmypW677TZNnjxZeXl5OnjwoJo1a6ZFixa5bJ+cnKwbbrjhsmNxAICygwIEAH7jyJEjeuihh3Tq1Cm99tprevnll3XixAn17dtXp06dknT+H9qvv/66Hn74Yc2ePVvDhw/XihUrNGnSJEnSjBkzVKtWLXXp0kWLFi1SSEhIiccqLCzUihUr9Pvf/152u13333+/Tp06pa+//tplu/j4ePXt21fZ2dl67bXX9OKLLyouLk4DBw5Ufn6+srKy9NBDD2nTpk168sknNWPGDPn7+2vQoEE6fPjwFf3827dv17FjxzR9+nQNHz5cFStW1ODBg7Vx40Y9+eSTmj17toYNG6ZNmzbphRdeKN5v4cKFGj16tG644QbNmDFDgwcP1oIFCzRhwgRFRESoZcuWWrFihcuxVqxYIYfDoTvvvPOKYgQA+C5GsADgN2bMmCGHw6F58+YpICBAknTLLbeoe/fu+uCDD/TUU09py5YtqlOnjh5++GFVqFBBbdu2VdWqVXXmzBlJUlRUlOx2u4KDgy853vXtt98qJSVFvXv3liS1atVKTZo00T//+U/9/ve/L95u5syZqlatmubMmSM/Pz9JUmhoqJ544gnt379fP/zwg06cOKHly5crMjJSkhQTE6P77rtP27ZtU+PGjT3++QsKCjRx4kTVr19f0vkuRZUqVfTUU08pJiZGktSuXTslJCRo4cKFkqSioiJNnz5dPXr00Msvv1z8Xrm5uVq2bJny8vLUu3dvvfDCCzpx4kTxOS7Lly/XXXfdpapVq3ocHwDAt9EBAYDf2Lx5s9q1ayeHw6GCggIVFBQoICBAMTEx2rRpkySpffv2Onr0qO6///7iE9nvuece9e/f/4qOtWTJEtWvX18NGzZURkaGMjIydNddd2nr1q0unYsdO3aoc+fOxcWHJLVo0ULffPONbrrpJm3fvl3h4eHFxYck+fn5adWqVXrooYeuKCaHw+Fynkzt2rX10UcfKSYmRj///LO+++47zZ8/Xzt37lR+fr6k812jX375Rd27d3d5rwEDBmjFihWy2+26++67VaVKleIuyO7du3X48GHdf//9VxQfAMC30QEBgN9IS0vTypUrtXLlSrfXgoODJUm///3vVVRUpAULFmjGjBl6++23VadOHT355JO6++67PTrO6dOn9e9//1v5+flq06aN2+uLFi3Ss88+WxzTddddd8mYL/X6lbjuuutks9lc1j777DO9+eabSkxMVPXq1RUZGSmHw+Fy/Av7XkxAQIB+97vf6bPPPtPjjz+uZcuWqX79+sVdFQBA+UABAgC/ERgYqFtvvVV//vOf3V6rVOm/vzZ79uypnj176uzZs9qwYYPef/99jRkzRjExMapdu/Zlj7NixQrl5+drxowZCgoKcnntnXfe0fLlyzV69Gg5HA4FBga6XD74gv/85z+KjIxUYGBgiSe679q1SwEBAYqIiJB0/pyT/5WVlXXZOLdv366nnnpK/fr106OPPqrQ0FBJ0t/+9rfik9wvxP/bGNPS0rR37161atVK/v7+6t27t5YtW6bdu3drzZo1euSRRy57fABA2cIIFgD8Rtu2bXXo0CHdcMMNat68uZo3b66bbrpJ8+bN09q1ayVJTzzxhB5//HFJ5wuWu+66S8OGDVNhYaFSUlIkSRUqXPpX7NKlS9WqVSv16NFD7dq1c3n07dtX6enpWrVqlaTz53OsX7/e5epY+/fv12OPPaYff/xRMTExOnHihPbv31/8el5enkaMGKFPPvlE0vkORFJSkksMO3fuvGw+du3apaKiIo0cObK4+CgsLCweRysqKlKjRo1Uo0YNrVu3zmXfzz//XLGxscrNzZUktWnTRg0aNNDkyZN15swZ3XfffZc9PgCgbKEAAYDfGDZsmI4fP67Bgwfr66+/1vr16zVixAh9+eWXxedYtG/fXmvXrtXrr7+u7777TmvWrNHbb7+tBg0aFG8TFBSkuLg4bd26VTk5OS7H2L17tw4cOHDRca1u3bqpWrVqxSd5Dxs2TGfOnFFsbKy++eYbrV69Wk888YRuvPFGde7cWffff7/q1q2roUOHasWKFVq/fr1GjhypnJyc4i7Dbbfdpi+//FILFizQd999p7Fjx+rYsWOXzUeLFi0kSS+++KI2b96sr776Sn/+858VHx8v6XwXpWLFihoxYoTWrFmjCRMmaOPGjfr444/11ltvqW/fvsWja5LUu3dvbd26VbfccovCwsKu5H8aAEAZQAECAL8RGRmpjz/+WDabTWPHjtXIkSOVmpqqd955R3fccYck6aGHHtK4ceP07bffasiQIXrhhRfUuHFjzZkzR5UrV5YkDRw4UL/88oseffRR7dmzx+UYS5YsUcWKFV2udPW/7Ha77rrrLn3//ffat2+foqKi9I9//ENFRUUaNWqUXnzxRbVq1Urvv/++7Ha7AgICNH/+fEVHR+vll1/WX/7yF+Xm5uof//hH8QnlzzzzjG6//XZNnjxZI0eOVJUqVTy64WG7du30wgsvaNeuXYqNjdWrr76q66+/vvju8RfGsB5++GG99tpr2r59uwYPHqw5c+Zo4MCBevrpp13e77bbbpMkTj4HgHLK5nQ6nVYHAQAoP95//3198MEHWr9+vex2u9XhAABMxknoAABTLFu2TAcOHNCCBQv02GOPUXwAQDlFAQIAMEV8fLwWLlyo7t27KzY21upwAAAWYQQLAAAAgGbOnKnvvvtO//jHPy66zZkzZzRp0iR9++23kqTf/e53euaZZ1S1alWPj8NJ6AAAAEA5N2/ePE2bNu2y240cOVInTpwo3n7jxo2aOHHiFR2LESwAAACgnEpOTtZzzz2nHTt2qGHDhpfcdteuXdq6datWrlypxo0bSzp/ifZBgwZp9OjRHt2EV6IDAgAAAJRbe/fuVbVq1fTZZ5+pZcuWl9x2+/btqlWrVnHxIZ2/ea/NZiu+JLsn6IAAAAAAPqxbt26XfH3dunUXfe3222/X7bff7tFxkpOT3W4ga7fbVb16dSUmJnr0HlI5K0BaD51qdQgAAAC4hJ2zRlkdwkUVJTW1OoSLqGvKUbKzs0u8hLqfn59yc3M9fp9yVYAAAAAAZc2lOhze5HA4lJeX57aem5vLVbAAAAAAeFdoaKhSUlJc1vLy8pSWlubxCegSBQgAAADgkaJS+n9madOmjZKSknTs2LHitS1btkiSWrdu7fH7UIAAAAAAcFNYWKjU1FTl5ORIklq2bKnWrVtr1KhR2r17tzZv3qzx48frvvvuowMCAAAA4NokJiaqY8eOWrlypSTJZrNpxowZCg8PV//+/fXEE0+oc+fOmjBhwhW9r83pdDoNiLdU4ipYAAAApVtpvgpWbmIjq0MokV/YT1aHcEXogAAAAAAwDQUIAAAAANNwHxAAAADAA0UqN2cuGIoOCAAAAADTUIAAAAAAMA0jWAAAAIAHzLzpX1lGBwQAAACAaShAAAAAAJiGESwAAADAA4Xl5/7dhqIDAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaRjBAgAAADxQyAiWV9ABAQAAAGAaChAAAAAApmEECwAAAPAAV8HyDjogAAAAAExDAQIAAADANIxgAQAAAB4odDKC5Q10QAAAAACYhgIEAAAAgGkYwQIAAAA8UGR1AGUEHRAAAAAApqEAAQAAAGAaRrAAAAAADxRyI0KvoAMCAAAAwDQUIAAAAABMwwgWAAAA4IFCJrC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgAe4EaF30AEBAAAAYBoKEAAAAACmYQQLAAAA8EChbFaHUCbQAQEAAABgGgoQAAAAAKZhBAsAAADwQBE3IvQKOiAAAAAATEMBAgAAAMA0jGBZ5Nao+hp2bwc1DAtW2tlsLV6/W3PXbLvo9g1q19DSCQPc1o8knVbviR8aGKnvIsfGIr/GI8fGI8fGIr/GI8fm4ipY3kEBYoEWjcI0dWgvfbXjgGZ+tlGtmtTR8Hs7qILNptmrt5a4T9O6tSRJsW9+qryCguL1nLyCErcv78ixsciv8cix8cixsciv8cgxfBUFiAUG391e+xNS9fy81ZKkTXHHVKlCBQ24s43mr9uh3PxCt32ahYfo5C/p2nEwwexwfRI5Nhb5NR45Nh45Nhb5NR45hq/iHBCTVa5UUTdHhOubXYdc1r/edVD+Druim9Qpcb9m4bW0PyHVjBB9Hjk2Fvk1Hjk2Hjk2Fvk1Hjm2RqFspfLha+iAmCy8ZjXZK1fS8ZQzLusnUtMkSfVDamjzvuNu+zUNr6UjSac0b8yDalY3RGezcvX55jjN+myTCoqKzAjdZ5BjY5Ff45Fj45FjY5Ff45Fj+DLLC5CCggJ99dVX2r59u37++Wfl5eWpSpUqCg0NVUxMjHr06KFKlSwP02sCq/hJkjJz8lzWs3597l/F7rZPcGBV1azmL6fTqWnLNijxTIbaNqunAXfEqHaNAI2bu9r4wH0IOTYW+TUeOTYeOTYW+TUeOYYvs/Rf9sePH1dsbKySk5MVFRWlkJAQVatWTbm5udq3b5+WLFmi6dOn64MPPtD1119vZaheY7P92iZzlnwnm5K+fDiXk6shby/RseTTSj6TKUnaefCk8gsKNbxXB81etVVHkk4bFbLPIcfGIr/GI8fGI8fGIr/GI8fWKHL63rhTaWTpOSATJ05UeHi41q9frwULFuitt97S66+/rrfeeksLFizQt99+q7CwML344otWhulVZ7NzJUn+DtdvJqr++jwzJ9dtn9z8Qm2NP178y+KC9XuOSJIiwmsaEarPIsfGIr/GI8fGI8fGIr/GI8fwZZYWIDt27NDYsWMVGBhY4utBQUEaM2aMtm27+PWsfU1CapoKCotUN6S6y3rdWuef/5R4ym2f+rVrqHenFm6/ZByVzzew0jJzDInVV5FjY5Ff45Fj45FjY5Ff45Fj+DJLC5CgoCClpKRccpuff/5ZDofDpIiMl1dQqF2HEnR7qyYu692jI5SRlaO9R5Pc9gmpHqDn/thN3aMjXNbviGmqzOxc7TuebGjMvoYcG4v8Go8cG48cG4v8Go8cW8Pqq11xFSwv6NOnj5555hmNHDlS7dq1U1hYmOx2u/Ly8pScnKytW7dqypQp6tOnj5Vhet0Hq7Zq1sjeen3Q3Vrx3V61bBSmP/WI0bTl65WbXyh/h12NwoJ1IjVdaZnZ2nEgQdv2n9DoPp3l8Kuso0mn1al5Qz10W7TeWvqtzma5t1nLO3JsLPJrPHJsPHJsLPJrPHIMX2VzOi9y9pIJnE6n3nnnHc2dO1dZWVlur/v7++vhhx/WX/7yF1WocO3NmtZDp17ze3hL15aNNaTnLapfu4ZS0s/pk39/r/nrdkqSbo4I1/ujH9D4D9fo881xkqQAh12De96i21o2Vs1q/kpITdeCb3Zq2cY9Vv4YpRo5Nhb5NR45Nh45Nhb5NV5ZzPHOWaOsDuGidh6vZ3UIJWpdz/2Sy6WZpQXIBfn5+dq3b5+Sk5OVnZ0th8Oh0NBQRUZGym53v4zc1SpNBQgAAADcleYCZNvxBlaHUKI29Y5aHcIVKRU32KhcubJatGhhdRgAAAAADGbpSegAAAAAypdS0QEBAAAASjtuROgddEAAAAAAmIYCBAAAAIBpGMECAAAAPOCLN/0rjeiAAAAAADANBQgAAAAA0zCCBQAAAHig0Ml3995AFgEAAACYhgIEAAAAgGkYwQIAAAA8UMR3915BFgEAAACYhgIEAAAAgGkYwQIAAAA8wI0IvYMOCAAAAADTUIAAAAAAMA0jWAAAAIAHuBGhd5BFAAAAAKahAAEAAABgGkawAAAAAA8UcRUsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOCBQr679wqyCAAAAMA0FCAAAAAATMMIFgAAAOABbkToHWQRAAAAgGkoQAAAAACYhhEsAAAAwANFfHfvFWQRAAAAgGkoQAAAAACYhhEsAAAAwAOFTpvVIZQJdEAAAAAAmIYCBAAAAIBpKEAAAAAAmIZzQAAAAAAPFPLdvVeQRQAAAKCcKioq0rRp09SpUye1bNlSAwcO1LFjxy66/YkTJzRkyBC1bdtWHTp00KRJk5SdnX1Fx6QAAQAAAMqpmTNnauHChZo0aZIWLVokm82m2NhY5eXluW179uxZ9e3bV+np6frggw/07rvvas+ePRo+fPgVHZMCBAAAAPBAkbNCqXxcrby8PM2ZM0cjRoxQly5dFBkZqalTpyo5OVlr1651237ZsmXKzMzUO++8oxYtWqh58+aaOnWqNm3apO3bt3t8XAoQAAAAoByKj4/XuXPn1L59++K1oKAgRUVFadu2bW7bHzlyRI0aNVJwcHDxWlhYmGrUqKGtW7d6fFxOQgcAAAB8WLdu3S75+rp160pcT0pKknS+iPhfISEhSkxMdNu+Vq1aSk1NVWFhoSpWrChJyszMVHp6uk6dOuVxvHRAAAAAAA8UqkKpfFytCyeP2+12l3U/Pz/l5ua6bX/33XcrPT1dr7zyis6dO6eMjAyNHz9eNputxHNGLoYOCAAAAODDLtbhuByHwyHp/LkgF/4sSbm5uapSpYrb9vXr19f06dP1wgsv6OOPP5bD4dAjjzyim266SQEBAR4flwIEAAAAKIcujF6lpKSoXr16xespKSmKjIwscZ8uXbroP//5j1JTUxUYGCiHw6Fbb71V999/v8fHZQQLAAAA8ECh01YqH1crMjJSAQEB2rJlS/FaRkaG4uLiFBMT47b9jh071K9fP+Xl5alWrVpyOBzaunWrzpw5o1tvvdXj49IBAQAAAMohu92ufv36acqUKQoODladOnU0efJkhYaGqkePHiosLNTp06eLOx2NGzfWwYMH9corr+jRRx/ViRMnNHbsWD300EOqW7eux8elAAEAAADKqZEjR6qgoEDjxo1TTk6O2rRpo9mzZ8tutyshIUHdunXTq6++qvvvv1/Vq1fXe++9p1dffVX33HOPatSooYceekhDhw69omPanE6n06Cfp9RpPXSq1SEAAADgEnbOGmV1CBf14UHPx4zM1D9ik9UhXBHOAQEAAABgGgoQAAAAAKbhHBAAAADAA4VOvrv3BrIIAAAAwDQUIAAAAABMwwgWAAAA4IEiXf1N//BfdEAAAAAAmIYCBAAAAIBpGMECAAAAPMBVsLyDLAIAAAAwDQUIAAAAANMwggUAAAB4oJDv7r2CLAIAAAAwDQUIAAAAANMwggUAAAB4oMjJjQi9gQ4IAAAAANNQgAAAAAAwDSNYAAAAgAe4CpZ3kEUAAAAApqEAAQAAAGAaRrAAAAAADxQ5+e7eG8giAAAAANNQgAAAAAAwDSNYAAAAgAcKxY0IvYEOCAAAAADTUIAAAAAAMA0jWAAAAIAHuAqWd5BFAAAAAKahAAEAAABgGkawAAAAAA9wFSzvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AGuguUdZBEAAACAaShAAAAAAJiGESwAAADAA4WMYHkFWQQAAABgGgoQAAAAAKZhBAsAAADwQBE3IvQKOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BFgEAAACYhgIEAAAAgGkYwQIAAAA8UOTkKljeQAcEAAAAgGkoQAAAAACYhhEsAAAAwAOFfHfvFRQgFrk1qr6G3dtBDcOClXY2W4vX79bcNdsuun2D2jW0dMIAt/UjSafVe+KHBkbqu8ixsciv8cix8cixsciv8cgxfBEFiAVaNArT1KG99NWOA5r52Ua1alJHw+/toAo2m2av3lriPk3r1pIkxb75qfIKCorXc/IKSty+vCPHxiK/xiPHxiPHxiK/xiPH8FUUIBYYfHd77U9I1fPzVkuSNsUdU6UKFTTgzjaav26HcvML3fZpFh6ik7+ka8fBBLPD9Unk2Fjk13jk2Hjk2Fjk13jk2HxcBcs7GGQzWeVKFXVzRLi+2XXIZf3rXQfl77ArukmdEvdrFl5L+xNSzQjR55FjY5Ff45Fj45FjY5Ff45Fj+DI6ICYLr1lN9sqVdDzljMv6idQ0SVL9kBravO+4235Nw2vpSNIpzRvzoJrVDdHZrFx9vjlOsz7bpIKiIjNC9xnk2Fjk13jk2Hjk2Fjk13jkGL6MAsRkgVX8JEmZOXku61m/PvevYnfbJziwqmpW85fT6dS0ZRuUeCZDbZvV04A7YlS7RoDGzV1tfOA+hBwbi/wajxwbjxwbi/wajxxbo4jhIa+gADGZzfbr7KDTWeLrJX35cC4nV0PeXqJjyaeVfCZTkrTz4EnlFxRqeK8Omr1qq44knTYqZJ9Djo1Ffo1Hjo1Hjo1Ffo1HjuHLKONMdjY7V5Lk73D9ZqLqr88zc3Ld9snNL9TW+OPFvywuWL/niCQpIrymEaH6LHJsLPJrPHJsPHJsLPJrPHIMX2Z5B+SRRx75bxV/GR999JHB0RgvITVNBYVFqhtS3WW9bq3zz39KPOW2T/3aNRTTtK5Wb4vXuf9ptToqn/+fLy0zx7B4fRE5Nhb5NR45Nh45Nhb5NR45tkYhV8HyCss7ILfccou2bdumU6dOqU6dOpd8lAV5BYXadShBt7dq4rLePTpCGVk52ns0yW2fkOoBeu6P3dQ9OsJl/Y6YpsrMztW+48mGxuxryLGxyK/xyLHxyLGxyK/xyDF8meUdkGHDhqlq1aqaNm2a/v73vys8PNzqkAz3waqtmjWyt14fdLdWfLdXLRuF6U89YjRt+Xrl5hfK32FXo7BgnUhNV1pmtnYcSNC2/Sc0uk9nOfwq62jSaXVq3lAP3Ratt5Z+q7NZ7m3W8o4cG4v8Go8cG48cG4v8Go8cw1fZnM6LnL1kskGDBql69eqaMmWKYcdoPXSqYe99pbq2bKwhPW9R/do1lJJ+Tp/8+3vNX7dTknRzRLjeH/2Axn+4Rp9vjpMkBTjsGtzzFt3WsrFqVvNXQmq6FnyzU8s27rHyxyjVyLGxyK/xyLHxyLGxyK/xymKOd84aZXUIF/WXXX2tDqFEb0f/0+oQrkipKUCSk5MVFxenrl27GnaM0lSAAAAAwB0FyJXztQLE8hGsC2rXrq3atWtbHQYAAAAAA5WaAgQAAAAozYqcll+/qUwgiwAAAABMQwECAAAAwDSMYAEAAAAeKBQ3IvQGOiAAAAAATEMBAgAAAMA0jGABAAAAHihyMoLlDXRAAAAAAJiGAgQAAACAaRjBAgAAADzAjQi9gywCAAAAMA0FCAAAAADTMIIFAAAAeKCIGxF6BR0QAAAAAKahAAEAAABgGkawAAAAAA8UciNCr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHWQRAAAAgGkoQAAAAACYhhEsAAAAwANFXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHigSIxgeQMdEAAAAACmoQABAAAAYBpGsAAAAAAPcBUs76ADAgAAAJRTRUVFmjZtmjp16qSWLVtq4MCBOnbs2EW3T01N1ejRo9WuXTu1a9dOf/nLX5SUlHRFx6QAAQAAAMqpmTNnauHChZo0aZIWLVokm82m2NhY5eXllbj9qFGjlJiYqLlz52ru3LlKSkrSsGHDruiYFCAAAACAB4qcFUrl42rl5eVpzpw5GjFihLp06aLIyEhNnTpVycnJWrt2rdv2GRkZ2rZtm2JjYxUVFaWoqCg99thj2rt3r86cOePxcSlAAAAAgHIoPj5e586dU/v27YvXgoKCFBUVpW3btrlt7+fnp6pVq2r58uXKzMxUZmamVqxYoQYNGqhatWoeH5eT0AEAAIBy6MK5G2FhYS7rISEhSkxMdNvez89PL7/8sl588UXFxMTIZrOpVq1amj9/vipU8LyvQQECAAAAeKC0XgWrW7dul3x93bp1Ja5nZ2dLkux2u8u6n5+f0tPT3bZ3Op3av3+/oqOjNWjQIBUWFmrq1KkaPny4/vnPfyogIMCjeClAAAAAgHLI4XBIOn8uyIU/S1Jubq6qVKnitv2XX36pBQsW6F//+ldxsfHuu++qa9euWrJkifr37+/RcSlAAAAAAB92sQ7H5VwYvUpJSVG9evWK11NSUhQZGem2/Y4dO9SwYUOXTke1atXUsGFDHT161OPjchI6AAAA4IEi2Url42pFRkYqICBAW7ZsKV7LyMhQXFycYmJi3LYPCwvTsWPHlJubW7yWnZ2thIQE1a9f3+PjUoAAAAAA5ZDdble/fv00ZcoUrVu3TvHx8Ro1apRCQ0PVo0cPFRYWKjU1VTk5OZKk++67T5L0xBNPKD4+vnh7u92u+++/3+PjUoAAAAAA5dTIkSPVp08fjRs3Tn379lXFihU1e/Zs2e12JSYmqmPHjlq5cqWk81fHWrBggZxOp/r3768///nPqly5sv75z38qKCjI42PanE6n06gfqLRpPXSq1SEAAADgEnbOGmV1CBf1h43DrQ6hRMs6vGN1CFeEDggAAAAA01CAAAAAADANl+EFAAAAPFBab0Toa+iAAAAAADANBQgAAAAA0zCCBQAAAHiAESzvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AFGsLyDDggAAAAA01CAAAAAADANI1gAAACAB4rECJY30AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAHmAEyzvogAAAAAAwDQUIAAAAANMwggUAAAB4gBEs76ADAgAAAMA0FCAAAAAATMMIFgAAAOABRrC8gw4IAAAAANNQgAAAAAAwDSNYAAAAgAecjGB5BR0QAAAAAKahAAEAAABgGkawAAAAAA8UiREsb6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaRjBAgAAADzAjQi9gw4IAAAAANNQgAAAAAAwDSNYAAAAgAe4CpZ30AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BBwQAAACAaShAAAAAAJiGESwAAADAA06n1RGUDXRAAAAAAJiGAgQAAACAaRjBAgAAADxQJK6C5Q10QAAAAACYhgIEAAAAgGkYwQIAAAA84ORGhF5BBwQAAACAaShAAAAAAJiGESyL3BpVX8Pu7aCGYcFKO5utxet3a+6abRfdvkHtGlo6YYDb+pGk0+o98UMDI/Vd5NhY5Nd45Nh45NhY5Nd45NhcRYxgeQUFiAVaNArT1KG99NWOA5r52Ua1alJHw+/toAo2m2av3lriPk3r1pIkxb75qfIKCorXc/IKSty+vCPHxiK/xiPHxiPHxiK/xiPH8FUUIBYYfHd77U9I1fPzVkuSNsUdU6UKFTTgzjaav26HcvML3fZpFh6ik7+ka8fBBLPD9Unk2Fjk13jk2Hjk2Fjk13jkGL6Kc0BMVrlSRd0cEa5vdh1yWf9610H5O+yKblKnxP2ahdfS/oRUM0L0eeTYWOTXeOTYeOTYWOTXeOTYGk5n6Xz4GjogJguvWU32ypV0POWMy/qJ1DRJUv2QGtq877jbfk3Da+lI0inNG/OgmtUN0dmsXH2+OU6zPtukgqIiM0L3GeTYWOTXeOTYeOTYWOTXeOQYvszyAuTIkSP64osvlJ6erk6dOqlLly4ur2dmZurll1/Wq6++alGE3hVYxU+SlJmT57Ke9etz/yp2t32CA6uqZjV/OZ1OTVu2QYlnMtS2WT0NuCNGtWsEaNzc1cYH7kPIsbHIr/HIsfHIsbHIr/HIMXyZpQXIjh079Oijj6p27dpyOp36+OOP1b17d73xxhuy28//xcnJydHy5cvLTAFis/169YSL9MtK+vLhXE6uhry9RMeSTyv5TKYkaefBk8ovKNTwXh00e9VWHUk6bVTIPoccG4v8Go8cG48cG4v8Go8cW4MbEXqHpeeAvPHGG+rTp4/WrFmjr776Sm+++aY2btyoIUOGKD8/38rQDHM2O1eS5O9w/Wai6q/PM3Ny3fbJzS/U1vjjxb8sLli/54gkKSK8phGh+ixybCzyazxybDxybCzyazxyDF9maQGyf/9+9evXr/j5XXfdpffff1+7du3S2LFjLYzMOAmpaSooLFLdkOou63VrnX/+U+Ipt33q166h3p1auP2ScVQ+38BKy8wxJFZfRY6NRX6NR46NR46NRX6NR47hyywtQAICAnTmjOvJUzfffLMmT56sNWvWlJmxq/+VV1CoXYcSdHurJi7r3aMjlJGVo71Hk9z2CakeoOf+2E3doyNc1u+IaarM7FztO55saMy+hhwbi/wajxwbjxwbi/wajxxbw+m0lcqHr7H0HJAuXbroxRdf1IQJExQVFaXKlStLkrp3765nn31WkyZNUmJiopUhGuKDVVs1a2RvvT7obq34bq9aNgrTn3rEaNry9crNL5S/w65GYcE6kZqutMxs7TiQoG37T2h0n85y+FXW0aTT6tS8oR66LVpvLf1WZ7Pc26zlHTk2Fvk1Hjk2Hjk2Fvk1HjmGr7I5ndZdPTg9PV2jRo3Sd999p7///e/q3Lmzy+sLFizQK6+8osLCQu3bt++aj9d66NRrfg9v6dqysYb0vEX1a9dQSvo5ffLv7zV/3U5J0s0R4Xp/9AMa/+Eafb45TpIU4LBrcM9bdFvLxqpZzV8Jqela8M1OLdu4x8ofo1Qjx8Yiv8Yjx8Yjx8Yiv8YrizneOWuU1SFcVPPPxlsdQol+vHei1SFcEUsLkAuOHz+uGjVqKDAw0O21I0eO6KuvvtLgwYOv+TilqQABAACAu9JcgNy4YoLVIZRob68JVodwRSy/D4gk1atX76KvNWzY0CvFBwAAAADrWXoSOgAAAIDypVR0QAAAAIDSzvoTF8oGOiAAAAAATEMBAgAAAMA0jGABAAAAHvDFm/6VRnRAAAAAAJiGAgQAAACAaRjBAgAAADzACJZ30AEBAAAAYBoKEAAAAACmYQQLAAAA8AD3IfQOOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BBwQAAACAaShAAAAAAJiGESwAAADAE1wGyyvogAAAAAAwDQUIAAAAANMwggUAAAB4gKtgeQcdEAAAAACmoQABAAAAyqmioiJNmzZNnTp1UsuWLTVw4EAdO3asxG2nT5+uZs2alfh45plnPD4mBQgAAADgAaezdD6uxcyZM7Vw4UJNmjRJixYtks1mU2xsrPLy8ty2HThwoDZs2ODyeOKJJ+RwONS/f3+Pj0kBAgAAAJRDeXl5mjNnjkaMGKEuXbooMjJSU6dOVXJystauXeu2vb+/v2rVqlX8yM7O1t///nc9/fTTioyM9Pi4FCAAAABAORQfH69z586pffv2xWtBQUGKiorStm3bLrv/a6+9poiICD344INXdFyuggUAAAB4oLReBatbt26XfH3dunUlriclJUmSwsLCXNZDQkKUmJh4yff88ccftW7dOn344YeqUOHKehp0QAAAAIByKDs7W5Jkt9td1v38/JSbm3vJfefNm6eWLVu6dE88RQcEAAAA8GEX63BcjsPhkHT+XJALf5ak3NxcValS5aL7ZWVlae3atRo/fvxVHZcOCAAAAOAJp610Pq7ShdGrlJQUl/WUlBSFhoZedL/169erqKhIPXr0uKrjUoAAAAAA5VBkZKQCAgK0ZcuW4rWMjAzFxcUpJibmovvt2LFDN954o4KCgq7quIxgAQAAAOWQ3W5Xv379NGXKFAUHB6tOnTqaPHmyQkND1aNHDxUWFur06dMKDAx0GdGKj49X06ZNr/q4dEAAAAAAD1h9w0EjbkQ4cuRI9enTR+PGjVPfvn1VsWJFzZ49W3a7XYmJierYsaNWrlzpss8vv/yi6tWrX/UxbU7ntYbtO1oPnWp1CAAAALiEnbNGWR3CRTX65ytWh1Cin/o+a3UIV4QOCAAAAADTcA4IAAAA4IlyMzdkLDogAAAAAExDAQIAAADANIxgAQAAAB5wXsNN//BfdEAAAAAAmIYCBAAAAIBpGMECAAAAPMFVsLyCDggAAAAA01CAAAAAADANI1gAAACAB7gKlnfQAQEAAABgGgoQAAAAAKZhBAsAAADwBFfB8go6IAAAAABMU646INf9fZPVIZRp6SsjrA6hzCtYUcvqEMq0Sr1SrQ6hzOMzDAAoVwUIAAAAcPW4CpY3MIIFAAAAwDQUIAAAAABMwwgWAAAA4AmuguUVdEAAAAAAmIYCBAAAAIBpGMECAAAAPMEIllfQAQEAAABgGgoQAAAAAKZhBAsAAADwhJMbEXoDHRAAAAAApqEAAQAAAGAaRrAAAAAADzi5CpZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8AQjWF5BBwQAAACAaShAAAAAAJiGESwAAADAE9yI0CvogAAAAAAwDQUIAAAAANMwggUAAAB4wMZVsLyCDggAAAAA01CAAAAAADANI1gAAACAJxjB8go6IAAAAABMQwECAAAAwDSMYAEAAACe4EaEXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMATXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANI1gAAACAJ5w2qyMoE+iAAAAAADANBQgAAAAA0zCCBQAAAHjAxlWwvIIOCAAAAADTUIAAAAAAMA0jWAAAAIAnGMHyCjogAAAAAExDB8QkMXe20p9fekj1osKVnpqhL/7+lRa+tvyi21eqXEl9nuypHn+6TbXqXqdfEk7rmwXrtfC15SrILyjebtHP7ys4tLrb/v8XFqszyWne/0FKqfY1IzQ0oocaBoToTN45LT2xVR/+9J8St727TmuNb97nou81cfen+vLnXZKkVV2f0XV+gW7b3PXNKzqVl+md4H3UrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt/AZ9gafI6NRX6NR47hiyhATBB1S1O9uOIp/WfRJs19fqFu6hipP0/qqwoVKmjBK0tL3GfoWwPU409d9PGkJTqw7ZCatG6kR8Y/oJD6tfTmoFmSpOoh1RQcWl2zRs/Tvu8OuOyfceqs4T9XadG8ej290foRrU38UbMOrlWrGg00NKKHKsimuT/92237jSn7NfC7WS5rNpv07I33y7+Snzamns9lsD1A1/kFauq+L/Vj2nGX7dPyswz7eXxBi0Zhmjq0l77acUAzP9uoVk3qaPi9HVTBZtPs1VtL3Kdp3VqSpNg3P1VewX+L6Jy8ghK3L0/4DFuDz7GxyK/xyDF8FQWICR554QEd/v6oXu8/XZK0fc33qlS5oh586j4tfvML5eXkuWwfWCNAPQf30AdPf6xPp3wmSdr1zR5J0mN/e0Szn/5Y6b9kqEl0Q0nShqVblHL8FxN/otIltsntOpCRqAk/fipJ2vzLQVWyVdCfGnXRgqMblFvk+ks1Lf+c0tLPuaw9WP9WNQiopUGb31Va/vnXmgaFSZL+lbxXSTlpxv8gPmTw3e21PyFVz89bLUnaFHdMlSpU0IA722j+uh3KzS9026dZeIhO/pKuHQcTzA631OMzbA0+x8Yiv8Yjx/BVV3QOSGpqqiZMmKBHH31UTz31lObNm6ft27crOzvbqPh8XmV7JbW47UZtWLbFZf3bxZtVNbCKmneKdNvHv1pVffH3tfrus+0u6wkHfpYkhTUKkSQ1btVAZ89kluvio7KtoloHN9K/kve6rK9L2iP/Sn5qVaPBZd/jOnuAhkT00JLjW7Q3/b+/kJsGhikjP5t/uP1G5UoVdXNEuL7Zdchl/etdB+XvsCu6SZ0S92sWXkv7E1LNCNGn8Bm2Bp9jY5Ff45Fj+LIr6oA8++yz2rBhgyIiIpSQkKDPP/9cTqdTFSpUUKNGjXTTTTepefPmat68uSIjI1W5cmWj4vYZYY1qy+5XWSd/LR4u+PlQkiQpvOn12rF2t8trSUdTNH34B27v1fEP7ZSfV6CEA4mSpMYtGyjzzDmNX/xXRXdrrgoVK2jzFzv07uh5Op2UZswPVMrUqRose4VKOp51ymU94dfn9fxrasupQyXtWuyxiO4qchbp3YNrXdabBl2vs/nZer3Vw2pzXWNVsNm0IXW/psZ/qVO55WfE7bfCa1aTvXIlHU8547J+IjVNklQ/pIY27zvutl/T8Fo6knRK88Y8qGZ1Q3Q2K1efb47TrM82qaCoyIzQSyU+w9bgc2ws8ms8cmwNbkToHVdUgOzatUtjxozRwIEDJUlZWVnau3evfvzxR/3444/atm2bli1bJkmy2+3avXv3pd5OkpSbm6uDBw+qSZMmcjgc2rdvn+bPn6/k5GRFRESof//+Cg0NvYofrXTwr+4vSTqX4dolyjp7/nnVoCoevU/H+9up+yOdtWzaSmWmnR+vaNyqgWqGX6eVH3ytJW99oXo3hKv/xAf1xr8namjrscrJyvXiT1I6BVZ2SJLOFeS4rGcVnh9r86/kd8n9a9j99fvro/Xx0Q3K/M17NA0MU4gjSMsTtumfxzaqgX8tDY7orr+3jVW/TdOVU5jvxZ/EdwRWOZ/TzN+MDmb9+ty/it1tn+DAqqpZzV9Op1PTlm1Q4pkMtW1WTwPuiFHtGgEaN3e18YGXUnyGrcHn2Fjk13jkGL7sigoQPz8/RUVFFT+vWrWq2rRpozZt2hSvpaWlaffu3dqzZ89l3+/w4cMaMGCAUlNTdf3112vSpEkaNmyYwsPD1bhxY3399ddaunSpFixYoMaNG19JqKVGhQq2839wllwyFxVdvpTu1Lu9np4/Urv/E6fZT39cvD5l4Ezl5eTp8PdHJUl7NsTr2N4TemvDJHX/Uxd98e5X1xx/aWfThfyW/HrRRfJ+wX3hbVTBZtPCo5vcXntpz2LlFhbowNnzHafvzxzVT5nJ+qD9EN19fWstObHFbZ/ywGa73Gfafe1cTq6GvL1Ex5JPK/nM+Ssv7Tx4UvkFhRreq4Nmr9qqI0mnjQq5VOMzbA0+x8Yiv8Yjx/BlV3QOSPfu3RUXF3fJbapXr67OnTtr2LBhl32/v/3tb4qOjtby5ct18803a+jQobrnnnv0+eef6+2339aqVavUoUMHvfrqq1cSZqlyoVtRNaiqy3rVwPOdj3Ppl74STe9RPfXcwlHauyFez9/7uvL/5yoV+zYfKC4+Lti7ab8y086pccv6Xoi+9Lvwje9vvyWuWvH8Nz/nCi7dBbo99CZt+eVQ8Um7/+vHtBPF/3C7YHfacZ3Nz1ZEoO925a7V2ezzOfV3uH67VvXX55k57jnPzS/U1vjjxf/Bu2D9niOSpIjwmkaE6hP4DFuDz7GxyK/xyLFFnLbS+fAxV1SA9O7dW6tWrdKhQ5eeR/bU1q1b9cQTTygyMlJPPfWUcnNz1bdv3+KqvlKlShoyZIh27NjhleNZ4efDySosKNT1TVz/Y3/h+fG4i1+FYvi0gRryRn+tX7xZz939inLO/Xe8wr9aVd35566qd0O4236V7JWU/kv5mO9OyDqtgqJChVe9zmX9wvOfMlMuum+IX5CaBV2vr5N+dHstoJJD99S5WQ39Q9xeq1yhYrm+hGlCapoKCotUN6S6y3rdWuef/5R4ym2f+rVrqHenFm7/oXRUPt+ETcvMcdunvOAzbA0+x8Yiv8Yjx/BlV1SA/N///Z/27NmjBx54QM8884xWrlypY8eOXfXBHQ6HcnLOf9hr1qyp//u//5Ofn+u3gBkZGQoMdL+Jlq/Iz83X7m/3qeMf2rmsd+7TXmfPZCp+a8nF3MBX/qj7Hr9Li6d+oZf7TnXpfEhSQV6BRr4zSA89dZ/L+q292shR1U8//Nv1ijplVV5Rgb4/c1RdQ290We8WepMy8rMVl37iovtGVa8rSfrhjPtnOL+oUGOj7lX/Rp1d1ruE3CBHRbt2nP7JC9H7pryCQu06lKDbWzVxWe8eHaGMrBztPZrktk9I9QA998du6h4d4bJ+R0xTZWbnat/xZENjLs34DFuDz7GxyK/xyDF82RWdAzJp0iTt27dPe/fu1apVq7Rs2TLZbDb5+/srKipKN910k8aOHevx+3Xs2FEvvfSSJk2apMaNG+vFF18sfs3pdGrr1q2aOHGiunfvfiVhljoLXl6i19c+r+cXjdbqud8o6tZmeuCv9+qDpz9WXk6eqgZWUf2ocP18OFnpv2SoccsGenBsL+3fdkjffrJJN7Rz/UVxLC5BWWez9cnkz9Tv+T46k5Ku7at3qWGL+vrT+P/T5i92aNc6929Ey6o5h/+lGW0G6tVWffVZwg61qF5P/Rp20oz9a5RbVCD/in5qGBCihKzTLmMqTQJqK7cwXyez3eddc4vy9Y8j6zWoye06lZepzakH1SQwVLFNumlDSry2nTps5o9Y6nywaqtmjeyt1wfdrRXf7VXLRmH6U48YTVu+Xrn5hfJ32NUoLFgnUtOVlpmtHQcStG3/CY3u01kOv8o6mnRanZo31EO3Reutpd/qbDm4YMKl8Bm2Bp9jY5Ff45FjC3AVLK+wOZ2XOcPxIoqKinT48GHt3btXe/bsUVxcnOLj47Vz506P3+P06dMaMmSI6tatqzfeeMPltS+//FJPPvmkOnXqpKlTpyogIOBqwnTRo8ID1/weV6vDfW31pwn/p/Bm1+vUydP6bOZqLX7zC0lSiy5ReuNfEzX5z+/oqw//rf4TH1S/5/tc9L2e7Dpeu/8TJ5vNpnuG3qGeQ+7Q9Y1DlXHqrL755wZ9NP4Tt5sbmiF9ZcTlNzLIbSFRio3opvr+tZSak6FPj2/WgqMbJEmtgxvq3baxmvjjYn158r+fz7FR96pr7Rt1179KPsfIJpt612un3nXbqU7VYKXnZ+mrn3/Qe4e+drsxnFkKVtSy5Lgl6dqysYb0vEX1a9dQSvo5ffLv7zV/3fn83hwRrvdHP6DxH67R55vPnzcW4LBrcM9bdFvLxqpZzV8Jqela8M1OLdt4+QtWmKVSL+uujc9n2Bpl8XNcmpBf45XFHO+cNcrqEC6q0VtvWh1CiX56YrTVIVyRqy5ASuJ0Ov97VYYrkJaWpurVq7usnT59WikpKYqMdL9R39WysgApD6wsQMqL0vaPt7LGygKkvOAzDOByKECunK8VIFc0gnU5V1N8SHIrPiQpODhYwcHB1xgRAAAA4CWMYHnFFZ2EDgAAAADXggIEAAAAgGm8OoIFAAAAlFU2RrC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4gBsRegcdEAAAAACmoQABAAAAYBoKEAAAAMATTlvpfFyDoqIiTZs2TZ06dVLLli01cOBAHTt27KLb5+fn64033lCnTp3UqlUr9evXT/v27buiY1KAAAAAAOXUzJkztXDhQk2aNEmLFi2SzWZTbGys8vLyStx+woQJWrx4sV566SUtWbJE1atXV2xsrM6ePevxMSlAAAAAgHIoLy9Pc+bM0YgRI9SlSxdFRkZq6tSpSk5O1tq1a922P3HihBYvXqxXX31Vt912mxo3bqxXXnlFdrtde/bs8fi4FCAAAACAJ5yl9HGV4uPjde7cObVv3754LSgoSFFRUdq2bZvb9hs2bFBQUJA6d+7ssv0333yjW265xePjchleAAAAwId169btkq+vW7euxPWkpCRJUlhYmMt6SEiIEhMT3bY/evSo6tatq6+++krvvfeekpOTFRUVpaefflqNGzf2OF46IAAAAEA5lJ2dLUmy2+0u635+fsrNzXXbPjMzU8ePH9fMmTM1evRozZo1S5UqVdIf//hHnTp1yuPj0gEBAAAAPFBab0R4sQ7H5TgcDknnzwW58GdJys3NVZUqVdy2r1y5ss6ePaupU6cWdzymTp2qLl26aNmyZRo0aJBHx6UDAgAAAJRDF0avUlJSXNZTUlIUGhrqtn1oaKgqVarkMm7lcDhUt25dJSQkeHxcChAAAACgHIqMjFRAQIC2bNlSvJaRkaG4uDjFxMS4bR8TE6OCggL9+OOPxWs5OTk6ceKE6tev7/FxGcECAAAAPFFKR7Cult1uV79+/TRlyhQFBwerTp06mjx5skJDQ9WjRw8VFhbq9OnTCgwMlMPhUExMjG699VY99dRTevHFF1W9enVNmzZNFStWVK9evTw+Lh0QAAAAoJwaOXKk+vTpo3Hjxqlv376qWLGiZs+eLbvdrsTERHXs2FErV64s3n769Olq27atHn/8cfXp00eZmZn66KOPFBwc7PExbU6ns4zVchfXo8IDVodQpqWvjLA6hDKvYEUtq0Mo0yr1SrU6hDKPzzCAy9k5a5TVIVxUxKtTrQ6hRAefKb05KwkjWAAAAIAHSutVsHwNI1gAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaShAAAAAAJiGAgQAAACAaShAAAAAAJiGq2ABAAAAnuAqWF5BBwQAAACAaShAAAAAAJiGESwAAADAA9yI0DvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsryhXBcipwbdaHULZtsLqAMq+7eNnWR0CcE1iVgy1OgQAgMUYwQIAAABgmnLVAQEAAACuGiNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMAD3IjQO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AGuguUddEAAAAAAmIYCBAAAAIBpGMECAAAAPMEIllfQAQEAAABgGgoQAAAAAKZhBAsAAADwBCNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMAD3IjQO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANI1gAAACAB7gRoXfQAQEAAABgGgoQAAAAAKZhBAsAAADwBCNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMADXAXLO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANBQgAAAAA03AOCAAAAOABm9UBlBF0QAAAAACYhgIEAAAAgGkYwbLIrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt9Fjq2XmCL1+rM0Y5LUNtrqaMomcnxt+D1hLPJrPHJsMi7D6xUUIBZo0ShMU4f20lc7DmjmZxvVqkkdDb+3gyrYbJq9emuJ+zStW0uSFPvmp8orKChez8krKHH78o4cW+9kkhQ7RjqbaRO/sY1Bjq8NvyeMRX6NR47hqyhALDD47vban5Cq5+etliRtijumShUqaMCdbTR/3Q7l5he67dMsPEQnf0nXjoMJZofrk8ixdYqKpOWrpb/NsjqSsoscewe/J4xFfo1HjuGrOAfEZJUrVdTNEeH6Ztchl/Wvdx2Uv8Ou6CZ1StyvWXgt7U9INSNEn0eOrbX/sDRxqnTfndLrz1kdTdlEjq8dvyeMRX6NR46tYXOWzoevoQNisvCa1WSvXEnHU864rJ9ITZMk1Q+poc37jrvt1zS8lo4kndK8MQ+qWd0Qnc3K1eeb4zTrs00qKCoyI3SfQY6tFVZbWvOxFBoibd1ldTRlEzm+dvyeMBb5NR45hi8rtQXIPffco/fee09hYWFWh+JVgVX8JEmZOXku61m/PvevYnfbJziwqmpW85fT6dS0ZRuUeCZDbZvV04A7YlS7RoDGzV1tfOA+hBxbq3qQpCCroyjbyPG14/eEsciv8cgxfJmlBcjy5csv+tqxY8e0atUqBQcHS5Luu+8+c4IymM326y1snCX3y0r68uFcTq6GvL1Ex5JPK/lMpiRp58GTyi8o1PBeHTR71VYdSTptVMg+hxwDuBx+TxiL/BqPHFvEB8edSiNLC5CJEycqJydHkuQs4S/Q3/72N0nn/5KVlQLkbHauJMnf4frNRNVfn2fm5Lrtk5tfqK3x7m3U9XuOaHivDooIr8kvjP9BjgFcDr8njEV+jUeO4cssPQl96dKlioqKUrt27fSf//xH8fHxxY8qVapo7dq1io+P1759+6wM06sSUtNUUFikuiHVXdbr1jr//KfEU2771K9dQ707tXD7JeOofL5+TMvMMSRWX0WOAVwOvyeMRX6NR47hyywtQBo2bKhFixapRYsW6tWrl1auXGllOKbIKyjUrkMJur1VE5f17tERysjK0d6jSW77hFQP0HN/7Kbu0REu63fENFVmdq72HU82NGZfQ44BXA6/J4xFfo1Hji3iLKUPH2P5SeiVKlXS6NGj1alTJz311FNat26dJkyYYHVYhvpg1VbNGtlbrw+6Wyu+26uWjcL0px4xmrZ8vXLzC+XvsKtRWLBOpKYrLTNbOw4kaNv+Exrdp7McfpV1NOm0OjVvqIdui9ZbS7/V2Sz3Nmt5R44BXA6/J4xFfo1HjuGrbM6STr6wSEZGhiZOnKjt27fr1KlTWrVqlerWreu19289dKrX3utadW3ZWEN63qL6tWsoJf2cPvn395q/bqck6eaIcL0/+gGN/3CNPt8cJ0kKcNg1uOctuq1lY9Ws5q+E1HQt+Ganlm3cY+WPUaqVxRxvH+9bd57bukvq/4RNH77lVNtoq6Mpm3wtxzETh1odgouy+HuiNCG/xiuLOd45a5TVIVxUy5Gl59+S/+uHaaU3ZyUpVQXIBcuXL9fSpUs1ZcoUhYSEeO19S1MBAlwNXytAgN8qbQUIgNKnNBcgrUaUzn9Lfj+99OasJJaPYJXkvvvuKzNXvQIAAADwX5aehA4AAACgfCmVHRAAAACg1Cl1Jy74JjogAAAAAExDAQIAAADANIxgAQAAAB6wMYLlFXRAAAAAAJiGAgQAAAAop4qKijRt2jR16tRJLVu21MCBA3Xs2LGLbr9s2TI1a9bM7XGpfX6LESwAAADAE2VwBGvmzJlauHChXn31VdWuXVuTJ09WbGysvvjiC9ntdrft9+/fr7Zt2+rNN990WQ8ODvb4mHRAAAAAgHIoLy9Pc+bM0YgRI9SlSxdFRkZq6tSpSk5O1tq1a0vc58CBA4qMjFStWrVcHhUrVvT4uBQgAAAAQDkUHx+vc+fOqX379sVrQUFBioqK0rZt20rcZ//+/WrSpMk1HZcCBAAAAPCAzVk6H1crKSlJkhQWFuayHhISosTERLftT58+rV9++UXbtm1Tz5491bFjRw0fPlxHjhy5ouNyDggAAADgw7p163bJ19etW1fienZ2tiS5nevh5+en9PR0t+0PHDggSapYsaJef/11ZWVlaebMmfrjH/+ozz//XDVr1vQoXgoQAAAAoBxyOBySzp8LcuHPkpSbm6sqVaq4bd++fXtt3bpV1apVK15755131LVrVy1dulSPPfaYR8elAAEAAAA8UUqvgnWxDsflXBi9SklJUb169YrXU1JSFBkZWeI+/1t8SFLVqlUVHh6u5ORkj4/LOSAAAABAORQZGamAgABt2bKleC0jI0NxcXGKiYlx237BggVq166dcnJyitcyMzN19OjRKzoxnQIEAAAAKIfsdrv69eunKVOmaN26dYqPj9eoUaMUGhqqHj16qLCwUKmpqcUFR9euXeV0OjV27FgdPHhQP/74o0aMGKHg4GD94Q9/8Pi4FCAAAACAJ5yl9HENRo4cqT59+mjcuHHq27evKlasqNmzZ8tutysxMVEdO3bUypUrJZ0f2frwww917tw59e3bVwMGDFBgYKA++ugjl3NILodzQAAAAIByqmLFihozZozGjBnj9lp4eLj279/vsnbDDTdo9uzZ13RMOiAAAAAATEMHBAAAAPDAtdz0D/9FBwQAAACAaShAAAAAAJiGESwAAADAE4xgeQUdEAAAAACmoQABAAAAYBpGsAAAAAAP2JzMYHkDHRAAAAAApqEAAQAAAGAaRrAAAAAATzCB5RV0QAAAAACYhgIEAAAAgGkYwQIAAAA8YGMEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABroLlHXRAAAAAAJiGAgQAAACAaRjBAgAAADzBCJZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwTbnqgFTqlWp1CMA1af99H6tDKNMKVtSyOgQAAMq8clWAAAAAAFeLq2B5ByNYAAAAAExDAQIAAADANIxgAQAAAJ5wMoPlDXRAAAAAAJiGAgQAAACAaRjBAgAAADzAVbC8gw4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAHrAVWR1B2UAHBAAAAIBpKEAAAAAAmIYRLAAAAMATXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHjAxgiWV9ABAQAAAGAaChAAAAAApmEECwAAAPCEkxksb6ADAgAAAMA0FCAAAAAATMMIFgAAAOABroLlHXRAAAAAAJiGAgQAAACAaRjBAgAAADzBCJZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAnnAyg+UNdEAAAAAAmIYCBAAAAIBpGMECAAAAPMBVsLyDDggAAAAA01CAAAAAADANI1gAAACAJxjB8go6IAAAAABMQwECAAAAwDSMYAEAAAAe4CpY3kEHBAAAAIBpKEAAAAAAmIYRLAAAAMATRcxgeQMdEAAAAACmoQNikvY1IzQ0oocaBoToTN45LT2xVR/+9J8St727TmuNb97nou81cfen+vLnXZKkVV2f0XV+gW7b3PXNKzqVl+md4H0A+TUeOTbfrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt9Fjo1Ffo1HjuGLKEBM0Lx6Pb3R+hGtTfxRsw6uVasaDTQ0oocqyKa5P/3bbfuNKfs18LtZLms2m/TsjffLv5KfNqYekCQF2wN0nV+gpu77Uj+mHXfZPi0/y7Cfp7Qhv8Yjx+Zr0ShMU4f20lc7DmjmZxvVqkkdDb+3gyrYbJq9emuJ+zStW0uSFPvmp8orKChez8krKHH78o4cG4v8Go8cW4AJLK+gADFBbJPbdSAjURN+/FSStPmXg6pkq6A/NeqiBUc3KLfI9S99Wv45paWfc1l7sP6tahBQS4M2v6u0/POvNQ0KkyT9K3mvknLSjP9BSinyazxybL7Bd7fX/oRUPT9vtSRpU9wxVapQQQPubKP563YoN7/QbZ9m4SE6+Uu6dhxMMDtcn0SOjUV+jUeO4as4B8RglW0V1Tq4kf6VvNdlfV3SHvlX8lOrGg0u+x7X2QM0JKKHlhzfor3p//2F0TQwTBn52eX6H27k13jk2HyVK1XUzRHh+mbXIZf1r3cdlL/DrugmdUrcr1l4Le1PSDUjRJ9Hjo1Ffo1HjuHL6IAYrE7VYNkrVNLxrFMu6wm/Pq/nX1NbTh0qaddij0V0V5GzSO8eXOuy3jToep3Nz9brrR5Wm+saq4LNpg2p+zU1/kudyj3r3R+klCK/xiPH5guvWU32ypV0POWMy/qJ1DRJUv2QGtq877jbfk3Da+lI0inNG/OgmtUN0dmsXH2+OU6zPtukgqIiM0L3GeTYWOTXeOTYGtyI0DssLUAWL16se++9V3a7vXht8+bNmjNnjpKSkhQREaGhQ4eqSZMmFkZ5bQIrOyRJ5wpyXNazCvMkSf6V/C65fw27v35/fbQ+PrpBmb95j6aBYQpxBGl5wjb989hGNfCvpcER3fX3trHqt2m6cgrzvfiTlE7k13jk2HyBVc7nNDMnz2U969fn/lXsbvsEB1ZVzWr+cjqdmrZsgxLPZKhts3oacEeMatcI0Li5q40P3IeQY2ORX+ORY/gySwuQ559/Xl27dtV1110nSdqwYYNiY2PVoUMHdezYUXv27FHv3r01d+5ctW7d2spQr5pNtvN/uEjFXOS8dCl9X3gbVbDZtPDoJrfXXtqzWLmFBTpwNlGS9P2Zo/opM1kftB+iu69vrSUntlxT7L6A/BqPHJvPZruQ85JzW9KXlOdycjXk7SU6lnxayWfOXz1s58GTyi8o1PBeHTR71VYdSTptVMg+hxwbi/wajxzDl1l6DojzN39pZs6cqT/96U/64IMPNHbsWH300Ud66KGHNGXKFIsivHYXvvH97bfEVSue/2biXEHuJfe/PfQmbfnlUPFJu//rx7QTxf9wu2B32nGdzc9WRGDotYTtM8iv8cix+c5mn8+pv8P1G8yqvz7PzHHPeW5+obbGHy/+R8UF6/cckSRFhNc0IlSfRY6NRX6NR44t4nSWzoePKVUnoR87dky9evVyWXvwwQcVFxdnUUTXLiHrtAqKChVe9TqX9QvPf8pMuei+IX5BahZ0vb5O+tHttYBKDt1T52Y19A9xe61yhYrl5hKm5Nd45Nh8CalpKigsUt2Q6i7rdWudf/5T4im3ferXrqHenVq4/WPEUfl8ozstM8dtn/KMHBuL/BqPHMOXWVqAFLcPf9WgQQNlZbn+o+PMmTMKDHS/SZmvyCsq0Pdnjqpr6I0u691Cb1JGfrbi0k9cdN+o6nUlST+cOeb2Wn5RocZG3av+jTq7rHcJuUGOinbtOP2TF6Iv/civ8cix+fIKCrXrUIJub+V6/lv36AhlZOVo79Ekt31CqgfouT92U/foCJf1O2KaKjM7V/uOJxsas68hx8Yiv8Yjx/Bllp4D4nQ61a1bNzVs2FCNGzeW3W7X5MmTNX/+fFWuXFk7d+7UxIkT1aVLFyvDvGZzDv9LM9oM1Kut+uqzhB1qUb2e+jXspBn71yi3qED+Ff3UMCBECVmnXcZUmgTUVm5hvk5mu89j5hbl6x9H1mtQk9t1Ki9Tm1MPqklgqGKbdNOGlHhtO3XYzB/RUuTXeOTYfB+s2qpZI3vr9UF3a8V3e9WyUZj+1CNG05avV25+ofwddjUKC9aJ1HSlZWZrx4EEbdt/QqP7dJbDr7KOJp1Wp+YN9dBt0Xpr6bc6m3XpUbnyiBwbi/wajxybj6tgeYfN+dsTMUz0888/a//+/Tpw4EDx/z969Ki2b98uh8Oh6OhoNWvWTLNmzVKNGjWu+XhtVz/rhaivzm0hUYqN6Kb6/rWUmpOhT49v1oKjGyRJrYMb6t22sZr442J9eXJn8T5jo+5V19o36q5/vVrie9pkU+967dS7bjvVqRqs9PwsffXzD3rv0NduN4Yr68iv8cpDjgtW1DL9mJfStWVjDel5i+rXrqGU9HP65N/fa/668/m9OSJc749+QOM/XKPPN58fUw1w2DW45y26rWVj1azmr4TUdC34ZqeWbdxj5Y9RqpFjY5Ff45XFHO+cNcrqEC6q652vWx1Cif615imrQ7gilhYgJcnPz1flypUlSfv371fTpk3dRrWulpUFCIDSr7QVIABQHlGAXDlfK0BK1UnokoqLD0lq1qyZ14oPAAAA4Jo4S+njGhQVFWnatGnq1KmTWrZsqYEDB+rYMfdzN0vy+eefq1mzZkpISLiiY5a6AgQAAACAOWbOnKmFCxdq0qRJWrRokWw2m2JjY5WXl3fJ/U6ePKmJEyde1TEpQAAAAIByKC8vT3PmzNGIESPUpUsXRUZGaurUqUpOTtbatWsvul9RUZHGjBmjG2+88aLbXAoFCAAAAOABm9NZKh9XKz4+XufOnVP79u2L14KCghQVFaVt27ZddL93331X+fn5Gjx48FUd19LL8AIAAAC4Nt26dbvk6+vWrStxPSnp/P1iwsLCXNZDQkKUmJhY4j67d+/WnDlztHjxYiUnX929Y+iAAAAAAOVQdna2JMlut7us+/n5KTfX/b4wWVlZ+utf/6q//vWvatCgwVUflw4IAAAA4IkiqwMo2cU6HJfjcDgknT8X5MKfJSk3N1dVqlRx237SpElq0KCBHnrooasL9FcUIAAAAEA5dGH0KiUlRfXq1SteT0lJUWRkpNv2S5Yskd1uV3R0tCSpsLBQktSzZ0/de++9evHFFz06LgUIAAAAUA5FRkYqICBAW7ZsKS5AMjIyFBcXp379+rlt/9VXX7k8/+GHHzRmzBi99957aty4scfHpQABAAAAPHAtV5wqjex2u/r166cpU6YoODhYderU0eTJkxUaGqoePXqosLBQp0+fVmBgoBwOh+rXr++y/4WT2K+//npdd911Hh+Xk9ABAACAcmrkyJHq06ePxo0bp759+6pixYqaPXu27Ha7EhMT1bFjR61cudKrx7Q5nWWslLuEtquftToEAKVYwYpaVocAAOXezlmjrA7horrd/qrVIZRo3TfPWB3CFWEECwAAAPBEufna3liMYAEAAAAwDQUIAAAAANMwggUAAAB4ovycOm0oOiAAAAAATEMBAgAAAMA0jGABAAAAHrAxgeUVdEAAAAAAmIYCBAAAAIBpGMECAAAAPMFVsLyCDggAAAAA01CAAAAAADANI1gAAACAB2xFVkdQNtABAQAAAGAaChAAAAAApmEECwAAAPAEV8HyCjogAAAAAExDAQIAAADANIxgAQAAAJ5gAssr6IAAAAAAMA0FCAAAAADTMIIFAAAAeMDGVbC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAniiyOoCygQ4IAAAAANNQgAAAAAAwDSNYAAAAgAe4EaF30AEBAAAAYBoKEAAAAACmYQQLAAAA8AQjWF5BBwQAAACAaShAAAAAAJiGESwAAADAE4xgeUW5KkAKVtSyOgQAAACgXGMECwAAAIBpylUHBAAAALhqRVYHUDbQAQEAAABgGgoQAAAAAKZhBAsAAADwgI2rYHkFHRAAAAAApqEAAQAAAGAaRrAAAAAATzCC5RV0QAAAAACYhgIEAAAAgGkYwQIAAAA8wQiWV9ABAQAAAGAaChAAAAAApmEECwAAAPAEI1heQQcEAAAAgGkoQAAAAACYhhEsAAAAwBNFVgdQNtABAQAAAGAaChAAAAAApmEECwAAAPCAjatgeQUdEAAAAACmoQABAAAAYBpGsAAAAABPMILlFXRAAAAAAJiGAgQAAACAaRjBAgAAADxRxAiWN9ABAQAAAGAaChAAAAAApmEECwAAAPAEV8HyCjogAAAAAExDAQIAAADANIxgAQAAAJ5gBMsr6IAAAAAAMA0FCAAAAADTMIIFAAAAeIIRLK+gAwIAAADANBQgAAAAAEzDCBYAAADgiSJGsLyBDggAAAAA01CAAAAAADANI1gAAACAJ5xFVkdQJtABAQAAAGAaChAAAAAApmEECwAAAPAENyL0CjogAAAAAExDAQIAAADANIxgAQAAAJ7gRoReQQFikVuj6mvYvR3UMCxYaWeztXj9bs1ds+2i2zeoXUNLJwxwWz+SdFq9J35oYKS+ixwbi/wajxwbjxwbi/wajxzDF1GAWKBFozBNHdpLX+04oJmfbVSrJnU0/N4OqmCzafbqrSXu07RuLUlS7JufKq+goHg9J6+gxO3LO3JsLPJrPHJsPHJsLPJrPHIMX0UBYoHBd7fX/oRUPT9vtSRpU9wxVapQQQPubKP563YoN7/QbZ9m4SE6+Uu6dhxMMDtcn0SOjUV+jUeOjUeOjUV+jUeOLcBVsLyCk9BNVrlSRd0cEa5vdh1yWf9610H5O+yKblKnxP2ahdfS/oRUM0L0eeTYWOTXeOTYeOTYWOTXeOQYvowOiMnCa1aTvXIlHU8547J+IjVNklQ/pIY27zvutl/T8Fo6knRK88Y8qGZ1Q3Q2K1efb47TrM82qaCoyIzQfQY5Nhb5NR45Nh45Nhb5NR45hi+zvAD54YcftGXLFj322GOSpM2bN2vevHlKSEhQvXr1NHDgQMXExFgcpfcEVvGTJGXm5LmsZ/363L+K3W2f4MCqqlnNX06nU9OWbVDimQy1bVZPA+6IUe0aARo3d7XxgfsQcmws8ms8cmw8cmws8ms8cmwRRrC8wtICZPXq1Ro9erRuvfVWPfbYY/rXv/6lYcOGqXPnzurSpYsOHDig/v37a8aMGeratauVoXqNzWY7/4eLfIBL+vLhXE6uhry9RMeSTyv5TKYkaefBk8ovKNTwXh00e9VWHUk6bVTIPoccG4v8Go8cG48cG4v8Go8cw1uKioo0Y8YMffrpp8rIyNDNN9+s8ePHq379+iVuv2fPHk2ePFm7d++Wn5+f7rjjDv31r39VUFCQx8e09ByQGTNm6PHHH9cHH3wgSZo1a5aGDBmiv//97xozZozef/99Pf7445o2bZqVYXrV2excSZK/w/Wbiaq/Ps/MyXXbJze/UFvjjxf/srhg/Z4jkqSI8JpGhOqzyLGxyK/xyLHxyLGxyK/xyDG8ZebMmVq4cKEmTZqkRYsWyWazKTY2Vnl5eW7bpqSk6M9//rPq1aunZcuWaebMmdq5c6eeeuqpKzqmpQXI8ePHdc899xQ/T0hI0J133umyTc+ePXX48GGzQzNMQmqaCgqLVDekust63Vrnn/+UeMptn/q1a6h3pxZuv2Qclc83sNIycwyJ1VeRY2ORX+ORY+ORY2ORX+ORY4s4naXzcZXy8vI0Z84cjRgxQl26dFFkZKSmTp2q5ORkrV271m37kydPqlOnTho/frwaNGig1q1b64EHHtB33313Rce1tACpW7eu/vOf/xQ/v+GGGxQfH++yze7du1W7dm2zQzNMXkGhdh1K0O2tmrisd4+OUEZWjvYeTXLbJ6R6gJ77Yzd1j45wWb8jpqkys3O173iyoTH7GnJsLPJrPHJsPHJsLPJrPHIMb4iPj9e5c+fUvn374rWgoCBFRUVp2zb3G1pGR0frzTffVKVK54vWQ4cOadmyZerQocMVHdfSc0BiY2P13HPPKSkpST179tSwYcP09NNPKzc3VxEREfrhhx/0zjvv6PHHH7cyTK/7YNVWzRrZW68Pulsrvturlo3C9KceMZq2fL1y8wvl77CrUViwTqSmKy0zWzsOJGjb/hMa3aezHH6VdTTptDo1b6iHbovWW0u/1dks9zZreUeOjUV+jUeOjUeOjUV+jUeOcUG3bt0u+fq6detKXE9KOl+ohoWFuayHhIQoMTHxku9555136ujRo6pTp45mzpx5BdFKNqfT2tP5V6xYoWnTpunkyZOy2Wz633D8/f01aNAgDR061CvHaj10qlfexxu6tmysIT1vUf3aNZSSfk6f/Pt7zV+3U5J0c0S43h/9gMZ/uEafb46TJAU47Brc8xbd1rKxalbzV0JquhZ8s1PLNu6x8sco1cixsciv8cix8cixsciv8cpijnfOGmV1CBd1V9hwq0MoUV5U/CVfv1gBsmLFCo0dO1b79u1ThQr/HYwaO3asUlJSNG/evIu+548//qicnBxNmTJFp06d0ooVK+Tv7+9RvJYXIBf89NNPOnr0qDIzM1W5cmWFhoYqKipKfn5+XjtGaSpAAAAA4I4C5MqtSnznqvZbs2aNRo4cqR9++EEOh6N4/S9/+Yvy8vI0a9asy75HSkqKunTpoldffVX33XefR8e1/D4gFzRq1EiNGjWyOgwAAACgXLgwepWSkqJ69eoVr6ekpCgyMtJt+8OHDyshIUFdunQpXgsJCVG1atWUnOz5OUSWnoQOAAAA+Ayrr3bl5atgRUZGKiAgQFu2bCley8jIUFxcXIk3Al+/fr3+8pe/KDPzv5dyPn78uM6cOaPGjRt7fFwKEAAAAKAcstvt6tevn6ZMmaJ169YpPj5eo0aNUmhoqHr06KHCwkKlpqYqJ+f8JZp79eqlwMBAjRkzRgcPHtT27ds1cuRItWjR4opuGk4BAgAAAJRTI0eOVJ8+fTRu3Dj17dtXFStW1OzZs2W325WYmKiOHTtq5cqVkqQaNWroo48+UlFRkfr27avhw4crKipKs2fPVsWKFT0+Zqk5Cd0MnIQOAABQupXqk9Bre+fKrN62KvnyJ4uXJnRAAAAAAJiGAgQAAACAaUrNZXgBAACAUq2o3Jy5YCg6IAAAAABMQwECAAAAwDSMYAEAAAAecDqLrA6hTKADAgAAAMA0FCAAAAAATMMIFgAAAOAJroLlFXRAAAAAAJiGAgQAAACAaRjBAgAAADzhZATLG+iAAAAAADANBQgAAAAA0zCCBQAAAHiiiBsRegMdEAAAAACmoQABAAAAYBpGsAAAAABPcBUsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABJ1fB8go6IAAAAABMQwECAAAAwDSMYAEAAACe4CpYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMATRYxgeQMdEAAAAACmoQABAAAAYBpGsAAAAABPOLkRoTfQAQEAAABgGgoQAAAAAKZhBAsAAADwgJOrYHkFHRAAAAAApqEAAQAAAGAaRrAAAAAAT3AVLK+gAwIAAADANBQgAAAAAEzDCBYAAADgAa6C5R10QAAAAACYhgIEAAAAgGkYwQIAAAA8wVWwvIIOCAAAAADTUIAAAAAAMI3N6XRyOj8AAAAAU9ABAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaCpBSpqioSNOmTVOnTp3UsmVLDRw4UMeOHbM6rDJr5syZeuSRR6wOo0xJS0vTCy+8oM6dO6t169bq27evtm/fbnVYZcqpU6c0ZswYtW/fXtHR0Xrsscd06NAhq8Mqk44cOaLo6GgtXbrU6lDKlJMnT6pZs2Zuj08//dTq0MqU5cuX6/e//72aN2+uu+++W6tWrbI6JEASBUipM3PmTC1cuFCTJk3SokWLZLPZFBsbq7y8PKtDK3PmzZunadOmWR1GmTN69Gj98MMPevPNN7V48WLdeOONevTRR3X48GGrQyszhg4dqhMnTuj999/X4sWL5XA4NGDAAGVnZ1sdWpmSn5+vv/71r8rKyrI6lDJn//798vPz0/r167Vhw4bixz333GN1aGXGihUr9Oyzz+rBBx/UF198od///vcaPXq0du3aZXVoAAVIaZKXl6c5c+ZoxIgR6tKliyIjIzV16lQlJydr7dq1VodXZiQnJ2vQoEF6++231bBhQ6vDKVOOHTumjRs3avz48YqJiVGjRo303HPPqXbt2vriiy+sDq9MOHPmjMLDw/XSSy+pefPmaty4sYYNG6bU1FQdPHjQ6vDKlOnTp8vf39/qMMqkAwcOqGHDhgoJCVGtWrWKHw6Hw+rQygSn06m3335b/fv3V//+/VW/fn0NHz5ct956q7Zu3Wp1eAAFSGkSHx+vc+fOqX379sVrQUFBioqK0rZt2yyMrGzZu3evqlWrps8++0wtW7a0OpwypUaNGnrvvfd00003Fa/ZbDY5nU6lp6dbGFnZUaNGDb355puKiIiQJP3yyy+aPXu2QkND1aRJE4ujKzu2bdumRYsW6fXXX7c6lDJp//79fF4N9NNPP+nkyZNuHaXZs2dr8ODBFkUF/FclqwPAfyUlJUmSwsLCXNZDQkKUmJhoRUhl0u23367bb7/d6jDKpKCgIHXp0sVlbdWqVTp+/Lg6duxoUVRl1/PPP69PPvlEdrtds2bNUtWqVa0OqUzIyMjQ2LFjNW7cOLffx/COAwcOqFatWvrjH/+oo0ePqn79+ho2bJg6depkdWhlwtGjRyVJWVlZevTRRxUXF6fw8HANHTqU//6hVKADUopcmN+22+0u635+fsrNzbUiJOCa7NixQ88++6y6devGf/QM0L9/fy1ZskT33nuvhg8frr1791odUpkwYcIEtWrVivMRDJKXl6ejR48qMzNTTzzxhN577z01b95csbGx+u6776wOr0zIzMyUJD311FPq2bOn5syZow4dOmjYsGHkGKUCHZBS5MLsa15ensscbG5urqpUqWJVWMBV+frrr/XXv/5VLVu21Jtvvml1OGXShRGWl156Sd9//73mz5+vV1991eKofNvy5cu1fft2ff7551aHUmbZ7XZt27ZNlSpVKv7C7aabbtLhw4c1e/Zs3XLLLRZH6PsqV64sSXr00Uf1hz/8QZJ0ww03KC4uTnPnziXHsBwdkFLkQqs/JSXFZT0lJUWhoaFWhARclfnz52vEiBHq3Lmz3n//fU4s9aJTp07piy++UGFhYfFahQoV1LhxY7ffHbhyS5Ys0alTp3TbbbcpOjpa0dHRkqTx48fr7rvvtji6sqNq1apu3f6mTZsqOTnZoojKlgv/ZmjatKnLepMmTZSQkGBFSIALCpBSJDIyUgEBAdqyZUvxWkZGhuLi4hQTE2NhZIDnFixYoJdeekkPP/yw3nrrLbd/ZODapKSk6Mknn3S5kk1+fr7i4uLUuHFjCyMrG6ZMmaKVK1dq+fLlxQ9JGjlypN577z1rgysj4uPjFR0d7XZ/oD179nBiupdERUXJ399fP/zwg8v6gQMHVK9ePYuiAv6LEaxSxG63q1+/fpoyZYqCg4NVp04dTZ48WaGhoerRo4fV4QGXdeTIEb3yyivq0aOHBg8erFOnThW/5nA4FBgYaGF0ZUNkZKQ6duyoiRMnatKkSQoKCtK7776rjIwMDRgwwOrwfF7t2rVLXL/uuutUp04dk6Mpm5o2baqIiAhNnDhR48ePV40aNfTJJ5/o+++/1+LFi60Or0xwOBwaNGiQ3nnnHdWuXVstWrTQl19+qY0bN2revHlWhwdQgJQ2I0eOVEFBgcaNG6ecnBy1adNGs2fP5ltk+IQ1a9YoPz9fa9eudbt3zR/+8Ae99tprFkVWdthsNr311lt644039MQTT+js2bOKiYnRxx9/rOuvv97q8IDLqlChgt59911NmTJFTzzxhDIyMhQVFaW5c+eqWbNmVodXZgwbNkxVqlQpvp9Y48aNNX36dLVr187q0ADZnE6n0+ogAAAAAJQPnAMCAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAD4mNzdXUVFRio6O1ksvvWR1OAAAXBEKEADwMTabTR9++KFatGih+fPn68iRI1aHBACAxyhAAMDH2O12tWnTRoMGDZIk7d271+KIAADwHAUIAPioRo0aSZL27dtncSQAAHiOAgQAfNT7778vSYqPj7c4EgAAPEcBAgA+aMOGDfrnP/+patWqKS4uzupwAADwGAUIAPiYjIwMPfvss+rWrZv69u2r06dPKzk52eqwAADwCAUIAPiYiRMnqqCgQJMmTVJUVJQkxrAAAL6DAgQAfMjq1av1xRdf6OWXX1ZwcHBxAcKJ6AAAX0EBAgA+IjU1VePHj9eDDz6orl27SpLq1q2roKAgzgMBAPgMChAA8BHPP/+8qlWrpqefftpl/YYbbmAECwDgMyhAAMAHfPrpp/r222/1t7/9TVWrVnV5LSoqSsePH1dmZqZF0QEA4Dmb0+l0Wh0EAAAAgPKBDggAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADDN/wOpUE87FNLINAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "filenames": { "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week43_30_3.png" } }, "output_type": "display_data" } ], "source": [ "# import necessary packages\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.neural_network import MLPClassifier\n", "from sklearn.metrics import accuracy_score\n", "import seaborn as sns\n", "\n", "# ensure the same random numbers appear every time\n", "np.random.seed(0)\n", "\n", "# Design matrix\n", "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", "\n", "# The XOR gate\n", "yXOR = np.array( [ 0, 1 ,1, 0])\n", "# The OR gate\n", "yOR = np.array( [ 0, 1 ,1, 1])\n", "# The AND gate\n", "yAND = np.array( [ 0, 0 ,0, 1])\n", "\n", "# Defining the neural network\n", "n_hidden_neurons = 2\n", "\n", "eta_vals = np.logspace(-5, 1, 7)\n", "lmbd_vals = np.logspace(-5, 1, 7)\n", "# store models for later use\n", "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", "epochs = 100\n", "\n", "for i, eta in enumerate(eta_vals):\n", " for j, lmbd in enumerate(lmbd_vals):\n", " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n", " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n", " dnn.fit(X, yXOR)\n", " DNN_scikit[i][j] = dnn\n", " print(\"Learning rate = \", eta)\n", " print(\"Lambda = \", lmbd)\n", " print(\"Accuracy score on data set: \", dnn.score(X, yXOR))\n", " print()\n", "\n", "sns.set()\n", "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "for i in range(len(eta_vals)):\n", " for j in range(len(lmbd_vals)):\n", " dnn = DNN_scikit[i][j]\n", " test_pred = dnn.predict(X)\n", " test_accuracy[i][j] = accuracy_score(yXOR, test_pred)\n", "\n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Test Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "9f870fcf", "metadata": { "editable": true }, "source": [ "## Building a neural network code\n", "\n", "Here we present a flexible object oriented codebase\n", "for a feed forward neural network, along with a demonstration of how\n", "to use it. Before we get into the details of the neural network, we\n", "will first present some implementations of various schedulers, cost\n", "functions and activation functions that can be used together with the\n", "neural network.\n", "\n", "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023." ] }, { "cell_type": "markdown", "id": "75623483", "metadata": { "editable": true }, "source": [ "### Learning rate methods\n", "\n", "The code below shows object oriented implementations of the Constant,\n", "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n", "of the classes belong to the shared abstract Scheduler class, and\n", "share the update_change() and reset() methods allowing for any of the\n", "schedulers to be seamlessly used during the training stage, as will\n", "later be shown in the fit() method of the neural\n", "network. Update_change() only has one parameter, the gradient\n", "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n", "from the weights. The reset() function takes no parameters, and resets\n", "the desired variables. For Constant and Momentum, reset does nothing." ] }, { "cell_type": "code", "execution_count": 3, "id": "7884cf44", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "\n", "class Scheduler:\n", " \"\"\"\n", " Abstract class for Schedulers\n", " \"\"\"\n", "\n", " def __init__(self, eta):\n", " self.eta = eta\n", "\n", " # should be overwritten\n", " def update_change(self, gradient):\n", " raise NotImplementedError\n", "\n", " # overwritten if needed\n", " def reset(self):\n", " pass\n", "\n", "\n", "class Constant(Scheduler):\n", " def __init__(self, eta):\n", " super().__init__(eta)\n", "\n", " def update_change(self, gradient):\n", " return self.eta * gradient\n", " \n", " def reset(self):\n", " pass\n", "\n", "\n", "class Momentum(Scheduler):\n", " def __init__(self, eta: float, momentum: float):\n", " super().__init__(eta)\n", " self.momentum = momentum\n", " self.change = 0\n", "\n", " def update_change(self, gradient):\n", " self.change = self.momentum * self.change + self.eta * gradient\n", " return self.change\n", "\n", " def reset(self):\n", " pass\n", "\n", "\n", "class Adagrad(Scheduler):\n", " def __init__(self, eta):\n", " super().__init__(eta)\n", " self.G_t = None\n", "\n", " def update_change(self, gradient):\n", " delta = 1e-8 # avoid division ny zero\n", "\n", " if self.G_t is None:\n", " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", "\n", " self.G_t += gradient @ gradient.T\n", "\n", " G_t_inverse = 1 / (\n", " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", " )\n", " return self.eta * gradient * G_t_inverse\n", "\n", " def reset(self):\n", " self.G_t = None\n", "\n", "\n", "class AdagradMomentum(Scheduler):\n", " def __init__(self, eta, momentum):\n", " super().__init__(eta)\n", " self.G_t = None\n", " self.momentum = momentum\n", " self.change = 0\n", "\n", " def update_change(self, gradient):\n", " delta = 1e-8 # avoid division ny zero\n", "\n", " if self.G_t is None:\n", " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", "\n", " self.G_t += gradient @ gradient.T\n", "\n", " G_t_inverse = 1 / (\n", " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", " )\n", " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n", " return self.change\n", "\n", " def reset(self):\n", " self.G_t = None\n", "\n", "\n", "class RMS_prop(Scheduler):\n", " def __init__(self, eta, rho):\n", " super().__init__(eta)\n", " self.rho = rho\n", " self.second = 0.0\n", "\n", " def update_change(self, gradient):\n", " delta = 1e-8 # avoid division ny zero\n", " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n", " return self.eta * gradient / (np.sqrt(self.second + delta))\n", "\n", " def reset(self):\n", " self.second = 0.0\n", "\n", "\n", "class Adam(Scheduler):\n", " def __init__(self, eta, rho, rho2):\n", " super().__init__(eta)\n", " self.rho = rho\n", " self.rho2 = rho2\n", " self.moment = 0\n", " self.second = 0\n", " self.n_epochs = 1\n", "\n", " def update_change(self, gradient):\n", " delta = 1e-8 # avoid division ny zero\n", "\n", " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n", " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n", "\n", " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n", " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n", "\n", " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n", "\n", " def reset(self):\n", " self.n_epochs += 1\n", " self.moment = 0\n", " self.second = 0" ] }, { "cell_type": "markdown", "id": "a9747db3", "metadata": { "editable": true }, "source": [ "### Usage of the above learning rate schedulers\n", "\n", "To initalize a scheduler, simply create the object and pass in the\n", "necessary parameters such as the learning rate and the momentum as\n", "shown below. As the Scheduler class is an abstract class it should not\n", "called directly, and will raise an error upon usage." ] }, { "cell_type": "code", "execution_count": 4, "id": "b394ebc2", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)" ] }, { "cell_type": "markdown", "id": "734ce228", "metadata": { "editable": true }, "source": [ "Here is a small example for how a segment of code using schedulers\n", "could look. Switching out the schedulers is simple." ] }, { "cell_type": "code", "execution_count": 5, "id": "51396f90", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Before scheduler:\n", "weights=array([[1., 1., 1.],\n", " [1., 1., 1.],\n", " [1., 1., 1.]])\n", "\n", "After scheduler:\n", "weights=array([[0.993993 , 0.993993 , 0.99399301],\n", " [0.99399308, 0.99399315, 0.99399301],\n", " [0.99399301, 0.99399309, 0.99399301]])\n" ] } ], "source": [ "weights = np.ones((3,3))\n", "print(f\"Before scheduler:\\n{weights=}\")\n", "\n", "epochs = 10\n", "for e in range(epochs):\n", " gradient = np.random.rand(3, 3)\n", " change = adam_scheduler.update_change(gradient)\n", " weights = weights - change\n", " adam_scheduler.reset()\n", "\n", "print(f\"\\nAfter scheduler:\\n{weights=}\")" ] }, { "cell_type": "markdown", "id": "44646fc6", "metadata": { "editable": true }, "source": [ "### Cost functions\n", "\n", "Here we discuss cost functions that can be used when creating the\n", "neural network. Every cost function takes the target vector as its\n", "parameter, and returns a function valued only at $x$ such that it may\n", "easily be differentiated." ] }, { "cell_type": "code", "execution_count": 6, "id": "912285f7", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "\n", "def CostOLS(target):\n", " \n", " def func(X):\n", " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n", "\n", " return func\n", "\n", "\n", "def CostLogReg(target):\n", "\n", " def func(X):\n", " \n", " return -(1.0 / target.shape[0]) * np.sum(\n", " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n", " )\n", "\n", " return func\n", "\n", "\n", "def CostCrossEntropy(target):\n", " \n", " def func(X):\n", " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n", "\n", " return func" ] }, { "cell_type": "markdown", "id": "9297b955", "metadata": { "editable": true }, "source": [ "Below we give a short example of how these cost function may be used\n", "to obtain results if you wish to test them out on your own using\n", "AutoGrad's automatics differentiation." ] }, { "cell_type": "code", "execution_count": 7, "id": "39943643", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Derivative of cost function CostCrossEntropy valued at a:\n", "[[-0.08333333]\n", " [-0.13333333]\n", " [-0.16666667]]\n" ] } ], "source": [ "from autograd import grad\n", "\n", "target = np.array([[1, 2, 3]]).T\n", "a = np.array([[4, 5, 6]]).T\n", "\n", "cost_func = CostCrossEntropy\n", "cost_func_derivative = grad(cost_func(target))\n", "\n", "valued_at_a = cost_func_derivative(a)\n", "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")" ] }, { "cell_type": "markdown", "id": "53f4d1d2", "metadata": { "editable": true }, "source": [ "### Activation functions\n", "\n", "Finally, before we look at the neural network, we will look at the\n", "activation functions which can be specified between the hidden layers\n", "and as the output function. Each function can be valued for any given\n", "vector or matrix X, and can be differentiated via derivate()." ] }, { "cell_type": "code", "execution_count": 8, "id": "90222b1a", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import elementwise_grad\n", "\n", "def identity(X):\n", " return X\n", "\n", "\n", "def sigmoid(X):\n", " try:\n", " return 1.0 / (1 + np.exp(-X))\n", " except FloatingPointError:\n", " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n", "\n", "\n", "def softmax(X):\n", " X = X - np.max(X, axis=-1, keepdims=True)\n", " delta = 10e-10\n", " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n", "\n", "\n", "def RELU(X):\n", " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n", "\n", "\n", "def LRELU(X):\n", " delta = 10e-4\n", " return np.where(X > np.zeros(X.shape), X, delta * X)\n", "\n", "\n", "def derivate(func):\n", " if func.__name__ == \"RELU\":\n", "\n", " def func(X):\n", " return np.where(X > 0, 1, 0)\n", "\n", " return func\n", "\n", " elif func.__name__ == \"LRELU\":\n", "\n", " def func(X):\n", " delta = 10e-4\n", " return np.where(X > 0, 1, delta)\n", "\n", " return func\n", "\n", " else:\n", " return elementwise_grad(func)" ] }, { "cell_type": "markdown", "id": "f3dd3611", "metadata": { "editable": true }, "source": [ "Below follows a short demonstration of how to use an activation\n", "function. The derivative of the activation function will be important\n", "when calculating the output delta term during backpropagation. Note\n", "that derivate() can also be used for cost functions for a more\n", "generalized approach." ] }, { "cell_type": "code", "execution_count": 9, "id": "ded43dca", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Input to activation function:\n", "[[4]\n", " [5]\n", " [6]]\n", "\n", "Output from sigmoid activation function:\n", "[[0.98201379]\n", " [0.99330715]\n", " [0.99752738]]\n", "\n", "Derivative of sigmoid activation function valued at z:\n", "[[0.19824029]\n", " [0.19721923]\n", " [0.19683648]]\n" ] } ], "source": [ "z = np.array([[4, 5, 6]]).T\n", "print(f\"Input to activation function:\\n{z}\")\n", "\n", "act_func = sigmoid\n", "a = act_func(z)\n", "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n", "\n", "act_func_derivative = derivate(act_func)\n", "valued_at_z = act_func_derivative(a)\n", "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")" ] }, { "cell_type": "markdown", "id": "771c53a1", "metadata": { "editable": true }, "source": [ "### The Neural Network\n", "\n", "Now that we have gotten a good understanding of the implementation of\n", "some important components, we can take a look at an object oriented\n", "implementation of a feed forward neural network. The feed forward\n", "neural network has been implemented as a class named FFNN, which can\n", "be initiated as a regressor or classifier dependant on the choice of\n", "cost function. The FFNN can have any number of input nodes, hidden\n", "layers with any amount of hidden nodes, and any amount of output nodes\n", "meaning it can perform multiclass classification as well as binary\n", "classification and regression problems. Although there is a lot of\n", "code present, it makes for an easy to use and generalizeable interface\n", "for creating many types of neural networks as will be demonstrated\n", "below." ] }, { "cell_type": "code", "execution_count": 10, "id": "56ef870b", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import math\n", "import autograd.numpy as np\n", "import sys\n", "import warnings\n", "from autograd import grad, elementwise_grad\n", "from random import random, seed\n", "from copy import deepcopy, copy\n", "from typing import Tuple, Callable\n", "from sklearn.utils import resample\n", "\n", "warnings.simplefilter(\"error\")\n", "\n", "\n", "class FFNN:\n", " \"\"\"\n", " Description:\n", " ------------\n", " Feed Forward Neural Network with interface enabling flexible design of a\n", " nerual networks architecture and the specification of activation function\n", " in the hidden layers and output layer respectively. This model can be used\n", " for both regression and classification problems, depending on the output function.\n", "\n", " Attributes:\n", " ------------\n", " I dimensions (tuple[int]): A list of positive integers, which specifies the\n", " number of nodes in each of the networks layers. The first integer in the array\n", " defines the number of nodes in the input layer, the second integer defines number\n", " of nodes in the first hidden layer and so on until the last number, which\n", " specifies the number of nodes in the output layer.\n", " II hidden_func (Callable): The activation function for the hidden layers\n", " III output_func (Callable): The activation function for the output layer\n", " IV cost_func (Callable): Our cost function\n", " V seed (int): Sets random seed, makes results reproducible\n", " \"\"\"\n", "\n", " def __init__(\n", " self,\n", " dimensions: tuple[int],\n", " hidden_func: Callable = sigmoid,\n", " output_func: Callable = lambda x: x,\n", " cost_func: Callable = CostOLS,\n", " seed: int = None,\n", " ):\n", " self.dimensions = dimensions\n", " self.hidden_func = hidden_func\n", " self.output_func = output_func\n", " self.cost_func = cost_func\n", " self.seed = seed\n", " self.weights = list()\n", " self.schedulers_weight = list()\n", " self.schedulers_bias = list()\n", " self.a_matrices = list()\n", " self.z_matrices = list()\n", " self.classification = None\n", "\n", " self.reset_weights()\n", " self._set_classification()\n", "\n", " def fit(\n", " self,\n", " X: np.ndarray,\n", " t: np.ndarray,\n", " scheduler: Scheduler,\n", " batches: int = 1,\n", " epochs: int = 100,\n", " lam: float = 0,\n", " X_val: np.ndarray = None,\n", " t_val: np.ndarray = None,\n", " ):\n", " \"\"\"\n", " Description:\n", " ------------\n", " This function performs the training the neural network by performing the feedforward and backpropagation\n", " algorithm to update the networks weights.\n", "\n", " Parameters:\n", " ------------\n", " I X (np.ndarray) : training data\n", " II t (np.ndarray) : target data\n", " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n", " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n", "\n", " Optional Parameters:\n", " ------------\n", " V batches (int) : number of batches the datasets are split into, default equal to 1\n", " VI epochs (int) : number of iterations used to train the network, default equal to 100\n", " VII lam (float) : regularization hyperparameter lambda\n", " VIII X_val (np.ndarray) : validation set\n", " IX t_val (np.ndarray) : validation target set\n", "\n", " Returns:\n", " ------------\n", " I scores (dict) : A dictionary containing the performance metrics of the model.\n", " The number of the metrics depends on the parameters passed to the fit-function.\n", "\n", " \"\"\"\n", "\n", " # setup \n", " if self.seed is not None:\n", " np.random.seed(self.seed)\n", "\n", " val_set = False\n", " if X_val is not None and t_val is not None:\n", " val_set = True\n", "\n", " # creating arrays for score metrics\n", " train_errors = np.empty(epochs)\n", " train_errors.fill(np.nan)\n", " val_errors = np.empty(epochs)\n", " val_errors.fill(np.nan)\n", "\n", " train_accs = np.empty(epochs)\n", " train_accs.fill(np.nan)\n", " val_accs = np.empty(epochs)\n", " val_accs.fill(np.nan)\n", "\n", " self.schedulers_weight = list()\n", " self.schedulers_bias = list()\n", "\n", " batch_size = X.shape[0] // batches\n", "\n", " X, t = resample(X, t)\n", "\n", " # this function returns a function valued only at X\n", " cost_function_train = self.cost_func(t)\n", " if val_set:\n", " cost_function_val = self.cost_func(t_val)\n", "\n", " # create schedulers for each weight matrix\n", " for i in range(len(self.weights)):\n", " self.schedulers_weight.append(copy(scheduler))\n", " self.schedulers_bias.append(copy(scheduler))\n", "\n", " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n", "\n", " try:\n", " for e in range(epochs):\n", " for i in range(batches):\n", " # allows for minibatch gradient descent\n", " if i == batches - 1:\n", " # If the for loop has reached the last batch, take all thats left\n", " X_batch = X[i * batch_size :, :]\n", " t_batch = t[i * batch_size :, :]\n", " else:\n", " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n", " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n", "\n", " self._feedforward(X_batch)\n", " self._backpropagate(X_batch, t_batch, lam)\n", "\n", " # reset schedulers for each epoch (some schedulers pass in this call)\n", " for scheduler in self.schedulers_weight:\n", " scheduler.reset()\n", "\n", " for scheduler in self.schedulers_bias:\n", " scheduler.reset()\n", "\n", " # computing performance metrics\n", " pred_train = self.predict(X)\n", " train_error = cost_function_train(pred_train)\n", "\n", " train_errors[e] = train_error\n", " if val_set:\n", " \n", " pred_val = self.predict(X_val)\n", " val_error = cost_function_val(pred_val)\n", " val_errors[e] = val_error\n", "\n", " if self.classification:\n", " train_acc = self._accuracy(self.predict(X), t)\n", " train_accs[e] = train_acc\n", " if val_set:\n", " val_acc = self._accuracy(pred_val, t_val)\n", " val_accs[e] = val_acc\n", "\n", " # printing progress bar\n", " progression = e / epochs\n", " print_length = self._progress_bar(\n", " progression,\n", " train_error=train_errors[e],\n", " train_acc=train_accs[e],\n", " val_error=val_errors[e],\n", " val_acc=val_accs[e],\n", " )\n", " except KeyboardInterrupt:\n", " # allows for stopping training at any point and seeing the result\n", " pass\n", "\n", " # visualization of training progression (similiar to tensorflow progression bar)\n", " sys.stdout.write(\"\\r\" + \" \" * print_length)\n", " sys.stdout.flush()\n", " self._progress_bar(\n", " 1,\n", " train_error=train_errors[e],\n", " train_acc=train_accs[e],\n", " val_error=val_errors[e],\n", " val_acc=val_accs[e],\n", " )\n", " sys.stdout.write(\"\")\n", "\n", " # return performance metrics for the entire run\n", " scores = dict()\n", "\n", " scores[\"train_errors\"] = train_errors\n", "\n", " if val_set:\n", " scores[\"val_errors\"] = val_errors\n", "\n", " if self.classification:\n", " scores[\"train_accs\"] = train_accs\n", "\n", " if val_set:\n", " scores[\"val_accs\"] = val_accs\n", "\n", " return scores\n", "\n", " def predict(self, X: np.ndarray, *, threshold=0.5):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Performs prediction after training of the network has been finished.\n", "\n", " Parameters:\n", " ------------\n", " I X (np.ndarray): The design matrix, with n rows of p features each\n", "\n", " Optional Parameters:\n", " ------------\n", " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n", " in classification problems\n", "\n", " Returns:\n", " ------------\n", " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", " This vector is thresholded if regression=False, meaning that classification results\n", " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n", "\n", " \"\"\"\n", "\n", " predict = self._feedforward(X)\n", "\n", " if self.classification:\n", " return np.where(predict > threshold, 1, 0)\n", " else:\n", " return predict\n", "\n", " def reset_weights(self):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Resets/Reinitializes the weights in order to train the network for a new problem.\n", "\n", " \"\"\"\n", " if self.seed is not None:\n", " np.random.seed(self.seed)\n", "\n", " self.weights = list()\n", " for i in range(len(self.dimensions) - 1):\n", " weight_array = np.random.randn(\n", " self.dimensions[i] + 1, self.dimensions[i + 1]\n", " )\n", " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n", "\n", " self.weights.append(weight_array)\n", "\n", " def _feedforward(self, X: np.ndarray):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Calculates the activation of each layer starting at the input and ending at the output.\n", " Each following activation is calculated from a weighted sum of each of the preceeding\n", " activations (except in the case of the input layer).\n", "\n", " Parameters:\n", " ------------\n", " I X (np.ndarray): The design matrix, with n rows of p features each\n", "\n", " Returns:\n", " ------------\n", " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", " \"\"\"\n", "\n", " # reset matrices\n", " self.a_matrices = list()\n", " self.z_matrices = list()\n", "\n", " # if X is just a vector, make it into a matrix\n", " if len(X.shape) == 1:\n", " X = X.reshape((1, X.shape[0]))\n", "\n", " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n", " # to add bias to our data\n", " bias = np.ones((X.shape[0], 1)) * 0.01\n", " X = np.hstack([bias, X])\n", "\n", " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n", " # exponent indicates layer number).\n", " a = X\n", " self.a_matrices.append(a)\n", " self.z_matrices.append(a)\n", "\n", " # The feed forward algorithm\n", " for i in range(len(self.weights)):\n", " if i < len(self.weights) - 1:\n", " z = a @ self.weights[i]\n", " self.z_matrices.append(z)\n", " a = self.hidden_func(z)\n", " # bias column again added to the data here\n", " bias = np.ones((a.shape[0], 1)) * 0.01\n", " a = np.hstack([bias, a])\n", " self.a_matrices.append(a)\n", " else:\n", " try:\n", " # a^L, the nodes in our output layers\n", " z = a @ self.weights[i]\n", " a = self.output_func(z)\n", " self.a_matrices.append(a)\n", " self.z_matrices.append(z)\n", " except Exception as OverflowError:\n", " print(\n", " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n", " )\n", "\n", " # this will be a^L\n", " return a\n", "\n", " def _backpropagate(self, X, t, lam):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Performs the backpropagation algorithm. In other words, this method\n", " calculates the gradient of all the layers starting at the\n", " output layer, and moving from right to left accumulates the gradient until\n", " the input layer is reached. Each layers respective weights are updated while\n", " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n", "\n", " Parameters:\n", " ------------\n", " I X (np.ndarray): The design matrix, with n rows of p features each.\n", " II t (np.ndarray): The target vector, with n rows of p targets.\n", " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n", "\n", " Returns:\n", " ------------\n", " No return value.\n", "\n", " \"\"\"\n", " out_derivative = derivate(self.output_func)\n", " hidden_derivative = derivate(self.hidden_func)\n", "\n", " for i in range(len(self.weights) - 1, -1, -1):\n", " # delta terms for output\n", " if i == len(self.weights) - 1:\n", " # for multi-class classification\n", " if (\n", " self.output_func.__name__ == \"softmax\"\n", " ):\n", " delta_matrix = self.a_matrices[i + 1] - t\n", " # for single class classification\n", " else:\n", " cost_func_derivative = grad(self.cost_func(t))\n", " delta_matrix = out_derivative(\n", " self.z_matrices[i + 1]\n", " ) * cost_func_derivative(self.a_matrices[i + 1])\n", "\n", " # delta terms for hidden layer\n", " else:\n", " delta_matrix = (\n", " self.weights[i + 1][1:, :] @ delta_matrix.T\n", " ).T * hidden_derivative(self.z_matrices[i + 1])\n", "\n", " # calculate gradient\n", " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n", " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n", " 1, delta_matrix.shape[1]\n", " )\n", "\n", " # regularization term\n", " gradient_weights += self.weights[i][1:, :] * lam\n", "\n", " # use scheduler\n", " update_matrix = np.vstack(\n", " [\n", " self.schedulers_bias[i].update_change(gradient_bias),\n", " self.schedulers_weight[i].update_change(gradient_weights),\n", " ]\n", " )\n", "\n", " # update weights and bias\n", " self.weights[i] -= update_matrix\n", "\n", " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Calculates accuracy of given prediction to target\n", "\n", " Parameters:\n", " ------------\n", " I prediction (np.ndarray): vector of predicitons output network\n", " (1s and 0s in case of classification, and real numbers in case of regression)\n", " II target (np.ndarray): vector of true values (What the network ideally should predict)\n", "\n", " Returns:\n", " ------------\n", " A floating point number representing the percentage of correctly classified instances.\n", " \"\"\"\n", " assert prediction.size == target.size\n", " return np.average((target == prediction))\n", " def _set_classification(self):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Decides if FFNN acts as classifier (True) og regressor (False),\n", " sets self.classification during init()\n", " \"\"\"\n", " self.classification = False\n", " if (\n", " self.cost_func.__name__ == \"CostLogReg\"\n", " or self.cost_func.__name__ == \"CostCrossEntropy\"\n", " ):\n", " self.classification = True\n", "\n", " def _progress_bar(self, progression, **kwargs):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Displays progress of training\n", " \"\"\"\n", " print_length = 40\n", " num_equals = int(progression * print_length)\n", " num_not = print_length - num_equals\n", " arrow = \">\" if num_equals > 0 else \"\"\n", " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n", " perc_print = self._format(progression * 100, decimals=5)\n", " line = f\" {bar} {perc_print}% \"\n", "\n", " for key in kwargs:\n", " if not np.isnan(kwargs[key]):\n", " value = self._format(kwargs[key], decimals=4)\n", " line += f\"| {key}: {value} \"\n", " sys.stdout.write(\"\\r\" + line)\n", " sys.stdout.flush()\n", " return len(line)\n", "\n", " def _format(self, value, decimals=4):\n", " \"\"\"\n", " Description:\n", " ------------\n", " Formats decimal numbers for progress bar\n", " \"\"\"\n", " if value > 0:\n", " v = value\n", " elif value < 0:\n", " v = -10 * value\n", " else:\n", " v = 1\n", " n = 1 + math.floor(math.log10(v))\n", " if n >= decimals - 1:\n", " return str(round(value))\n", " return f\"{value:.{decimals-n-1}f}\"" ] }, { "cell_type": "markdown", "id": "cefab2ac", "metadata": { "editable": true }, "source": [ "Before we make a model, we will quickly generate a dataset we can use\n", "for our linear regression problem as shown below" ] }, { "cell_type": "code", "execution_count": 11, "id": "6d5b49a1", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from sklearn.model_selection import train_test_split\n", "\n", "def SkrankeFunction(x, y):\n", " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n", "\n", "def create_X(x, y, n):\n", " if len(x.shape) > 1:\n", " x = np.ravel(x)\n", " y = np.ravel(y)\n", "\n", " N = len(x)\n", " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n", " X = np.ones((N, l))\n", "\n", " for i in range(1, n + 1):\n", " q = int((i) * (i + 1) / 2)\n", " for k in range(i + 1):\n", " X[:, q + k] = (x ** (i - k)) * (y**k)\n", "\n", " return X\n", "\n", "step=0.5\n", "x = np.arange(0, 1, step)\n", "y = np.arange(0, 1, step)\n", "x, y = np.meshgrid(x, y)\n", "target = SkrankeFunction(x, y)\n", "target = target.reshape(target.shape[0], 1)\n", "\n", "poly_degree=3\n", "X = create_X(x, y, poly_degree)\n", "\n", "X_train, X_test, t_train, t_test = train_test_split(X, target)" ] }, { "cell_type": "markdown", "id": "f857ae2b", "metadata": { "editable": true }, "source": [ "Now that we have our dataset ready for the regression, we can create\n", "our regressor. Note that with the seed parameter, we can make sure our\n", "results stay the same every time we run the neural network. For\n", "inititialization, we simply specify the dimensions (we wish the amount\n", "of input nodes to be equal to the datapoints, and the output to\n", "predict one value)." ] }, { "cell_type": "code", "execution_count": 12, "id": "2a92f970", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", "output_nodes = 1\n", "\n", "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)" ] }, { "cell_type": "markdown", "id": "28236066", "metadata": { "editable": true }, "source": [ "We then fit our model with our training data using the scheduler of our choice." ] }, { "cell_type": "code", "execution_count": 13, "id": "b3c6c87e", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Constant: Eta=0.001, Lambda=0\n", "\r", " [----------------------------------------] 0.000% | train_error: 3.69 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.000% | train_error: 3.67 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.000% | train_error: 3.65 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.000% | train_error: 3.64 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.000% | train_error: 3.62 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.000% | train_error: 3.60 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.000% | train_error: 3.58 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.000% | train_error: 3.57 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.000% | train_error: 3.55 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.000% | train_error: 3.53 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.00% | train_error: 3.52 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.00% | train_error: 3.50 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.00% | train_error: 3.48 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.00% | train_error: 3.47 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.00% | train_error: 3.45 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.00% | train_error: 3.43 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.00% | train_error: 3.42 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.00% | train_error: 3.40 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.00% | train_error: 3.38 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.00% | train_error: 3.37 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.00% | train_error: 3.35 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.00% | train_error: 3.34 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.00% | train_error: 3.32 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.00% | train_error: 3.31 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.00% | train_error: 3.29 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.00% | train_error: 3.27 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.00% | train_error: 3.26 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.00% | train_error: 3.24 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.00% | train_error: 3.23 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.00% | train_error: 3.21 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.00% | train_error: 3.20 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.00% | train_error: 3.18 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.00% | train_error: 3.17 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.00% | train_error: 3.15 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.00% | train_error: 3.14 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.00% | train_error: 3.12 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.00% | train_error: 3.11 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.00% | train_error: 3.09 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.00% | train_error: 3.08 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.00% | train_error: 3.06 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.00% | train_error: 3.05 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.00% | train_error: 3.03 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.00% | train_error: 3.02 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.00% | train_error: 3.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.00% | train_error: 2.99 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.00% | train_error: 2.98 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.00% | train_error: 2.96 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.00% | train_error: 2.95 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.00% | train_error: 2.93 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.00% | train_error: 2.92 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.00% | train_error: 2.91 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.00% | train_error: 2.89 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.00% | train_error: 2.88 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.00% | train_error: 2.86 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.00% | train_error: 2.85 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.00% | train_error: 2.84 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.00% | train_error: 2.82 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.00% | train_error: 2.81 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.00% | train_error: 2.80 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.00% | train_error: 2.78 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.00% | train_error: 2.77 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.00% | train_error: 2.76 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.00% | train_error: 2.74 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.00% | train_error: 2.73 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.00% | train_error: 2.72 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.00% | train_error: 2.70 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.00% | train_error: 2.69 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.00% | train_error: 2.68 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.00% | train_error: 2.67 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.00% | train_error: 2.65 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.00% | train_error: 2.64 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.00% | train_error: 2.63 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.00% | train_error: 2.62 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.00% | train_error: 2.60 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.00% | train_error: 2.59 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.00% | train_error: 2.58 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.00% | train_error: 2.57 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.00% | train_error: 2.55 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.00% | train_error: 2.54 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.00% | train_error: 2.53 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.00% | train_error: 2.52 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.00% | train_error: 2.51 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.00% | train_error: 2.49 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.00% | train_error: 2.48 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.00% | train_error: 2.47 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.00% | train_error: 2.46 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.00% | train_error: 2.45 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.00% | train_error: 2.44 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.00% | train_error: 2.42 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.00% | train_error: 2.41 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.00% | train_error: 2.40 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.00% | train_error: 2.39 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.00% | train_error: 2.38 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.00% | train_error: 2.37 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.00% | train_error: 2.36 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.00% | train_error: 2.34 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.00% | train_error: 2.33 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.00% | train_error: 2.32 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.00% | train_error: 2.31 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.00% | train_error: 2.30 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================================>] 100.0% | train_error: 2.30 " ] } ], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", "\n", "scheduler = Constant(eta=1e-3)\n", "scores = linear_regression.fit(X_train, t_train, scheduler)" ] }, { "cell_type": "markdown", "id": "f5b6209f", "metadata": { "editable": true }, "source": [ "Due to the progress bar we can see the MSE (train_error) throughout\n", "the FFNN's training. Note that the fit() function has some optional\n", "parameters with defualt arguments. For example, the regularization\n", "hyperparameter can be left ignored if not needed, and equally the FFNN\n", "will by default run for 100 epochs. These can easily be changed, such\n", "as for example:" ] }, { "cell_type": "code", "execution_count": 14, "id": "59d5742b", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Constant: Eta=0.001, Lambda=0.0001\n", "\r", " [----------------------------------------] 0.000% | train_error: 3.69 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.1000% | train_error: 3.67 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.2000% | train_error: 3.65 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.3000% | train_error: 3.64 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.4000% | train_error: 3.62 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.5000% | train_error: 3.60 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.6000% | train_error: 3.58 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.7000% | train_error: 3.57 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.8000% | train_error: 3.55 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.9000% | train_error: 3.53 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.000% | train_error: 3.52 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.100% | train_error: 3.50 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.200% | train_error: 3.48 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.300% | train_error: 3.47 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.400% | train_error: 3.45 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.500% | train_error: 3.43 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.600% | train_error: 3.42 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.700% | train_error: 3.40 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.800% | train_error: 3.38 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.900% | train_error: 3.37 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.000% | train_error: 3.35 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.100% | train_error: 3.34 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.200% | train_error: 3.32 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.300% | train_error: 3.31 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.400% | train_error: 3.29 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.500% | train_error: 3.27 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.600% | train_error: 3.26 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.700% | train_error: 3.24 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.800% | train_error: 3.23 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.900% | train_error: 3.21 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.000% | train_error: 3.20 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.100% | train_error: 3.18 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.200% | train_error: 3.17 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.300% | train_error: 3.15 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.400% | train_error: 3.14 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.500% | train_error: 3.12 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.600% | train_error: 3.11 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.700% | train_error: 3.09 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.800% | train_error: 3.08 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.900% | train_error: 3.06 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.000% | train_error: 3.05 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.100% | train_error: 3.03 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.200% | train_error: 3.02 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.300% | train_error: 3.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.400% | train_error: 2.99 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.500% | train_error: 2.98 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.600% | train_error: 2.96 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.700% | train_error: 2.95 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.800% | train_error: 2.93 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.900% | train_error: 2.92 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.000% | train_error: 2.91 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.100% | train_error: 2.89 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.200% | train_error: 2.88 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.300% | train_error: 2.86 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.400% | train_error: 2.85 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.500% | train_error: 2.84 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.600% | train_error: 2.82 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.700% | train_error: 2.81 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.800% | train_error: 2.80 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.900% | train_error: 2.78 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.000% | train_error: 2.77 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.100% | train_error: 2.76 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.200% | train_error: 2.74 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.300% | train_error: 2.73 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.400% | train_error: 2.72 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.500% | train_error: 2.70 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.600% | train_error: 2.69 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.700% | train_error: 2.68 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.800% | train_error: 2.67 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.900% | train_error: 2.65 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.000% | train_error: 2.64 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.100% | train_error: 2.63 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.200% | train_error: 2.62 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.300% | train_error: 2.60 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.400% | train_error: 2.59 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.500% | train_error: 2.58 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.600% | train_error: 2.57 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.700% | train_error: 2.55 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.800% | train_error: 2.54 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.900% | train_error: 2.53 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.000% | train_error: 2.52 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.100% | train_error: 2.51 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.200% | train_error: 2.49 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.300% | train_error: 2.48 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.400% | train_error: 2.47 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.500% | train_error: 2.46 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.600% | train_error: 2.45 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.700% | train_error: 2.44 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.800% | train_error: 2.42 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.900% | train_error: 2.41 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.000% | train_error: 2.40 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.100% | train_error: 2.39 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.200% | train_error: 2.38 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.300% | train_error: 2.37 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.400% | train_error: 2.36 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.500% | train_error: 2.34 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.600% | train_error: 2.33 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.700% | train_error: 2.32 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.800% | train_error: 2.31 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.900% | train_error: 2.30 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.00% | train_error: 2.29 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.10% | train_error: 2.28 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.20% | train_error: 2.27 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.30% | train_error: 2.26 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.40% | train_error: 2.25 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.50% | train_error: 2.23 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.60% | train_error: 2.22 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.70% | train_error: 2.21 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.80% | train_error: 2.20 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.90% | train_error: 2.19 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.00% | train_error: 2.18 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.10% | train_error: 2.17 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.20% | train_error: 2.16 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.30% | train_error: 2.15 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.40% | train_error: 2.14 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.50% | train_error: 2.13 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.60% | train_error: 2.12 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.70% | train_error: 2.11 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.80% | train_error: 2.10 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.90% | train_error: 2.09 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.00% | train_error: 2.08 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.10% | train_error: 2.07 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.20% | train_error: 2.06 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.30% | train_error: 2.05 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.40% | train_error: 2.04 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.50% | train_error: 2.03 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.60% | train_error: 2.02 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.70% | train_error: 2.01 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.80% | train_error: 2.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.90% | train_error: 1.99 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.00% | train_error: 1.98 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.10% | train_error: 1.97 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.20% | train_error: 1.96 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.30% | train_error: 1.96 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.40% | train_error: 1.95 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.50% | train_error: 1.94 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.60% | train_error: 1.93 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.70% | train_error: 1.92 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.80% | train_error: 1.91 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.90% | train_error: 1.90 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.00% | train_error: 1.89 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.10% | train_error: 1.88 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.20% | train_error: 1.87 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.30% | train_error: 1.86 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.40% | train_error: 1.86 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.50% | train_error: 1.85 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.60% | train_error: 1.84 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.70% | train_error: 1.83 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.80% | train_error: 1.82 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.90% | train_error: 1.81 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.00% | train_error: 1.80 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.10% | train_error: 1.79 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.20% | train_error: 1.79 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.30% | train_error: 1.78 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.40% | train_error: 1.77 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.50% | train_error: 1.76 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.60% | train_error: 1.75 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.70% | train_error: 1.74 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.80% | train_error: 1.74 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.90% | train_error: 1.73 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.00% | train_error: 1.72 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.10% | train_error: 1.71 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.20% | train_error: 1.70 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.30% | train_error: 1.69 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.40% | train_error: 1.69 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.50% | train_error: 1.68 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.60% | train_error: 1.67 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.70% | train_error: 1.66 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.80% | train_error: 1.65 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.90% | train_error: 1.65 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.00% | train_error: 1.64 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.10% | train_error: 1.63 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.20% | train_error: 1.62 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.30% | train_error: 1.62 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.40% | train_error: 1.61 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.50% | train_error: 1.60 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.60% | train_error: 1.59 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.70% | train_error: 1.59 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.80% | train_error: 1.58 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.90% | train_error: 1.57 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.00% | train_error: 1.56 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.10% | train_error: 1.56 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.20% | train_error: 1.55 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.30% | train_error: 1.54 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.40% | train_error: 1.53 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.50% | train_error: 1.53 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.60% | train_error: 1.52 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.70% | train_error: 1.51 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.80% | train_error: 1.50 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.90% | train_error: 1.50 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.00% | train_error: 1.49 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.10% | train_error: 1.48 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.20% | train_error: 1.48 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.30% | train_error: 1.47 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.40% | train_error: 1.46 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.50% | train_error: 1.46 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.60% | train_error: 1.45 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.70% | train_error: 1.44 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.80% | train_error: 1.43 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.90% | train_error: 1.43 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.00% | train_error: 1.42 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.10% | train_error: 1.41 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.20% | train_error: 1.41 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.30% | train_error: 1.40 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.40% | train_error: 1.39 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.50% | train_error: 1.39 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.60% | train_error: 1.38 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.70% | train_error: 1.37 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.80% | train_error: 1.37 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.90% | train_error: 1.36 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.00% | train_error: 1.35 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.10% | train_error: 1.35 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.20% | train_error: 1.34 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.30% | train_error: 1.34 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.40% | train_error: 1.33 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.50% | train_error: 1.32 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.60% | train_error: 1.32 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.70% | train_error: 1.31 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.80% | train_error: 1.30 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.90% | train_error: 1.30 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.00% | train_error: 1.29 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.10% | train_error: 1.29 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.20% | train_error: 1.28 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.30% | train_error: 1.27 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.40% | train_error: 1.27 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.50% | train_error: 1.26 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.60% | train_error: 1.26 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.70% | train_error: 1.25 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.80% | train_error: 1.24 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.90% | train_error: 1.24 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.00% | train_error: 1.23 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.10% | train_error: 1.23 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.20% | train_error: 1.22 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.30% | train_error: 1.21 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.40% | train_error: 1.21 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.50% | train_error: 1.20 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.60% | train_error: 1.20 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.70% | train_error: 1.19 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.80% | train_error: 1.19 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.90% | train_error: 1.18 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.00% | train_error: 1.17 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.10% | train_error: 1.17 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.20% | train_error: 1.16 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.30% | train_error: 1.16 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.40% | train_error: 1.15 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.50% | train_error: 1.15 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.60% | train_error: 1.14 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.70% | train_error: 1.14 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.80% | train_error: 1.13 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.90% | train_error: 1.13 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.00% | train_error: 1.12 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.10% | train_error: 1.11 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.20% | train_error: 1.11 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.30% | train_error: 1.10 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.40% | train_error: 1.10 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.50% | train_error: 1.09 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.60% | train_error: 1.09 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.70% | train_error: 1.08 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.80% | train_error: 1.08 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.90% | train_error: 1.07 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.00% | train_error: 1.07 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.10% | train_error: 1.06 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.20% | train_error: 1.06 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.30% | train_error: 1.05 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.40% | train_error: 1.05 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.50% | train_error: 1.04 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.60% | train_error: 1.04 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.70% | train_error: 1.03 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.80% | train_error: 1.03 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.90% | train_error: 1.02 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.00% | train_error: 1.02 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.10% | train_error: 1.01 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.20% | train_error: 1.01 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.30% | train_error: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.40% | train_error: 0.999 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.50% | train_error: 0.994 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.60% | train_error: 0.990 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.70% | train_error: 0.985 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.80% | train_error: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.90% | train_error: 0.976 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.00% | train_error: 0.971 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.10% | train_error: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.20% | train_error: 0.962 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.30% | train_error: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.40% | train_error: 0.953 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.50% | train_error: 0.948 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.60% | train_error: 0.944 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.70% | train_error: 0.939 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.80% | train_error: 0.935 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.90% | train_error: 0.930 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.00% | train_error: 0.926 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.10% | train_error: 0.922 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.20% | train_error: 0.917 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.30% | train_error: 0.913 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.40% | train_error: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.50% | train_error: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.60% | train_error: 0.900 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.70% | train_error: 0.896 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.80% | train_error: 0.891 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.90% | train_error: 0.887 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.00% | train_error: 0.883 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.10% | train_error: 0.879 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.20% | train_error: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.30% | train_error: 0.870 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.40% | train_error: 0.866 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.50% | train_error: 0.862 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.60% | train_error: 0.858 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.70% | train_error: 0.854 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.80% | train_error: 0.850 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.90% | train_error: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.00% | train_error: 0.842 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.10% | train_error: 0.838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.20% | train_error: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.30% | train_error: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.40% | train_error: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.50% | train_error: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.60% | train_error: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.70% | train_error: 0.814 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.80% | train_error: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.90% | train_error: 0.807 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.00% | train_error: 0.803 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.10% | train_error: 0.799 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.20% | train_error: 0.795 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.30% | train_error: 0.792 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.40% | train_error: 0.788 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.50% | train_error: 0.784 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.60% | train_error: 0.780 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.70% | train_error: 0.777 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.80% | train_error: 0.773 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.90% | train_error: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.00% | train_error: 0.766 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.10% | train_error: 0.762 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.20% | train_error: 0.759 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.30% | train_error: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.40% | train_error: 0.751 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.50% | train_error: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.60% | train_error: 0.744 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.70% | train_error: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.80% | train_error: 0.737 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.90% | train_error: 0.734 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.00% | train_error: 0.730 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.10% | train_error: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.20% | train_error: 0.723 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.30% | train_error: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.40% | train_error: 0.717 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.50% | train_error: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.60% | train_error: 0.710 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.70% | train_error: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.80% | train_error: 0.703 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.90% | train_error: 0.700 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.00% | train_error: 0.696 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.10% | train_error: 0.693 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.20% | train_error: 0.690 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.30% | train_error: 0.687 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.40% | train_error: 0.683 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.50% | train_error: 0.680 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.60% | train_error: 0.677 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.70% | train_error: 0.674 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.80% | train_error: 0.670 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.90% | train_error: 0.667 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.00% | train_error: 0.664 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.10% | train_error: 0.661 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.20% | train_error: 0.658 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.30% | train_error: 0.655 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.40% | train_error: 0.652 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.50% | train_error: 0.649 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.60% | train_error: 0.646 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.70% | train_error: 0.642 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.80% | train_error: 0.639 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.90% | train_error: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.00% | train_error: 0.633 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.10% | train_error: 0.630 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.20% | train_error: 0.627 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.30% | train_error: 0.624 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.40% | train_error: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.50% | train_error: 0.619 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.60% | train_error: 0.616 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.70% | train_error: 0.613 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.80% | train_error: 0.610 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.90% | train_error: 0.607 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.00% | train_error: 0.604 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.10% | train_error: 0.601 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.20% | train_error: 0.598 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.30% | train_error: 0.596 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.40% | train_error: 0.593 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.50% | train_error: 0.590 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.60% | train_error: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.70% | train_error: 0.584 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.80% | train_error: 0.582 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.90% | train_error: 0.579 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.00% | train_error: 0.576 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.10% | train_error: 0.573 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.20% | train_error: 0.571 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.30% | train_error: 0.568 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.40% | train_error: 0.565 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.50% | train_error: 0.563 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.60% | train_error: 0.560 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.70% | train_error: 0.557 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.80% | train_error: 0.555 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.90% | train_error: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.00% | train_error: 0.549 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.10% | train_error: 0.547 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.20% | train_error: 0.544 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.30% | train_error: 0.542 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.40% | train_error: 0.539 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.50% | train_error: 0.537 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.60% | train_error: 0.534 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.70% | train_error: 0.532 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.80% | train_error: 0.529 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.90% | train_error: 0.527 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.00% | train_error: 0.524 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.10% | train_error: 0.522 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.20% | train_error: 0.519 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.30% | train_error: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.40% | train_error: 0.514 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.50% | train_error: 0.512 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.60% | train_error: 0.509 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.70% | train_error: 0.507 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.80% | train_error: 0.505 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.90% | train_error: 0.502 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.00% | train_error: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.10% | train_error: 0.498 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.20% | train_error: 0.495 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.30% | train_error: 0.493 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.40% | train_error: 0.491 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.50% | train_error: 0.488 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.60% | train_error: 0.486 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.70% | train_error: 0.484 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.80% | train_error: 0.481 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.90% | train_error: 0.479 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.00% | train_error: 0.477 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.10% | train_error: 0.475 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.20% | train_error: 0.472 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.30% | train_error: 0.470 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.40% | train_error: 0.468 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.50% | train_error: 0.466 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.60% | train_error: 0.463 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.70% | train_error: 0.461 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.80% | train_error: 0.459 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.90% | train_error: 0.457 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.00% | train_error: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.10% | train_error: 0.453 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.20% | train_error: 0.451 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.30% | train_error: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.40% | train_error: 0.446 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.50% | train_error: 0.444 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.60% | train_error: 0.442 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.70% | train_error: 0.440 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.80% | train_error: 0.438 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.90% | train_error: 0.436 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.00% | train_error: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.10% | train_error: 0.432 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.20% | train_error: 0.430 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.30% | train_error: 0.428 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.40% | train_error: 0.426 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.50% | train_error: 0.424 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.60% | train_error: 0.422 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.70% | train_error: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.80% | train_error: 0.418 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.90% | train_error: 0.416 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.00% | train_error: 0.414 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.10% | train_error: 0.412 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.20% | train_error: 0.410 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.30% | train_error: 0.408 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.40% | train_error: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.50% | train_error: 0.404 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.60% | train_error: 0.402 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.70% | train_error: 0.400 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.80% | train_error: 0.398 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.90% | train_error: 0.397 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.00% | train_error: 0.395 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.10% | train_error: 0.393 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.20% | train_error: 0.391 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.30% | train_error: 0.389 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.40% | train_error: 0.387 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.50% | train_error: 0.386 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.60% | train_error: 0.384 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.70% | train_error: 0.382 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.80% | train_error: 0.380 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.90% | train_error: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.00% | train_error: 0.377 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.10% | train_error: 0.375 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.20% | train_error: 0.373 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.30% | train_error: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.40% | train_error: 0.370 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.50% | train_error: 0.368 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.60% | train_error: 0.366 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.70% | train_error: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.80% | train_error: 0.363 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.90% | train_error: 0.361 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.00% | train_error: 0.359 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.10% | train_error: 0.358 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.20% | train_error: 0.356 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.30% | train_error: 0.354 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.40% | train_error: 0.353 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.50% | train_error: 0.351 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.60% | train_error: 0.349 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.70% | train_error: 0.348 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.80% | train_error: 0.346 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.90% | train_error: 0.344 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.00% | train_error: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.10% | train_error: 0.341 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.20% | train_error: 0.339 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.30% | train_error: 0.338 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.40% | train_error: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.50% | train_error: 0.335 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.60% | train_error: 0.333 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.70% | train_error: 0.332 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.80% | train_error: 0.330 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.90% | train_error: 0.328 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.00% | train_error: 0.327 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.10% | train_error: 0.325 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.20% | train_error: 0.324 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.30% | train_error: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.40% | train_error: 0.321 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.50% | train_error: 0.319 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.60% | train_error: 0.318 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.70% | train_error: 0.316 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.80% | train_error: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.90% | train_error: 0.313 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.00% | train_error: 0.312 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.10% | train_error: 0.310 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.20% | train_error: 0.309 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.30% | train_error: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.40% | train_error: 0.306 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.50% | train_error: 0.305 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.60% | train_error: 0.303 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.70% | train_error: 0.302 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.80% | train_error: 0.300 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.90% | train_error: 0.299 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.00% | train_error: 0.298 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.10% | train_error: 0.296 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.20% | train_error: 0.295 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.30% | train_error: 0.293 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.40% | train_error: 0.292 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.50% | train_error: 0.291 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.60% | train_error: 0.289 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.70% | train_error: 0.288 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.80% | train_error: 0.287 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.90% | train_error: 0.285 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.00% | train_error: 0.284 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.10% | train_error: 0.283 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.20% | train_error: 0.281 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.30% | train_error: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.40% | train_error: 0.279 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.50% | train_error: 0.277 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.60% | train_error: 0.276 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.70% | train_error: 0.275 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.80% | train_error: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.90% | train_error: 0.272 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.00% | train_error: 0.271 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.10% | train_error: 0.270 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.20% | train_error: 0.268 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.30% | train_error: 0.267 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.40% | train_error: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.50% | train_error: 0.265 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.60% | train_error: 0.263 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.70% | train_error: 0.262 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.80% | train_error: 0.261 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.90% | train_error: 0.260 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.00% | train_error: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.10% | train_error: 0.257 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.20% | train_error: 0.256 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.30% | train_error: 0.255 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.40% | train_error: 0.254 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.50% | train_error: 0.253 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.60% | train_error: 0.251 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.70% | train_error: 0.250 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.80% | train_error: 0.249 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.90% | train_error: 0.248 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.00% | train_error: 0.247 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.10% | train_error: 0.246 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.20% | train_error: 0.244 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.30% | train_error: 0.243 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.40% | train_error: 0.242 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.50% | train_error: 0.241 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.60% | train_error: 0.240 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.70% | train_error: 0.239 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.80% | train_error: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.90% | train_error: 0.237 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.00% | train_error: 0.235 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.10% | train_error: 0.234 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.20% | train_error: 0.233 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.30% | train_error: 0.232 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.40% | train_error: 0.231 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.50% | train_error: 0.230 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.60% | train_error: 0.229 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.70% | train_error: 0.228 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.80% | train_error: 0.227 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.90% | train_error: 0.226 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.00% | train_error: 0.225 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.10% | train_error: 0.224 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.20% | train_error: 0.223 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.30% | train_error: 0.222 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.40% | train_error: 0.221 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.50% | train_error: 0.219 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.60% | train_error: 0.218 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.70% | train_error: 0.217 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.80% | train_error: 0.216 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.90% | train_error: 0.215 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.00% | train_error: 0.214 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.10% | train_error: 0.213 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.20% | train_error: 0.212 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.30% | train_error: 0.211 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.40% | train_error: 0.210 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.50% | train_error: 0.209 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.60% | train_error: 0.209 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.70% | train_error: 0.208 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.80% | train_error: 0.207 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.90% | train_error: 0.206 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.00% | train_error: 0.205 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.10% | train_error: 0.204 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.20% | train_error: 0.203 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.30% | train_error: 0.202 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.40% | train_error: 0.201 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.50% | train_error: 0.200 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.60% | train_error: 0.199 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.70% | train_error: 0.198 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.80% | train_error: 0.197 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.90% | train_error: 0.196 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.00% | train_error: 0.195 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.10% | train_error: 0.194 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.20% | train_error: 0.194 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.30% | train_error: 0.193 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.40% | train_error: 0.192 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.50% | train_error: 0.191 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.60% | train_error: 0.190 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.70% | train_error: 0.189 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.80% | train_error: 0.188 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.90% | train_error: 0.187 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.00% | train_error: 0.186 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.10% | train_error: 0.186 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.20% | train_error: 0.185 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.30% | train_error: 0.184 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.40% | train_error: 0.183 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.50% | train_error: 0.182 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.60% | train_error: 0.181 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.70% | train_error: 0.180 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.80% | train_error: 0.180 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.90% | train_error: 0.179 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.00% | train_error: 0.178 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.10% | train_error: 0.177 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.20% | train_error: 0.176 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.30% | train_error: 0.176 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.40% | train_error: 0.175 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.50% | train_error: 0.174 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.60% | train_error: 0.173 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.70% | train_error: 0.172 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.80% | train_error: 0.171 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.90% | train_error: 0.171 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.00% | train_error: 0.170 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.10% | train_error: 0.169 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.20% | train_error: 0.168 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.30% | train_error: 0.168 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.40% | train_error: 0.167 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.50% | train_error: 0.166 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.60% | train_error: 0.165 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.70% | train_error: 0.164 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.80% | train_error: 0.164 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.90% | train_error: 0.163 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.00% | train_error: 0.162 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.10% | train_error: 0.161 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.20% | train_error: 0.161 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.30% | train_error: 0.160 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.40% | train_error: 0.159 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.50% | train_error: 0.158 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.60% | train_error: 0.158 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.70% | train_error: 0.157 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.80% | train_error: 0.156 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.90% | train_error: 0.156 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.00% | train_error: 0.155 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.10% | train_error: 0.154 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.20% | train_error: 0.153 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.30% | train_error: 0.153 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.40% | train_error: 0.152 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.50% | train_error: 0.151 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.60% | train_error: 0.151 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.70% | train_error: 0.150 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.80% | train_error: 0.149 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.90% | train_error: 0.149 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.00% | train_error: 0.148 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.10% | train_error: 0.147 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.20% | train_error: 0.146 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.30% | train_error: 0.146 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.40% | train_error: 0.145 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.50% | train_error: 0.144 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.60% | train_error: 0.144 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.70% | train_error: 0.143 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.80% | train_error: 0.142 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.90% | train_error: 0.142 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.00% | train_error: 0.141 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.10% | train_error: 0.141 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.20% | train_error: 0.140 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.30% | train_error: 0.139 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.40% | train_error: 0.139 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.50% | train_error: 0.138 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.60% | train_error: 0.137 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.70% | train_error: 0.137 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.80% | train_error: 0.136 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.90% | train_error: 0.135 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.00% | train_error: 0.135 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.10% | train_error: 0.134 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.20% | train_error: 0.134 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.30% | train_error: 0.133 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.40% | train_error: 0.132 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.50% | train_error: 0.132 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.60% | train_error: 0.131 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.70% | train_error: 0.131 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.80% | train_error: 0.130 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.90% | train_error: 0.129 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.00% | train_error: 0.129 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.10% | train_error: 0.128 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.20% | train_error: 0.128 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.30% | train_error: 0.127 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.40% | train_error: 0.126 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.50% | train_error: 0.126 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.60% | train_error: 0.125 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.70% | train_error: 0.125 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.80% | train_error: 0.124 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.90% | train_error: 0.124 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.00% | train_error: 0.123 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.10% | train_error: 0.122 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.20% | train_error: 0.122 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.30% | train_error: 0.121 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.40% | train_error: 0.121 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.50% | train_error: 0.120 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.60% | train_error: 0.120 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.70% | train_error: 0.119 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.80% | train_error: 0.119 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.90% | train_error: 0.118 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.00% | train_error: 0.117 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.10% | train_error: 0.117 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.20% | train_error: 0.116 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.30% | train_error: 0.116 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.40% | train_error: 0.115 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.50% | train_error: 0.115 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.60% | train_error: 0.114 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.70% | train_error: 0.114 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.80% | train_error: 0.113 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.90% | train_error: 0.113 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.00% | train_error: 0.112 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.10% | train_error: 0.112 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.20% | train_error: 0.111 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.30% | train_error: 0.111 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.40% | train_error: 0.110 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.50% | train_error: 0.110 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.60% | train_error: 0.109 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.70% | train_error: 0.109 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.80% | train_error: 0.108 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.90% | train_error: 0.108 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.00% | train_error: 0.107 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.10% | train_error: 0.107 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.20% | train_error: 0.106 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.30% | train_error: 0.106 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.40% | train_error: 0.105 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.50% | train_error: 0.105 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.60% | train_error: 0.104 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.70% | train_error: 0.104 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.80% | train_error: 0.103 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.90% | train_error: 0.103 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.00% | train_error: 0.102 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.10% | train_error: 0.102 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.20% | train_error: 0.101 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.30% | train_error: 0.101 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.40% | train_error: 0.101 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.50% | train_error: 0.100 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.60% | train_error: 0.0996 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.70% | train_error: 0.0992 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.80% | train_error: 0.0987 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.90% | train_error: 0.0983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.00% | train_error: 0.0978 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.10% | train_error: 0.0974 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.20% | train_error: 0.0969 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.30% | train_error: 0.0965 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.40% | train_error: 0.0961 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.50% | train_error: 0.0956 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.60% | train_error: 0.0952 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.70% | train_error: 0.0948 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.80% | train_error: 0.0943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.90% | train_error: 0.0939 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.00% | train_error: 0.0935 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.10% | train_error: 0.0930 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.20% | train_error: 0.0926 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.30% | train_error: 0.0922 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.40% | train_error: 0.0918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.50% | train_error: 0.0914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.60% | train_error: 0.0910 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.70% | train_error: 0.0905 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.80% | train_error: 0.0901 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.90% | train_error: 0.0897 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.00% | train_error: 0.0893 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.10% | train_error: 0.0889 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.20% | train_error: 0.0885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.30% | train_error: 0.0881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.40% | train_error: 0.0877 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.50% | train_error: 0.0873 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.60% | train_error: 0.0869 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.70% | train_error: 0.0865 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.80% | train_error: 0.0861 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.90% | train_error: 0.0858 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.00% | train_error: 0.0854 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.10% | train_error: 0.0850 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.20% | train_error: 0.0846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.30% | train_error: 0.0842 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.40% | train_error: 0.0838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.50% | train_error: 0.0835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.60% | train_error: 0.0831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.70% | train_error: 0.0827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.80% | train_error: 0.0823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.90% | train_error: 0.0820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.00% | train_error: 0.0816 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.10% | train_error: 0.0812 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.20% | train_error: 0.0809 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.30% | train_error: 0.0805 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.40% | train_error: 0.0801 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.50% | train_error: 0.0798 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.60% | train_error: 0.0794 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.70% | train_error: 0.0791 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.80% | train_error: 0.0787 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.90% | train_error: 0.0784 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.00% | train_error: 0.0780 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.10% | train_error: 0.0776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.20% | train_error: 0.0773 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.30% | train_error: 0.0770 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.40% | train_error: 0.0766 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.50% | train_error: 0.0763 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.60% | train_error: 0.0759 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.70% | train_error: 0.0756 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.80% | train_error: 0.0752 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.90% | train_error: 0.0749 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.00% | train_error: 0.0746 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.10% | train_error: 0.0742 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.20% | train_error: 0.0739 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.30% | train_error: 0.0736 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.40% | train_error: 0.0732 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.50% | train_error: 0.0729 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.60% | train_error: 0.0726 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.70% | train_error: 0.0723 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.80% | train_error: 0.0719 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.90% | train_error: 0.0716 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.00% | train_error: 0.0713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.10% | train_error: 0.0710 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.20% | train_error: 0.0707 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.30% | train_error: 0.0703 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.40% | train_error: 0.0700 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.50% | train_error: 0.0697 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.60% | train_error: 0.0694 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.70% | train_error: 0.0691 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.80% | train_error: 0.0688 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.90% | train_error: 0.0685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.00% | train_error: 0.0682 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.10% | train_error: 0.0679 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.20% | train_error: 0.0676 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.30% | train_error: 0.0673 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.40% | train_error: 0.0670 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.50% | train_error: 0.0667 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.60% | train_error: 0.0664 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.70% | train_error: 0.0661 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.80% | train_error: 0.0658 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.90% | train_error: 0.0655 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.00% | train_error: 0.0652 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.10% | train_error: 0.0649 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.20% | train_error: 0.0646 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.30% | train_error: 0.0643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.40% | train_error: 0.0641 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.50% | train_error: 0.0638 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.60% | train_error: 0.0635 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.70% | train_error: 0.0632 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.80% | train_error: 0.0629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.90% | train_error: 0.0626 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.00% | train_error: 0.0624 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.10% | train_error: 0.0621 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.20% | train_error: 0.0618 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.30% | train_error: 0.0615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.40% | train_error: 0.0613 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.50% | train_error: 0.0610 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.60% | train_error: 0.0607 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.70% | train_error: 0.0605 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.80% | train_error: 0.0602 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.90% | train_error: 0.0599 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.00% | train_error: 0.0597 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.10% | train_error: 0.0594 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.20% | train_error: 0.0591 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.30% | train_error: 0.0589 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.40% | train_error: 0.0586 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.50% | train_error: 0.0584 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.60% | train_error: 0.0581 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.70% | train_error: 0.0578 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.80% | train_error: 0.0576 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.90% | train_error: 0.0573 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.00% | train_error: 0.0571 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.10% | train_error: 0.0568 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.20% | train_error: 0.0566 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.30% | train_error: 0.0563 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.40% | train_error: 0.0561 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.50% | train_error: 0.0558 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.60% | train_error: 0.0556 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.70% | train_error: 0.0553 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.80% | train_error: 0.0551 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.90% | train_error: 0.0549 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.00% | train_error: 0.0546 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.10% | train_error: 0.0544 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.20% | train_error: 0.0541 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.30% | train_error: 0.0539 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.40% | train_error: 0.0537 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.50% | train_error: 0.0534 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.60% | train_error: 0.0532 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.70% | train_error: 0.0530 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.80% | train_error: 0.0527 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.90% | train_error: 0.0525 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.00% | train_error: 0.0523 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.10% | train_error: 0.0520 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.20% | train_error: 0.0518 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.30% | train_error: 0.0516 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.40% | train_error: 0.0514 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.50% | train_error: 0.0511 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.60% | train_error: 0.0509 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.70% | train_error: 0.0507 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.80% | train_error: 0.0505 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.90% | train_error: 0.0503 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.00% | train_error: 0.0500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.10% | train_error: 0.0498 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.20% | train_error: 0.0496 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.30% | train_error: 0.0494 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.40% | train_error: 0.0492 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.50% | train_error: 0.0490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.60% | train_error: 0.0487 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.70% | train_error: 0.0485 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.80% | train_error: 0.0483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.90% | train_error: 0.0481 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.00% | train_error: 0.0479 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.10% | train_error: 0.0477 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.20% | train_error: 0.0475 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.30% | train_error: 0.0473 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.40% | train_error: 0.0471 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.50% | train_error: 0.0469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.60% | train_error: 0.0467 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.70% | train_error: 0.0465 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.80% | train_error: 0.0463 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.90% | train_error: 0.0461 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.00% | train_error: 0.0459 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.10% | train_error: 0.0457 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.20% | train_error: 0.0455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.30% | train_error: 0.0453 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.40% | train_error: 0.0451 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.50% | train_error: 0.0449 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.60% | train_error: 0.0447 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.70% | train_error: 0.0445 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.80% | train_error: 0.0443 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.90% | train_error: 0.0441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.00% | train_error: 0.0439 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.10% | train_error: 0.0437 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.20% | train_error: 0.0435 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.30% | train_error: 0.0434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.40% | train_error: 0.0432 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.50% | train_error: 0.0430 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.60% | train_error: 0.0428 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.70% | train_error: 0.0426 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.80% | train_error: 0.0424 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.90% | train_error: 0.0423 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.00% | train_error: 0.0421 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.10% | train_error: 0.0419 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.20% | train_error: 0.0417 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.30% | train_error: 0.0415 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.40% | train_error: 0.0414 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.50% | train_error: 0.0412 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.60% | train_error: 0.0410 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.70% | train_error: 0.0408 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.80% | train_error: 0.0407 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.90% | train_error: 0.0405 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.00% | train_error: 0.0403 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.10% | train_error: 0.0401 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.20% | train_error: 0.0400 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.30% | train_error: 0.0398 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.40% | train_error: 0.0396 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.50% | train_error: 0.0395 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.60% | train_error: 0.0393 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.70% | train_error: 0.0391 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.80% | train_error: 0.0389 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.90% | train_error: 0.0388 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.00% | train_error: 0.0386 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.10% | train_error: 0.0385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.20% | train_error: 0.0383 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.30% | train_error: 0.0381 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.40% | train_error: 0.0380 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.50% | train_error: 0.0378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.60% | train_error: 0.0376 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.70% | train_error: 0.0375 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.80% | train_error: 0.0373 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.90% | train_error: 0.0372 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.00% | train_error: 0.0370 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.10% | train_error: 0.0369 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.20% | train_error: 0.0367 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.30% | train_error: 0.0365 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.40% | train_error: 0.0364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.50% | train_error: 0.0362 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.60% | train_error: 0.0361 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.70% | train_error: 0.0359 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.80% | train_error: 0.0358 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.90% | train_error: 0.0356 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================================>] 100.0% | train_error: 0.0356 " ] } ], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", "\n", "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)" ] }, { "cell_type": "markdown", "id": "13e834ae", "metadata": { "editable": true }, "source": [ "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", "\n", "Let us then switch to a binary classification. We use a binary\n", "classification dataset, and follow a similar setup to the regression\n", "case." ] }, { "cell_type": "code", "execution_count": 15, "id": "ab390311", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "from sklearn.preprocessing import MinMaxScaler\n", "\n", "wisconsin = load_breast_cancer()\n", "X = wisconsin.data\n", "target = wisconsin.target\n", "target = target.reshape(target.shape[0], 1)\n", "\n", "X_train, X_val, t_train, t_val = train_test_split(X, target)\n", "\n", "scaler = MinMaxScaler()\n", "scaler.fit(X_train)\n", "X_train = scaler.transform(X_train)\n", "X_val = scaler.transform(X_val)" ] }, { "cell_type": "code", "execution_count": 16, "id": "f2d67c2e", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", "output_nodes = 1\n", "\n", "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)" ] }, { "cell_type": "markdown", "id": "e350f60e", "metadata": { "editable": true }, "source": [ "We will now make use of our validation data by passing it into our fit function as a keyword argument" ] }, { "cell_type": "code", "execution_count": 17, "id": "211a328d", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adam: Eta=0.001, Lambda=0" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\r", " [----------------------------------------] 0.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.1000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.2000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.3000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.4000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.5000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.6000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.7000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.8000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.9000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.900% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.000% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.100% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.200% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.300% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.400% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.500% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.600% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.700% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.800% | train_error: 12.9 | train_acc: 0.376 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.900% | train_error: 13.0 | train_acc: 0.373 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.00% | train_error: 13.0 | train_acc: 0.373 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.10% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.20% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.30% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.40% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.50% | train_error: 13.0 | train_acc: 0.371 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.60% | train_error: 13.0 | train_acc: 0.371 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.70% | train_error: 13.0 | train_acc: 0.371 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.80% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.90% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.00% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.10% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.20% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.30% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.40% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.50% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.60% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.70% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.80% | train_error: 13.1 | train_acc: 0.369 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.90% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.00% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.10% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.20% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.30% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.40% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.50% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.60% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.70% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.80% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.90% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.00% | train_error: 13.1 | train_acc: 0.366 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.10% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.20% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.30% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.40% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.50% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.60% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.70% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.8 | val_acc: 0.385 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.80% | train_error: 13.3 | train_acc: 0.359 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.90% | train_error: 13.4 | train_acc: 0.354 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.00% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.0 | val_acc: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.10% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.0 | val_acc: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.20% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.0 | val_acc: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.30% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.0 | val_acc: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.40% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.0 | val_acc: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.50% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.0 | val_acc: 0.371 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.60% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.70% | train_error: 13.4 | train_acc: 0.354 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.80% | train_error: 13.4 | train_acc: 0.352 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.90% | train_error: 13.4 | train_acc: 0.352 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.00% | train_error: 13.5 | train_acc: 0.347 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.10% | train_error: 13.6 | train_acc: 0.345 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.20% | train_error: 13.6 | train_acc: 0.345 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.30% | train_error: 13.8 | train_acc: 0.336 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.40% | train_error: 13.8 | train_acc: 0.336 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.50% | train_error: 13.8 | train_acc: 0.336 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.60% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.70% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.80% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.90% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.00% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.10% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.20% | train_error: 13.8 | train_acc: 0.333 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.30% | train_error: 14.0 | train_acc: 0.326 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.40% | train_error: 14.0 | train_acc: 0.326 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.50% | train_error: 14.0 | train_acc: 0.326 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.60% | train_error: 14.0 | train_acc: 0.326 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.70% | train_error: 14.1 | train_acc: 0.322 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.80% | train_error: 14.1 | train_acc: 0.322 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.90% | train_error: 14.1 | train_acc: 0.319 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.00% | train_error: 14.1 | train_acc: 0.319 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.10% | train_error: 14.1 | train_acc: 0.319 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.20% | train_error: 14.2 | train_acc: 0.317 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.30% | train_error: 14.2 | train_acc: 0.317 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.40% | train_error: 14.2 | train_acc: 0.317 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.50% | train_error: 14.2 | train_acc: 0.317 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.60% | train_error: 14.3 | train_acc: 0.312 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.70% | train_error: 14.4 | train_acc: 0.308 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.80% | train_error: 14.4 | train_acc: 0.308 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.90% | train_error: 14.4 | train_acc: 0.308 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.00% | train_error: 14.4 | train_acc: 0.308 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.10% | train_error: 14.4 | train_acc: 0.305 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.20% | train_error: 14.4 | train_acc: 0.305 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.30% | train_error: 14.4 | train_acc: 0.305 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.40% | train_error: 14.5 | train_acc: 0.298 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.50% | train_error: 14.5 | train_acc: 0.298 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.60% | train_error: 14.6 | train_acc: 0.296 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.70% | train_error: 14.6 | train_acc: 0.293 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.80% | train_error: 14.7 | train_acc: 0.291 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.90% | train_error: 14.7 | train_acc: 0.291 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.00% | train_error: 14.7 | train_acc: 0.291 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.10% | train_error: 14.7 | train_acc: 0.291 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.20% | train_error: 14.7 | train_acc: 0.291 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.30% | train_error: 14.8 | train_acc: 0.286 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.40% | train_error: 14.8 | train_acc: 0.284 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.50% | train_error: 14.8 | train_acc: 0.284 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.60% | train_error: 14.9 | train_acc: 0.282 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.70% | train_error: 14.9 | train_acc: 0.282 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.80% | train_error: 14.9 | train_acc: 0.282 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.90% | train_error: 14.9 | train_acc: 0.282 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.00% | train_error: 14.9 | train_acc: 0.279 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.10% | train_error: 15.1 | train_acc: 0.272 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.20% | train_error: 15.1 | train_acc: 0.272 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.30% | train_error: 15.1 | train_acc: 0.272 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.40% | train_error: 15.1 | train_acc: 0.270 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.50% | train_error: 15.1 | train_acc: 0.270 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.60% | train_error: 15.4 | train_acc: 0.258 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.70% | train_error: 15.4 | train_acc: 0.258 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.80% | train_error: 15.4 | train_acc: 0.258 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.90% | train_error: 15.4 | train_acc: 0.258 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.00% | train_error: 15.4 | train_acc: 0.258 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.10% | train_error: 15.4 | train_acc: 0.258 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.20% | train_error: 15.3 | train_acc: 0.261 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.30% | train_error: 15.5 | train_acc: 0.254 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.40% | train_error: 15.5 | train_acc: 0.254 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.50% | train_error: 15.5 | train_acc: 0.254 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.60% | train_error: 15.5 | train_acc: 0.254 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.70% | train_error: 15.4 | train_acc: 0.256 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.80% | train_error: 15.4 | train_acc: 0.256 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.90% | train_error: 15.4 | train_acc: 0.256 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.00% | train_error: 15.4 | train_acc: 0.256 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.10% | train_error: 15.6 | train_acc: 0.249 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.20% | train_error: 15.6 | train_acc: 0.249 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.30% | train_error: 15.6 | train_acc: 0.249 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.40% | train_error: 15.7 | train_acc: 0.244 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.50% | train_error: 15.7 | train_acc: 0.242 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.60% | train_error: 15.7 | train_acc: 0.242 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.70% | train_error: 15.7 | train_acc: 0.242 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.80% | train_error: 15.7 | train_acc: 0.242 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.90% | train_error: 15.8 | train_acc: 0.239 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.00% | train_error: 15.9 | train_acc: 0.235 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.10% | train_error: 15.9 | train_acc: 0.235 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.20% | train_error: 15.9 | train_acc: 0.232 | val_error: 14.6 | val_acc: 0.294 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.30% | train_error: 15.9 | train_acc: 0.232 | val_error: 14.6 | val_acc: 0.294 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.40% | train_error: 15.9 | train_acc: 0.232 | val_error: 14.6 | val_acc: 0.294 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.50% | train_error: 15.9 | train_acc: 0.232 | val_error: 14.9 | val_acc: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.60% | train_error: 15.9 | train_acc: 0.232 | val_error: 14.8 | val_acc: 0.287 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.70% | train_error: 15.9 | train_acc: 0.232 | val_error: 14.8 | val_acc: 0.287 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.80% | train_error: 16.0 | train_acc: 0.230 | val_error: 14.8 | val_acc: 0.287 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.90% | train_error: 16.0 | train_acc: 0.230 | val_error: 14.9 | val_acc: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.00% | train_error: 16.0 | train_acc: 0.230 | val_error: 14.9 | val_acc: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.10% | train_error: 16.0 | train_acc: 0.230 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.20% | train_error: 16.0 | train_acc: 0.230 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.30% | train_error: 16.0 | train_acc: 0.230 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.40% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.50% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.60% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.70% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.80% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.90% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.00% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.10% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.20% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.30% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.40% | train_error: 16.0 | train_acc: 0.228 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.50% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.00% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.10% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.20% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.30% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.40% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.50% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.00% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.10% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.20% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.30% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.40% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.50% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.00% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.10% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.20% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.30% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.40% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.50% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.00% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.10% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.20% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.30% | train_error: 16.0 | train_acc: 0.230 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.40% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.50% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.00% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.10% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.20% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.30% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.40% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.50% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.60% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.70% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.80% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.90% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.8 | val_acc: 0.238 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.00% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.10% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.20% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.30% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.40% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.50% | train_error: 15.7 | train_acc: 0.242 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.60% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.70% | train_error: 15.5 | train_acc: 0.251 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.80% | train_error: 15.5 | train_acc: 0.251 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.90% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.00% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.10% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.20% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.30% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.40% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.50% | train_error: 15.6 | train_acc: 0.246 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.00% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.10% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.20% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.30% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.40% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.50% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.60% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.70% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.80% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.90% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.7 | val_acc: 0.245 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.00% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.10% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.20% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.30% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.40% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.50% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.60% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.70% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.80% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.90% | train_error: 15.9 | train_acc: 0.235 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.00% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.10% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.20% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.30% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.40% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.50% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.60% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.70% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.80% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.90% | train_error: 15.8 | train_acc: 0.237 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.00% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.10% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.20% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.5 | val_acc: 0.252 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.30% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.40% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.50% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.60% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.70% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.80% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.90% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.00% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.10% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.20% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.30% | train_error: 15.7 | train_acc: 0.244 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.40% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.4 | val_acc: 0.259 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.50% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.60% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.70% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.80% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.90% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.00% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.10% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.20% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.30% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.40% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.50% | train_error: 15.6 | train_acc: 0.249 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.60% | train_error: 15.4 | train_acc: 0.256 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.70% | train_error: 15.3 | train_acc: 0.261 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.80% | train_error: 15.3 | train_acc: 0.261 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.90% | train_error: 15.4 | train_acc: 0.258 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.00% | train_error: 15.4 | train_acc: 0.258 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.10% | train_error: 15.4 | train_acc: 0.258 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.20% | train_error: 15.4 | train_acc: 0.258 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.30% | train_error: 15.2 | train_acc: 0.268 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.40% | train_error: 15.2 | train_acc: 0.268 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.50% | train_error: 15.2 | train_acc: 0.268 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.60% | train_error: 15.2 | train_acc: 0.268 | val_error: 15.2 | val_acc: 0.266 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.70% | train_error: 15.2 | train_acc: 0.268 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.80% | train_error: 15.2 | train_acc: 0.268 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.90% | train_error: 15.1 | train_acc: 0.272 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.00% | train_error: 15.1 | train_acc: 0.272 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.10% | train_error: 15.1 | train_acc: 0.272 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.20% | train_error: 15.1 | train_acc: 0.272 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.30% | train_error: 15.1 | train_acc: 0.272 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.40% | train_error: 15.1 | train_acc: 0.272 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.50% | train_error: 15.0 | train_acc: 0.277 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.60% | train_error: 15.0 | train_acc: 0.277 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.70% | train_error: 15.0 | train_acc: 0.277 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.80% | train_error: 15.0 | train_acc: 0.277 | val_error: 15.1 | val_acc: 0.273 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.90% | train_error: 15.0 | train_acc: 0.277 | val_error: 14.9 | val_acc: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.00% | train_error: 15.0 | train_acc: 0.277 | val_error: 14.9 | val_acc: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.10% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.9 | val_acc: 0.280 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.20% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.6 | val_acc: 0.294 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.30% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.40% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.5 | val_acc: 0.301 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.50% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.60% | train_error: 15.0 | train_acc: 0.277 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.70% | train_error: 15.0 | train_acc: 0.277 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.80% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.3 | val_acc: 0.308 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.90% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.00% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.10% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.20% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.30% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.40% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.50% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.60% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.70% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.80% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.90% | train_error: 14.9 | train_acc: 0.279 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.00% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.10% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.20% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.30% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.40% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.50% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.60% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.70% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.80% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.90% | train_error: 14.9 | train_acc: 0.282 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.00% | train_error: 14.8 | train_acc: 0.284 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.10% | train_error: 14.8 | train_acc: 0.284 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.20% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.30% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.40% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.50% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.60% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.70% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.80% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.90% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.00% | train_error: 14.7 | train_acc: 0.289 | val_error: 14.2 | val_acc: 0.315 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.10% | train_error: 14.7 | train_acc: 0.291 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.20% | train_error: 14.6 | train_acc: 0.296 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.30% | train_error: 14.6 | train_acc: 0.296 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.40% | train_error: 14.5 | train_acc: 0.300 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.50% | train_error: 14.5 | train_acc: 0.300 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.60% | train_error: 14.4 | train_acc: 0.308 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.70% | train_error: 14.4 | train_acc: 0.308 | val_error: 14.1 | val_acc: 0.322 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.80% | train_error: 14.4 | train_acc: 0.308 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.90% | train_error: 14.4 | train_acc: 0.308 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.00% | train_error: 14.3 | train_acc: 0.310 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.10% | train_error: 14.3 | train_acc: 0.310 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.20% | train_error: 14.3 | train_acc: 0.312 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.30% | train_error: 14.2 | train_acc: 0.315 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.40% | train_error: 14.2 | train_acc: 0.315 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.50% | train_error: 14.2 | train_acc: 0.315 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.60% | train_error: 14.2 | train_acc: 0.315 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.70% | train_error: 14.2 | train_acc: 0.315 | val_error: 13.9 | val_acc: 0.329 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.80% | train_error: 14.1 | train_acc: 0.319 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.90% | train_error: 14.1 | train_acc: 0.319 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.00% | train_error: 14.1 | train_acc: 0.322 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.10% | train_error: 14.1 | train_acc: 0.322 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.20% | train_error: 14.1 | train_acc: 0.322 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.30% | train_error: 14.0 | train_acc: 0.324 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.40% | train_error: 14.0 | train_acc: 0.324 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.50% | train_error: 14.0 | train_acc: 0.324 | val_error: 13.8 | val_acc: 0.336 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.60% | train_error: 14.0 | train_acc: 0.326 | val_error: 13.6 | val_acc: 0.343 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.70% | train_error: 13.9 | train_acc: 0.329 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.80% | train_error: 13.9 | train_acc: 0.329 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.90% | train_error: 13.7 | train_acc: 0.338 | val_error: 13.5 | val_acc: 0.350 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.00% | train_error: 13.6 | train_acc: 0.343 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.10% | train_error: 13.6 | train_acc: 0.343 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.20% | train_error: 13.6 | train_acc: 0.343 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.30% | train_error: 13.6 | train_acc: 0.343 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.40% | train_error: 13.6 | train_acc: 0.343 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.50% | train_error: 13.5 | train_acc: 0.347 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.60% | train_error: 13.5 | train_acc: 0.347 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.70% | train_error: 13.5 | train_acc: 0.347 | val_error: 13.3 | val_acc: 0.357 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.80% | train_error: 13.5 | train_acc: 0.347 | val_error: 13.2 | val_acc: 0.364 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.90% | train_error: 13.4 | train_acc: 0.352 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.00% | train_error: 13.4 | train_acc: 0.352 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.10% | train_error: 13.4 | train_acc: 0.352 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.20% | train_error: 13.4 | train_acc: 0.352 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.30% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.40% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.50% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.60% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.70% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.80% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.90% | train_error: 13.2 | train_acc: 0.362 | val_error: 12.9 | val_acc: 0.378 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.00% | train_error: 13.2 | train_acc: 0.364 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.10% | train_error: 13.2 | train_acc: 0.364 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.20% | train_error: 13.0 | train_acc: 0.373 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.30% | train_error: 13.0 | train_acc: 0.373 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.40% | train_error: 13.0 | train_acc: 0.371 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.50% | train_error: 13.0 | train_acc: 0.371 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.60% | train_error: 13.0 | train_acc: 0.371 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.70% | train_error: 12.8 | train_acc: 0.380 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.80% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.90% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.00% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.10% | train_error: 12.7 | train_acc: 0.385 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.20% | train_error: 12.7 | train_acc: 0.385 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.30% | train_error: 12.7 | train_acc: 0.385 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.40% | train_error: 12.7 | train_acc: 0.385 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.50% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.60% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.70% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.80% | train_error: 12.8 | train_acc: 0.380 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.90% | train_error: 12.8 | train_acc: 0.380 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.00% | train_error: 12.8 | train_acc: 0.380 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.10% | train_error: 12.8 | train_acc: 0.380 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.20% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.30% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.40% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.50% | train_error: 12.8 | train_acc: 0.383 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.60% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.70% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.80% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.90% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.00% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.10% | train_error: 12.6 | train_acc: 0.390 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.20% | train_error: 12.6 | train_acc: 0.392 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.30% | train_error: 12.6 | train_acc: 0.392 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.40% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.50% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.60% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.70% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.80% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.90% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.00% | train_error: 12.6 | train_acc: 0.394 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.10% | train_error: 12.5 | train_acc: 0.397 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.20% | train_error: 12.5 | train_acc: 0.397 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.30% | train_error: 12.5 | train_acc: 0.397 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.40% | train_error: 12.5 | train_acc: 0.397 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.50% | train_error: 12.5 | train_acc: 0.397 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.60% | train_error: 12.4 | train_acc: 0.401 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.70% | train_error: 12.4 | train_acc: 0.404 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.80% | train_error: 12.2 | train_acc: 0.411 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.90% | train_error: 12.1 | train_acc: 0.415 | val_error: 12.2 | val_acc: 0.413 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.00% | train_error: 12.1 | train_acc: 0.415 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.10% | train_error: 12.1 | train_acc: 0.415 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.20% | train_error: 12.1 | train_acc: 0.415 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.30% | train_error: 12.1 | train_acc: 0.418 | val_error: 12.2 | val_acc: 0.413 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.40% | train_error: 11.7 | train_acc: 0.437 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.50% | train_error: 11.7 | train_acc: 0.434 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.60% | train_error: 11.7 | train_acc: 0.434 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.70% | train_error: 11.7 | train_acc: 0.437 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.80% | train_error: 11.7 | train_acc: 0.437 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.90% | train_error: 11.7 | train_acc: 0.437 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.00% | train_error: 11.7 | train_acc: 0.437 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.10% | train_error: 11.7 | train_acc: 0.437 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.20% | train_error: 11.7 | train_acc: 0.437 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.30% | train_error: 11.7 | train_acc: 0.437 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.40% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.50% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.60% | train_error: 11.6 | train_acc: 0.441 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.70% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.80% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.90% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.00% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.10% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.20% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.30% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.40% | train_error: 11.3 | train_acc: 0.453 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.50% | train_error: 11.4 | train_acc: 0.451 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.60% | train_error: 11.4 | train_acc: 0.451 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.70% | train_error: 11.4 | train_acc: 0.451 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.80% | train_error: 11.4 | train_acc: 0.451 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.90% | train_error: 11.3 | train_acc: 0.455 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.00% | train_error: 11.2 | train_acc: 0.460 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.10% | train_error: 10.9 | train_acc: 0.472 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.20% | train_error: 10.9 | train_acc: 0.472 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.30% | train_error: 10.9 | train_acc: 0.472 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.40% | train_error: 10.9 | train_acc: 0.472 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.50% | train_error: 10.9 | train_acc: 0.472 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.60% | train_error: 10.9 | train_acc: 0.472 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.70% | train_error: 10.8 | train_acc: 0.477 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.80% | train_error: 10.8 | train_acc: 0.477 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.90% | train_error: 10.8 | train_acc: 0.477 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.00% | train_error: 10.7 | train_acc: 0.484 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.10% | train_error: 10.7 | train_acc: 0.484 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.20% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.30% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.40% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.50% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.60% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.70% | train_error: 10.6 | train_acc: 0.488 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.80% | train_error: 10.6 | train_acc: 0.488 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.90% | train_error: 10.6 | train_acc: 0.491 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.00% | train_error: 10.5 | train_acc: 0.493 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.10% | train_error: 10.5 | train_acc: 0.493 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.20% | train_error: 10.5 | train_acc: 0.493 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.30% | train_error: 10.5 | train_acc: 0.493 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.40% | train_error: 10.5 | train_acc: 0.493 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.50% | train_error: 10.5 | train_acc: 0.493 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.60% | train_error: 10.3 | train_acc: 0.502 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.70% | train_error: 10.3 | train_acc: 0.502 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.80% | train_error: 10.3 | train_acc: 0.502 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.90% | train_error: 10.3 | train_acc: 0.502 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.00% | train_error: 10.2 | train_acc: 0.507 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.10% | train_error: 10.2 | train_acc: 0.507 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.20% | train_error: 10.2 | train_acc: 0.509 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.30% | train_error: 10.2 | train_acc: 0.509 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.40% | train_error: 10.2 | train_acc: 0.509 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.50% | train_error: 10.2 | train_acc: 0.509 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.60% | train_error: 10.1 | train_acc: 0.512 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.70% | train_error: 10.1 | train_acc: 0.512 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.80% | train_error: 10.1 | train_acc: 0.512 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.90% | train_error: 10.1 | train_acc: 0.512 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.00% | train_error: 10.0 | train_acc: 0.516 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.10% | train_error: 9.97 | train_acc: 0.519 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.20% | train_error: 9.97 | train_acc: 0.519 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.30% | train_error: 9.88 | train_acc: 0.523 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.40% | train_error: 9.88 | train_acc: 0.523 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.50% | train_error: 9.78 | train_acc: 0.528 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.60% | train_error: 9.68 | train_acc: 0.533 | val_error: 10.3 | val_acc: 0.503 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.70% | train_error: 9.68 | train_acc: 0.533 | val_error: 10.00 | val_acc: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.80% | train_error: 9.53 | train_acc: 0.540 | val_error: 10.00 | val_acc: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.90% | train_error: 9.44 | train_acc: 0.545 | val_error: 10.00 | val_acc: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.00% | train_error: 9.39 | train_acc: 0.547 | val_error: 10.00 | val_acc: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.10% | train_error: 9.29 | train_acc: 0.552 | val_error: 10.00 | val_acc: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.20% | train_error: 9.29 | train_acc: 0.552 | val_error: 10.00 | val_acc: 0.517 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.30% | train_error: 9.24 | train_acc: 0.554 | val_error: 9.85 | val_acc: 0.524 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.40% | train_error: 9.24 | train_acc: 0.554 | val_error: 9.85 | val_acc: 0.524 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.50% | train_error: 8.85 | train_acc: 0.573 | val_error: 9.85 | val_acc: 0.524 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.60% | train_error: 8.85 | train_acc: 0.573 | val_error: 9.85 | val_acc: 0.524 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.70% | train_error: 8.76 | train_acc: 0.577 | val_error: 9.56 | val_acc: 0.538 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.80% | train_error: 8.76 | train_acc: 0.577 | val_error: 9.56 | val_acc: 0.538 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.90% | train_error: 8.76 | train_acc: 0.577 | val_error: 9.56 | val_acc: 0.538 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.00% | train_error: 8.76 | train_acc: 0.577 | val_error: 9.56 | val_acc: 0.538 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.10% | train_error: 8.71 | train_acc: 0.580 | val_error: 9.56 | val_acc: 0.538 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.20% | train_error: 8.71 | train_acc: 0.580 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.30% | train_error: 8.71 | train_acc: 0.580 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.40% | train_error: 8.56 | train_acc: 0.587 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.50% | train_error: 8.56 | train_acc: 0.587 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.60% | train_error: 8.51 | train_acc: 0.589 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.70% | train_error: 8.46 | train_acc: 0.592 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.80% | train_error: 8.32 | train_acc: 0.599 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.90% | train_error: 8.22 | train_acc: 0.603 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.00% | train_error: 8.12 | train_acc: 0.608 | val_error: 9.42 | val_acc: 0.545 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.10% | train_error: 8.08 | train_acc: 0.610 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.20% | train_error: 8.03 | train_acc: 0.613 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.30% | train_error: 8.03 | train_acc: 0.613 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.40% | train_error: 8.03 | train_acc: 0.613 | val_error: 9.13 | val_acc: 0.559 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.50% | train_error: 8.03 | train_acc: 0.613 | val_error: 9.13 | val_acc: 0.559 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.60% | train_error: 8.03 | train_acc: 0.613 | val_error: 8.98 | val_acc: 0.566 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.70% | train_error: 8.03 | train_acc: 0.613 | val_error: 8.98 | val_acc: 0.566 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.80% | train_error: 8.03 | train_acc: 0.613 | val_error: 8.84 | val_acc: 0.573 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.90% | train_error: 7.98 | train_acc: 0.615 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.00% | train_error: 7.98 | train_acc: 0.615 | val_error: 8.55 | val_acc: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.10% | train_error: 7.88 | train_acc: 0.620 | val_error: 8.55 | val_acc: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.20% | train_error: 7.88 | train_acc: 0.620 | val_error: 8.55 | val_acc: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.30% | train_error: 7.88 | train_acc: 0.620 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.40% | train_error: 7.88 | train_acc: 0.620 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.50% | train_error: 7.88 | train_acc: 0.620 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.60% | train_error: 7.73 | train_acc: 0.627 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.70% | train_error: 7.73 | train_acc: 0.627 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.80% | train_error: 7.73 | train_acc: 0.627 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.90% | train_error: 7.73 | train_acc: 0.627 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.00% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.55 | val_acc: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.10% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.55 | val_acc: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.20% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.41 | val_acc: 0.594 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.30% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.41 | val_acc: 0.594 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.40% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.26 | val_acc: 0.601 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.50% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.12 | val_acc: 0.608 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.60% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.12 | val_acc: 0.608 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.70% | train_error: 7.69 | train_acc: 0.629 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.80% | train_error: 7.54 | train_acc: 0.636 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.90% | train_error: 7.54 | train_acc: 0.636 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.00% | train_error: 7.54 | train_acc: 0.636 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.10% | train_error: 7.44 | train_acc: 0.641 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.20% | train_error: 7.44 | train_acc: 0.641 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.30% | train_error: 7.30 | train_acc: 0.648 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.40% | train_error: 7.30 | train_acc: 0.648 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.50% | train_error: 7.30 | train_acc: 0.648 | val_error: 7.83 | val_acc: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.60% | train_error: 7.10 | train_acc: 0.657 | val_error: 7.83 | val_acc: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.70% | train_error: 7.01 | train_acc: 0.662 | val_error: 7.83 | val_acc: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.80% | train_error: 6.91 | train_acc: 0.667 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.90% | train_error: 6.91 | train_acc: 0.667 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.00% | train_error: 6.81 | train_acc: 0.671 | val_error: 7.83 | val_acc: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.10% | train_error: 6.81 | train_acc: 0.671 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.20% | train_error: 6.81 | train_acc: 0.671 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.30% | train_error: 6.81 | train_acc: 0.671 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.40% | train_error: 6.76 | train_acc: 0.674 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.50% | train_error: 6.76 | train_acc: 0.674 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.60% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.70% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.80% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.90% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.00% | train_error: 6.91 | train_acc: 0.667 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.10% | train_error: 6.91 | train_acc: 0.667 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.20% | train_error: 6.86 | train_acc: 0.669 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.30% | train_error: 6.86 | train_acc: 0.669 | val_error: 7.68 | val_acc: 0.629 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.40% | train_error: 6.86 | train_acc: 0.669 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.50% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.60% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.70% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.80% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.90% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.00% | train_error: 6.71 | train_acc: 0.676 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.10% | train_error: 6.66 | train_acc: 0.678 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.20% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.30% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.39 | val_acc: 0.643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.40% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.39 | val_acc: 0.643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.50% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.39 | val_acc: 0.643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.60% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.39 | val_acc: 0.643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.70% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.25 | val_acc: 0.650 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.80% | train_error: 6.52 | train_acc: 0.685 | val_error: 7.25 | val_acc: 0.650 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.90% | train_error: 6.52 | train_acc: 0.685 | val_error: 7.25 | val_acc: 0.650 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.00% | train_error: 6.52 | train_acc: 0.685 | val_error: 7.25 | val_acc: 0.650 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.10% | train_error: 6.52 | train_acc: 0.685 | val_error: 7.25 | val_acc: 0.650 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.20% | train_error: 6.47 | train_acc: 0.688 | val_error: 7.25 | val_acc: 0.650 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.30% | train_error: 6.42 | train_acc: 0.690 | val_error: 7.10 | val_acc: 0.657 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.40% | train_error: 6.42 | train_acc: 0.690 | val_error: 7.10 | val_acc: 0.657 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.50% | train_error: 6.42 | train_acc: 0.690 | val_error: 7.10 | val_acc: 0.657 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.60% | train_error: 6.42 | train_acc: 0.690 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.70% | train_error: 6.37 | train_acc: 0.692 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.80% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.90% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.00% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.10% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.20% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.30% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.40% | train_error: 6.32 | train_acc: 0.695 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.50% | train_error: 6.23 | train_acc: 0.700 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.60% | train_error: 6.23 | train_acc: 0.700 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.70% | train_error: 6.23 | train_acc: 0.700 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.80% | train_error: 6.23 | train_acc: 0.700 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.90% | train_error: 6.18 | train_acc: 0.702 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.00% | train_error: 5.98 | train_acc: 0.711 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.10% | train_error: 5.93 | train_acc: 0.714 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.20% | train_error: 5.84 | train_acc: 0.718 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.30% | train_error: 5.84 | train_acc: 0.718 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.40% | train_error: 5.84 | train_acc: 0.718 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.50% | train_error: 5.79 | train_acc: 0.721 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.60% | train_error: 5.64 | train_acc: 0.728 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.70% | train_error: 5.64 | train_acc: 0.728 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.80% | train_error: 5.59 | train_acc: 0.730 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.90% | train_error: 5.59 | train_acc: 0.730 | val_error: 6.38 | val_acc: 0.692 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.00% | train_error: 5.59 | train_acc: 0.730 | val_error: 6.38 | val_acc: 0.692 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.10% | train_error: 5.50 | train_acc: 0.735 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.20% | train_error: 5.30 | train_acc: 0.744 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.30% | train_error: 5.30 | train_acc: 0.744 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.40% | train_error: 5.30 | train_acc: 0.744 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.50% | train_error: 5.16 | train_acc: 0.751 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.60% | train_error: 5.11 | train_acc: 0.754 | val_error: 6.09 | val_acc: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.70% | train_error: 5.06 | train_acc: 0.756 | val_error: 5.94 | val_acc: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.80% | train_error: 5.06 | train_acc: 0.756 | val_error: 5.94 | val_acc: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.90% | train_error: 5.06 | train_acc: 0.756 | val_error: 5.94 | val_acc: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.00% | train_error: 5.06 | train_acc: 0.756 | val_error: 5.94 | val_acc: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.10% | train_error: 5.01 | train_acc: 0.758 | val_error: 5.94 | val_acc: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.20% | train_error: 5.01 | train_acc: 0.758 | val_error: 5.94 | val_acc: 0.713 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.30% | train_error: 4.91 | train_acc: 0.763 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.40% | train_error: 4.91 | train_acc: 0.763 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.50% | train_error: 4.91 | train_acc: 0.763 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.60% | train_error: 4.91 | train_acc: 0.763 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.70% | train_error: 4.91 | train_acc: 0.763 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.80% | train_error: 4.91 | train_acc: 0.763 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.90% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.00% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.10% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.80 | val_acc: 0.720 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.20% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.30% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.40% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.50% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.60% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.70% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.80% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.90% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.00% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.10% | train_error: 4.67 | train_acc: 0.775 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.20% | train_error: 4.62 | train_acc: 0.777 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.30% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.22 | val_acc: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.40% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.22 | val_acc: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.50% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.22 | val_acc: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.60% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.22 | val_acc: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.70% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.22 | val_acc: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.80% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.22 | val_acc: 0.748 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.90% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.00% | train_error: 4.52 | train_acc: 0.782 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.10% | train_error: 4.52 | train_acc: 0.782 | val_error: 4.93 | val_acc: 0.762 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.20% | train_error: 4.43 | train_acc: 0.786 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.30% | train_error: 4.38 | train_acc: 0.789 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.40% | train_error: 4.13 | train_acc: 0.800 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.50% | train_error: 4.13 | train_acc: 0.800 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.60% | train_error: 4.13 | train_acc: 0.800 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.70% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.80% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.90% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.00% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.10% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.49 | val_acc: 0.783 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.20% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.49 | val_acc: 0.783 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.30% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.49 | val_acc: 0.783 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.40% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.49 | val_acc: 0.783 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.50% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.60% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.70% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.80% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.90% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.00% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.10% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.20% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.30% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.40% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.50% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.60% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.70% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.80% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.90% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.00% | train_error: 4.28 | train_acc: 0.793 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.10% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.20% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.30% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.40% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.50% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.60% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.70% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.80% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.90% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.00% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.10% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.20% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.30% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.40% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.50% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.60% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.70% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.80% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.90% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.00% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.10% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.20% | train_error: 4.33 | train_acc: 0.791 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.30% | train_error: 4.33 | train_acc: 0.791 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.40% | train_error: 4.33 | train_acc: 0.791 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.50% | train_error: 4.33 | train_acc: 0.791 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.60% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.70% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.80% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.90% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.00% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.10% | train_error: 4.23 | train_acc: 0.796 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.20% | train_error: 4.18 | train_acc: 0.798 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.30% | train_error: 4.18 | train_acc: 0.798 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.40% | train_error: 4.18 | train_acc: 0.798 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.50% | train_error: 4.18 | train_acc: 0.798 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.60% | train_error: 4.13 | train_acc: 0.800 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.70% | train_error: 4.13 | train_acc: 0.800 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.80% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.90% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.00% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.10% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.20% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.30% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.40% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.50% | train_error: 4.09 | train_acc: 0.803 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.60% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.70% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.80% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.90% | train_error: 3.99 | train_acc: 0.808 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.00% | train_error: 3.89 | train_acc: 0.812 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.10% | train_error: 3.89 | train_acc: 0.812 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.20% | train_error: 3.89 | train_acc: 0.812 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.30% | train_error: 3.89 | train_acc: 0.812 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.40% | train_error: 3.79 | train_acc: 0.817 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.50% | train_error: 3.79 | train_acc: 0.817 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.60% | train_error: 3.89 | train_acc: 0.812 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.70% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.80% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.90% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.00% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.10% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.20% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.30% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.40% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.50% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.60% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.70% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.80% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.90% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.00% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.10% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.20% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.30% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.40% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.50% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.60% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.70% | train_error: 3.70 | train_acc: 0.822 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.80% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.90% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.00% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.10% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.20% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.30% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.40% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.50% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.60% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.70% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.80% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.90% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.00% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.10% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.20% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.30% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.40% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.50% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.60% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.70% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.80% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.90% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.00% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.10% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.20% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.30% | train_error: 3.60 | train_acc: 0.826 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.40% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.50% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.60% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.70% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.80% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.90% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.00% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.10% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.20% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.30% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.40% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.50% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.60% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.70% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.80% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.90% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.00% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.10% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.20% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.30% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.40% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.50% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.60% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.70% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.80% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.90% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.00% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.10% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.20% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.30% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.40% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.50% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.60% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.70% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.80% | train_error: 3.65 | train_acc: 0.824 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.90% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.00% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.10% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.20% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.30% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.40% | train_error: 3.55 | train_acc: 0.829 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.50% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.60% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.70% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.80% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.90% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.00% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.10% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.20% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.30% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.40% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.50% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.60% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.70% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.80% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.90% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.00% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.10% | train_error: 3.50 | train_acc: 0.831 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.20% | train_error: 3.45 | train_acc: 0.833 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.30% | train_error: 3.45 | train_acc: 0.833 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.40% | train_error: 3.41 | train_acc: 0.836 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.50% | train_error: 3.36 | train_acc: 0.838 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.60% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.70% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.80% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.90% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.00% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.10% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.20% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.30% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.40% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.50% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.60% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.70% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.80% | train_error: 3.26 | train_acc: 0.843 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.90% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================================>] 100.0% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.77 | val_acc: 0.818 " ] } ], "source": [ "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", "\n", "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" ] }, { "cell_type": "markdown", "id": "8425a8fb", "metadata": { "editable": true }, "source": [ "Finally, we will create a neural network with 2 hidden layers with activation functions." ] }, { "cell_type": "code", "execution_count": 18, "id": "1070e9eb", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", "hidden_nodes1 = 100\n", "hidden_nodes2 = 30\n", "output_nodes = 1\n", "\n", "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", "\n", "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)" ] }, { "cell_type": "code", "execution_count": 19, "id": "c7171327", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adam: Eta=0.0001, Lambda=0\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.1000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.2000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.3000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.4000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.5000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.6000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.7000% | train_error: 11.5 | train_acc: 0.444 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.8000% | train_error: 11.5 | train_acc: 0.444 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.9000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.100% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.200% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.300% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.400% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.500% | train_error: 11.4 | train_acc: 0.451 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.600% | train_error: 11.5 | train_acc: 0.444 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.700% | train_error: 11.5 | train_acc: 0.444 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.800% | train_error: 11.6 | train_acc: 0.441 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.900% | train_error: 11.6 | train_acc: 0.441 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.000% | train_error: 11.6 | train_acc: 0.441 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.100% | train_error: 11.6 | train_acc: 0.439 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.200% | train_error: 11.6 | train_acc: 0.439 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.300% | train_error: 11.8 | train_acc: 0.430 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.400% | train_error: 11.9 | train_acc: 0.425 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.500% | train_error: 11.9 | train_acc: 0.425 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.600% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.700% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.800% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.900% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.000% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.100% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.200% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.300% | train_error: 11.9 | train_acc: 0.425 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.400% | train_error: 11.9 | train_acc: 0.427 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.500% | train_error: 11.9 | train_acc: 0.427 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.600% | train_error: 11.8 | train_acc: 0.432 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.700% | train_error: 11.6 | train_acc: 0.439 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.800% | train_error: 11.6 | train_acc: 0.439 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.900% | train_error: 11.7 | train_acc: 0.434 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.000% | train_error: 11.7 | train_acc: 0.434 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.100% | train_error: 11.8 | train_acc: 0.430 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.200% | train_error: 11.9 | train_acc: 0.425 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.300% | train_error: 12.0 | train_acc: 0.423 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.400% | train_error: 11.9 | train_acc: 0.427 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.500% | train_error: 11.8 | train_acc: 0.430 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.600% | train_error: 11.8 | train_acc: 0.432 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.700% | train_error: 11.8 | train_acc: 0.430 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.800% | train_error: 11.7 | train_acc: 0.434 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.900% | train_error: 11.5 | train_acc: 0.446 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.000% | train_error: 11.5 | train_acc: 0.446 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.100% | train_error: 11.5 | train_acc: 0.446 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.200% | train_error: 11.5 | train_acc: 0.446 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.300% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.400% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.500% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.0 | val_acc: 0.469 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.600% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.700% | train_error: 11.5 | train_acc: 0.444 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.800% | train_error: 11.9 | train_acc: 0.425 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.900% | train_error: 11.9 | train_acc: 0.425 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.000% | train_error: 12.0 | train_acc: 0.420 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.100% | train_error: 12.0 | train_acc: 0.420 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.200% | train_error: 12.1 | train_acc: 0.418 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.300% | train_error: 12.1 | train_acc: 0.418 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.400% | train_error: 11.9 | train_acc: 0.425 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.500% | train_error: 11.9 | train_acc: 0.427 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.600% | train_error: 11.8 | train_acc: 0.432 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.700% | train_error: 11.5 | train_acc: 0.444 | val_error: 12.2 | val_acc: 0.413 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.800% | train_error: 11.4 | train_acc: 0.448 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.900% | train_error: 11.2 | train_acc: 0.458 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.000% | train_error: 11.2 | train_acc: 0.458 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.100% | train_error: 11.3 | train_acc: 0.455 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.200% | train_error: 11.3 | train_acc: 0.455 | val_error: 12.6 | val_acc: 0.392 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.300% | train_error: 11.3 | train_acc: 0.455 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.400% | train_error: 11.3 | train_acc: 0.453 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.500% | train_error: 11.3 | train_acc: 0.455 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.600% | train_error: 11.5 | train_acc: 0.446 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.700% | train_error: 11.3 | train_acc: 0.453 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.800% | train_error: 11.4 | train_acc: 0.448 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.900% | train_error: 11.4 | train_acc: 0.448 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.000% | train_error: 11.4 | train_acc: 0.451 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.100% | train_error: 11.4 | train_acc: 0.448 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.200% | train_error: 11.3 | train_acc: 0.453 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.300% | train_error: 11.3 | train_acc: 0.453 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.400% | train_error: 11.4 | train_acc: 0.451 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.500% | train_error: 11.4 | train_acc: 0.451 | val_error: 12.5 | val_acc: 0.399 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.600% | train_error: 11.4 | train_acc: 0.451 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.700% | train_error: 11.2 | train_acc: 0.458 | val_error: 12.3 | val_acc: 0.406 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.800% | train_error: 11.2 | train_acc: 0.458 | val_error: 12.2 | val_acc: 0.413 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.900% | train_error: 11.2 | train_acc: 0.458 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.000% | train_error: 11.2 | train_acc: 0.458 | val_error: 12.0 | val_acc: 0.420 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.100% | train_error: 11.2 | train_acc: 0.458 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.200% | train_error: 11.2 | train_acc: 0.458 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.300% | train_error: 11.2 | train_acc: 0.460 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.400% | train_error: 11.0 | train_acc: 0.469 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.500% | train_error: 11.0 | train_acc: 0.469 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.600% | train_error: 10.8 | train_acc: 0.479 | val_error: 11.9 | val_acc: 0.427 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.700% | train_error: 10.7 | train_acc: 0.484 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.800% | train_error: 10.7 | train_acc: 0.484 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.900% | train_error: 10.7 | train_acc: 0.484 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.00% | train_error: 10.7 | train_acc: 0.484 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.10% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.7 | val_acc: 0.434 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.20% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.30% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.6 | val_acc: 0.441 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.40% | train_error: 10.7 | train_acc: 0.486 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.50% | train_error: 10.4 | train_acc: 0.500 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.60% | train_error: 10.4 | train_acc: 0.500 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.70% | train_error: 10.4 | train_acc: 0.500 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.80% | train_error: 10.4 | train_acc: 0.500 | val_error: 11.4 | val_acc: 0.448 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.90% | train_error: 10.2 | train_acc: 0.507 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.00% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.10% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.20% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.3 | val_acc: 0.455 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.30% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.40% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.50% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.60% | train_error: 9.83 | train_acc: 0.526 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.70% | train_error: 9.73 | train_acc: 0.531 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.80% | train_error: 9.63 | train_acc: 0.535 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.90% | train_error: 9.63 | train_acc: 0.535 | val_error: 11.2 | val_acc: 0.462 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.00% | train_error: 9.53 | train_acc: 0.540 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.10% | train_error: 9.53 | train_acc: 0.540 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.20% | train_error: 9.49 | train_acc: 0.542 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.30% | train_error: 9.49 | train_acc: 0.542 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.40% | train_error: 9.44 | train_acc: 0.545 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.50% | train_error: 9.44 | train_acc: 0.545 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.60% | train_error: 9.39 | train_acc: 0.547 | val_error: 10.9 | val_acc: 0.476 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.70% | train_error: 9.39 | train_acc: 0.547 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.80% | train_error: 9.39 | train_acc: 0.547 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.90% | train_error: 9.34 | train_acc: 0.549 | val_error: 10.7 | val_acc: 0.483 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.00% | train_error: 9.10 | train_acc: 0.561 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.10% | train_error: 9.05 | train_acc: 0.563 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.20% | train_error: 9.05 | train_acc: 0.563 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.30% | train_error: 8.80 | train_acc: 0.575 | val_error: 10.6 | val_acc: 0.490 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.40% | train_error: 8.76 | train_acc: 0.577 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.50% | train_error: 8.56 | train_acc: 0.587 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.60% | train_error: 8.56 | train_acc: 0.587 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.70% | train_error: 8.42 | train_acc: 0.594 | val_error: 10.4 | val_acc: 0.497 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.80% | train_error: 8.42 | train_acc: 0.594 | val_error: 10.3 | val_acc: 0.503 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.90% | train_error: 8.42 | train_acc: 0.594 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.00% | train_error: 8.42 | train_acc: 0.594 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.10% | train_error: 8.32 | train_acc: 0.599 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.20% | train_error: 8.32 | train_acc: 0.599 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.30% | train_error: 8.22 | train_acc: 0.603 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.40% | train_error: 8.22 | train_acc: 0.603 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.50% | train_error: 8.22 | train_acc: 0.603 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.60% | train_error: 8.22 | train_acc: 0.603 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.70% | train_error: 8.17 | train_acc: 0.606 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.80% | train_error: 8.17 | train_acc: 0.606 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.90% | train_error: 8.17 | train_acc: 0.606 | val_error: 10.1 | val_acc: 0.510 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.00% | train_error: 8.08 | train_acc: 0.610 | val_error: 9.85 | val_acc: 0.524 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.10% | train_error: 7.98 | train_acc: 0.615 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.20% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.30% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.40% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.50% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.60% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.70% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.71 | val_acc: 0.531 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.80% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.56 | val_acc: 0.538 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.90% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.00% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.10% | train_error: 7.93 | train_acc: 0.617 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.20% | train_error: 7.88 | train_acc: 0.620 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.30% | train_error: 7.69 | train_acc: 0.629 | val_error: 9.27 | val_acc: 0.552 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.40% | train_error: 7.69 | train_acc: 0.629 | val_error: 9.13 | val_acc: 0.559 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.50% | train_error: 7.69 | train_acc: 0.629 | val_error: 9.13 | val_acc: 0.559 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.60% | train_error: 7.69 | train_acc: 0.629 | val_error: 9.13 | val_acc: 0.559 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.70% | train_error: 7.69 | train_acc: 0.629 | val_error: 8.98 | val_acc: 0.566 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.80% | train_error: 7.59 | train_acc: 0.634 | val_error: 8.84 | val_acc: 0.573 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.90% | train_error: 7.49 | train_acc: 0.638 | val_error: 8.84 | val_acc: 0.573 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.00% | train_error: 7.39 | train_acc: 0.643 | val_error: 8.70 | val_acc: 0.580 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.10% | train_error: 7.39 | train_acc: 0.643 | val_error: 8.55 | val_acc: 0.587 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.20% | train_error: 7.30 | train_acc: 0.648 | val_error: 8.41 | val_acc: 0.594 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.30% | train_error: 7.30 | train_acc: 0.648 | val_error: 8.26 | val_acc: 0.601 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.40% | train_error: 7.15 | train_acc: 0.655 | val_error: 8.26 | val_acc: 0.601 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.50% | train_error: 7.05 | train_acc: 0.660 | val_error: 8.12 | val_acc: 0.608 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.60% | train_error: 7.05 | train_acc: 0.660 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.70% | train_error: 6.91 | train_acc: 0.667 | val_error: 7.97 | val_acc: 0.615 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.80% | train_error: 6.86 | train_acc: 0.669 | val_error: 7.83 | val_acc: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.90% | train_error: 6.86 | train_acc: 0.669 | val_error: 7.83 | val_acc: 0.622 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.00% | train_error: 6.86 | train_acc: 0.669 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.10% | train_error: 6.76 | train_acc: 0.674 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.20% | train_error: 6.76 | train_acc: 0.674 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.30% | train_error: 6.62 | train_acc: 0.681 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.40% | train_error: 6.57 | train_acc: 0.683 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.50% | train_error: 6.47 | train_acc: 0.688 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.60% | train_error: 6.47 | train_acc: 0.688 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.70% | train_error: 6.37 | train_acc: 0.692 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.80% | train_error: 6.32 | train_acc: 0.695 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.90% | train_error: 6.28 | train_acc: 0.697 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.00% | train_error: 6.28 | train_acc: 0.697 | val_error: 7.54 | val_acc: 0.636 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.10% | train_error: 6.28 | train_acc: 0.697 | val_error: 7.39 | val_acc: 0.643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.20% | train_error: 6.28 | train_acc: 0.697 | val_error: 7.39 | val_acc: 0.643 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.30% | train_error: 6.28 | train_acc: 0.697 | val_error: 7.10 | val_acc: 0.657 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.40% | train_error: 6.18 | train_acc: 0.702 | val_error: 7.10 | val_acc: 0.657 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.50% | train_error: 6.18 | train_acc: 0.702 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.60% | train_error: 6.13 | train_acc: 0.704 | val_error: 6.81 | val_acc: 0.671 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.70% | train_error: 6.13 | train_acc: 0.704 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.80% | train_error: 6.13 | train_acc: 0.704 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.90% | train_error: 6.13 | train_acc: 0.704 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.00% | train_error: 6.13 | train_acc: 0.704 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.10% | train_error: 6.08 | train_acc: 0.707 | val_error: 6.38 | val_acc: 0.692 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.20% | train_error: 6.03 | train_acc: 0.709 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.30% | train_error: 5.64 | train_acc: 0.728 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.40% | train_error: 5.69 | train_acc: 0.725 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.50% | train_error: 5.69 | train_acc: 0.725 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.60% | train_error: 5.59 | train_acc: 0.730 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.70% | train_error: 5.59 | train_acc: 0.730 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.80% | train_error: 5.55 | train_acc: 0.732 | val_error: 6.67 | val_acc: 0.678 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.90% | train_error: 5.50 | train_acc: 0.735 | val_error: 6.52 | val_acc: 0.685 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.00% | train_error: 5.35 | train_acc: 0.742 | val_error: 6.38 | val_acc: 0.692 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.10% | train_error: 5.35 | train_acc: 0.742 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.20% | train_error: 5.30 | train_acc: 0.744 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.30% | train_error: 5.21 | train_acc: 0.749 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.40% | train_error: 5.06 | train_acc: 0.756 | val_error: 6.23 | val_acc: 0.699 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.50% | train_error: 5.06 | train_acc: 0.756 | val_error: 6.09 | val_acc: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.60% | train_error: 5.06 | train_acc: 0.756 | val_error: 6.09 | val_acc: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.70% | train_error: 5.01 | train_acc: 0.758 | val_error: 6.09 | val_acc: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.80% | train_error: 5.01 | train_acc: 0.758 | val_error: 6.09 | val_acc: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.90% | train_error: 4.96 | train_acc: 0.761 | val_error: 6.09 | val_acc: 0.706 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.00% | train_error: 4.96 | train_acc: 0.761 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.10% | train_error: 4.72 | train_acc: 0.772 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.20% | train_error: 4.62 | train_acc: 0.777 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.30% | train_error: 4.52 | train_acc: 0.782 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.40% | train_error: 4.62 | train_acc: 0.777 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.50% | train_error: 4.52 | train_acc: 0.782 | val_error: 5.51 | val_acc: 0.734 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.60% | train_error: 4.57 | train_acc: 0.779 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.70% | train_error: 4.52 | train_acc: 0.782 | val_error: 5.65 | val_acc: 0.727 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.80% | train_error: 4.33 | train_acc: 0.791 | val_error: 5.51 | val_acc: 0.734 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.90% | train_error: 4.33 | train_acc: 0.791 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.00% | train_error: 4.33 | train_acc: 0.791 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.10% | train_error: 4.33 | train_acc: 0.791 | val_error: 5.36 | val_acc: 0.741 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.20% | train_error: 4.33 | train_acc: 0.791 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.30% | train_error: 4.33 | train_acc: 0.791 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.40% | train_error: 4.28 | train_acc: 0.793 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.50% | train_error: 4.28 | train_acc: 0.793 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.60% | train_error: 4.13 | train_acc: 0.800 | val_error: 5.07 | val_acc: 0.755 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.70% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.93 | val_acc: 0.762 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.80% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.90% | train_error: 4.04 | train_acc: 0.805 | val_error: 4.78 | val_acc: 0.769 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.00% | train_error: 3.89 | train_acc: 0.812 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.10% | train_error: 3.84 | train_acc: 0.815 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.20% | train_error: 3.79 | train_acc: 0.817 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.30% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.40% | train_error: 3.70 | train_acc: 0.822 | val_error: 4.49 | val_acc: 0.783 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.50% | train_error: 3.65 | train_acc: 0.824 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.60% | train_error: 3.55 | train_acc: 0.829 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.70% | train_error: 3.55 | train_acc: 0.829 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.80% | train_error: 3.50 | train_acc: 0.831 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.90% | train_error: 3.45 | train_acc: 0.833 | val_error: 4.64 | val_acc: 0.776 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.00% | train_error: 3.21 | train_acc: 0.845 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.10% | train_error: 3.21 | train_acc: 0.845 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.20% | train_error: 3.11 | train_acc: 0.850 | val_error: 4.20 | val_acc: 0.797 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.30% | train_error: 3.11 | train_acc: 0.850 | val_error: 4.35 | val_acc: 0.790 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.40% | train_error: 3.11 | train_acc: 0.850 | val_error: 4.06 | val_acc: 0.804 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.50% | train_error: 3.11 | train_acc: 0.850 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.60% | train_error: 3.11 | train_acc: 0.850 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.70% | train_error: 3.11 | train_acc: 0.850 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.80% | train_error: 3.11 | train_acc: 0.850 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.90% | train_error: 3.11 | train_acc: 0.850 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.00% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.10% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.20% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.30% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.40% | train_error: 3.21 | train_acc: 0.845 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.50% | train_error: 3.02 | train_acc: 0.854 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.60% | train_error: 2.77 | train_acc: 0.866 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.70% | train_error: 2.77 | train_acc: 0.866 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.80% | train_error: 2.72 | train_acc: 0.869 | val_error: 3.91 | val_acc: 0.811 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.90% | train_error: 2.68 | train_acc: 0.871 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.00% | train_error: 2.53 | train_acc: 0.878 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.10% | train_error: 2.48 | train_acc: 0.880 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.20% | train_error: 2.48 | train_acc: 0.880 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.30% | train_error: 2.38 | train_acc: 0.885 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.40% | train_error: 2.38 | train_acc: 0.885 | val_error: 3.77 | val_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.50% | train_error: 2.38 | train_acc: 0.885 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.60% | train_error: 2.29 | train_acc: 0.890 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.70% | train_error: 2.24 | train_acc: 0.892 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.80% | train_error: 2.24 | train_acc: 0.892 | val_error: 3.62 | val_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.90% | train_error: 2.19 | train_acc: 0.894 | val_error: 3.48 | val_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.00% | train_error: 2.19 | train_acc: 0.894 | val_error: 3.33 | val_acc: 0.839 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.10% | train_error: 2.19 | train_acc: 0.894 | val_error: 3.19 | val_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.20% | train_error: 2.19 | train_acc: 0.894 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.30% | train_error: 2.19 | train_acc: 0.894 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.40% | train_error: 2.14 | train_acc: 0.897 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.50% | train_error: 2.09 | train_acc: 0.899 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.60% | train_error: 2.04 | train_acc: 0.901 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.70% | train_error: 2.09 | train_acc: 0.899 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.80% | train_error: 2.09 | train_acc: 0.899 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.90% | train_error: 2.09 | train_acc: 0.899 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.00% | train_error: 2.09 | train_acc: 0.899 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.10% | train_error: 2.09 | train_acc: 0.899 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.20% | train_error: 2.09 | train_acc: 0.899 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.30% | train_error: 2.04 | train_acc: 0.901 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.40% | train_error: 2.04 | train_acc: 0.901 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.50% | train_error: 2.04 | train_acc: 0.901 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.60% | train_error: 2.04 | train_acc: 0.901 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.70% | train_error: 2.04 | train_acc: 0.901 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.80% | train_error: 1.99 | train_acc: 0.904 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.90% | train_error: 1.95 | train_acc: 0.906 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.00% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.10% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.20% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.19 | val_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.30% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.19 | val_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.40% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.19 | val_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.50% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.19 | val_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.60% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.19 | val_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.70% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.80% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.90% | train_error: 1.90 | train_acc: 0.908 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.00% | train_error: 1.75 | train_acc: 0.915 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.10% | train_error: 1.75 | train_acc: 0.915 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.20% | train_error: 1.61 | train_acc: 0.923 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.30% | train_error: 1.61 | train_acc: 0.923 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.40% | train_error: 1.56 | train_acc: 0.925 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.50% | train_error: 1.56 | train_acc: 0.925 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.60% | train_error: 1.56 | train_acc: 0.925 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.70% | train_error: 1.51 | train_acc: 0.927 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.80% | train_error: 1.51 | train_acc: 0.927 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.90% | train_error: 1.51 | train_acc: 0.927 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.00% | train_error: 1.51 | train_acc: 0.927 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.10% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.20% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.30% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.40% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.50% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.60% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.70% | train_error: 1.41 | train_acc: 0.932 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.80% | train_error: 1.31 | train_acc: 0.937 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.90% | train_error: 1.26 | train_acc: 0.939 | val_error: 3.04 | val_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.00% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.10% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.20% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.30% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.90 | val_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.40% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.50% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.60% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.70% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.80% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.90% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.00% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.10% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.20% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.30% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.40% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.50% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.60% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.70% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.80% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.90% | train_error: 1.22 | train_acc: 0.941 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.00% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.10% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.20% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.75 | val_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.30% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.40% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.50% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.60% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.70% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.80% | train_error: 1.02 | train_acc: 0.951 | val_error: 2.61 | val_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.90% | train_error: 0.876 | train_acc: 0.958 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.00% | train_error: 0.876 | train_acc: 0.958 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.10% | train_error: 0.778 | train_acc: 0.962 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.20% | train_error: 0.778 | train_acc: 0.962 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.30% | train_error: 0.778 | train_acc: 0.962 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.40% | train_error: 0.730 | train_acc: 0.965 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.50% | train_error: 0.730 | train_acc: 0.965 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.60% | train_error: 0.730 | train_acc: 0.965 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.70% | train_error: 0.730 | train_acc: 0.965 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.80% | train_error: 0.730 | train_acc: 0.965 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.90% | train_error: 0.730 | train_acc: 0.965 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.00% | train_error: 0.681 | train_acc: 0.967 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.10% | train_error: 0.681 | train_acc: 0.967 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.20% | train_error: 0.681 | train_acc: 0.967 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.30% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.40% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.50% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.60% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.70% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.80% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.90% | train_error: 0.632 | train_acc: 0.969 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.00% | train_error: 0.535 | train_acc: 0.974 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.10% | train_error: 0.535 | train_acc: 0.974 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.20% | train_error: 0.486 | train_acc: 0.977 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.30% | train_error: 0.341 | train_acc: 0.984 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.40% | train_error: 0.486 | train_acc: 0.977 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.50% | train_error: 0.341 | train_acc: 0.984 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.60% | train_error: 0.486 | train_acc: 0.977 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.70% | train_error: 0.341 | train_acc: 0.984 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.80% | train_error: 0.486 | train_acc: 0.977 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.90% | train_error: 0.341 | train_acc: 0.984 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.00% | train_error: 0.486 | train_acc: 0.977 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.10% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.20% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.30% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.40% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.50% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.60% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.70% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.80% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.90% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.00% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.10% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.20% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.30% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.40% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.50% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.60% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.70% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.80% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.90% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.00% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.10% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.20% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.30% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.40% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.50% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.60% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.70% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.80% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.90% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.00% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.10% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.20% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.30% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.40% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.50% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.60% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.70% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.80% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.90% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.00% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.10% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.20% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.30% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.40% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.50% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.60% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.70% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.80% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.90% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.00% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.10% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.20% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.46 | val_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.30% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.40% | train_error: 0.292 | train_acc: 0.986 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.50% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.60% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.70% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.80% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.90% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.00% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.32 | val_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.10% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.20% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.30% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.40% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.50% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.60% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.70% | train_error: 0.243 | train_acc: 0.988 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.80% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.90% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.00% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.10% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.20% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.30% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.40% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.50% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.60% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.70% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.80% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.90% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.10% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.30% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.50% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.60% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.70% | train_error: 0.195 | train_acc: 0.991 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.80% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.90% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.10% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.30% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.50% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.60% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.70% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.80% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.17 | val_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 2.03 | val_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.60% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.80% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.10% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.30% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.50% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.60% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.70% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.80% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.90% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.10% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.30% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.50% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.60% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.70% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.80% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.90% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.00% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.10% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.20% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.30% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.40% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.50% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.60% | train_error: 0.146 | train_acc: 0.993 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.70% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.90% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.10% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.30% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.50% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.70% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.88 | val_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.90% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.10% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.30% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.50% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.70% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.90% | train_error: 0.0486 | train_acc: 0.998 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.59 | val_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.00% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.10% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.20% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.30% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.40% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.50% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.60% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.70% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.80% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.90% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================================>] 100.0% | train_error: 0.0973 | train_acc: 0.995 | val_error: 1.74 | val_acc: 0.916 " ] } ], "source": [ "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", "\n", "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" ] }, { "cell_type": "markdown", "id": "9c37cc8a", "metadata": { "editable": true }, "source": [ "### Multiclass classification\n", "\n", "Finally, we will demonstrate the use case of multiclass classification\n", "using our FFNN with the famous MNIST dataset, which contain images of\n", "digits between the range of 0 to 9." ] }, { "cell_type": "code", "execution_count": 20, "id": "f33e3872", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adam: Eta=0.0001, Lambda=0\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.1000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.2000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.3000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.4000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.5000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.6000% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.7000% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.8000% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.9000% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.000% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.100% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.200% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.300% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.400% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.500% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.600% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.700% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.800% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.900% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.000% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.100% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.200% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.300% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.400% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.500% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.600% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.700% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.800% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.900% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.000% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.100% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.200% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.300% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.400% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.500% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.600% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.700% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.800% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.900% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.000% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.100% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.200% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.300% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.400% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.500% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.600% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.700% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.800% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.900% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.000% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.100% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.200% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.300% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.400% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.500% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.600% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.700% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.800% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.900% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.000% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.100% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.200% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.300% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.400% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.500% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.600% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.700% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.800% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.900% | train_error: 1.85 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.000% | train_error: 1.85 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.100% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.200% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.300% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.400% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.500% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.600% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.700% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.800% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.900% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.000% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.100% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.200% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.300% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.400% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.500% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.600% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.700% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.800% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.900% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.000% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.100% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.200% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.300% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.400% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.500% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.600% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.700% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.800% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.900% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.20% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.30% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.40% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.50% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.20% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.30% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.40% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.50% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.70% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.80% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.90% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.00% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.10% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.20% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.30% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.40% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.50% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.60% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.10% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.20% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.30% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.40% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.50% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.60% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.70% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.80% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.90% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.00% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.10% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.20% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.30% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.40% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.50% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.60% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.70% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.80% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.90% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.00% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.10% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.20% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.30% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.40% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.50% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.60% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.70% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.80% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.90% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.00% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.10% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.20% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.30% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.40% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.50% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.20% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.30% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.40% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.50% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.20% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.30% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.40% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.50% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.60% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.90% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.10% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.20% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.30% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.40% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.50% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.80% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.20% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.30% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.40% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.50% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.20% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.30% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.40% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.50% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.60% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.70% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.80% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.90% | train_error: 1.88 | train_acc: 0.818 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.00% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.10% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.20% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.30% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.40% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.50% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.60% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.70% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.80% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.90% | train_error: 1.88 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.00% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.10% | train_error: 1.87 | train_acc: 0.819 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.20% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.30% | train_error: 1.87 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.40% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.50% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.60% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.70% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.80% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.90% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.00% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.10% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.20% | train_error: 1.86 | train_acc: 0.820 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.30% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.40% | train_error: 1.86 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.50% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.60% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.70% | train_error: 1.85 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.80% | train_error: 1.85 | train_acc: 0.821 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.90% | train_error: 1.85 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.00% | train_error: 1.85 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.10% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.20% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.30% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.40% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.50% | train_error: 1.84 | train_acc: 0.822 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.60% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.70% | train_error: 1.84 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.80% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.90% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.00% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.10% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.20% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.30% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.40% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.50% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.60% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.70% | train_error: 1.83 | train_acc: 0.823 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.80% | train_error: 1.83 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.90% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.00% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.10% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.20% | train_error: 1.82 | train_acc: 0.824 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.30% | train_error: 1.82 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.40% | train_error: 1.82 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.50% | train_error: 1.82 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.60% | train_error: 1.82 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.70% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.80% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.90% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.00% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.10% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.20% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.30% | train_error: 1.81 | train_acc: 0.825 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.40% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.50% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.60% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.70% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.80% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.90% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.00% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.10% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.20% | train_error: 1.80 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.30% | train_error: 1.80 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.40% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.50% | train_error: 1.80 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.60% | train_error: 1.80 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.70% | train_error: 1.80 | train_acc: 0.826 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.80% | train_error: 1.79 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.90% | train_error: 1.79 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.00% | train_error: 1.79 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.10% | train_error: 1.79 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.20% | train_error: 1.79 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.30% | train_error: 1.79 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.40% | train_error: 1.79 | train_acc: 0.827 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.50% | train_error: 1.79 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.60% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.70% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.80% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.90% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.00% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.10% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.20% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.30% | train_error: 1.78 | train_acc: 0.828 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.40% | train_error: 1.78 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.50% | train_error: 1.78 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.60% | train_error: 1.77 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.70% | train_error: 1.78 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.80% | train_error: 1.78 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.90% | train_error: 1.77 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.00% | train_error: 1.77 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.10% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.20% | train_error: 1.77 | train_acc: 0.829 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.30% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.40% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.50% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.60% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.70% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.80% | train_error: 1.76 | train_acc: 0.830 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.90% | train_error: 1.76 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.00% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.10% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.20% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.30% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.40% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.50% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.60% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.70% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.80% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.90% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.00% | train_error: 1.75 | train_acc: 0.831 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.10% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.20% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.30% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.40% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.50% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.60% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.70% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.80% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.90% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.00% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.10% | train_error: 1.73 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.20% | train_error: 1.73 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.30% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.40% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.50% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.60% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.70% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.80% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.90% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.00% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.10% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.20% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.30% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.40% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.50% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.60% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.70% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.80% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.90% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.00% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.10% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.20% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.30% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.40% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.50% | train_error: 1.74 | train_acc: 0.832 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.60% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.70% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.80% | train_error: 1.73 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.90% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.00% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.10% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.20% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.30% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.40% | train_error: 1.73 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.50% | train_error: 1.73 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.60% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.70% | train_error: 1.73 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.80% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.90% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.00% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.10% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.20% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.30% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.40% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.50% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.60% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.70% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.80% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.90% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.00% | train_error: 1.73 | train_acc: 0.833 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.10% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.20% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.30% | train_error: 1.72 | train_acc: 0.834 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.40% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.50% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.60% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.70% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.80% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.90% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.00% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.10% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.20% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.30% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.40% | train_error: 1.71 | train_acc: 0.835 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.50% | train_error: 1.70 | train_acc: 0.836 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.60% | train_error: 1.70 | train_acc: 0.836 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.70% | train_error: 1.70 | train_acc: 0.836 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.80% | train_error: 1.70 | train_acc: 0.836 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.90% | train_error: 1.70 | train_acc: 0.836 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.00% | train_error: 1.70 | train_acc: 0.836 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.10% | train_error: 1.69 | train_acc: 0.837 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.20% | train_error: 1.69 | train_acc: 0.837 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.30% | train_error: 1.69 | train_acc: 0.837 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.40% | train_error: 1.69 | train_acc: 0.837 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.50% | train_error: 1.69 | train_acc: 0.837 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.60% | train_error: 1.69 | train_acc: 0.837 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.70% | train_error: 1.68 | train_acc: 0.838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.80% | train_error: 1.68 | train_acc: 0.838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.90% | train_error: 1.68 | train_acc: 0.838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.00% | train_error: 1.68 | train_acc: 0.838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.10% | train_error: 1.68 | train_acc: 0.838 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.20% | train_error: 1.67 | train_acc: 0.839 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.30% | train_error: 1.67 | train_acc: 0.839 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.40% | train_error: 1.67 | train_acc: 0.839 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.50% | train_error: 1.67 | train_acc: 0.839 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.60% | train_error: 1.66 | train_acc: 0.840 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.70% | train_error: 1.66 | train_acc: 0.840 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.80% | train_error: 1.65 | train_acc: 0.840 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.90% | train_error: 1.65 | train_acc: 0.841 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.00% | train_error: 1.65 | train_acc: 0.841 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.10% | train_error: 1.64 | train_acc: 0.841 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.20% | train_error: 1.64 | train_acc: 0.842 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.30% | train_error: 1.62 | train_acc: 0.843 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.40% | train_error: 1.62 | train_acc: 0.843 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.50% | train_error: 1.62 | train_acc: 0.843 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.60% | train_error: 1.62 | train_acc: 0.844 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.70% | train_error: 1.61 | train_acc: 0.844 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.80% | train_error: 1.61 | train_acc: 0.845 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.90% | train_error: 1.60 | train_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.00% | train_error: 1.59 | train_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.10% | train_error: 1.59 | train_acc: 0.846 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.20% | train_error: 1.59 | train_acc: 0.847 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.30% | train_error: 1.58 | train_acc: 0.847 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.40% | train_error: 1.58 | train_acc: 0.847 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.50% | train_error: 1.58 | train_acc: 0.848 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.60% | train_error: 1.57 | train_acc: 0.848 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.70% | train_error: 1.57 | train_acc: 0.849 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.80% | train_error: 1.56 | train_acc: 0.849 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.90% | train_error: 1.56 | train_acc: 0.850 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.00% | train_error: 1.55 | train_acc: 0.850 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.10% | train_error: 1.55 | train_acc: 0.851 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.20% | train_error: 1.55 | train_acc: 0.851 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.30% | train_error: 1.54 | train_acc: 0.852 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.40% | train_error: 1.53 | train_acc: 0.852 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.50% | train_error: 1.53 | train_acc: 0.852 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.60% | train_error: 1.53 | train_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.70% | train_error: 1.52 | train_acc: 0.853 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.80% | train_error: 1.52 | train_acc: 0.854 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.90% | train_error: 1.52 | train_acc: 0.854 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.00% | train_error: 1.51 | train_acc: 0.854 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.10% | train_error: 1.51 | train_acc: 0.854 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.20% | train_error: 1.50 | train_acc: 0.855 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.30% | train_error: 1.50 | train_acc: 0.855 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.40% | train_error: 1.49 | train_acc: 0.856 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.50% | train_error: 1.47 | train_acc: 0.858 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.60% | train_error: 1.47 | train_acc: 0.858 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.70% | train_error: 1.47 | train_acc: 0.858 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.80% | train_error: 1.46 | train_acc: 0.859 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.90% | train_error: 1.46 | train_acc: 0.859 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.00% | train_error: 1.46 | train_acc: 0.859 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.10% | train_error: 1.46 | train_acc: 0.859 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.20% | train_error: 1.46 | train_acc: 0.859 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.30% | train_error: 1.46 | train_acc: 0.859 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.40% | train_error: 1.45 | train_acc: 0.860 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.50% | train_error: 1.44 | train_acc: 0.861 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.60% | train_error: 1.44 | train_acc: 0.861 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.70% | train_error: 1.44 | train_acc: 0.861 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.80% | train_error: 1.44 | train_acc: 0.861 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.90% | train_error: 1.43 | train_acc: 0.862 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.00% | train_error: 1.42 | train_acc: 0.863 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.10% | train_error: 1.41 | train_acc: 0.864 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.20% | train_error: 1.40 | train_acc: 0.865 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.30% | train_error: 1.39 | train_acc: 0.866 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.40% | train_error: 1.39 | train_acc: 0.866 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.50% | train_error: 1.39 | train_acc: 0.866 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.60% | train_error: 1.39 | train_acc: 0.866 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.70% | train_error: 1.38 | train_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.80% | train_error: 1.38 | train_acc: 0.867 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.90% | train_error: 1.37 | train_acc: 0.868 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.00% | train_error: 1.36 | train_acc: 0.869 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.10% | train_error: 1.35 | train_acc: 0.869 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.20% | train_error: 1.35 | train_acc: 0.870 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.30% | train_error: 1.34 | train_acc: 0.870 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.40% | train_error: 1.34 | train_acc: 0.870 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.50% | train_error: 1.34 | train_acc: 0.870 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.60% | train_error: 1.34 | train_acc: 0.871 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.70% | train_error: 1.33 | train_acc: 0.872 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.80% | train_error: 1.32 | train_acc: 0.873 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.90% | train_error: 1.31 | train_acc: 0.873 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.00% | train_error: 1.31 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.10% | train_error: 1.31 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.20% | train_error: 1.31 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.30% | train_error: 1.31 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.40% | train_error: 1.31 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.50% | train_error: 1.31 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.60% | train_error: 1.30 | train_acc: 0.874 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.70% | train_error: 1.30 | train_acc: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.80% | train_error: 1.30 | train_acc: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.90% | train_error: 1.29 | train_acc: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.00% | train_error: 1.29 | train_acc: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.10% | train_error: 1.29 | train_acc: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.20% | train_error: 1.29 | train_acc: 0.875 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.30% | train_error: 1.29 | train_acc: 0.876 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.40% | train_error: 1.28 | train_acc: 0.876 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.50% | train_error: 1.28 | train_acc: 0.876 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.60% | train_error: 1.28 | train_acc: 0.877 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.70% | train_error: 1.28 | train_acc: 0.877 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.80% | train_error: 1.27 | train_acc: 0.877 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.90% | train_error: 1.27 | train_acc: 0.877 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.00% | train_error: 1.27 | train_acc: 0.877 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.10% | train_error: 1.27 | train_acc: 0.878 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.20% | train_error: 1.26 | train_acc: 0.878 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.30% | train_error: 1.26 | train_acc: 0.878 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.40% | train_error: 1.26 | train_acc: 0.879 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.50% | train_error: 1.25 | train_acc: 0.879 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.60% | train_error: 1.25 | train_acc: 0.880 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.70% | train_error: 1.24 | train_acc: 0.880 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.80% | train_error: 1.24 | train_acc: 0.880 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.90% | train_error: 1.24 | train_acc: 0.880 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.00% | train_error: 1.23 | train_acc: 0.881 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.10% | train_error: 1.22 | train_acc: 0.882 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.20% | train_error: 1.22 | train_acc: 0.882 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.30% | train_error: 1.21 | train_acc: 0.883 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.40% | train_error: 1.21 | train_acc: 0.883 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.50% | train_error: 1.21 | train_acc: 0.883 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.60% | train_error: 1.21 | train_acc: 0.883 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.70% | train_error: 1.21 | train_acc: 0.884 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.80% | train_error: 1.20 | train_acc: 0.884 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.90% | train_error: 1.20 | train_acc: 0.884 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.00% | train_error: 1.20 | train_acc: 0.884 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.10% | train_error: 1.20 | train_acc: 0.884 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.20% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.30% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.40% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.50% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.60% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.70% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.80% | train_error: 1.18 | train_acc: 0.886 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.90% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.00% | train_error: 1.19 | train_acc: 0.885 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.10% | train_error: 1.18 | train_acc: 0.886 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.20% | train_error: 1.18 | train_acc: 0.886 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.30% | train_error: 1.17 | train_acc: 0.887 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.40% | train_error: 1.17 | train_acc: 0.887 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.50% | train_error: 1.17 | train_acc: 0.887 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.60% | train_error: 1.16 | train_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.70% | train_error: 1.16 | train_acc: 0.888 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.80% | train_error: 1.15 | train_acc: 0.889 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.90% | train_error: 1.15 | train_acc: 0.889 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.00% | train_error: 1.14 | train_acc: 0.890 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.10% | train_error: 1.14 | train_acc: 0.890 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.20% | train_error: 1.14 | train_acc: 0.890 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.30% | train_error: 1.14 | train_acc: 0.890 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.40% | train_error: 1.13 | train_acc: 0.891 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.50% | train_error: 1.13 | train_acc: 0.891 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.60% | train_error: 1.13 | train_acc: 0.891 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.70% | train_error: 1.12 | train_acc: 0.891 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.80% | train_error: 1.13 | train_acc: 0.891 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.90% | train_error: 1.12 | train_acc: 0.892 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.00% | train_error: 1.11 | train_acc: 0.893 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.10% | train_error: 1.11 | train_acc: 0.893 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.20% | train_error: 1.09 | train_acc: 0.894 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.30% | train_error: 1.09 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.40% | train_error: 1.09 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.50% | train_error: 1.09 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.60% | train_error: 1.09 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.70% | train_error: 1.09 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.80% | train_error: 1.08 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.90% | train_error: 1.08 | train_acc: 0.895 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.00% | train_error: 1.08 | train_acc: 0.896 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.10% | train_error: 1.08 | train_acc: 0.896 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.20% | train_error: 1.08 | train_acc: 0.896 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.30% | train_error: 1.07 | train_acc: 0.896 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.40% | train_error: 1.07 | train_acc: 0.897 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.50% | train_error: 1.06 | train_acc: 0.898 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.60% | train_error: 1.05 | train_acc: 0.898 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.70% | train_error: 1.05 | train_acc: 0.899 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.80% | train_error: 1.04 | train_acc: 0.899 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.90% | train_error: 1.04 | train_acc: 0.899 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.00% | train_error: 1.04 | train_acc: 0.899 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.10% | train_error: 1.04 | train_acc: 0.900 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.20% | train_error: 1.03 | train_acc: 0.900 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.30% | train_error: 1.03 | train_acc: 0.901 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.40% | train_error: 1.03 | train_acc: 0.901 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.50% | train_error: 1.02 | train_acc: 0.901 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.60% | train_error: 1.02 | train_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.70% | train_error: 1.01 | train_acc: 0.902 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.80% | train_error: 1.01 | train_acc: 0.903 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.90% | train_error: 1.00 | train_acc: 0.903 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.00% | train_error: 1.00 | train_acc: 0.903 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.10% | train_error: 1.00 | train_acc: 0.903 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.20% | train_error: 1.00 | train_acc: 0.903 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.30% | train_error: 0.999 | train_acc: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.40% | train_error: 0.998 | train_acc: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.50% | train_error: 0.998 | train_acc: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.60% | train_error: 0.994 | train_acc: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.70% | train_error: 0.994 | train_acc: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.80% | train_error: 0.991 | train_acc: 0.904 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.90% | train_error: 0.987 | train_acc: 0.905 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.00% | train_error: 0.986 | train_acc: 0.905 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.10% | train_error: 0.978 | train_acc: 0.906 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.20% | train_error: 0.976 | train_acc: 0.906 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.30% | train_error: 0.976 | train_acc: 0.906 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.40% | train_error: 0.973 | train_acc: 0.906 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.50% | train_error: 0.972 | train_acc: 0.906 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.60% | train_error: 0.968 | train_acc: 0.907 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.70% | train_error: 0.968 | train_acc: 0.907 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.80% | train_error: 0.965 | train_acc: 0.907 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.90% | train_error: 0.966 | train_acc: 0.907 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.00% | train_error: 0.962 | train_acc: 0.907 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.10% | train_error: 0.954 | train_acc: 0.908 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.20% | train_error: 0.951 | train_acc: 0.908 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.30% | train_error: 0.948 | train_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.40% | train_error: 0.946 | train_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.50% | train_error: 0.943 | train_acc: 0.909 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.60% | train_error: 0.935 | train_acc: 0.910 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.70% | train_error: 0.926 | train_acc: 0.911 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.80% | train_error: 0.926 | train_acc: 0.911 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.90% | train_error: 0.926 | train_acc: 0.911 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.00% | train_error: 0.914 | train_acc: 0.912 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.10% | train_error: 0.914 | train_acc: 0.912 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.20% | train_error: 0.906 | train_acc: 0.913 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.30% | train_error: 0.903 | train_acc: 0.913 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.40% | train_error: 0.900 | train_acc: 0.913 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.50% | train_error: 0.895 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.60% | train_error: 0.894 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.70% | train_error: 0.891 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.80% | train_error: 0.893 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.90% | train_error: 0.893 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.00% | train_error: 0.890 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.10% | train_error: 0.889 | train_acc: 0.914 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.20% | train_error: 0.883 | train_acc: 0.915 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.30% | train_error: 0.880 | train_acc: 0.915 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.40% | train_error: 0.878 | train_acc: 0.915 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.50% | train_error: 0.876 | train_acc: 0.915 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.60% | train_error: 0.875 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.70% | train_error: 0.874 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.80% | train_error: 0.875 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.90% | train_error: 0.873 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.00% | train_error: 0.871 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.10% | train_error: 0.868 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.20% | train_error: 0.866 | train_acc: 0.916 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.30% | train_error: 0.864 | train_acc: 0.917 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.40% | train_error: 0.855 | train_acc: 0.918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.50% | train_error: 0.851 | train_acc: 0.918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.60% | train_error: 0.850 | train_acc: 0.918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.70% | train_error: 0.850 | train_acc: 0.918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.80% | train_error: 0.848 | train_acc: 0.918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.90% | train_error: 0.844 | train_acc: 0.919 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.00% | train_error: 0.845 | train_acc: 0.918 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.10% | train_error: 0.842 | train_acc: 0.919 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.20% | train_error: 0.841 | train_acc: 0.919 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.30% | train_error: 0.841 | train_acc: 0.919 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.40% | train_error: 0.836 | train_acc: 0.919 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.50% | train_error: 0.821 | train_acc: 0.921 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.60% | train_error: 0.808 | train_acc: 0.922 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.70% | train_error: 0.805 | train_acc: 0.922 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.80% | train_error: 0.800 | train_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.90% | train_error: 0.799 | train_acc: 0.923 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.00% | train_error: 0.792 | train_acc: 0.924 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.10% | train_error: 0.791 | train_acc: 0.924 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.20% | train_error: 0.782 | train_acc: 0.925 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.30% | train_error: 0.774 | train_acc: 0.925 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.40% | train_error: 0.766 | train_acc: 0.926 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.50% | train_error: 0.763 | train_acc: 0.926 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.60% | train_error: 0.757 | train_acc: 0.927 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.70% | train_error: 0.753 | train_acc: 0.927 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.80% | train_error: 0.754 | train_acc: 0.927 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.90% | train_error: 0.747 | train_acc: 0.928 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.00% | train_error: 0.740 | train_acc: 0.929 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.10% | train_error: 0.746 | train_acc: 0.928 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.20% | train_error: 0.737 | train_acc: 0.929 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.30% | train_error: 0.744 | train_acc: 0.928 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.40% | train_error: 0.736 | train_acc: 0.929 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.50% | train_error: 0.745 | train_acc: 0.928 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.60% | train_error: 0.737 | train_acc: 0.929 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.70% | train_error: 0.736 | train_acc: 0.929 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.80% | train_error: 0.724 | train_acc: 0.930 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.90% | train_error: 0.722 | train_acc: 0.930 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.00% | train_error: 0.718 | train_acc: 0.931 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.10% | train_error: 0.718 | train_acc: 0.931 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.20% | train_error: 0.717 | train_acc: 0.931 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.30% | train_error: 0.712 | train_acc: 0.931 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.40% | train_error: 0.713 | train_acc: 0.931 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.50% | train_error: 0.710 | train_acc: 0.931 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.60% | train_error: 0.708 | train_acc: 0.932 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.70% | train_error: 0.705 | train_acc: 0.932 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.80% | train_error: 0.702 | train_acc: 0.932 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.90% | train_error: 0.701 | train_acc: 0.932 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.00% | train_error: 0.695 | train_acc: 0.933 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.10% | train_error: 0.694 | train_acc: 0.933 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.20% | train_error: 0.691 | train_acc: 0.933 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.30% | train_error: 0.687 | train_acc: 0.934 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.40% | train_error: 0.683 | train_acc: 0.934 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.50% | train_error: 0.682 | train_acc: 0.934 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.60% | train_error: 0.677 | train_acc: 0.935 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.70% | train_error: 0.672 | train_acc: 0.935 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.80% | train_error: 0.669 | train_acc: 0.935 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.90% | train_error: 0.669 | train_acc: 0.935 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.00% | train_error: 0.668 | train_acc: 0.936 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.10% | train_error: 0.665 | train_acc: 0.936 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.20% | train_error: 0.658 | train_acc: 0.936 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.30% | train_error: 0.655 | train_acc: 0.937 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.40% | train_error: 0.654 | train_acc: 0.937 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.50% | train_error: 0.657 | train_acc: 0.937 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.60% | train_error: 0.652 | train_acc: 0.937 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.70% | train_error: 0.647 | train_acc: 0.938 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.80% | train_error: 0.645 | train_acc: 0.938 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.90% | train_error: 0.645 | train_acc: 0.938 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.00% | train_error: 0.630 | train_acc: 0.939 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.10% | train_error: 0.637 | train_acc: 0.939 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.20% | train_error: 0.624 | train_acc: 0.940 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.30% | train_error: 0.630 | train_acc: 0.939 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.40% | train_error: 0.618 | train_acc: 0.940 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.50% | train_error: 0.623 | train_acc: 0.940 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.60% | train_error: 0.611 | train_acc: 0.941 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.70% | train_error: 0.612 | train_acc: 0.941 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.80% | train_error: 0.614 | train_acc: 0.941 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.90% | train_error: 0.608 | train_acc: 0.941 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.00% | train_error: 0.605 | train_acc: 0.942 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.10% | train_error: 0.609 | train_acc: 0.941 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.20% | train_error: 0.604 | train_acc: 0.942 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.30% | train_error: 0.601 | train_acc: 0.942 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.40% | train_error: 0.600 | train_acc: 0.942 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.50% | train_error: 0.601 | train_acc: 0.942 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.60% | train_error: 0.599 | train_acc: 0.942 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.70% | train_error: 0.593 | train_acc: 0.943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.80% | train_error: 0.592 | train_acc: 0.943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.90% | train_error: 0.593 | train_acc: 0.943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.00% | train_error: 0.589 | train_acc: 0.943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.10% | train_error: 0.590 | train_acc: 0.943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.20% | train_error: 0.585 | train_acc: 0.944 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.30% | train_error: 0.590 | train_acc: 0.943 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.40% | train_error: 0.580 | train_acc: 0.944 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.50% | train_error: 0.579 | train_acc: 0.944 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.60% | train_error: 0.574 | train_acc: 0.945 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.70% | train_error: 0.578 | train_acc: 0.944 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.80% | train_error: 0.560 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.90% | train_error: 0.566 | train_acc: 0.945 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.00% | train_error: 0.564 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.10% | train_error: 0.563 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.20% | train_error: 0.559 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.30% | train_error: 0.560 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.40% | train_error: 0.549 | train_acc: 0.947 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.50% | train_error: 0.563 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.60% | train_error: 0.542 | train_acc: 0.948 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.70% | train_error: 0.558 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.80% | train_error: 0.542 | train_acc: 0.948 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.90% | train_error: 0.556 | train_acc: 0.946 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.00% | train_error: 0.536 | train_acc: 0.948 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.10% | train_error: 0.549 | train_acc: 0.947 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.20% | train_error: 0.529 | train_acc: 0.949 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.30% | train_error: 0.536 | train_acc: 0.948 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.40% | train_error: 0.534 | train_acc: 0.949 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.50% | train_error: 0.532 | train_acc: 0.949 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.60% | train_error: 0.528 | train_acc: 0.949 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.70% | train_error: 0.526 | train_acc: 0.949 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.80% | train_error: 0.515 | train_acc: 0.950 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.90% | train_error: 0.517 | train_acc: 0.950 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.00% | train_error: 0.510 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.10% | train_error: 0.511 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.20% | train_error: 0.510 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.30% | train_error: 0.504 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.40% | train_error: 0.510 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.50% | train_error: 0.507 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.60% | train_error: 0.509 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.70% | train_error: 0.499 | train_acc: 0.952 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.80% | train_error: 0.506 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.90% | train_error: 0.499 | train_acc: 0.952 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.00% | train_error: 0.504 | train_acc: 0.951 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.10% | train_error: 0.503 | train_acc: 0.952 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.20% | train_error: 0.495 | train_acc: 0.952 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.30% | train_error: 0.495 | train_acc: 0.952 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.40% | train_error: 0.488 | train_acc: 0.953 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.50% | train_error: 0.482 | train_acc: 0.953 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.60% | train_error: 0.473 | train_acc: 0.954 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.70% | train_error: 0.476 | train_acc: 0.954 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.80% | train_error: 0.477 | train_acc: 0.954 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.90% | train_error: 0.468 | train_acc: 0.955 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.00% | train_error: 0.472 | train_acc: 0.954 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.10% | train_error: 0.466 | train_acc: 0.955 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.20% | train_error: 0.473 | train_acc: 0.954 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.30% | train_error: 0.464 | train_acc: 0.955 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.40% | train_error: 0.467 | train_acc: 0.955 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.50% | train_error: 0.458 | train_acc: 0.956 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.60% | train_error: 0.461 | train_acc: 0.955 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.70% | train_error: 0.449 | train_acc: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.80% | train_error: 0.464 | train_acc: 0.955 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.90% | train_error: 0.446 | train_acc: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.00% | train_error: 0.456 | train_acc: 0.956 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.10% | train_error: 0.449 | train_acc: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.20% | train_error: 0.454 | train_acc: 0.956 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.30% | train_error: 0.446 | train_acc: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.40% | train_error: 0.443 | train_acc: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.50% | train_error: 0.443 | train_acc: 0.957 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.60% | train_error: 0.429 | train_acc: 0.959 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.70% | train_error: 0.434 | train_acc: 0.958 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.80% | train_error: 0.427 | train_acc: 0.959 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.90% | train_error: 0.422 | train_acc: 0.959 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.00% | train_error: 0.419 | train_acc: 0.960 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.10% | train_error: 0.424 | train_acc: 0.959 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.20% | train_error: 0.424 | train_acc: 0.959 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.30% | train_error: 0.422 | train_acc: 0.959 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.40% | train_error: 0.417 | train_acc: 0.960 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.50% | train_error: 0.413 | train_acc: 0.960 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.60% | train_error: 0.408 | train_acc: 0.961 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.70% | train_error: 0.401 | train_acc: 0.961 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.80% | train_error: 0.402 | train_acc: 0.961 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.90% | train_error: 0.396 | train_acc: 0.962 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.00% | train_error: 0.402 | train_acc: 0.961 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.10% | train_error: 0.399 | train_acc: 0.962 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.20% | train_error: 0.401 | train_acc: 0.961 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.30% | train_error: 0.389 | train_acc: 0.962 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.40% | train_error: 0.397 | train_acc: 0.962 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.50% | train_error: 0.386 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.60% | train_error: 0.389 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.70% | train_error: 0.386 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.80% | train_error: 0.385 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.90% | train_error: 0.385 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.00% | train_error: 0.382 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.10% | train_error: 0.378 | train_acc: 0.963 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.20% | train_error: 0.374 | train_acc: 0.964 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.30% | train_error: 0.372 | train_acc: 0.964 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.40% | train_error: 0.371 | train_acc: 0.964 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.50% | train_error: 0.369 | train_acc: 0.964 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.60% | train_error: 0.364 | train_acc: 0.965 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.70% | train_error: 0.372 | train_acc: 0.964 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.80% | train_error: 0.368 | train_acc: 0.964 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.90% | train_error: 0.362 | train_acc: 0.965 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.00% | train_error: 0.364 | train_acc: 0.965 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.10% | train_error: 0.355 | train_acc: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.20% | train_error: 0.356 | train_acc: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.30% | train_error: 0.349 | train_acc: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.40% | train_error: 0.341 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.50% | train_error: 0.347 | train_acc: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.60% | train_error: 0.349 | train_acc: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.70% | train_error: 0.345 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.80% | train_error: 0.346 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.90% | train_error: 0.338 | train_acc: 0.968 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.00% | train_error: 0.348 | train_acc: 0.966 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.10% | train_error: 0.344 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.20% | train_error: 0.346 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.30% | train_error: 0.340 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.40% | train_error: 0.339 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.50% | train_error: 0.336 | train_acc: 0.968 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.60% | train_error: 0.343 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.70% | train_error: 0.336 | train_acc: 0.968 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.80% | train_error: 0.339 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.90% | train_error: 0.330 | train_acc: 0.968 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.00% | train_error: 0.338 | train_acc: 0.967 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.10% | train_error: 0.328 | train_acc: 0.969 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.20% | train_error: 0.326 | train_acc: 0.969 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.30% | train_error: 0.317 | train_acc: 0.969 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.40% | train_error: 0.329 | train_acc: 0.968 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.50% | train_error: 0.317 | train_acc: 0.969 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.60% | train_error: 0.316 | train_acc: 0.970 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.70% | train_error: 0.318 | train_acc: 0.969 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.80% | train_error: 0.315 | train_acc: 0.970 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.90% | train_error: 0.309 | train_acc: 0.970 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.00% | train_error: 0.308 | train_acc: 0.970 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.10% | train_error: 0.296 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.20% | train_error: 0.302 | train_acc: 0.971 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.30% | train_error: 0.298 | train_acc: 0.971 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.40% | train_error: 0.300 | train_acc: 0.971 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.50% | train_error: 0.296 | train_acc: 0.971 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.60% | train_error: 0.291 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.70% | train_error: 0.287 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.80% | train_error: 0.283 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.90% | train_error: 0.280 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.00% | train_error: 0.285 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.10% | train_error: 0.277 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.20% | train_error: 0.292 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.30% | train_error: 0.289 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.40% | train_error: 0.292 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.50% | train_error: 0.287 | train_acc: 0.972 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.60% | train_error: 0.285 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.70% | train_error: 0.280 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.80% | train_error: 0.283 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.90% | train_error: 0.277 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.00% | train_error: 0.285 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.10% | train_error: 0.278 | train_acc: 0.973 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.20% | train_error: 0.266 | train_acc: 0.974 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.30% | train_error: 0.264 | train_acc: 0.975 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.40% | train_error: 0.271 | train_acc: 0.974 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.50% | train_error: 0.265 | train_acc: 0.974 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.60% | train_error: 0.265 | train_acc: 0.974 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.70% | train_error: 0.258 | train_acc: 0.975 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.80% | train_error: 0.263 | train_acc: 0.975 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.90% | train_error: 0.251 | train_acc: 0.976 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.00% | train_error: 0.248 | train_acc: 0.976 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.10% | train_error: 0.248 | train_acc: 0.976 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.20% | train_error: 0.250 | train_acc: 0.976 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.30% | train_error: 0.243 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.40% | train_error: 0.241 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.50% | train_error: 0.239 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.60% | train_error: 0.240 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.70% | train_error: 0.239 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.80% | train_error: 0.239 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.90% | train_error: 0.235 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.00% | train_error: 0.233 | train_acc: 0.978 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.10% | train_error: 0.234 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.20% | train_error: 0.240 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.30% | train_error: 0.238 | train_acc: 0.977 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.40% | train_error: 0.226 | train_acc: 0.978 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.50% | train_error: 0.226 | train_acc: 0.978 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.60% | train_error: 0.229 | train_acc: 0.978 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.70% | train_error: 0.226 | train_acc: 0.978 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.80% | train_error: 0.218 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.90% | train_error: 0.219 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.00% | train_error: 0.220 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.10% | train_error: 0.216 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.20% | train_error: 0.217 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.30% | train_error: 0.216 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.40% | train_error: 0.216 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.50% | train_error: 0.213 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.60% | train_error: 0.213 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.70% | train_error: 0.213 | train_acc: 0.979 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.80% | train_error: 0.205 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.90% | train_error: 0.210 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.00% | train_error: 0.211 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.10% | train_error: 0.209 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.20% | train_error: 0.206 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.30% | train_error: 0.204 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.40% | train_error: 0.210 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.50% | train_error: 0.198 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.60% | train_error: 0.198 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.70% | train_error: 0.202 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.80% | train_error: 0.201 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.90% | train_error: 0.202 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.00% | train_error: 0.195 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.10% | train_error: 0.203 | train_acc: 0.980 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.20% | train_error: 0.197 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.30% | train_error: 0.201 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.40% | train_error: 0.187 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.50% | train_error: 0.197 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.60% | train_error: 0.187 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.70% | train_error: 0.195 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.80% | train_error: 0.195 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.90% | train_error: 0.190 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.00% | train_error: 0.190 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.10% | train_error: 0.194 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.20% | train_error: 0.187 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.30% | train_error: 0.190 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.40% | train_error: 0.190 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.50% | train_error: 0.195 | train_acc: 0.981 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.60% | train_error: 0.188 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.70% | train_error: 0.188 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.80% | train_error: 0.190 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.90% | train_error: 0.191 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.00% | train_error: 0.188 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.10% | train_error: 0.189 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.20% | train_error: 0.175 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.30% | train_error: 0.189 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.40% | train_error: 0.183 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.50% | train_error: 0.189 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.60% | train_error: 0.181 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.70% | train_error: 0.188 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.80% | train_error: 0.179 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.90% | train_error: 0.190 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.00% | train_error: 0.186 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.10% | train_error: 0.185 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.20% | train_error: 0.182 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.30% | train_error: 0.182 | train_acc: 0.982 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.40% | train_error: 0.171 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.50% | train_error: 0.178 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.60% | train_error: 0.170 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.70% | train_error: 0.181 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.80% | train_error: 0.166 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.90% | train_error: 0.175 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.00% | train_error: 0.173 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.10% | train_error: 0.173 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.20% | train_error: 0.171 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.30% | train_error: 0.168 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.40% | train_error: 0.167 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.50% | train_error: 0.174 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.60% | train_error: 0.158 | train_acc: 0.985 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.70% | train_error: 0.173 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.80% | train_error: 0.164 | train_acc: 0.984 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.90% | train_error: 0.175 | train_acc: 0.983 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================================>] 100.0% | train_error: 0.175 | train_acc: 0.983 " ] } ], "source": [ "from sklearn.datasets import load_digits\n", "\n", "def onehot(target: np.ndarray):\n", " onehot = np.zeros((target.size, target.max() + 1))\n", " onehot[np.arange(target.size), target] = 1\n", " return onehot\n", "\n", "digits = load_digits()\n", "\n", "X = digits.data\n", "target = digits.target\n", "target = onehot(target)\n", "\n", "input_nodes = 64\n", "hidden_nodes1 = 100\n", "hidden_nodes2 = 30\n", "output_nodes = 10\n", "\n", "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", "\n", "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n", "\n", "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", "\n", "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", "scores = multiclass.fit(X, target, scheduler, epochs=1000)" ] }, { "cell_type": "markdown", "id": "98281e67", "metadata": { "editable": true }, "source": [ "## Testing the XOR gate and other gates\n", "\n", "Let us now use our code to test the XOR gate." ] }, { "cell_type": "code", "execution_count": 21, "id": "24031566", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Adam: Eta=0.1, Lambda=0\n", "\r", " [----------------------------------------] 0.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.1000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.2000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.3000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.4000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.5000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.6000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.7000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.8000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 0.9000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 1.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [----------------------------------------] 2.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 2.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 3.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [>---------------------------------------] 4.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 5.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 6.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=>--------------------------------------] 7.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 7.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 8.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.000% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.100% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.200% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.300% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.400% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.500% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.600% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.700% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.800% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==>-------------------------------------] 9.900% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 10.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 11.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===>------------------------------------] 12.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 12.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 13.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====>-----------------------------------] 14.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 15.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 16.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====>----------------------------------] 17.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 17.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 18.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======>---------------------------------] 19.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 20.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 21.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======>--------------------------------] 22.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 22.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 23.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========>-------------------------------] 24.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 25.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 26.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========>------------------------------] 27.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 27.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 28.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========>-----------------------------] 29.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 30.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 31.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========>----------------------------] 32.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 32.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 33.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============>---------------------------] 34.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 35.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 36.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============>--------------------------] 37.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 37.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 38.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============>-------------------------] 39.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 40.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 41.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============>------------------------] 42.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 42.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 43.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================>-----------------------] 44.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 45.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 46.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================>----------------------] 47.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 47.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 48.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.10% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.30% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.50% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.70% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================>---------------------] 49.90% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 50.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 51.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================>--------------------] 52.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 52.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 53.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================>-------------------] 54.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 55.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 56.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================>------------------] 57.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 57.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 58.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================>-----------------] 59.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 60.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 61.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================>----------------] 62.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 62.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 63.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [========================>---------------] 64.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 65.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 66.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=========================>--------------] 67.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 67.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 68.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==========================>-------------] 69.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 70.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 71.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===========================>------------] 72.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 72.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 73.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.60% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.80% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [============================>-----------] 74.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.00% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.20% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.40% | train_error: 10.4 | train_acc: 0.500 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 75.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 76.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=============================>----------] 77.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 77.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 78.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==============================>---------] 79.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 80.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 81.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===============================>--------] 82.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 82.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 83.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [================================>-------] 84.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 85.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 86.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=================================>------] 87.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 87.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 88.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [==================================>-----] 89.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 90.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 91.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [===================================>----] 92.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 92.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 93.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [====================================>---] 94.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 95.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 96.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=====================================>--] 97.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 97.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 98.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.00% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.10% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.20% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.30% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.40% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.50% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.60% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.70% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.80% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [======================================>-] 99.90% | train_error: -0.0000000010 | train_acc: 1.00 " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " " ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", " [=======================================>] 100.0% | train_error: -0.0000000010 | train_acc: 1.00 " ] } ], "source": [ "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", "\n", "# The XOR gate\n", "yXOR = np.array( [[ 0], [1] ,[1], [0]])\n", "\n", "input_nodes = X.shape[1]\n", "output_nodes = 1\n", "\n", "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)\n", "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", "scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)\n", "scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)" ] }, { "cell_type": "markdown", "id": "8302830f", "metadata": { "editable": true }, "source": [ "Not bad, but the results depend strongly on the learning reate. Try different learning rates." ] }, { "cell_type": "markdown", "id": "9003b71a", "metadata": { "editable": true }, "source": [ "## Lecture Thursday October 26" ] }, { "cell_type": "markdown", "id": "4446e61e", "metadata": { "editable": true }, "source": [ "## Developing a code for doing neural networks with back propagation\n", "\n", "We repeat some of the elements discussed last week. The first part of\n", "the material for Thursday was contained in the slides for last\n", "week as well. We will repeat some of the topics here before we move into\n", "applications to differential equations and other examples.\n", "\n", "One can identify a set of key steps when using neural networks to solve supervised learning problems: \n", "\n", "1. Collect and pre-process data \n", "\n", "2. Define model and architecture \n", "\n", "3. Choose cost function and optimizer \n", "\n", "4. Train the model \n", "\n", "5. Evaluate model performance on test data \n", "\n", "6. Adjust hyperparameters (if necessary, network architecture)" ] }, { "cell_type": "markdown", "id": "de6e80a8", "metadata": { "editable": true }, "source": [ "## Collect and pre-process data\n", "\n", "Here we will be using the MNIST dataset, which is readily available through the **scikit-learn**\n", "package. You may also find it for example [here](http://yann.lecun.com/exdb/mnist/). \n", "The *MNIST* (Modified National Institute of Standards and Technology) database is a large database\n", "of handwritten digits that is commonly used for training various image processing systems. \n", "The MNIST dataset consists of 70 000 images of size $28\\times 28$ pixels, each labeled from 0 to 9. \n", "The scikit-learn dataset we will use consists of a selection of 1797 images of size $8\\times 8$ collected and processed from this database. \n", "\n", "To feed data into a feed-forward neural network we need to represent\n", "the inputs as a design/feature matrix $X = (n_{inputs}, n_{features})$. Each\n", "row represents an *input*, in this case a handwritten digit, and\n", "each column represents a *feature*, in this case a pixel. The\n", "correct answers, also known as *labels* or *targets* are\n", "represented as a 1D array of integers \n", "$Y = (n_{inputs}) = (5, 3, 1, 8,...)$.\n", "\n", "As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from\n", "measurements of height (in m) \n", "and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example: \n", "\n", "$$ X = \\begin{bmatrix}\n", "1.85 & 81\\\\\n", "1.71 & 65\\\\\n", "1.95 & 103\\\\\n", "1.55 & 42\\\\\n", "1.63 & 56\n", "\\end{bmatrix} ,$$ \n", "\n", "and the targets would be: \n", "\n", "$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$ \n", "\n", "Since each input image is a 2D matrix, we need to flatten the image\n", "(i.e. \"unravel\" the 2D matrix into a 1D array) to turn the data into a\n", "design/feature matrix. This means we lose all spatial information in the\n", "image, such as locality and translational invariance. More complicated\n", "architectures such as Convolutional Neural Networks can take advantage\n", "of such information, and are most commonly applied when analyzing\n", "images." ] }, { "cell_type": "code", "execution_count": 22, "id": "011b021a", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "inputs = (n_inputs, pixel_width, pixel_height) = (1797, 8, 8)\n", "labels = (n_inputs) = (1797,)\n", "X = (n_inputs, n_features) = (1797, 64)\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7YAAADICAYAAADcOn20AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAQvUlEQVR4nO3df4zXdR0H8Cd0pGspV85VFoOEojDgZoOt7YKDcoWmXKy2/iGOajahifyBMZcBA6s5l4dALP3Do2ClpIJtTVfLQ02suev69UfCxmU/5hzOu5ViZ/rtD8dN4kfKveF83z0eG5v35XvP7/vz9fX+fr7Pfb53jGs0Go0AAABApcaP9AIAAABgOBRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1cZssV26dGmWLl067Jx7770306dPz9/+9rdhZy1cuDBr1659w993991354orrkhLS0sWLVqUXbt2pdFoDHs9jF6jYf5//etfZ/r06Sf9s3Xr1mGvidHJ/DPWjYY9kCQvv/xybr/99lx22WWZNWtWrrrqquzdu3fYa2F0M/+jV9NIL4Dh2b17d2688cYsXbo0n/jEJ/Kb3/wmGzduzIsvvpgvf/nLI708OGMuueSS3HXXXcfd3tnZmT/84Q+54oorRmBVcHaYf0i++93vZseOHbn22mszc+bM7Nu3L9dff33Gjx+fK6+8cqSXB2eU+T+eYlu5e+65J5deemm+8Y1vJEk+9rGPpa+vL7t27VJsGdXe/va3p6Wl5ZjbfvGLX2T//v3ZvHlz3v/+94/MwuAsMP+Mdc8//3x27tyZZcuW5eqrr07y6nugP/3pT9m5c+eYfWPP2GD+T2zMfhT59dq9e3eWLFmSlpaWzJo1K4sXL87Pfvaz4+7X09OT9vb2zJw5M1deeeVx9/n3v/+dm2++OfPnz89HPvKRE97nfy1dujQLFy485X0GBwdz3nnnHXPbO97xjvT397++A4RTeLPP/2u9+OKL2bRpU9ra2vLpT3/6dX8fnIz5Z6x7M++Bc845J3fddVeWL19+zO0TJkzI4ODgGzhKODHzXx9XbE9h165d2bRpU772ta/l61//evr7+3PHHXdkzZo1aWlpyUUXXTR03xtvvDHXXHNNZsyYkfvuuy+rV6/O+eefn9bW1jQajaxcuTI9PT259tprM3Xq1Pz85z/P6tWrMzg4mPb29hM+/rp16/7vcC5btiw33HBD9u7dm4ULF6a3tzf33XffSTPh9aph/l+rq6srzzzzTHbs2DHcQwfzz5j3Zt8DTU1N+dCHPpQkaTQaOXz4cO6999489thj2bhxY9HngrHH/NdJsT2Fv/71r/nSl76UlStXDt32vve9L0uWLElPT88xQ71y5cqhjwLMmzcvfX192bp1a1pbW/PYY4/lkUceya233prLL788SfLxj388R44cyS233JLPfOYzaWo6/n/FtGnT/u8aFy1alMcffzzXX3/90G2tra254YYbTvu4Ialj/o8aHBzMD3/4w1x++eWZPHny6R4yDDH/jHU17YGf/vSnWbNmTZJk/vz5Q48Dp8v810mxPYWjv53sn//8Z/r6+tLX15f9+/cnSV566aVj7rto0aJjvv7kJz+ZLVu25Pnnn8/+/fszbty4zJ8/P//5z3+G7rNw4cLcf//9OXDgQD784Q+f1hqvueaa9PT0ZM2aNZk1a1b+/Oc/Z+vWrVm1alW2bduWcePGnVYu1DD/Rz3wwAM5fPhwvvKVrwwrB44y/4x1Ne2B2bNnZ+fOnTl06FBuu+22fOELX8hPfvKTnHPOOcPKZewy/3VSbE/hqaeeyje/+c08/vjjaWpqysUXX5zp06cnyXH/nM6FF154zNcXXHBBGo1G/vWvf6W/vz+NRiOXXnrpCR/nmWeeOa2h7unpyaOPPppNmzbl85//fJJk7ty5mTRpUr761a+mu7s7CxYseMO5kLz55/+1HnzwwXzgAx8Y+lgODJf5Z6yraQ9Mnjw5kydPzpw5czJp0qR0dHTkwQcfzFVXXTWsXMYu818nxfYkXnnllVx99dWZMGFC7r777syYMSNNTU05ePBg7r///uPuPzAwkHPPPXfo68OHD+ctb3lLJk6cmPPOOy9ve9vb8oMf/OCEj3W6Hx37xz/+kSTHbZY5c+YkSQ4cOKDYclpqmP+jXnrppfzqV79ytYpizD9jXQ174Nlnn83DDz+cefPm5YILLhi6febMmUmSp59++rRywfzXy29FPonnnnsuhw4dyuc+97nMmjVr6PPvDz/8cJJXh/61HnnkkaH/fuWVV/LAAw9k9uzZOffcczN37ty88MILaTQamTlz5tCfAwcOZNu2bcd8NOGNuPjii5MkTzzxxDG39/T0JHn1ZwHgdNQw/0c9+eSTOXLkSD760Y8OKweOMv+MdTXsgRdeeCFr167N7t27T7iWo1fX4I0y//Ua01dsn3766XR1dR13+7Rp09La2pr3vve92bVrV9797nfn/PPPz6OPPjr0GyePHDlyzPd0dnbm5Zdfznve85786Ec/yqFDh3LnnXcmefUHuefMmZMVK1ZkxYoVmTp1an7/+99ny5YtaW1tzTvf+c4Tru/gwYMZHBzMjBkzTvj3M2bMyKc+9al85zvfycDAQGbPnp2DBw9my5YtueSSS3LZZZcN49lhtKt9/o968sknkyRTp059o08BY5j5Z6yrfQ9MmjQp7e3t2bZtW8aPH5+ZM2fmj3/8Y7Zv357W1tbMmzdvGM8Oo535H53GdLF96qmn8u1vf/u42z/72c+mtbU13/ve93LTTTdl7dq1eetb35pp06Zl+/bt+da3vpUnnngiS5cuHfqem266KTfffHP+8pe/5IMf/GDuuOOOzJ07N0kyfvz43H777dm8eXO+//3v59lnn8273vWudHR0HPPb1v7Xhg0b8ve//z2//OUvT3qfW265Jdu3b8+Pf/zj3HbbbbnooouyZMmSrFy5MhMmTBjGs8NoNxrmP3n1Iz9JMnHixNN5GhijzD9j3WjYAxs3bsyUKVNyzz33ZMuWLbnwwgvzxS9+MStWrPDLMzkl8z86jWv8709AAwAAQEX8jC0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFStaaQXUEpXV1fRvPXr1xfNa25uLpqXJJ2dnUXz2traiuZx9nR3dxfNK72f9uzZUzQvSQYGBormPfTQQ0Xz7Keza+/evUXzVq1aVTTvTCi976dMmVI0j5Pr6+srmlf6/UDpc0Dp1+skmThxYtG83t7eonn206n19/cXzXuz74HSx5uY2RNxxRYAAICqKbYAAABUTbEFAACgaootAAAAVVNsAQAAqJpiCwAAQNUUWwAAAKqm2AIAAFA1xRYAAICqKbYAAABUTbEFAACgaootAAAAVVNsAQAAqJpiCwAAQNUUWwAAAKqm2AIAAFA1xRYAAICqKbYAAABUrWmkHri7u7to3vLly4vmLV68uGhec3Nz0bwkaW9vL5rX399fNI+z57rrriuaV3oWOjo6iuYlyebNm4vmnYk9ysn19fUVzSv9eliDPXv2FM0r/TrCyb3Zn+sdO3YUzXvooYeK5iXlzwHeA51dpZ/v0q+Hpc8ppdeXJF1dXUXz1q9fXzRvJLhiCwAAQNUUWwAAAKqm2AIAAFA1xRYAAICqKbYAAABUTbEFAACgaootAAAAVVNsAQAAqJpiCwAAQNUUWwAAAKqm2AIAAFA1xRYAAICqKbYAAABUTbEFAACgaootAAAAVVNsAQAAqJpiCwAAQNUUWwAAAKqm2AIAAFC1cY1GozESD3zdddcVzevr6yuat2fPnqJ5bW1tRfOSpLm5uWhe6WPm7Ck9/6Vna9++fUXzkmTZsmVF8/r7+4vmcXZ1dnYWzWtpaSmat2DBgqJ5STJ//vyied3d3UXz4KjS7/mSpLe3t2ie+edMOhM9oPR5qvR5dCS4YgsAAEDVFFsAAACqptgCAABQNcUWAACAqim2AAAAVE2xBQAAoGqKLQAAAFVTbAEAAKiaYgsAAEDVFFsAAACqptgCAABQNcUWAACAqim2AAAAVE2xBQAAoGqKLQAAAFVTbAEAAKiaYgsAAEDVFFsAAACqptgCAABQtaaReuApU6YUzevr6yuat379+qJ5+/btK5qXJL/97W+LZ1Kn/v7+onml99O6deuK5iVJc3Nz0bzSx1z6NY5T6+joKJpX+hxwJpQ+r5Q+5hqeQ86OlpaW4pldXV1F80qfR0ufozi10ufw9vb2onlnQmdn50gv4U3HFVsAAACqptgCAABQNcUWAACAqim2AAAAVE2xBQAAoGqKLQAAAFVTbAEAAKiaYgsAAEDVFFsAAACqptgCAABQNcUWAACAqim2AAAAVE2xBQAAoGqKLQAAAFVTbAEAAKiaYgsAAEDVFFsAAACqptgCAABQNcUWAACAqo1rNBqNkV5ECS0tLUXzfve73xXNW7ZsWdG8JOnq6iqeydmxd+/eonnt7e1F88aidevWFc1bv3590bzRpre3t2heW1tb0byBgYGieWdC6fNK6ZmdMmVK0Tx4rdLzVfo82tnZWTSPUxuL55Q777yzaF5HR0fRvJHgii0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqjau0Wg0RnoRJbS0tIz0Ek6pubm5eGbpY+7s7Cyax8l1d3cXzduzZ0/RvN7e3qJ5fX19RfOS8ms8E3uUkyu9BxYsWFA0r7TFixcXzyy976EmbW1tI72EUyr9Gjfa9Pf3j/QSTqn0e4IzMa+l31udifdqZ5srtgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVaxrpBZTS3NxcNK+tra1o3vr164vmJeWPufQaS69vNCk9XwMDA0Xzurq6iua1t7cXzUvMV+1K74FVq1YVzdu8eXPRvOXLlxfNg9fau3dv0bzJkycXzevt7S2adyYyz8T7NE5u3759RfPWrVtXNG/Dhg1F8zo6OormJeXPK/39/UXzRuJ9miu2AAAAVE2xBQAAoGqKLQAAAFVTbAEAAKiaYgsAAEDVFFsAAACqptgCAABQNcUWAACAqim2AAAAVE2xBQAAoGqKLQAAAFVTbAEAAKiaYgsAAEDVFFsAAACqptgCAABQNcUWAACAqim2AAAAVE2xBQAAoGqKLQAAAFVrGukFlLJ69eqiee3t7UXzNmzYUDQvSRYvXlw0r7m5uWgeZ89zzz1XNG9gYKBoXkdHR9E8ONNmz55dNK/06zW81q233lo0b9++fUXzJk6cWDQvKX9ecZ46u+bPn180r62trWhe6T3V399fNC9JVq1aVTRvNPQAV2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqjau0Wg0RnoRAAAAcLpcsQUAAKBqii0AAABVU2wBAACommILAABA1RRbAAAAqqbYAgAAUDXFFgAAgKoptgAAAFRNsQUAAKBq/wUlNg0p+guplwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "filenames": { "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/week43_72_1.png" } }, "output_type": "display_data" } ], "source": [ "# import necessary packages\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn import datasets\n", "\n", "\n", "# ensure the same random numbers appear every time\n", "np.random.seed(0)\n", "\n", "# display images in notebook\n", "%matplotlib inline\n", "plt.rcParams['figure.figsize'] = (12,12)\n", "\n", "\n", "# download MNIST dataset\n", "digits = datasets.load_digits()\n", "\n", "# define inputs and labels\n", "inputs = digits.images\n", "labels = digits.target\n", "\n", "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", "\n", "\n", "# flatten the image\n", "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", "n_inputs = len(inputs)\n", "inputs = inputs.reshape(n_inputs, -1)\n", "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", "\n", "\n", "# choose some random images to display\n", "indices = np.arange(n_inputs)\n", "random_indices = np.random.choice(indices, size=5)\n", "\n", "for i, image in enumerate(digits.images[random_indices]):\n", " plt.subplot(1, 5, i+1)\n", " plt.axis('off')\n", " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "4de43fe8", "metadata": { "editable": true }, "source": [ "## Train and test datasets\n", "\n", "Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions. \n", "\n", "We will reserve $80 \\%$ of our dataset for training and $20 \\%$ for testing. \n", "\n", "It is important that the train and test datasets are drawn randomly from our dataset, to ensure\n", "no bias in the sampling. \n", "Say you are taking measurements of weather data to predict the weather in the coming 5 days.\n", "You don't want to train your model on measurements taken from the hours 00.00 to 12.00, and then test it on data\n", "collected from 12.00 to 24.00." ] }, { "cell_type": "code", "execution_count": 23, "id": "d9281e06", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of training images: 1437\n", "Number of test images: 360\n" ] } ], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "# one-liner from scikit-learn library\n", "train_size = 0.8\n", "test_size = 1 - train_size\n", "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", " test_size=test_size)\n", "\n", "# equivalently in numpy\n", "def train_test_split_numpy(inputs, labels, train_size, test_size):\n", " n_inputs = len(inputs)\n", " inputs_shuffled = inputs.copy()\n", " labels_shuffled = labels.copy()\n", " \n", " np.random.shuffle(inputs_shuffled)\n", " np.random.shuffle(labels_shuffled)\n", " \n", " train_end = int(n_inputs*train_size)\n", " X_train, X_test = inputs_shuffled[:train_end], inputs_shuffled[train_end:]\n", " Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]\n", " \n", " return X_train, X_test, Y_train, Y_test\n", "\n", "#X_train, X_test, Y_train, Y_test = train_test_split_numpy(inputs, labels, train_size, test_size)\n", "\n", "print(\"Number of training images: \" + str(len(X_train)))\n", "print(\"Number of test images: \" + str(len(X_test)))" ] }, { "cell_type": "markdown", "id": "5d86b543", "metadata": { "editable": true }, "source": [ "## Define model and architecture\n", "\n", "Our simple feed-forward neural network will consist of an *input* layer, a single *hidden* layer and an *output* layer. The activation $y$ of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have \n", "\n", "$$ z = \\sum_{i=1}^n w_i a_i ,$$\n", "\n", "$$ y = f(z) ,$$\n", "\n", "where $f$ is the activation function, $a_i$ represents input from neuron $i$ in the preceding layer\n", "and $w_i$ is the weight to input $i$. \n", "The activation of the neurons in the input layer is just the features (e.g. a pixel value). \n", "\n", "The simplest activation function for a neuron is the *Heaviside* function:\n", "\n", "$$ f(z) = \n", "\\begin{cases}\n", "1, & z > 0\\\\\n", "0, & \\text{otherwise}\n", "\\end{cases}\n", "$$\n", "\n", "A feed-forward neural network with this activation is known as a *perceptron*. \n", "For a binary classifier (i.e. two classes, 0 or 1, dog or not-dog) we can also use this in our output layer. \n", "This activation can be generalized to $k$ classes (using e.g. the *one-against-all* strategy), \n", "and we call these architectures *multiclass perceptrons*. \n", "\n", "However, it is now common to use the terms Single Layer Perceptron (SLP) (1 hidden layer) and \n", "Multilayer Perceptron (MLP) (2 or more hidden layers) to refer to feed-forward neural networks with any activation function. \n", "\n", "Typical choices for activation functions include the sigmoid function, hyperbolic tangent, and Rectified Linear Unit (ReLU). \n", "We will be using the sigmoid function $\\sigma(x)$: \n", "\n", "$$ f(x) = \\sigma(x) = \\frac{1}{1 + e^{-x}} ,$$\n", "\n", "which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions." ] }, { "cell_type": "markdown", "id": "02df2616", "metadata": { "editable": true }, "source": [ "## Layers\n", "\n", "* Input \n", "\n", "Since each input image has 8x8 = 64 pixels or features, we have an input layer of 64 neurons. \n", "\n", "* Hidden layer\n", "\n", "We will use 50 neurons in the hidden layer receiving input from the neurons in the input layer. \n", "Since each neuron in the hidden layer is connected to the 64 inputs we have 64x50 = 3200 weights to the hidden layer. \n", "\n", "* Output\n", "\n", "If we were building a binary classifier, it would be sufficient with a single neuron in the output layer,\n", "which could output 0 or 1 according to the Heaviside function. This would be an example of a *hard* classifier, meaning it outputs the class of the input directly. However, if we are dealing with noisy data it is often beneficial to use a *soft* classifier, which outputs the probability of being in class 0 or 1. \n", "\n", "For a soft binary classifier, we could use a single neuron and interpret the output as either being the probability of being in class 0 or the probability of being in class 1. Alternatively we could use 2 neurons, and interpret each neuron as the probability of being in each class. \n", "\n", "Since we are doing multiclass classification, with 10 categories, it is natural to use 10 neurons in the output layer. We number the neurons $j = 0,1,...,9$. The activation of each output neuron $j$ will be according to the *softmax* function: \n", "\n", "$$ P(\\text{class $j$} \\mid \\text{input $\\boldsymbol{a}$}) = \\frac{\\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_j)}}\n", "{\\sum_{c=0}^{9} \\exp{(\\boldsymbol{a}^T \\boldsymbol{w}_c)}} ,$$ \n", "\n", "i.e. each neuron $j$ outputs the probability of being in class $j$ given an input from the hidden layer $\\boldsymbol{a}$, with $\\boldsymbol{w}_j$ the weights of neuron $j$ to the inputs. \n", "The denominator is a normalization factor to ensure the outputs (probabilities) sum up to 1. \n", "The exponent is just the weighted sum of inputs as before: \n", "\n", "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i+b_j.$$ \n", "\n", "Since each neuron in the output layer is connected to the 50 inputs from the hidden layer we have 50x10 = 500\n", "weights to the output layer." ] }, { "cell_type": "markdown", "id": "0a17cfeb", "metadata": { "editable": true }, "source": [ "## Weights and biases\n", "\n", "Typically weights are initialized with small values distributed around zero, drawn from a uniform\n", "or normal distribution. Setting all weights to zero means all neurons give the same output, making the network useless. \n", "\n", "Adding a bias value to the weighted sum of inputs allows the neural network to represent a greater range\n", "of values. Without it, any input with the value 0 will be mapped to zero (before being passed through the activation). The bias unit has an output of 1, and a weight to each neuron $j$, $b_j$: \n", "\n", "$$ z_j = \\sum_{i=1}^n w_ {ij} a_i + b_j.$$ \n", "\n", "The bias weights $\\boldsymbol{b}$ are often initialized to zero, but a small value like $0.01$ ensures all neurons have some output which can be backpropagated in the first training cycle." ] }, { "cell_type": "code", "execution_count": 24, "id": "94dd542a", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# building our neural network\n", "\n", "n_inputs, n_features = X_train.shape\n", "n_hidden_neurons = 50\n", "n_categories = 10\n", "\n", "# we make the weights normally distributed using numpy.random.randn\n", "\n", "# weights and bias in the hidden layer\n", "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n", "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n", "\n", "# weights and bias in the output layer\n", "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n", "output_bias = np.zeros(n_categories) + 0.01" ] }, { "cell_type": "markdown", "id": "7c13f0a8", "metadata": { "editable": true }, "source": [ "## Feed-forward pass\n", "\n", "Denote $F$ the number of features, $H$ the number of hidden neurons and $C$ the number of categories. \n", "For each input image we calculate a weighted sum of input features (pixel values) to each neuron $j$ in the hidden layer $l$: \n", "\n", "$$ z_{j}^{l} = \\sum_{i=1}^{F} w_{ij}^{l} x_i + b_{j}^{l},$$\n", "\n", "this is then passed through our activation function \n", "\n", "$$ a_{j}^{l} = f(z_{j}^{l}) .$$ \n", "\n", "We calculate a weighted sum of inputs (activations in the hidden layer) to each neuron $j$ in the output layer: \n", "\n", "$$ z_{j}^{L} = \\sum_{i=1}^{H} w_{ij}^{L} a_{i}^{l} + b_{j}^{L}.$$ \n", "\n", "Finally we calculate the output of neuron $j$ in the output layer using the softmax function: \n", "\n", "$$ a_{j}^{L} = \\frac{\\exp{(z_j^{L})}}\n", "{\\sum_{c=0}^{C-1} \\exp{(z_c^{L})}} .$$" ] }, { "cell_type": "markdown", "id": "d1bae3d4", "metadata": { "editable": true }, "source": [ "## Matrix multiplications\n", "\n", "Since our data has the dimensions $X = (n_{inputs}, n_{features})$ and our weights to the hidden\n", "layer have the dimensions \n", "$W_{hidden} = (n_{features}, n_{hidden})$,\n", "we can easily feed the network all our training data in one go by taking the matrix product \n", "\n", "$$ X W^{h} = (n_{inputs}, n_{hidden}),$$ \n", "\n", "and obtain a matrix that holds the weighted sum of inputs to the hidden layer\n", "for each input image and each hidden neuron. \n", "We also add the bias to obtain a matrix of weighted sums to the hidden layer $Z^{h}$: \n", "\n", "$$ \\boldsymbol{z}^{l} = \\boldsymbol{X} \\boldsymbol{W}^{l} + \\boldsymbol{b}^{l} ,$$\n", "\n", "meaning the same bias (1D array with size equal number of hidden neurons) is added to each input image. \n", "This is then passed through the activation: \n", "\n", "$$ \\boldsymbol{a}^{l} = f(\\boldsymbol{z}^l) .$$ \n", "\n", "This is fed to the output layer: \n", "\n", "$$ \\boldsymbol{z}^{L} = \\boldsymbol{a}^{L} \\boldsymbol{W}^{L} + \\boldsymbol{b}^{L} .$$\n", "\n", "Finally we receive our output values for each image and each category by passing it through the softmax function: \n", "\n", "$$ output = softmax (\\boldsymbol{z}^{L}) = (n_{inputs}, n_{categories}) .$$" ] }, { "cell_type": "code", "execution_count": 25, "id": "20c03478", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "probabilities = (n_inputs, n_categories) = (1437, 10)" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "probability that image 0 is in category 0,1,2,...,9 = \n", "[5.41511965e-04 2.17174962e-03 8.84355903e-03 1.44970586e-03\n", " 1.10378326e-04 5.08318298e-09 2.03256632e-04 1.92507116e-03\n", " 9.84443254e-01 3.11507992e-04]\n", "probabilities sum up to: 1.0\n", "\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "predictions = (n_inputs) = (1437,)\n", "prediction for image 0: 8\n", "correct label for image 0: 6\n" ] } ], "source": [ "# setup the feed-forward pass, subscript h = hidden layer\n", "\n", "def sigmoid(x):\n", " return 1/(1 + np.exp(-x))\n", "\n", "def feed_forward(X):\n", " # weighted sum of inputs to the hidden layer\n", " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", " # activation in the hidden layer\n", " a_h = sigmoid(z_h)\n", " \n", " # weighted sum of inputs to the output layer\n", " z_o = np.matmul(a_h, output_weights) + output_bias\n", " # softmax output\n", " # axis 0 holds each input and axis 1 the probabilities of each category\n", " exp_term = np.exp(z_o)\n", " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", " \n", " return probabilities\n", "\n", "probabilities = feed_forward(X_train)\n", "print(\"probabilities = (n_inputs, n_categories) = \" + str(probabilities.shape))\n", "print(\"probability that image 0 is in category 0,1,2,...,9 = \\n\" + str(probabilities[0]))\n", "print(\"probabilities sum up to: \" + str(probabilities[0].sum()))\n", "print()\n", "\n", "# we obtain a prediction by taking the class with the highest likelihood\n", "def predict(X):\n", " probabilities = feed_forward(X)\n", " return np.argmax(probabilities, axis=1)\n", "\n", "predictions = predict(X_train)\n", "print(\"predictions = (n_inputs) = \" + str(predictions.shape))\n", "print(\"prediction for image 0: \" + str(predictions[0]))\n", "print(\"correct label for image 0: \" + str(Y_train[0]))" ] }, { "cell_type": "markdown", "id": "42c28741", "metadata": { "editable": true }, "source": [ "## Choose cost function and optimizer\n", "\n", "To measure how well our neural network is doing we need to introduce a cost function. \n", "We will call the function that gives the error of a single sample output the *loss* function, and the function\n", "that gives the total error of our network across all samples the *cost* function.\n", "A typical choice for multiclass classification is the *cross-entropy* loss, also known as the negative log likelihood. \n", "\n", "In *multiclass* classification it is common to treat each integer label as a so called *one-hot* vector: \n", "\n", "$$ y = 5 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 0, 0, 0, 0, 1, 0, 0, 0, 0) ,$$ \n", "\n", "$$ y = 1 \\quad \\rightarrow \\quad \\boldsymbol{y} = (0, 1, 0, 0, 0, 0, 0, 0, 0, 0) ,$$ \n", "\n", "i.e. a binary bit string of length $C$, where $C = 10$ is the number of classes in the MNIST dataset. \n", "\n", "Let $y_{ic}$ denote the $c$-th component of the $i$-th one-hot vector. \n", "We define the cost function $\\mathcal{C}$ as a sum over the cross-entropy loss for each point $\\boldsymbol{x}_i$ in the dataset.\n", "\n", "In the one-hot representation only one of the terms in the loss function is non-zero, namely the\n", "probability of the correct category $c'$ \n", "(i.e. the category $c'$ such that $y_{ic'} = 1$). This means that the cross entropy loss only punishes you for how wrong\n", "you got the correct label. The probability of category $c$ is given by the softmax function. The vector $\\boldsymbol{\\theta}$ represents the parameters of our network, i.e. all the weights and biases." ] }, { "cell_type": "markdown", "id": "62954b00", "metadata": { "editable": true }, "source": [ "## Optimizing the cost function\n", "\n", "The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is *gradient descent* and its generalizations. The idea behind gradient descent\n", "is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a *local* minimum of the cost function. \n", "Each parameter $\\theta$ is iteratively adjusted according to the rule \n", "\n", "$$ \\theta_{i+1} = \\theta_i - \\eta \\nabla \\mathcal{C}(\\theta_i) ,$$\n", "\n", "where $\\eta$ is known as the *learning rate*, which controls how big a step we take towards the minimum. \n", "This update can be repeated for any number of iterations, or until we are satisfied with the result. \n", "\n", "A simple and effective improvement is a variant called *Batch Gradient Descent*. \n", "Instead of calculating the gradient on the whole dataset, we calculate an approximation of the gradient\n", "on a subset of the data called a *minibatch*. \n", "If there are $N$ data points and we have a minibatch size of $M$, the total number of batches\n", "is $N/M$. \n", "We denote each minibatch $B_k$, with $k = 1, 2,...,N/M$. The gradient then becomes: \n", "\n", "$$ \\nabla \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\nabla \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", "\\frac{1}{M} \\sum_{i \\in B_k} \\nabla \\mathcal{L}_i(\\theta) ,$$\n", "\n", "i.e. instead of averaging the loss over the entire dataset, we average over a minibatch. \n", "\n", "This has two important benefits: \n", "1. Introducing stochasticity decreases the chance that the algorithm becomes stuck in a local minima. \n", "\n", "2. It significantly speeds up the calculation, since we do not have to use the entire dataset to calculate the gradient. \n", "\n", "The various optmization methods, with codes and algorithms, are discussed in our lectures on [Gradient descent approaches](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html)." ] }, { "cell_type": "markdown", "id": "9f94aff8", "metadata": { "editable": true }, "source": [ "## Regularization\n", "\n", "It is common to add an extra term to the cost function, proportional\n", "to the size of the weights. This is equivalent to constraining the\n", "size of the weights, so that they do not grow out of control.\n", "Constraining the size of the weights means that the weights cannot\n", "grow arbitrarily large to fit the training data, and in this way\n", "reduces *overfitting*.\n", "\n", "We will measure the size of the weights using the so called *L2-norm*, meaning our cost function becomes: \n", "\n", "$$ \\mathcal{C}(\\theta) = \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) \\quad \\rightarrow \\quad\n", "\\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}_i(\\theta) + \\lambda \\lvert \\lvert \\boldsymbol{w} \\rvert \\rvert_2^2 \n", "= \\frac{1}{N} \\sum_{i=1}^N \\mathcal{L}(\\theta) + \\lambda \\sum_{ij} w_{ij}^2,$$ \n", "\n", "i.e. we sum up all the weights squared. The factor $\\lambda$ is known as a regularization parameter.\n", "\n", "In order to train the model, we need to calculate the derivative of\n", "the cost function with respect to every bias and weight in the\n", "network. In total our network has $(64 + 1)\\times 50=3250$ weights in\n", "the hidden layer and $(50 + 1)\\times 10=510$ weights to the output\n", "layer ($+1$ for the bias), and the gradient must be calculated for\n", "every parameter. We use the *backpropagation* algorithm discussed\n", "above. This is a clever use of the chain rule that allows us to\n", "calculate the gradient efficently." ] }, { "cell_type": "markdown", "id": "bf73f8e5", "metadata": { "editable": true }, "source": [ "## Matrix multiplication\n", "\n", "To more efficently train our network these equations are implemented using matrix operations. \n", "The error in the output layer is calculated simply as, with $\\boldsymbol{t}$ being our targets, \n", "\n", "$$ \\delta_L = \\boldsymbol{t} - \\boldsymbol{y} = (n_{inputs}, n_{categories}) .$$ \n", "\n", "The gradient for the output weights is calculated as \n", "\n", "$$ \\nabla W_{L} = \\boldsymbol{a}^T \\delta_L = (n_{hidden}, n_{categories}) ,$$\n", "\n", "where $\\boldsymbol{a} = (n_{inputs}, n_{hidden})$. This simply means that we are summing up the gradients for each input. \n", "Since we are going backwards we have to transpose the activation matrix. \n", "\n", "The gradient with respect to the output bias is then \n", "\n", "$$ \\nabla \\boldsymbol{b}_{L} = \\sum_{i=1}^{n_{inputs}} \\delta_L = (n_{categories}) .$$ \n", "\n", "The error in the hidden layer is \n", "\n", "$$ \\Delta_h = \\delta_L W_{L}^T \\circ f'(z_{h}) = \\delta_L W_{L}^T \\circ a_{h} \\circ (1 - a_{h}) = (n_{inputs}, n_{hidden}) ,$$ \n", "\n", "where $f'(a_{h})$ is the derivative of the activation in the hidden layer. The matrix products mean\n", "that we are summing up the products for each neuron in the output layer. The symbol $\\circ$ denotes\n", "the *Hadamard product*, meaning element-wise multiplication. \n", "\n", "This again gives us the gradients in the hidden layer: \n", "\n", "$$ \\nabla W_{h} = X^T \\delta_h = (n_{features}, n_{hidden}) ,$$ \n", "\n", "$$ \\nabla b_{h} = \\sum_{i=1}^{n_{inputs}} \\delta_h = (n_{hidden}) .$$" ] }, { "cell_type": "code", "execution_count": 26, "id": "9ffb0561", "metadata": { "collapsed": false, "editable": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Old accuracy on training data: 0.1440501043841336\n" ] }, { "ename": "RuntimeWarning", "evalue": "overflow encountered in exp", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mRuntimeWarning\u001b[0m Traceback (most recent call last)", "Input \u001b[0;32mIn [26]\u001b[0m, in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 53\u001b[0m lmbd \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0.01\u001b[39m\n\u001b[1;32m 54\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(\u001b[38;5;241m1000\u001b[39m):\n\u001b[1;32m 55\u001b[0m \u001b[38;5;66;03m# calculate gradients\u001b[39;00m\n\u001b[0;32m---> 56\u001b[0m dWo, dBo, dWh, dBh \u001b[38;5;241m=\u001b[39m \u001b[43mbackpropagation\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX_train\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mY_train_onehot\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 58\u001b[0m \u001b[38;5;66;03m# regularization term gradients\u001b[39;00m\n\u001b[1;32m 59\u001b[0m dWo \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m lmbd \u001b[38;5;241m*\u001b[39m output_weights\n", "Input \u001b[0;32mIn [26]\u001b[0m, in \u001b[0;36mbackpropagation\u001b[0;34m(X, Y)\u001b[0m\n\u001b[1;32m 32\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mbackpropagation\u001b[39m(X, Y):\n\u001b[0;32m---> 33\u001b[0m a_h, probabilities \u001b[38;5;241m=\u001b[39m \u001b[43mfeed_forward_train\u001b[49m\u001b[43m(\u001b[49m\u001b[43mX\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 35\u001b[0m \u001b[38;5;66;03m# error in the output layer\u001b[39;00m\n\u001b[1;32m 36\u001b[0m error_output \u001b[38;5;241m=\u001b[39m probabilities \u001b[38;5;241m-\u001b[39m Y\n", "Input \u001b[0;32mIn [26]\u001b[0m, in \u001b[0;36mfeed_forward_train\u001b[0;34m(X)\u001b[0m\n\u001b[1;32m 18\u001b[0m z_h \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mmatmul(X, hidden_weights) \u001b[38;5;241m+\u001b[39m hidden_bias\n\u001b[1;32m 19\u001b[0m \u001b[38;5;66;03m# activation in the hidden layer\u001b[39;00m\n\u001b[0;32m---> 20\u001b[0m a_h \u001b[38;5;241m=\u001b[39m \u001b[43msigmoid\u001b[49m\u001b[43m(\u001b[49m\u001b[43mz_h\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 22\u001b[0m \u001b[38;5;66;03m# weighted sum of inputs to the output layer\u001b[39;00m\n\u001b[1;32m 23\u001b[0m z_o \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mmatmul(a_h, output_weights) \u001b[38;5;241m+\u001b[39m output_bias\n", "Input \u001b[0;32mIn [25]\u001b[0m, in \u001b[0;36msigmoid\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21msigmoid\u001b[39m(x):\n\u001b[0;32m----> 4\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;241m1\u001b[39m\u001b[38;5;241m/\u001b[39m(\u001b[38;5;241m1\u001b[39m \u001b[38;5;241m+\u001b[39m \u001b[43mnp\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexp\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m-\u001b[39;49m\u001b[43mx\u001b[49m\u001b[43m)\u001b[49m)\n", "\u001b[0;31mRuntimeWarning\u001b[0m: overflow encountered in exp" ] } ], "source": [ "# to categorical turns our integer vector into a onehot representation\n", "from sklearn.metrics import accuracy_score\n", "\n", "# one-hot in numpy\n", "def to_categorical_numpy(integer_vector):\n", " n_inputs = len(integer_vector)\n", " n_categories = np.max(integer_vector) + 1\n", " onehot_vector = np.zeros((n_inputs, n_categories))\n", " onehot_vector[range(n_inputs), integer_vector] = 1\n", " \n", " return onehot_vector\n", "\n", "#Y_train_onehot, Y_test_onehot = to_categorical(Y_train), to_categorical(Y_test)\n", "Y_train_onehot, Y_test_onehot = to_categorical_numpy(Y_train), to_categorical_numpy(Y_test)\n", "\n", "def feed_forward_train(X):\n", " # weighted sum of inputs to the hidden layer\n", " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", " # activation in the hidden layer\n", " a_h = sigmoid(z_h)\n", " \n", " # weighted sum of inputs to the output layer\n", " z_o = np.matmul(a_h, output_weights) + output_bias\n", " # softmax output\n", " # axis 0 holds each input and axis 1 the probabilities of each category\n", " exp_term = np.exp(z_o)\n", " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", " \n", " # for backpropagation need activations in hidden and output layers\n", " return a_h, probabilities\n", "\n", "def backpropagation(X, Y):\n", " a_h, probabilities = feed_forward_train(X)\n", " \n", " # error in the output layer\n", " error_output = probabilities - Y\n", " # error in the hidden layer\n", " error_hidden = np.matmul(error_output, output_weights.T) * a_h * (1 - a_h)\n", " \n", " # gradients for the output layer\n", " output_weights_gradient = np.matmul(a_h.T, error_output)\n", " output_bias_gradient = np.sum(error_output, axis=0)\n", " \n", " # gradient for the hidden layer\n", " hidden_weights_gradient = np.matmul(X.T, error_hidden)\n", " hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", "\n", " return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient\n", "\n", "print(\"Old accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))\n", "\n", "eta = 0.01\n", "lmbd = 0.01\n", "for i in range(1000):\n", " # calculate gradients\n", " dWo, dBo, dWh, dBh = backpropagation(X_train, Y_train_onehot)\n", " \n", " # regularization term gradients\n", " dWo += lmbd * output_weights\n", " dWh += lmbd * hidden_weights\n", " \n", " # update weights and biases\n", " output_weights -= eta * dWo\n", " output_bias -= eta * dBo\n", " hidden_weights -= eta * dWh\n", " hidden_bias -= eta * dBh\n", "\n", "print(\"New accuracy on training data: \" + str(accuracy_score(predict(X_train), Y_train)))" ] }, { "cell_type": "markdown", "id": "3b19f8e5", "metadata": { "editable": true }, "source": [ "## Improving performance\n", "\n", "As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image. \n", "In order to obtain a network that does something useful, we will have to do a bit more work. \n", "\n", "The choice of *hyperparameters* such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a *grid-search* is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates $\\eta = 10^{-6}, 10^{-5},...,10^{-1}$ with different regularization parameters $\\lambda = 10^{-6},...,10^{-0}$. \n", "\n", "Next, we haven't implemented minibatching yet, which introduces stochasticity and is though to act as an important regularizer on the weights. We call a feed-forward + backward pass with a minibatch an *iteration*, and a full training period\n", "going through the entire dataset ($n/M$ batches) an *epoch*.\n", "\n", "If this does not improve network performance, you may want to consider altering the network architecture, adding more neurons or hidden layers. \n", "Andrew Ng goes through some of these considerations in this [video](https://youtu.be/F1ka6a13S9I). You can find a summary of the video [here](https://kevinzakka.github.io/2016/09/26/applying-deep-learning/)." ] }, { "cell_type": "markdown", "id": "443db3c1", "metadata": { "editable": true }, "source": [ "## Full object-oriented implementation\n", "\n", "It is very natural to think of the network as an object, with specific instances of the network\n", "being realizations of this object with different hyperparameters. An implementation using Python classes provides a clean structure and interface, and the full implementation of our neural network is given below." ] }, { "cell_type": "code", "execution_count": 27, "id": "5a7db6a1", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "class NeuralNetwork:\n", " def __init__(\n", " self,\n", " X_data,\n", " Y_data,\n", " n_hidden_neurons=50,\n", " n_categories=10,\n", " epochs=10,\n", " batch_size=100,\n", " eta=0.1,\n", " lmbd=0.0):\n", "\n", " self.X_data_full = X_data\n", " self.Y_data_full = Y_data\n", "\n", " self.n_inputs = X_data.shape[0]\n", " self.n_features = X_data.shape[1]\n", " self.n_hidden_neurons = n_hidden_neurons\n", " self.n_categories = n_categories\n", "\n", " self.epochs = epochs\n", " self.batch_size = batch_size\n", " self.iterations = self.n_inputs // self.batch_size\n", " self.eta = eta\n", " self.lmbd = lmbd\n", "\n", " self.create_biases_and_weights()\n", "\n", " def create_biases_and_weights(self):\n", " self.hidden_weights = np.random.randn(self.n_features, self.n_hidden_neurons)\n", " self.hidden_bias = np.zeros(self.n_hidden_neurons) + 0.01\n", "\n", " self.output_weights = np.random.randn(self.n_hidden_neurons, self.n_categories)\n", " self.output_bias = np.zeros(self.n_categories) + 0.01\n", "\n", " def feed_forward(self):\n", " # feed-forward for training\n", " self.z_h = np.matmul(self.X_data, self.hidden_weights) + self.hidden_bias\n", " self.a_h = sigmoid(self.z_h)\n", "\n", " self.z_o = np.matmul(self.a_h, self.output_weights) + self.output_bias\n", "\n", " exp_term = np.exp(self.z_o)\n", " self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", "\n", " def feed_forward_out(self, X):\n", " # feed-forward for output\n", " z_h = np.matmul(X, self.hidden_weights) + self.hidden_bias\n", " a_h = sigmoid(z_h)\n", "\n", " z_o = np.matmul(a_h, self.output_weights) + self.output_bias\n", " \n", " exp_term = np.exp(z_o)\n", " probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)\n", " return probabilities\n", "\n", " def backpropagation(self):\n", " error_output = self.probabilities - self.Y_data\n", " error_hidden = np.matmul(error_output, self.output_weights.T) * self.a_h * (1 - self.a_h)\n", "\n", " self.output_weights_gradient = np.matmul(self.a_h.T, error_output)\n", " self.output_bias_gradient = np.sum(error_output, axis=0)\n", "\n", " self.hidden_weights_gradient = np.matmul(self.X_data.T, error_hidden)\n", " self.hidden_bias_gradient = np.sum(error_hidden, axis=0)\n", "\n", " if self.lmbd > 0.0:\n", " self.output_weights_gradient += self.lmbd * self.output_weights\n", " self.hidden_weights_gradient += self.lmbd * self.hidden_weights\n", "\n", " self.output_weights -= self.eta * self.output_weights_gradient\n", " self.output_bias -= self.eta * self.output_bias_gradient\n", " self.hidden_weights -= self.eta * self.hidden_weights_gradient\n", " self.hidden_bias -= self.eta * self.hidden_bias_gradient\n", "\n", " def predict(self, X):\n", " probabilities = self.feed_forward_out(X)\n", " return np.argmax(probabilities, axis=1)\n", "\n", " def predict_probabilities(self, X):\n", " probabilities = self.feed_forward_out(X)\n", " return probabilities\n", "\n", " def train(self):\n", " data_indices = np.arange(self.n_inputs)\n", "\n", " for i in range(self.epochs):\n", " for j in range(self.iterations):\n", " # pick datapoints with replacement\n", " chosen_datapoints = np.random.choice(\n", " data_indices, size=self.batch_size, replace=False\n", " )\n", "\n", " # minibatch training data\n", " self.X_data = self.X_data_full[chosen_datapoints]\n", " self.Y_data = self.Y_data_full[chosen_datapoints]\n", "\n", " self.feed_forward()\n", " self.backpropagation()" ] }, { "cell_type": "markdown", "id": "fe73ada1", "metadata": { "editable": true }, "source": [ "## Evaluate model performance on test data\n", "\n", "To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data. \n", "We measure the performance of the network using the *accuracy* score. \n", "The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of $1$. \n", "\n", "$$ \\text{Accuracy} = \\frac{\\sum_{i=1}^n I(\\tilde{y}_i = y_i)}{n} ,$$ \n", "\n", "where $I$ is the indicator function, $1$ if $\\tilde{y}_i = y_i$ and $0$ otherwise." ] }, { "cell_type": "code", "execution_count": 28, "id": "3c71da93", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "epochs = 100\n", "batch_size = 100\n", "\n", "dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", "dnn.train()\n", "test_predict = dnn.predict(X_test)\n", "\n", "# accuracy score from scikit library\n", "print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", "\n", "# equivalent in numpy\n", "def accuracy_score_numpy(Y_test, Y_pred):\n", " return np.sum(Y_test == Y_pred) / len(Y_test)\n", "\n", "#print(\"Accuracy score on test set: \", accuracy_score_numpy(Y_test, test_predict))" ] }, { "cell_type": "markdown", "id": "c6d594b6", "metadata": { "editable": true }, "source": [ "## Adjust hyperparameters\n", "\n", "We now perform a grid search to find the optimal hyperparameters for the network. \n", "Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around $98\\%$ ($2\\%$ error rate)." ] }, { "cell_type": "code", "execution_count": 29, "id": "bb8c3746", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "eta_vals = np.logspace(-5, 1, 7)\n", "lmbd_vals = np.logspace(-5, 1, 7)\n", "# store the models for later use\n", "DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", "\n", "# grid search\n", "for i, eta in enumerate(eta_vals):\n", " for j, lmbd in enumerate(lmbd_vals):\n", " dnn = NeuralNetwork(X_train, Y_train_onehot, eta=eta, lmbd=lmbd, epochs=epochs, batch_size=batch_size,\n", " n_hidden_neurons=n_hidden_neurons, n_categories=n_categories)\n", " dnn.train()\n", " \n", " DNN_numpy[i][j] = dnn\n", " \n", " test_predict = dnn.predict(X_test)\n", " \n", " print(\"Learning rate = \", eta)\n", " print(\"Lambda = \", lmbd)\n", " print(\"Accuracy score on test set: \", accuracy_score(Y_test, test_predict))\n", " print()" ] }, { "cell_type": "markdown", "id": "c460a08f", "metadata": { "editable": true }, "source": [ "## Visualization" ] }, { "cell_type": "code", "execution_count": 30, "id": "ae408ce7", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# visual representation of grid search\n", "# uses seaborn heatmap, you can also do this with matplotlib imshow\n", "import seaborn as sns\n", "\n", "sns.set()\n", "\n", "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "\n", "for i in range(len(eta_vals)):\n", " for j in range(len(lmbd_vals)):\n", " dnn = DNN_numpy[i][j]\n", " \n", " train_pred = dnn.predict(X_train) \n", " test_pred = dnn.predict(X_test)\n", "\n", " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", "\n", " \n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Training Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()\n", "\n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Test Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "255351d8", "metadata": { "editable": true }, "source": [ "## scikit-learn implementation\n", "\n", "**scikit-learn** focuses more\n", "on traditional machine learning methods, such as regression,\n", "clustering, decision trees, etc. As such, it has only two types of\n", "neural networks: Multi Layer Perceptron outputting continuous values,\n", "*MPLRegressor*, and Multi Layer Perceptron outputting labels,\n", "*MLPClassifier*. We will see how simple it is to use these classes.\n", "\n", "**scikit-learn** implements a few improvements from our neural network,\n", "such as early stopping, a varying learning rate, different\n", "optimization methods, etc. We would therefore expect a better\n", "performance overall." ] }, { "cell_type": "code", "execution_count": 31, "id": "45102bf4", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "from sklearn.neural_network import MLPClassifier\n", "# store models for later use\n", "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", "\n", "for i, eta in enumerate(eta_vals):\n", " for j, lmbd in enumerate(lmbd_vals):\n", " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n", " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n", " dnn.fit(X_train, Y_train)\n", " \n", " DNN_scikit[i][j] = dnn\n", " \n", " print(\"Learning rate = \", eta)\n", " print(\"Lambda = \", lmbd)\n", " print(\"Accuracy score on test set: \", dnn.score(X_test, Y_test))\n", " print()" ] }, { "cell_type": "markdown", "id": "9ed166d7", "metadata": { "editable": true }, "source": [ "## Visualization" ] }, { "cell_type": "code", "execution_count": 32, "id": "e9fb271f", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# optional\n", "# visual representation of grid search\n", "# uses seaborn heatmap, could probably do this in matplotlib\n", "import seaborn as sns\n", "\n", "sns.set()\n", "\n", "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "\n", "for i in range(len(eta_vals)):\n", " for j in range(len(lmbd_vals)):\n", " dnn = DNN_scikit[i][j]\n", " \n", " train_pred = dnn.predict(X_train) \n", " test_pred = dnn.predict(X_test)\n", "\n", " train_accuracy[i][j] = accuracy_score(Y_train, train_pred)\n", " test_accuracy[i][j] = accuracy_score(Y_test, test_pred)\n", "\n", " \n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Training Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()\n", "\n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Test Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "6cbb4072", "metadata": { "editable": true }, "source": [ "## Building neural networks in Tensorflow and Keras\n", "\n", "Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn\n", "and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy\n", "and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer. \n", "\n", "In our previous example we used only one hidden layer, and in this we will use two. From this it should be quite\n", "clear how to build one using an arbitrary number of hidden layers, using data structures such as Python lists or\n", "NumPy arrays." ] }, { "cell_type": "markdown", "id": "2702932a", "metadata": { "editable": true }, "source": [ "## Tensorflow\n", "\n", "Tensorflow is an open source library machine learning library\n", "developed by the Google Brain team for internal use. It was released\n", "under the Apache 2.0 open source license in November 9, 2015.\n", "\n", "Tensorflow is a computational framework that allows you to construct\n", "machine learning models at different levels of abstraction, from\n", "high-level, object-oriented APIs like Keras, down to the C++ kernels\n", "that Tensorflow is built upon. The higher levels of abstraction are\n", "simpler to use, but less flexible, and our choice of implementation\n", "should reflect the problems we are trying to solve.\n", "\n", "[Tensorflow uses](https://www.tensorflow.org/guide/graphs) so-called graphs to represent your computation\n", "in terms of the dependencies between individual operations, such that you first build a Tensorflow *graph*\n", "to represent your model, and then create a Tensorflow *session* to run the graph.\n", "\n", "In this guide we will analyze the same data as we did in our NumPy and\n", "scikit-learn tutorial, gathered from the MNIST database of images. We\n", "will give an introduction to the lower level Python Application\n", "Program Interfaces (APIs), and see how we use them to build our graph.\n", "Then we will build (effectively) the same graph in Keras, to see just\n", "how simple solving a machine learning problem can be.\n", "\n", "To install tensorflow on Unix/Linux systems, use pip as" ] }, { "cell_type": "code", "execution_count": 33, "id": "a66fc7b6", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "pip3 install tensorflow" ] }, { "cell_type": "markdown", "id": "3c99a292", "metadata": { "editable": true }, "source": [ "and/or if you use **anaconda**, just write (or install from the graphical user interface)\n", "(current release of CPU-only TensorFlow)" ] }, { "cell_type": "code", "execution_count": 34, "id": "71e9f24b", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "conda create -n tf tensorflow\n", "conda activate tf" ] }, { "cell_type": "markdown", "id": "a623588f", "metadata": { "editable": true }, "source": [ "To install the current release of GPU TensorFlow" ] }, { "cell_type": "code", "execution_count": 35, "id": "fa19ee75", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "conda create -n tf-gpu tensorflow-gpu\n", "conda activate tf-gpu" ] }, { "cell_type": "markdown", "id": "de5fec4a", "metadata": { "editable": true }, "source": [ "## Using Keras\n", "\n", "Keras is a high level [neural network](https://en.wikipedia.org/wiki/Application_programming_interface)\n", "that supports Tensorflow, CTNK and Theano as backends. \n", "If you have Anaconda installed you may run the following command" ] }, { "cell_type": "code", "execution_count": 36, "id": "edb61bdb", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "conda install keras" ] }, { "cell_type": "markdown", "id": "fd72dabb", "metadata": { "editable": true }, "source": [ "You can look up the [instructions here](https://keras.io/) for more information.\n", "\n", "We will to a large extent use **keras** in our examples.." ] }, { "cell_type": "markdown", "id": "b8262ab5", "metadata": { "editable": true }, "source": [ "## Collect and pre-process data\n", "\n", "Let us look again at the MINST data set." ] }, { "cell_type": "code", "execution_count": 37, "id": "0b88258b", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# import necessary packages\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import tensorflow as tf\n", "from sklearn import datasets\n", "\n", "\n", "# ensure the same random numbers appear every time\n", "np.random.seed(0)\n", "\n", "# display images in notebook\n", "%matplotlib inline\n", "plt.rcParams['figure.figsize'] = (12,12)\n", "\n", "\n", "# download MNIST dataset\n", "digits = datasets.load_digits()\n", "\n", "# define inputs and labels\n", "inputs = digits.images\n", "labels = digits.target\n", "\n", "print(\"inputs = (n_inputs, pixel_width, pixel_height) = \" + str(inputs.shape))\n", "print(\"labels = (n_inputs) = \" + str(labels.shape))\n", "\n", "\n", "# flatten the image\n", "# the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64\n", "n_inputs = len(inputs)\n", "inputs = inputs.reshape(n_inputs, -1)\n", "print(\"X = (n_inputs, n_features) = \" + str(inputs.shape))\n", "\n", "\n", "# choose some random images to display\n", "indices = np.arange(n_inputs)\n", "random_indices = np.random.choice(indices, size=5)\n", "\n", "for i, image in enumerate(digits.images[random_indices]):\n", " plt.subplot(1, 5, i+1)\n", " plt.axis('off')\n", " plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')\n", " plt.title(\"Label: %d\" % digits.target[random_indices[i]])\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": 38, "id": "5000582f", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "from tensorflow.keras.layers import Input\n", "from tensorflow.keras.models import Sequential #This allows appending layers to existing models\n", "from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer\n", "from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)\n", "from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)\n", "from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function\n", "\n", "from sklearn.model_selection import train_test_split\n", "\n", "# one-hot representation of labels\n", "labels = to_categorical(labels)\n", "\n", "# split into train and test data\n", "train_size = 0.8\n", "test_size = 1 - train_size\n", "X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=train_size,\n", " test_size=test_size)" ] }, { "cell_type": "code", "execution_count": 39, "id": "a473cac3", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "\n", "epochs = 100\n", "batch_size = 100\n", "n_neurons_layer1 = 100\n", "n_neurons_layer2 = 50\n", "n_categories = 10\n", "eta_vals = np.logspace(-5, 1, 7)\n", "lmbd_vals = np.logspace(-5, 1, 7)\n", "def create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories, eta, lmbd):\n", " model = Sequential()\n", " model.add(Dense(n_neurons_layer1, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", " model.add(Dense(n_neurons_layer2, activation='sigmoid', kernel_regularizer=regularizers.l2(lmbd)))\n", " model.add(Dense(n_categories, activation='softmax'))\n", " \n", " sgd = optimizers.SGD(lr=eta)\n", " model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n", " \n", " return model" ] }, { "cell_type": "code", "execution_count": 40, "id": "041ff977", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", " \n", "for i, eta in enumerate(eta_vals):\n", " for j, lmbd in enumerate(lmbd_vals):\n", " DNN = create_neural_network_keras(n_neurons_layer1, n_neurons_layer2, n_categories,\n", " eta=eta, lmbd=lmbd)\n", " DNN.fit(X_train, Y_train, epochs=epochs, batch_size=batch_size, verbose=0)\n", " scores = DNN.evaluate(X_test, Y_test)\n", " \n", " DNN_keras[i][j] = DNN\n", " \n", " print(\"Learning rate = \", eta)\n", " print(\"Lambda = \", lmbd)\n", " print(\"Test accuracy: %.3f\" % scores[1])\n", " print()" ] }, { "cell_type": "code", "execution_count": 41, "id": "1821d39d", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# optional\n", "# visual representation of grid search\n", "# uses seaborn heatmap, could probably do this in matplotlib\n", "import seaborn as sns\n", "\n", "sns.set()\n", "\n", "train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", "\n", "for i in range(len(eta_vals)):\n", " for j in range(len(lmbd_vals)):\n", " DNN = DNN_keras[i][j]\n", "\n", " train_accuracy[i][j] = DNN.evaluate(X_train, Y_train)[1]\n", " test_accuracy[i][j] = DNN.evaluate(X_test, Y_test)[1]\n", "\n", " \n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(train_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Training Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()\n", "\n", "fig, ax = plt.subplots(figsize = (10, 10))\n", "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", "ax.set_title(\"Test Accuracy\")\n", "ax.set_ylabel(\"$\\eta$\")\n", "ax.set_xlabel(\"$\\lambda$\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "c8b43dbc", "metadata": { "editable": true }, "source": [ "## The Breast Cancer Data, now with Keras" ] }, { "cell_type": "code", "execution_count": 42, "id": "f902c476", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "\n", "import tensorflow as tf\n", "from tensorflow.keras.layers import Input\n", "from tensorflow.keras.models import Sequential #This allows appending layers to existing models\n", "from tensorflow.keras.layers import Dense #This allows defining the characteristics of a particular layer\n", "from tensorflow.keras import optimizers #This allows using whichever optimiser we want (sgd,adam,RMSprop)\n", "from tensorflow.keras import regularizers #This allows using whichever regularizer we want (l1,l2,l1_l2)\n", "from tensorflow.keras.utils import to_categorical #This allows using categorical cross entropy as the cost function\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from sklearn.model_selection import train_test_split as splitter\n", "from sklearn.datasets import load_breast_cancer\n", "import pickle\n", "import os \n", "\n", "\n", "\"\"\"Load breast cancer dataset\"\"\"\n", "\n", "np.random.seed(0) #create same seed for random number every time\n", "\n", "cancer=load_breast_cancer() #Download breast cancer dataset\n", "\n", "inputs=cancer.data #Feature matrix of 569 rows (samples) and 30 columns (parameters)\n", "outputs=cancer.target #Label array of 569 rows (0 for benign and 1 for malignant)\n", "labels=cancer.feature_names[0:30]\n", "\n", "print('The content of the breast cancer dataset is:') #Print information about the datasets\n", "print(labels)\n", "print('-------------------------')\n", "print(\"inputs = \" + str(inputs.shape))\n", "print(\"outputs = \" + str(outputs.shape))\n", "print(\"labels = \"+ str(labels.shape))\n", "\n", "x=inputs #Reassign the Feature and Label matrices to other variables\n", "y=outputs\n", "\n", "#%% \n", "\n", "# Visualisation of dataset (for correlation analysis)\n", "\n", "plt.figure()\n", "plt.scatter(x[:,0],x[:,2],s=40,c=y,cmap=plt.cm.Spectral)\n", "plt.xlabel('Mean radius',fontweight='bold')\n", "plt.ylabel('Mean perimeter',fontweight='bold')\n", "plt.show()\n", "\n", "plt.figure()\n", "plt.scatter(x[:,5],x[:,6],s=40,c=y, cmap=plt.cm.Spectral)\n", "plt.xlabel('Mean compactness',fontweight='bold')\n", "plt.ylabel('Mean concavity',fontweight='bold')\n", "plt.show()\n", "\n", "\n", "plt.figure()\n", "plt.scatter(x[:,0],x[:,1],s=40,c=y,cmap=plt.cm.Spectral)\n", "plt.xlabel('Mean radius',fontweight='bold')\n", "plt.ylabel('Mean texture',fontweight='bold')\n", "plt.show()\n", "\n", "plt.figure()\n", "plt.scatter(x[:,2],x[:,1],s=40,c=y,cmap=plt.cm.Spectral)\n", "plt.xlabel('Mean perimeter',fontweight='bold')\n", "plt.ylabel('Mean compactness',fontweight='bold')\n", "plt.show()\n", "\n", "\n", "# Generate training and testing datasets\n", "\n", "#Select features relevant to classification (texture,perimeter,compactness and symmetery) \n", "#and add to input matrix\n", "\n", "temp1=np.reshape(x[:,1],(len(x[:,1]),1))\n", "temp2=np.reshape(x[:,2],(len(x[:,2]),1))\n", "X=np.hstack((temp1,temp2)) \n", "temp=np.reshape(x[:,5],(len(x[:,5]),1))\n", "X=np.hstack((X,temp)) \n", "temp=np.reshape(x[:,8],(len(x[:,8]),1))\n", "X=np.hstack((X,temp)) \n", "\n", "X_train,X_test,y_train,y_test=splitter(X,y,test_size=0.1) #Split datasets into training and testing\n", "\n", "y_train=to_categorical(y_train) #Convert labels to categorical when using categorical cross entropy\n", "y_test=to_categorical(y_test)\n", "\n", "del temp1,temp2,temp\n", "\n", "# %%\n", "\n", "# Define tunable parameters\"\n", "\n", "eta=np.logspace(-3,-1,3) #Define vector of learning rates (parameter to SGD optimiser)\n", "lamda=0.01 #Define hyperparameter\n", "n_layers=2 #Define number of hidden layers in the model\n", "n_neuron=np.logspace(0,3,4,dtype=int) #Define number of neurons per layer\n", "epochs=100 #Number of reiterations over the input data\n", "batch_size=100 #Number of samples per gradient update\n", "\n", "# %%\n", "\n", "\"\"\"Define function to return Deep Neural Network model\"\"\"\n", "\n", "def NN_model(inputsize,n_layers,n_neuron,eta,lamda):\n", " model=Sequential() \n", " for i in range(n_layers): #Run loop to add hidden layers to the model\n", " if (i==0): #First layer requires input dimensions\n", " model.add(Dense(n_neuron,activation='relu',kernel_regularizer=regularizers.l2(lamda),input_dim=inputsize))\n", " else: #Subsequent layers are capable of automatic shape inferencing\n", " model.add(Dense(n_neuron,activation='relu',kernel_regularizer=regularizers.l2(lamda)))\n", " model.add(Dense(2,activation='softmax')) #2 outputs - ordered and disordered (softmax for prob)\n", " sgd=optimizers.SGD(lr=eta)\n", " model.compile(loss='categorical_crossentropy',optimizer=sgd,metrics=['accuracy'])\n", " return model\n", "\n", " \n", "Train_accuracy=np.zeros((len(n_neuron),len(eta))) #Define matrices to store accuracy scores as a function\n", "Test_accuracy=np.zeros((len(n_neuron),len(eta))) #of learning rate and number of hidden neurons for \n", "\n", "for i in range(len(n_neuron)): #run loops over hidden neurons and learning rates to calculate \n", " for j in range(len(eta)): #accuracy scores \n", " DNN_model=NN_model(X_train.shape[1],n_layers,n_neuron[i],eta[j],lamda)\n", " DNN_model.fit(X_train,y_train,epochs=epochs,batch_size=batch_size,verbose=1)\n", " Train_accuracy[i,j]=DNN_model.evaluate(X_train,y_train)[1]\n", " Test_accuracy[i,j]=DNN_model.evaluate(X_test,y_test)[1]\n", " \n", "\n", "def plot_data(x,y,data,title=None):\n", "\n", " # plot results\n", " fontsize=16\n", "\n", "\n", " fig = plt.figure()\n", " ax = fig.add_subplot(111)\n", " cax = ax.matshow(data, interpolation='nearest', vmin=0, vmax=1)\n", " \n", " cbar=fig.colorbar(cax)\n", " cbar.ax.set_ylabel('accuracy (%)',rotation=90,fontsize=fontsize)\n", " cbar.set_ticks([0,.2,.4,0.6,0.8,1.0])\n", " cbar.set_ticklabels(['0%','20%','40%','60%','80%','100%'])\n", "\n", " # put text on matrix elements\n", " for i, x_val in enumerate(np.arange(len(x))):\n", " for j, y_val in enumerate(np.arange(len(y))):\n", " c = \"${0:.1f}\\\\%$\".format( 100*data[j,i]) \n", " ax.text(x_val, y_val, c, va='center', ha='center')\n", "\n", " # convert axis vaues to to string labels\n", " x=[str(i) for i in x]\n", " y=[str(i) for i in y]\n", "\n", "\n", " ax.set_xticklabels(['']+x)\n", " ax.set_yticklabels(['']+y)\n", "\n", " ax.set_xlabel('$\\\\mathrm{learning\\\\ rate}$',fontsize=fontsize)\n", " ax.set_ylabel('$\\\\mathrm{hidden\\\\ neurons}$',fontsize=fontsize)\n", " if title is not None:\n", " ax.set_title(title)\n", "\n", " plt.tight_layout()\n", "\n", " plt.show()\n", " \n", "plot_data(eta,n_neuron,Train_accuracy, 'training')\n", "plot_data(eta,n_neuron,Test_accuracy, 'testing')" ] }, { "cell_type": "markdown", "id": "fa673f3a", "metadata": { "editable": true }, "source": [ "## Fine-tuning neural network hyperparameters\n", "\n", "The flexibility of neural networks is also one of their main\n", "drawbacks: there are many hyperparameters to tweak. Not only can you\n", "use any imaginable network topology (how neurons/nodes are interconnected),\n", "but even in a simple FFNN you can change the number of layers, the\n", "number of neurons per layer, the type of activation function to use in\n", "each layer, the weight initialization logic, the stochastic gradient optmized and much more. How do you\n", "know what combination of hyperparameters is the best for your task?\n", "\n", "* You can use grid search with cross-validation to find the right hyperparameters.\n", "\n", "However,since there are many hyperparameters to tune, and since\n", "training a neural network on a large dataset takes a lot of time, you\n", "will only be able to explore a tiny part of the hyperparameter space.\n", "\n", "* You can use randomized search.\n", "\n", "* Or use tools like [Oscar](http://oscar.calldesk.ai/), which implements more complex algorithms to help you find a good set of hyperparameters quickly." ] }, { "cell_type": "markdown", "id": "6c4727a9", "metadata": { "editable": true }, "source": [ "## Hidden layers\n", "\n", "For many problems you can start with just one or two hidden layers and it will work just fine.\n", "For the MNIST data set you ca easily get a high accuracy using just one hidden layer with a\n", "few hundred neurons.\n", "You can reach for this data set above 98% accuracy using two hidden layers with the same total amount of\n", "neurons, in roughly the same amount of training time. \n", "\n", "For more complex problems, you can gradually\n", "ramp up the number of hidden layers, until you start overfitting the training set. Very complex tasks, such\n", "as large image classification or speech recognition, typically require networks with dozens of layers\n", "and they need a huge amount\n", "of training data. However, you will rarely have to train such networks from scratch: it is much more\n", "common to reuse parts of a pretrained state-of-the-art network that performs a similar task." ] }, { "cell_type": "markdown", "id": "f24d5df1", "metadata": { "editable": true }, "source": [ "## Which activation function should I use?\n", "\n", "The Back propagation algorithm we derived above works by going from\n", "the output layer to the input layer, propagating the error gradient on\n", "the way. Once the algorithm has computed the gradient of the cost\n", "function with regards to each parameter in the network, it uses these\n", "gradients to update each parameter with a Gradient Descent (GD) step.\n", "\n", "Unfortunately for us, the gradients often get smaller and smaller as the\n", "algorithm progresses down to the first hidden layers. As a result, the\n", "GD update leaves the lower layer connection weights\n", "virtually unchanged, and training never converges to a good\n", "solution. This is known in the literature as \n", "**the vanishing gradients problem**. \n", "\n", "In other cases, the opposite can happen, namely the the gradients can grow bigger and\n", "bigger. The result is that many of the layers get large updates of the \n", "weights the\n", "algorithm diverges. This is the **exploding gradients problem**, which is\n", "mostly encountered in recurrent neural networks. More generally, deep\n", "neural networks suffer from unstable gradients, different layers may\n", "learn at widely different speeds" ] }, { "cell_type": "markdown", "id": "80cd048d", "metadata": { "editable": true }, "source": [ "## Is the Logistic activation function (Sigmoid) our choice?\n", "\n", "Although this unfortunate behavior has been empirically observed for\n", "quite a while (it was one of the reasons why deep neural networks were\n", "mostly abandoned for a long time), it is only around 2010 that\n", "significant progress was made in understanding it.\n", "\n", "A paper titled [Understanding the Difficulty of Training Deep\n", "Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio](http://proceedings.mlr.press/v9/glorot10a.html) found that\n", "the problems with the popular logistic\n", "sigmoid activation function and the weight initialization technique\n", "that was most popular at the time, namely random initialization using\n", "a normal distribution with a mean of 0 and a standard deviation of\n", "1. \n", "\n", "They showed that with this activation function and this\n", "initialization scheme, the variance of the outputs of each layer is\n", "much greater than the variance of its inputs. Going forward in the\n", "network, the variance keeps increasing after each layer until the\n", "activation function saturates at the top layers. This is actually made\n", "worse by the fact that the logistic function has a mean of 0.5, not 0\n", "(the hyperbolic tangent function has a mean of 0 and behaves slightly\n", "better than the logistic function in deep networks)." ] }, { "cell_type": "markdown", "id": "6863dde1", "metadata": { "editable": true }, "source": [ "## The derivative of the Logistic funtion\n", "\n", "Looking at the logistic activation function, when inputs become large\n", "(negative or positive), the function saturates at 0 or 1, with a\n", "derivative extremely close to 0. Thus when backpropagation kicks in,\n", "it has virtually no gradient to propagate back through the network,\n", "and what little gradient exists keeps getting diluted as\n", "backpropagation progresses down through the top layers, so there is\n", "really nothing left for the lower layers.\n", "\n", "In their paper, Glorot and Bengio propose a way to significantly\n", "alleviate this problem. We need the signal to flow properly in both\n", "directions: in the forward direction when making predictions, and in\n", "the reverse direction when backpropagating gradients. We don’t want\n", "the signal to die out, nor do we want it to explode and saturate. For\n", "the signal to flow properly, the authors argue that we need the\n", "variance of the outputs of each layer to be equal to the variance of\n", "its inputs, and we also need the gradients to have equal variance\n", "before and after flowing through a layer in the reverse direction.\n", "\n", "One of the insights in the 2010 paper by Glorot and Bengio was that\n", "the vanishing/exploding gradients problems were in part due to a poor\n", "choice of activation function. Until then most people had assumed that\n", "if Nature had chosen to use roughly sigmoid activation functions in\n", "biological neurons, they must be an excellent choice. But it turns out\n", "that other activation functions behave much better in deep neural\n", "networks, in particular the ReLU activation function, mostly because\n", "it does not saturate for positive values (and also because it is quite\n", "fast to compute)." ] }, { "cell_type": "markdown", "id": "95f03af7", "metadata": { "editable": true }, "source": [ "## The RELU function family\n", "\n", "The ReLU activation function suffers from a problem known as the dying\n", "ReLUs: during training, some neurons effectively die, meaning they\n", "stop outputting anything other than 0.\n", "\n", "In some cases, you may find that half of your network’s neurons are\n", "dead, especially if you used a large learning rate. During training,\n", "if a neuron’s weights get updated such that the weighted sum of the\n", "neuron’s inputs is negative, it will start outputting 0. When this\n", "happen, the neuron is unlikely to come back to life since the gradient\n", "of the ReLU function is 0 when its input is negative.\n", "\n", "To solve this problem, nowadays practitioners use a variant of the ReLU\n", "function, such as the leaky ReLU discussed above or the so-called\n", "exponential linear unit (ELU) function" ] }, { "cell_type": "markdown", "id": "620cac32", "metadata": { "editable": true }, "source": [ "$$\n", "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", "$$" ] }, { "cell_type": "markdown", "id": "2dd91f71", "metadata": { "editable": true }, "source": [ "## Which activation function should we use?\n", "\n", "In general it seems that the ELU activation function is better than\n", "the leaky ReLU function (and its variants), which is better than\n", "ReLU. ReLU performs better than $\\tanh$ which in turn performs better\n", "than the logistic function. \n", "\n", "If runtime\n", "performance is an issue, then you may opt for the leaky ReLU function over the \n", "ELU function If you don’t\n", "want to tweak yet another hyperparameter, you may just use the default\n", "$\\alpha$ of $0.01$ for the leaky ReLU, and $1$ for ELU. If you have\n", "spare time and computing power, you can use cross-validation or\n", "bootstrap to evaluate other activation functions." ] }, { "cell_type": "markdown", "id": "bfa3871c", "metadata": { "editable": true }, "source": [ "## More on activation functions, output layers\n", "\n", "In most cases you can use the ReLU activation function in the hidden layers (or one of its variants).\n", "\n", "It is a bit faster to compute than other activation functions, and the gradient descent optimization does in general not get stuck.\n", "\n", "**For the output layer:**\n", "\n", "* For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).\n", "\n", "* For regression tasks, you can simply use no activation function at all." ] }, { "cell_type": "markdown", "id": "9315a3fc", "metadata": { "editable": true }, "source": [ "## Batch Normalization\n", "\n", "Batch Normalization\n", "aims to address the vanishing/exploding gradients problems, and more generally the problem that the\n", "distribution of each layer’s inputs changes during training, as the parameters of the previous layers change.\n", "\n", "The technique consists of adding an operation in the model just before the activation function of each\n", "layer, simply zero-centering and normalizing the inputs, then scaling and shifting the result using two new\n", "parameters per layer (one for scaling, the other for shifting). In other words, this operation lets the model\n", "learn the optimal scale and mean of the inputs for each layer.\n", "In order to zero-center and normalize the inputs, the algorithm needs to estimate the inputs’ mean and\n", "standard deviation. It does so by evaluating the mean and standard deviation of the inputs over the current\n", "mini-batch, from this the name batch normalization." ] }, { "cell_type": "markdown", "id": "85eb5069", "metadata": { "editable": true }, "source": [ "## Dropout\n", "\n", "It is a fairly simple algorithm: at every training step, every neuron (including the input neurons but\n", "excluding the output neurons) has a probability $p$ of being temporarily dropped out, meaning it will be\n", "entirely ignored during this training step, but it may be active during the next step.\n", "\n", "The\n", "hyperparameter $p$ is called the dropout rate, and it is typically set to 50%. After training, the neurons are not dropped anymore.\n", " It is viewed as one of the most popular regularization techniques." ] }, { "cell_type": "markdown", "id": "d2d08622", "metadata": { "editable": true }, "source": [ "## Gradient Clipping\n", "\n", "A popular technique to lessen the exploding gradients problem is to simply clip the gradients during\n", "backpropagation so that they never exceed some threshold (this is mostly useful for recurrent neural\n", "networks).\n", "\n", "This technique is called Gradient Clipping.\n", "\n", "In general however, Batch\n", "Normalization is preferred." ] }, { "cell_type": "markdown", "id": "566bcca7", "metadata": { "editable": true }, "source": [ "## A very nice website on Neural Networks\n", "\n", "You may find this [website](https://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=0&networkShape=4,2&seed=0.29243&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false) very useful." ] }, { "cell_type": "markdown", "id": "aedf225e", "metadata": { "editable": true }, "source": [ "## A top-down perspective on Neural networks\n", "\n", "The first thing we would like to do is divide the data into two or three\n", "parts. A training set, a validation or dev (development) set, and a\n", "test set. The test set is the data on which we want to make\n", "predictions. The dev set is a subset of the training data we use to\n", "check how well we are doing out-of-sample, after training the model on\n", "the training dataset. We use the validation error as a proxy for the\n", "test error in order to make tweaks to our model. It is crucial that we\n", "do not use any of the test data to train the algorithm. This is a\n", "cardinal sin in ML. Then:\n", "\n", "* Estimate optimal error rate\n", "\n", "* Minimize underfitting (bias) on training data set.\n", "\n", "* Make sure you are not overfitting.\n", "\n", "If the validation and test sets are drawn from the same distributions,\n", "then a good performance on the validation set should lead to similarly\n", "good performance on the test set. \n", "\n", "However, sometimes\n", "the training data and test data differ in subtle ways because, for\n", "example, they are collected using slightly different methods, or\n", "because it is cheaper to collect data in one way versus another. In\n", "this case, there can be a mismatch between the training and test\n", "data. This can lead to the neural network overfitting these small\n", "differences between the test and training sets, and a poor performance\n", "on the test set despite having a good performance on the validation\n", "set. To rectify this, Andrew Ng suggests making two validation or dev\n", "sets, one constructed from the training data and one constructed from\n", "the test data. The difference between the performance of the algorithm\n", "on these two validation sets quantifies the train-test mismatch. This\n", "can serve as another important diagnostic when using DNNs for\n", "supervised learning." ] }, { "cell_type": "markdown", "id": "8723e9ba", "metadata": { "editable": true }, "source": [ "## Limitations of supervised learning with deep networks\n", "\n", "Like all statistical methods, supervised learning using neural\n", "networks has important limitations. This is especially important when\n", "one seeks to apply these methods, especially to physics problems. Like\n", "all tools, DNNs are not a universal solution. Often, the same or\n", "better performance on a task can be achieved by using a few\n", "hand-engineered features (or even a collection of random\n", "features). \n", "\n", "Here we list some of the important limitations of supervised neural network based models. \n", "\n", "* **Need labeled data**. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).\n", "\n", "* **Supervised neural networks are extremely data intensive.** DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs.\n", "\n", "* **Homogeneous data.** Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e. some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types.\n", "\n", "* **Many problems are not about prediction.** In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a *wrong* model. The model might or might not be useful for understanding the underlying science.\n", "\n", "Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems." ] }, { "cell_type": "markdown", "id": "b641b576", "metadata": { "editable": true }, "source": [ "## Solving ODEs with Deep Learning\n", "\n", "The Universal Approximation Theorem states that a neural network can\n", "approximate any function at a single hidden layer along with one input\n", "and output layer to any given precision.\n", "\n", "**Book on solving differential equations with ML methods.**\n", "\n", "[An Introduction to Neural Network Methods for Differential Equations](https://www.springer.com/gp/book/9789401798150), by Yadav and Kumar.\n", "\n", "**Master thesis on applying deep learning to problems in mechanics.**\n", "\n", "[Using Deep Reinforcement Learning for Active Flow Control](https://www.duo.uio.no/handle/10852/79212), by Marius Holm\n", "\n", "**Thanks to Kristine Baluka Hein.**\n", "\n", "The lectures on differential equations were developed by Kristine Baluka Hein, now PhD student at IFI.\n", "A great thanks to Kristine." ] }, { "cell_type": "markdown", "id": "e919d2c8", "metadata": { "editable": true }, "source": [ "## Ordinary Differential Equations\n", "\n", "An ordinary differential equation (ODE) is an equation involving functions having one variable.\n", "\n", "In general, an ordinary differential equation looks like" ] }, { "cell_type": "markdown", "id": "12b84451", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{ode} \\tag{1}\n", "f\\left(x, \\, g(x), \\, g'(x), \\, g''(x), \\, \\dots \\, , \\, g^{(n)}(x)\\right) = 0\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "88407501", "metadata": { "editable": true }, "source": [ "where $g(x)$ is the function to find, and $g^{(n)}(x)$ is the $n$-th derivative of $g(x)$.\n", "\n", "The $f\\left(x, g(x), g'(x), g''(x), \\, \\dots \\, , g^{(n)}(x)\\right)$ is just a way to write that there is an expression involving $x$ and $g(x), \\ g'(x), \\ g''(x), \\, \\dots \\, , \\text{ and } g^{(n)}(x)$ on the left side of the equality sign in ([1](#ode)).\n", "The highest order of derivative, that is the value of $n$, determines to the order of the equation.\n", "The equation is referred to as a $n$-th order ODE.\n", "Along with ([1](#ode)), some additional conditions of the function $g(x)$ are typically given\n", "for the solution to be unique." ] }, { "cell_type": "markdown", "id": "034cef5e", "metadata": { "editable": true }, "source": [ "## The trial solution\n", "\n", "Let the trial solution $g_t(x)$ be" ] }, { "cell_type": "markdown", "id": "86693b23", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation}\n", "\tg_t(x) = h_1(x) + h_2(x,N(x,P))\n", "\\label{_auto1} \\tag{2}\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "5c8e1418", "metadata": { "editable": true }, "source": [ "where $h_1(x)$ is a function that makes $g_t(x)$ satisfy a given set\n", "of conditions, $N(x,P)$ a neural network with weights and biases\n", "described by $P$ and $h_2(x, N(x,P))$ some expression involving the\n", "neural network. The role of the function $h_2(x, N(x,P))$, is to\n", "ensure that the output from $N(x,P)$ is zero when $g_t(x)$ is\n", "evaluated at the values of $x$ where the given conditions must be\n", "satisfied. The function $h_1(x)$ should alone make $g_t(x)$ satisfy\n", "the conditions.\n", "\n", "But what about the network $N(x,P)$?\n", "\n", "As described previously, an optimization method could be used to minimize the parameters of a neural network, that being its weights and biases, through backward propagation." ] }, { "cell_type": "markdown", "id": "4ca42ad5", "metadata": { "editable": true }, "source": [ "## Minimization process\n", "\n", "For the minimization to be defined, we need to have a cost function at hand to minimize.\n", "\n", "It is given that $f\\left(x, \\, g(x), \\, g'(x), \\, g''(x), \\, \\dots \\, , \\, g^{(n)}(x)\\right)$ should be equal to zero in ([1](#ode)).\n", "We can choose to consider the mean squared error as the cost function for an input $x$.\n", "Since we are looking at one input, the cost function is just $f$ squared.\n", "The cost function $c\\left(x, P \\right)$ can therefore be expressed as" ] }, { "cell_type": "markdown", "id": "a0808029", "metadata": { "editable": true }, "source": [ "$$\n", "C\\left(x, P\\right) = \\big(f\\left(x, \\, g(x), \\, g'(x), \\, g''(x), \\, \\dots \\, , \\, g^{(n)}(x)\\right)\\big)^2\n", "$$" ] }, { "cell_type": "markdown", "id": "8e242ee9", "metadata": { "editable": true }, "source": [ "If $N$ inputs are given as a vector $\\boldsymbol{x}$ with elements $x_i$ for $i = 1,\\dots,N$,\n", "the cost function becomes" ] }, { "cell_type": "markdown", "id": "6cb613c0", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{cost} \\tag{3}\n", "\tC\\left(\\boldsymbol{x}, P\\right) = \\frac{1}{N} \\sum_{i=1}^N \\big(f\\left(x_i, \\, g(x_i), \\, g'(x_i), \\, g''(x_i), \\, \\dots \\, , \\, g^{(n)}(x_i)\\right)\\big)^2\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "ff0ace15", "metadata": { "editable": true }, "source": [ "The neural net should then find the parameters $P$ that minimizes the cost function in\n", "([3](#cost)) for a set of $N$ training samples $x_i$." ] }, { "cell_type": "markdown", "id": "d77df59b", "metadata": { "editable": true }, "source": [ "## Minimizing the cost function using gradient descent and automatic differentiation\n", "\n", "To perform the minimization using gradient descent, the gradient of $C\\left(\\boldsymbol{x}, P\\right)$ is needed.\n", "It might happen so that finding an analytical expression of the gradient of $C(\\boldsymbol{x}, P)$ from ([3](#cost)) gets too messy, depending on which cost function one desires to use.\n", "\n", "Luckily, there exists libraries that makes the job for us through automatic differentiation.\n", "Automatic differentiation is a method of finding the derivatives numerically with very high precision." ] }, { "cell_type": "markdown", "id": "b34fce14", "metadata": { "editable": true }, "source": [ "## Example: Exponential decay\n", "\n", "An exponential decay of a quantity $g(x)$ is described by the equation" ] }, { "cell_type": "markdown", "id": "290ed299", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{solve_expdec} \\tag{4}\n", " g'(x) = -\\gamma g(x)\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "d386881a", "metadata": { "editable": true }, "source": [ "with $g(0) = g_0$ for some chosen initial value $g_0$.\n", "\n", "The analytical solution of ([4](#solve_expdec)) is" ] }, { "cell_type": "markdown", "id": "cf4aca15", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation}\n", " g(x) = g_0 \\exp\\left(-\\gamma x\\right)\n", "\\label{_auto2} \\tag{5}\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "ee5531c2", "metadata": { "editable": true }, "source": [ "Having an analytical solution at hand, it is possible to use it to compare how well a neural network finds a solution of ([4](#solve_expdec))." ] }, { "cell_type": "markdown", "id": "64a52bc0", "metadata": { "editable": true }, "source": [ "## The function to solve for\n", "\n", "The program will use a neural network to solve" ] }, { "cell_type": "markdown", "id": "e0154c0f", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{solveode} \\tag{6}\n", "g'(x) = -\\gamma g(x)\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "334e7f10", "metadata": { "editable": true }, "source": [ "where $g(0) = g_0$ with $\\gamma$ and $g_0$ being some chosen values.\n", "\n", "In this example, $\\gamma = 2$ and $g_0 = 10$." ] }, { "cell_type": "markdown", "id": "66c1d55c", "metadata": { "editable": true }, "source": [ "## The trial solution\n", "To begin with, a trial solution $g_t(t)$ must be chosen. A general trial solution for ordinary differential equations could be" ] }, { "cell_type": "markdown", "id": "f03c7338", "metadata": { "editable": true }, "source": [ "$$\n", "g_t(x, P) = h_1(x) + h_2(x, N(x, P))\n", "$$" ] }, { "cell_type": "markdown", "id": "ca114c87", "metadata": { "editable": true }, "source": [ "with $h_1(x)$ ensuring that $g_t(x)$ satisfies some conditions and $h_2(x,N(x, P))$ an expression involving $x$ and the output from the neural network $N(x,P)$ with $P $ being the collection of the weights and biases for each layer. For now, it is assumed that the network consists of one input layer, one hidden layer, and one output layer." ] }, { "cell_type": "markdown", "id": "801219a0", "metadata": { "editable": true }, "source": [ "## Setup of Network\n", "\n", "In this network, there are no weights and bias at the input layer, so $P = \\{ P_{\\text{hidden}}, P_{\\text{output}} \\}$.\n", "If there are $N_{\\text{hidden} }$ neurons in the hidden layer, then $P_{\\text{hidden}}$ is a $N_{\\text{hidden} } \\times (1 + N_{\\text{input}})$ matrix, given that there are $N_{\\text{input}}$ neurons in the input layer.\n", "\n", "The first column in $P_{\\text{hidden} }$ represents the bias for each neuron in the hidden layer and the second column represents the weights for each neuron in the hidden layer from the input layer.\n", "If there are $N_{\\text{output} }$ neurons in the output layer, then $P_{\\text{output}} $ is a $N_{\\text{output} } \\times (1 + N_{\\text{hidden} })$ matrix.\n", "\n", "Its first column represents the bias of each neuron and the remaining columns represents the weights to each neuron.\n", "\n", "It is given that $g(0) = g_0$. The trial solution must fulfill this condition to be a proper solution of ([6](#solveode)). A possible way to ensure that $g_t(0, P) = g_0$, is to let $F(N(x,P)) = x \\cdot N(x,P)$ and $A(x) = g_0$. This gives the following trial solution:" ] }, { "cell_type": "markdown", "id": "569486c4", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{trial} \\tag{7}\n", "g_t(x, P) = g_0 + x \\cdot N(x, P)\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "575831d8", "metadata": { "editable": true }, "source": [ "## Reformulating the problem\n", "\n", "We wish that our neural network manages to minimize a given cost function.\n", "\n", "A reformulation of out equation, ([6](#solveode)), must therefore be done,\n", "such that it describes the problem a neural network can solve for.\n", "\n", "The neural network must find the set of weights and biases $P$ such that the trial solution in ([7](#trial)) satisfies ([6](#solveode)).\n", "\n", "The trial solution" ] }, { "cell_type": "markdown", "id": "f8682123", "metadata": { "editable": true }, "source": [ "$$\n", "g_t(x, P) = g_0 + x \\cdot N(x, P)\n", "$$" ] }, { "cell_type": "markdown", "id": "e62b1005", "metadata": { "editable": true }, "source": [ "has been chosen such that it already solves the condition $g(0) = g_0$. What remains, is to find $P$ such that" ] }, { "cell_type": "markdown", "id": "4e33439d", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{nnmin} \\tag{8}\n", "g_t'(x, P) = - \\gamma g_t(x, P)\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "35159fc5", "metadata": { "editable": true }, "source": [ "is fulfilled as *best as possible*." ] }, { "cell_type": "markdown", "id": "03f74ead", "metadata": { "editable": true }, "source": [ "## More technicalities\n", "\n", "The left hand side and right hand side of ([8](#nnmin)) must be computed separately, and then the neural network must choose weights and biases, contained in $P$, such that the sides are equal as best as possible.\n", "This means that the absolute or squared difference between the sides must be as close to zero, ideally equal to zero.\n", "In this case, the difference squared shows to be an appropriate measurement of how erroneous the trial solution is with respect to $P$ of the neural network.\n", "\n", "This gives the following cost function our neural network must solve for:" ] }, { "cell_type": "markdown", "id": "3599c2de", "metadata": { "editable": true }, "source": [ "$$\n", "\\min_{P}\\Big\\{ \\big(g_t'(x, P) - ( -\\gamma g_t(x, P) \\big)^2 \\Big\\}\n", "$$" ] }, { "cell_type": "markdown", "id": "5c50baf7", "metadata": { "editable": true }, "source": [ "(the notation $\\min_{P}\\{ f(x, P) \\}$ means that we desire to find $P$ that yields the minimum of $f(x, P)$)\n", "\n", "or, in terms of weights and biases for the hidden and output layer in our network:" ] }, { "cell_type": "markdown", "id": "636cb74c", "metadata": { "editable": true }, "source": [ "$$\n", "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) \\big)^2 \\Big\\}\n", "$$" ] }, { "cell_type": "markdown", "id": "b4b301c2", "metadata": { "editable": true }, "source": [ "for an input value $x$." ] }, { "cell_type": "markdown", "id": "5d873576", "metadata": { "editable": true }, "source": [ "## More details\n", "\n", "If the neural network evaluates $g_t(x, P)$ at more values for $x$, say $N$ values $x_i$ for $i = 1, \\dots, N$, then the *total* error to minimize becomes" ] }, { "cell_type": "markdown", "id": "b744fd57", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{min} \\tag{9}\n", "\\min_{P}\\Big\\{\\frac{1}{N} \\sum_{i=1}^N \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2 \\Big\\}\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "63290336", "metadata": { "editable": true }, "source": [ "Letting $\\boldsymbol{x}$ be a vector with elements $x_i$ and $C(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes" ] }, { "cell_type": "markdown", "id": "f8790d32", "metadata": { "editable": true }, "source": [ "$$\n", "\\min_{P} C(\\boldsymbol{x}, P)\n", "$$" ] }, { "cell_type": "markdown", "id": "10ab609a", "metadata": { "editable": true }, "source": [ "In terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$, this could also be expressed as\n", "\n", "$$\n", "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }} C(\\boldsymbol{x}, \\{P_{\\text{hidden} }, P_{\\text{output} }\\})\n", "$$" ] }, { "cell_type": "markdown", "id": "06e2efca", "metadata": { "editable": true }, "source": [ "## A possible implementation of a neural network\n", "\n", "For simplicity, it is assumed that the input is an array $\\boldsymbol{x} = (x_1, \\dots, x_N)$ with $N$ elements. It is at these points the neural network should find $P$ such that it fulfills ([9](#min)).\n", "\n", "First, the neural network must feed forward the inputs.\n", "This means that $\\boldsymbol{x}s$ must be passed through an input layer, a hidden layer and a output layer. The input layer in this case, does not need to process the data any further.\n", "The input layer will consist of $N_{\\text{input} }$ neurons, passing its element to each neuron in the hidden layer. The number of neurons in the hidden layer will be $N_{\\text{hidden} }$." ] }, { "cell_type": "markdown", "id": "bc8a8e04", "metadata": { "editable": true }, "source": [ "## Technicalities\n", "\n", "For the $i$-th in the hidden layer with weight $w_i^{\\text{hidden} }$ and bias $b_i^{\\text{hidden} }$, the weighting from the $j$-th neuron at the input layer is:" ] }, { "cell_type": "markdown", "id": "6564e278", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "z_{i,j}^{\\text{hidden}} &= b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_j \\\\\n", "&=\n", "\\begin{pmatrix}\n", "b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 \\\\\n", "x_j\n", "\\end{pmatrix}\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "f62120f5", "metadata": { "editable": true }, "source": [ "## Final technicalities I\n", "\n", "The result after weighting the inputs at the $i$-th hidden neuron can be written as a vector:" ] }, { "cell_type": "markdown", "id": "4f875005", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "\\boldsymbol{z}_{i}^{\\text{hidden}} &= \\Big( b_i^{\\text{hidden}} + w_i^{\\text{hidden}}x_1 , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_2, \\ \\dots \\, , \\ b_i^{\\text{hidden}} + w_i^{\\text{hidden}} x_N\\Big) \\\\\n", "&=\n", "\\begin{pmatrix}\n", " b_i^{\\text{hidden}} & w_i^{\\text{hidden}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 & 1 & \\dots & 1 \\\\\n", "x_1 & x_2 & \\dots & x_N\n", "\\end{pmatrix} \\\\\n", "&= \\boldsymbol{p}_{i, \\text{hidden}}^T X\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "8748bf88", "metadata": { "editable": true }, "source": [ "## Final technicalities II\n", "\n", "The vector $\\boldsymbol{p}_{i, \\text{hidden}}^T$ constitutes each row in $P_{\\text{hidden} }$, which contains the weights for the neural network to minimize according to ([9](#min)).\n", "\n", "After having found $\\boldsymbol{z}_{i}^{\\text{hidden}} $ for every $i$-th neuron within the hidden layer, the vector will be sent to an activation function $a_i(\\boldsymbol{z})$.\n", "\n", "In this example, the sigmoid function has been chosen to be the activation function for each hidden neuron:" ] }, { "cell_type": "markdown", "id": "ce8b1154", "metadata": { "editable": true }, "source": [ "$$\n", "f(z) = \\frac{1}{1 + \\exp{(-z)}}\n", "$$" ] }, { "cell_type": "markdown", "id": "f6ae91fa", "metadata": { "editable": true }, "source": [ "It is possible to use other activations functions for the hidden layer also.\n", "\n", "The output $\\boldsymbol{x}_i^{\\text{hidden}}$ from each $i$-th hidden neuron is:\n", "\n", "$$\n", "\\boldsymbol{x}_i^{\\text{hidden} } = f\\big( \\boldsymbol{z}_{i}^{\\text{hidden}} \\big)\n", "$$\n", "\n", "The outputs $\\boldsymbol{x}_i^{\\text{hidden} } $ are then sent to the output layer.\n", "\n", "The output layer consists of one neuron in this case, and combines the\n", "output from each of the neurons in the hidden layers. The output layer\n", "combines the results from the hidden layer using some weights $w_i^{\\text{output}}$\n", "and biases $b_i^{\\text{output}}$. In this case,\n", "it is assumes that the number of neurons in the output layer is one." ] }, { "cell_type": "markdown", "id": "f33f3a2e", "metadata": { "editable": true }, "source": [ "## Final technicalities III\n", "\n", "The procedure of weighting the output neuron $j$ in the hidden layer to the $i$-th neuron in the output layer is similar as for the hidden layer described previously." ] }, { "cell_type": "markdown", "id": "d90fb703", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "z_{1,j}^{\\text{output}} & =\n", "\\begin{pmatrix}\n", "b_1^{\\text{output}} & \\boldsymbol{w}_1^{\\text{output}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 \\\\\n", "\\boldsymbol{x}_j^{\\text{hidden}}\n", "\\end{pmatrix}\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "0e4df2e9", "metadata": { "editable": true }, "source": [ "## Final technicalities IV\n", "\n", "Expressing $z_{1,j}^{\\text{output}}$ as a vector gives the following way of weighting the inputs from the hidden layer:" ] }, { "cell_type": "markdown", "id": "85cc6666", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{z}_{1}^{\\text{output}} =\n", "\\begin{pmatrix}\n", "b_1^{\\text{output}} & \\boldsymbol{w}_1^{\\text{output}}\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "1 & 1 & \\dots & 1 \\\\\n", "\\boldsymbol{x}_1^{\\text{hidden}} & \\boldsymbol{x}_2^{\\text{hidden}} & \\dots & \\boldsymbol{x}_N^{\\text{hidden}}\n", "\\end{pmatrix}\n", "$$" ] }, { "cell_type": "markdown", "id": "c4876b89", "metadata": { "editable": true }, "source": [ "In this case we seek a continuous range of values since we are approximating a function. This means that after computing $\\boldsymbol{z}_{1}^{\\text{output}}$ the neural network has finished its feed forward step, and $\\boldsymbol{z}_{1}^{\\text{output}}$ is the final output of the network." ] }, { "cell_type": "markdown", "id": "7e2cf00e", "metadata": { "editable": true }, "source": [ "## Back propagation\n", "\n", "The next step is to decide how the parameters should be changed such that they minimize the cost function.\n", "\n", "The chosen cost function for this problem is" ] }, { "cell_type": "markdown", "id": "a49b485d", "metadata": { "editable": true }, "source": [ "$$\n", "C(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", "$$" ] }, { "cell_type": "markdown", "id": "7c5a4275", "metadata": { "editable": true }, "source": [ "In order to minimize the cost function, an optimization method must be chosen.\n", "\n", "Here, gradient descent with a constant step size has been chosen." ] }, { "cell_type": "markdown", "id": "ee92110b", "metadata": { "editable": true }, "source": [ "## Gradient descent\n", "\n", "The idea of the gradient descent algorithm is to update parameters in\n", "a direction where the cost function decreases goes to a minimum.\n", "\n", "In general, the update of some parameters $\\boldsymbol{\\omega}$ given a cost\n", "function defined by some weights $\\boldsymbol{\\omega}$, $C(\\boldsymbol{x},\n", "\\boldsymbol{\\omega})$, goes as follows:" ] }, { "cell_type": "markdown", "id": "c76e461e", "metadata": { "editable": true }, "source": [ "$$\n", "\\boldsymbol{\\omega}_{\\text{new} } = \\boldsymbol{\\omega} - \\lambda \\nabla_{\\boldsymbol{\\omega}} C(\\boldsymbol{x}, \\boldsymbol{\\omega})\n", "$$" ] }, { "cell_type": "markdown", "id": "df9bc42a", "metadata": { "editable": true }, "source": [ "for a number of iterations or until $ \\big|\\big| \\boldsymbol{\\omega}_{\\text{new} } - \\boldsymbol{\\omega} \\big|\\big|$ becomes smaller than some given tolerance.\n", "\n", "The value of $\\lambda$ decides how large steps the algorithm must take\n", "in the direction of $ \\nabla_{\\boldsymbol{\\omega}} C(\\boldsymbol{x}, \\boldsymbol{\\omega})$.\n", "The notation $\\nabla_{\\boldsymbol{\\omega}}$ express the gradient with respect\n", "to the elements in $\\boldsymbol{\\omega}$.\n", "\n", "In our case, we have to minimize the cost function $C(\\boldsymbol{x}, P)$ with\n", "respect to the two sets of weights and biases, that is for the hidden\n", "layer $P_{\\text{hidden} }$ and for the output layer $P_{\\text{output}\n", "}$ .\n", "\n", "This means that $P_{\\text{hidden} }$ and $P_{\\text{output} }$ is updated by" ] }, { "cell_type": "markdown", "id": "418dcc35", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "P_{\\text{hidden},\\text{new}} &= P_{\\text{hidden}} - \\lambda \\nabla_{P_{\\text{hidden}}} C(\\boldsymbol{x}, P) \\\\\n", "P_{\\text{output},\\text{new}} &= P_{\\text{output}} - \\lambda \\nabla_{P_{\\text{output}}} C(\\boldsymbol{x}, P)\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "145ede1b", "metadata": { "editable": true }, "source": [ "## The code for solving the ODE" ] }, { "cell_type": "code", "execution_count": 43, "id": "98b44c55", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad, elementwise_grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import pyplot as plt\n", "\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "# Assuming one input, hidden, and output layer\n", "def neural_network(params, x):\n", "\n", " # Find the weights (including and biases) for the hidden and output layer.\n", " # Assume that params is a list of parameters for each layer.\n", " # The biases are the first element for each array in params,\n", " # and the weights are the remaning elements in each array in params.\n", "\n", " w_hidden = params[0]\n", " w_output = params[1]\n", "\n", " # Assumes input x being an one-dimensional array\n", " num_values = np.size(x)\n", " x = x.reshape(-1, num_values)\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", "\n", " ## Hidden layer:\n", "\n", " # Add a row of ones to include bias\n", " x_input = np.concatenate((np.ones((1,num_values)), x_input ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_input)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " ## Output layer:\n", "\n", " # Include bias:\n", " x_hidden = np.concatenate((np.ones((1,num_values)), x_hidden ), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_hidden)\n", " x_output = z_output\n", "\n", " return x_output\n", "\n", "# The trial solution using the deep neural network:\n", "def g_trial(x,params, g0 = 10):\n", " return g0 + x*neural_network(params,x)\n", "\n", "# The right side of the ODE:\n", "def g(x, g_trial, gamma = 2):\n", " return -gamma*g_trial\n", "\n", "# The cost function:\n", "def cost_function(P, x):\n", "\n", " # Evaluate the trial function with the current parameters P\n", " g_t = g_trial(x,P)\n", "\n", " # Find the derivative w.r.t x of the neural network\n", " d_net_out = elementwise_grad(neural_network,1)(P,x)\n", "\n", " # Find the derivative w.r.t x of the trial function\n", " d_g_t = elementwise_grad(g_trial,0)(x,P)\n", "\n", " # The right side of the ODE\n", " func = g(x, g_t)\n", "\n", " err_sqr = (d_g_t - func)**2\n", " cost_sum = np.sum(err_sqr)\n", "\n", " return cost_sum / np.size(err_sqr)\n", "\n", "# Solve the exponential decay ODE using neural network with one input, hidden, and output layer\n", "def solve_ode_neural_network(x, num_neurons_hidden, num_iter, lmb):\n", " ## Set up initial weights and biases\n", "\n", " # For the hidden layer\n", " p0 = npr.randn(num_neurons_hidden, 2 )\n", "\n", " # For the output layer\n", " p1 = npr.randn(1, num_neurons_hidden + 1 ) # +1 since bias is included\n", "\n", " P = [p0, p1]\n", "\n", " print('Initial cost: %g'%cost_function(P, x))\n", "\n", " ## Start finding the optimal weights using gradient descent\n", "\n", " # Find the Python function that represents the gradient of the cost function\n", " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", " cost_function_grad = grad(cost_function,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " # Evaluate the gradient at the current weights and biases in P.\n", " # The cost_grad consist now of two arrays;\n", " # one for the gradient w.r.t P_hidden and\n", " # one for the gradient w.r.t P_output\n", " cost_grad = cost_function_grad(P, x)\n", "\n", " P[0] = P[0] - lmb * cost_grad[0]\n", " P[1] = P[1] - lmb * cost_grad[1]\n", "\n", " print('Final cost: %g'%cost_function(P, x))\n", "\n", " return P\n", "\n", "def g_analytic(x, gamma = 2, g0 = 10):\n", " return g0*np.exp(-gamma*x)\n", "\n", "# Solve the given problem\n", "if __name__ == '__main__':\n", " # Set seed such that the weight are initialized\n", " # with same weights and biases for every run.\n", " npr.seed(15)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " N = 10\n", " x = np.linspace(0, 1, N)\n", "\n", " ## Set up the initial parameters\n", " num_hidden_neurons = 10\n", " num_iter = 10000\n", " lmb = 0.001\n", "\n", " # Use the network\n", " P = solve_ode_neural_network(x, num_hidden_neurons, num_iter, lmb)\n", "\n", " # Print the deviation from the trial solution and true solution\n", " res = g_trial(x,P)\n", " res_analytical = g_analytic(x)\n", "\n", " print('Max absolute difference: %g'%np.max(np.abs(res - res_analytical)))\n", "\n", " # Plot the results\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.title('Performance of neural network solving an ODE compared to the analytical solution')\n", " plt.plot(x, res_analytical)\n", " plt.plot(x, res[0,:])\n", " plt.legend(['analytical','nn'])\n", " plt.xlabel('x')\n", " plt.ylabel('g(x)')\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "47df9d6f", "metadata": { "editable": true }, "source": [ "## The network with one input layer, specified number of hidden layers, and one output layer\n", "\n", "It is also possible to extend the construction of our network into a more general one, allowing the network to contain more than one hidden layers.\n", "\n", "The number of neurons within each hidden layer are given as a list of integers in the program below." ] }, { "cell_type": "code", "execution_count": 44, "id": "1415bdf5", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad, elementwise_grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import pyplot as plt\n", "\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "# The neural network with one input layer and one output layer,\n", "# but with number of hidden layers specified by the user.\n", "def deep_neural_network(deep_params, x):\n", " # N_hidden is the number of hidden layers\n", "\n", " N_hidden = np.size(deep_params) - 1 # -1 since params consists of\n", " # parameters to all the hidden\n", " # layers AND the output layer.\n", "\n", " # Assumes input x being an one-dimensional array\n", " num_values = np.size(x)\n", " x = x.reshape(-1, num_values)\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", "\n", " # Due to multiple hidden layers, define a variable referencing to the\n", " # output of the previous layer:\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = deep_params[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = deep_params[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output\n", "\n", "# The trial solution using the deep neural network:\n", "def g_trial_deep(x,params, g0 = 10):\n", " return g0 + x*deep_neural_network(params, x)\n", "\n", "# The right side of the ODE:\n", "def g(x, g_trial, gamma = 2):\n", " return -gamma*g_trial\n", "\n", "# The same cost function as before, but calls deep_neural_network instead.\n", "def cost_function_deep(P, x):\n", "\n", " # Evaluate the trial function with the current parameters P\n", " g_t = g_trial_deep(x,P)\n", "\n", " # Find the derivative w.r.t x of the neural network\n", " d_net_out = elementwise_grad(deep_neural_network,1)(P,x)\n", "\n", " # Find the derivative w.r.t x of the trial function\n", " d_g_t = elementwise_grad(g_trial_deep,0)(x,P)\n", "\n", " # The right side of the ODE\n", " func = g(x, g_t)\n", "\n", " err_sqr = (d_g_t - func)**2\n", " cost_sum = np.sum(err_sqr)\n", "\n", " return cost_sum / np.size(err_sqr)\n", "\n", "# Solve the exponential decay ODE using neural network with one input and one output layer,\n", "# but with specified number of hidden layers from the user.\n", "def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):\n", " # num_hidden_neurons is now a list of number of neurons within each hidden layer\n", "\n", " # The number of elements in the list num_hidden_neurons thus represents\n", " # the number of hidden layers.\n", "\n", " # Find the number of hidden layers:\n", " N_hidden = np.size(num_neurons)\n", "\n", " ## Set up initial weights and biases\n", "\n", " # Initialize the list of parameters:\n", " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", "\n", " P[0] = npr.randn(num_neurons[0], 2 )\n", " for l in range(1,N_hidden):\n", " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias\n", "\n", " # For the output layer\n", " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", "\n", " print('Initial cost: %g'%cost_function_deep(P, x))\n", "\n", " ## Start finding the optimal weights using gradient descent\n", "\n", " # Find the Python function that represents the gradient of the cost function\n", " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", " cost_function_deep_grad = grad(cost_function_deep,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " # Evaluate the gradient at the current weights and biases in P.\n", " # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases\n", " # in the hidden layers and output layers evaluated at x.\n", " cost_deep_grad = cost_function_deep_grad(P, x)\n", "\n", " for l in range(N_hidden+1):\n", " P[l] = P[l] - lmb * cost_deep_grad[l]\n", "\n", " print('Final cost: %g'%cost_function_deep(P, x))\n", "\n", " return P\n", "\n", "def g_analytic(x, gamma = 2, g0 = 10):\n", " return g0*np.exp(-gamma*x)\n", "\n", "# Solve the given problem\n", "if __name__ == '__main__':\n", " npr.seed(15)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " N = 10\n", " x = np.linspace(0, 1, N)\n", "\n", " ## Set up the initial parameters\n", " num_hidden_neurons = np.array([10,10])\n", " num_iter = 10000\n", " lmb = 0.001\n", "\n", " P = solve_ode_deep_neural_network(x, num_hidden_neurons, num_iter, lmb)\n", "\n", " res = g_trial_deep(x,P)\n", " res_analytical = g_analytic(x)\n", "\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.title('Performance of a deep neural network solving an ODE compared to the analytical solution')\n", " plt.plot(x, res_analytical)\n", " plt.plot(x, res[0,:])\n", " plt.legend(['analytical','dnn'])\n", " plt.ylabel('g(x)')\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "b480dee1", "metadata": { "editable": true }, "source": [ "## Example: Population growth\n", "\n", "A logistic model of population growth assumes that a population converges toward an equilibrium.\n", "The population growth can be modeled by" ] }, { "cell_type": "markdown", "id": "3736290c", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{log} \\tag{10}\n", "\tg'(t) = \\alpha g(t)(A - g(t))\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "9fe04f20", "metadata": { "editable": true }, "source": [ "where $g(t)$ is the population density at time $t$, $\\alpha > 0$ the growth rate and $A > 0$ is the maximum population number in the environment.\n", "Also, at $t = 0$ the population has the size $g(0) = g_0$, where $g_0$ is some chosen constant.\n", "\n", "In this example, similar network as for the exponential decay using Autograd has been used to solve the equation. However, as the implementation might suffer from e.g numerical instability\n", "and high execution time (this might be more apparent in the examples solving PDEs),\n", "using a library like TensorFlow is recommended.\n", "Here, we stay with a more simple approach and implement for comparison, the simple forward Euler method." ] }, { "cell_type": "markdown", "id": "8090a04f", "metadata": { "editable": true }, "source": [ "## Setting up the problem\n", "\n", "Here, we will model a population $g(t)$ in an environment having carrying capacity $A$.\n", "The population follows the model" ] }, { "cell_type": "markdown", "id": "22a5d518", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{solveode_population} \\tag{11}\n", "g'(t) = \\alpha g(t)(A - g(t))\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "24312795", "metadata": { "editable": true }, "source": [ "where $g(0) = g_0$.\n", "\n", "In this example, we let $\\alpha = 2$, $A = 1$, and $g_0 = 1.2$." ] }, { "cell_type": "markdown", "id": "c90e2f79", "metadata": { "editable": true }, "source": [ "## The trial solution\n", "\n", "We will get a slightly different trial solution, as the boundary conditions are different\n", "compared to the case for exponential decay.\n", "\n", "A possible trial solution satisfying the condition $g(0) = g_0$ could be\n", "\n", "$$\n", "h_1(t) = g_0 + t \\cdot N(t,P)\n", "$$\n", "\n", "with $N(t,P)$ being the output from the neural network with weights and biases for each layer collected in the set $P$.\n", "\n", "The analytical solution is\n", "\n", "$$\n", "g(t) = \\frac{Ag_0}{g_0 + (A - g_0)\\exp(-\\alpha A t)}\n", "$$" ] }, { "cell_type": "markdown", "id": "112c3155", "metadata": { "editable": true }, "source": [ "## The program using Autograd\n", "\n", "The network will be the similar as for the exponential decay example, but with some small modifications for our problem." ] }, { "cell_type": "code", "execution_count": 45, "id": "73371f5c", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad, elementwise_grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import pyplot as plt\n", "\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "# Function to get the parameters.\n", "# Done such that one can easily change the paramaters after one's liking.\n", "def get_parameters():\n", " alpha = 2\n", " A = 1\n", " g0 = 1.2\n", " return alpha, A, g0\n", "\n", "def deep_neural_network(P, x):\n", " # N_hidden is the number of hidden layers\n", " N_hidden = np.size(P) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", "\n", " # Assumes input x being an one-dimensional array\n", " num_values = np.size(x)\n", " x = x.reshape(-1, num_values)\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", "\n", " # Due to multiple hidden layers, define a variable referencing to the\n", " # output of the previous layer:\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = P[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = P[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output\n", "\n", "\n", "def cost_function_deep(P, x):\n", "\n", " # Evaluate the trial function with the current parameters P\n", " g_t = g_trial_deep(x,P)\n", "\n", " # Find the derivative w.r.t x of the trial function\n", " d_g_t = elementwise_grad(g_trial_deep,0)(x,P)\n", "\n", " # The right side of the ODE\n", " func = f(x, g_t)\n", "\n", " err_sqr = (d_g_t - func)**2\n", " cost_sum = np.sum(err_sqr)\n", "\n", " return cost_sum / np.size(err_sqr)\n", "\n", "# The right side of the ODE:\n", "def f(x, g_trial):\n", " alpha,A, g0 = get_parameters()\n", " return alpha*g_trial*(A - g_trial)\n", "\n", "# The trial solution using the deep neural network:\n", "def g_trial_deep(x, params):\n", " alpha,A, g0 = get_parameters()\n", " return g0 + x*deep_neural_network(params,x)\n", "\n", "# The analytical solution:\n", "def g_analytic(t):\n", " alpha,A, g0 = get_parameters()\n", " return A*g0/(g0 + (A - g0)*np.exp(-alpha*A*t))\n", "\n", "def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):\n", " # num_hidden_neurons is now a list of number of neurons within each hidden layer\n", "\n", " # Find the number of hidden layers:\n", " N_hidden = np.size(num_neurons)\n", "\n", " ## Set up initial weigths and biases\n", "\n", " # Initialize the list of parameters:\n", " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", "\n", " P[0] = npr.randn(num_neurons[0], 2 )\n", " for l in range(1,N_hidden):\n", " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias\n", "\n", " # For the output layer\n", " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", "\n", " print('Initial cost: %g'%cost_function_deep(P, x))\n", "\n", " ## Start finding the optimal weigths using gradient descent\n", "\n", " # Find the Python function that represents the gradient of the cost function\n", " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", " cost_function_deep_grad = grad(cost_function_deep,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " # Evaluate the gradient at the current weights and biases in P.\n", " # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases\n", " # in the hidden layers and output layers evaluated at x.\n", " cost_deep_grad = cost_function_deep_grad(P, x)\n", "\n", " for l in range(N_hidden+1):\n", " P[l] = P[l] - lmb * cost_deep_grad[l]\n", "\n", " print('Final cost: %g'%cost_function_deep(P, x))\n", "\n", " return P\n", "\n", "if __name__ == '__main__':\n", " npr.seed(4155)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " Nt = 10\n", " T = 1\n", " t = np.linspace(0,T, Nt)\n", "\n", " ## Set up the initial parameters\n", " num_hidden_neurons = [100, 50, 25]\n", " num_iter = 1000\n", " lmb = 1e-3\n", "\n", " P = solve_ode_deep_neural_network(t, num_hidden_neurons, num_iter, lmb)\n", "\n", " g_dnn_ag = g_trial_deep(t,P)\n", " g_analytical = g_analytic(t)\n", "\n", " # Find the maximum absolute difference between the solutons:\n", " diff_ag = np.max(np.abs(g_dnn_ag - g_analytical))\n", " print(\"The max absolute difference between the solutions is: %g\"%diff_ag)\n", "\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.title('Performance of neural network solving an ODE compared to the analytical solution')\n", " plt.plot(t, g_analytical)\n", " plt.plot(t, g_dnn_ag[0,:])\n", " plt.legend(['analytical','nn'])\n", " plt.xlabel('t')\n", " plt.ylabel('g(t)')\n", "\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "9ce89121", "metadata": { "editable": true }, "source": [ "## Using forward Euler to solve the ODE\n", "\n", "A straightforward way of solving an ODE numerically, is to use Euler's method.\n", "\n", "Euler's method uses Taylor series to approximate the value at a function $f$ at a step $\\Delta x$ from $x$:\n", "\n", "$$\n", "f(x + \\Delta x) \\approx f(x) + \\Delta x f'(x)\n", "$$\n", "\n", "In our case, using Euler's method to approximate the value of $g$ at a step $\\Delta t$ from $t$ yields" ] }, { "cell_type": "markdown", "id": "dfe02266", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", " g(t + \\Delta t) &\\approx g(t) + \\Delta t g'(t) \\\\\n", " &= g(t) + \\Delta t \\big(\\alpha g(t)(A - g(t))\\big)\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "f2f12c48", "metadata": { "editable": true }, "source": [ "along with the condition that $g(0) = g_0$.\n", "\n", "Let $t_i = i \\cdot \\Delta t$ where $\\Delta t = \\frac{T}{N_t-1}$ where $T$ is the final time our solver must solve for and $N_t$ the number of values for $t \\in [0, T]$ for $i = 0, \\dots, N_t-1$.\n", "\n", "For $i \\geq 1$, we have that" ] }, { "cell_type": "markdown", "id": "ec46581e", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "t_i &= i\\Delta t \\\\\n", "&= (i - 1)\\Delta t + \\Delta t \\\\\n", "&= t_{i-1} + \\Delta t\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "02c7198e", "metadata": { "editable": true }, "source": [ "Now, if $g_i = g(t_i)$ then" ] }, { "cell_type": "markdown", "id": "90c2533d", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation}\n", " \\begin{aligned}\n", " g_i &= g(t_i) \\\\\n", " &= g(t_{i-1} + \\Delta t) \\\\\n", " &\\approx g(t_{i-1}) + \\Delta t \\big(\\alpha g(t_{i-1})(A - g(t_{i-1}))\\big) \\\\\n", " &= g_{i-1} + \\Delta t \\big(\\alpha g_{i-1}(A - g_{i-1})\\big)\n", " \\end{aligned}\n", "\\end{equation} \\label{odenum} \\tag{12}\n", "$$" ] }, { "cell_type": "markdown", "id": "f3dddcdf", "metadata": { "editable": true }, "source": [ "for $i \\geq 1$ and $g_0 = g(t_0) = g(0) = g_0$.\n", "\n", "Equation ([12](#odenum)) could be implemented in the following way,\n", "extending the program that uses the network using Autograd:" ] }, { "cell_type": "code", "execution_count": 46, "id": "52ee1308", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# Assume that all function definitions from the example program using Autograd\n", "# are located here.\n", "\n", "if __name__ == '__main__':\n", " npr.seed(4155)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " Nt = 10\n", " T = 1\n", " t = np.linspace(0,T, Nt)\n", "\n", " ## Set up the initial parameters\n", " num_hidden_neurons = [100,50,25]\n", " num_iter = 1000\n", " lmb = 1e-3\n", "\n", " P = solve_ode_deep_neural_network(t, num_hidden_neurons, num_iter, lmb)\n", "\n", " g_dnn_ag = g_trial_deep(t,P)\n", " g_analytical = g_analytic(t)\n", "\n", " # Find the maximum absolute difference between the solutons:\n", " diff_ag = np.max(np.abs(g_dnn_ag - g_analytical))\n", " print(\"The max absolute difference between the solutions is: %g\"%diff_ag)\n", "\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.title('Performance of neural network solving an ODE compared to the analytical solution')\n", " plt.plot(t, g_analytical)\n", " plt.plot(t, g_dnn_ag[0,:])\n", " plt.legend(['analytical','nn'])\n", " plt.xlabel('t')\n", " plt.ylabel('g(t)')\n", "\n", " ## Find an approximation to the funtion using forward Euler\n", "\n", " alpha, A, g0 = get_parameters()\n", " dt = T/(Nt - 1)\n", "\n", " # Perform forward Euler to solve the ODE\n", " g_euler = np.zeros(Nt)\n", " g_euler[0] = g0\n", "\n", " for i in range(1,Nt):\n", " g_euler[i] = g_euler[i-1] + dt*(alpha*g_euler[i-1]*(A - g_euler[i-1]))\n", "\n", " # Print the errors done by each method\n", " diff1 = np.max(np.abs(g_euler - g_analytical))\n", " diff2 = np.max(np.abs(g_dnn_ag[0,:] - g_analytical))\n", "\n", " print('Max absolute difference between Euler method and analytical: %g'%diff1)\n", " print('Max absolute difference between deep neural network and analytical: %g'%diff2)\n", "\n", " # Plot results\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.plot(t,g_euler)\n", " plt.plot(t,g_analytical)\n", " plt.plot(t,g_dnn_ag[0,:])\n", "\n", " plt.legend(['euler','analytical','dnn'])\n", " plt.xlabel('Time t')\n", " plt.ylabel('g(t)')\n", "\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "6537fc0c", "metadata": { "editable": true }, "source": [ "## Example: Solving the one dimensional Poisson equation\n", "\n", "The Poisson equation for $g(x)$ in one dimension is" ] }, { "cell_type": "markdown", "id": "bf0ace9e", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{poisson} \\tag{13}\n", " -g''(x) = f(x)\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "212fac99", "metadata": { "editable": true }, "source": [ "where $f(x)$ is a given function for $x \\in (0,1)$.\n", "\n", "The conditions that $g(x)$ is chosen to fulfill, are" ] }, { "cell_type": "markdown", "id": "fa2547d7", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", " g(0) &= 0 \\\\\n", " g(1) &= 0\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "id": "4d3aa4c3", "metadata": { "editable": true }, "source": [ "This equation can be solved numerically using programs where e.g Autograd and TensorFlow are used.\n", "The results from the networks can then be compared to the analytical solution.\n", "In addition, it could be interesting to see how a typical method for numerically solving second order ODEs compares to the neural networks." ] }, { "cell_type": "markdown", "id": "872d2747", "metadata": { "editable": true }, "source": [ "## The specific equation to solve for\n", "\n", "Here, the function $g(x)$ to solve for follows the equation" ] }, { "cell_type": "markdown", "id": "b51b983e", "metadata": { "editable": true }, "source": [ "$$\n", "-g''(x) = f(x),\\qquad x \\in (0,1)\n", "$$" ] }, { "cell_type": "markdown", "id": "f208cc09", "metadata": { "editable": true }, "source": [ "where $f(x)$ is a given function, along with the chosen conditions" ] }, { "cell_type": "markdown", "id": "d584ffd3", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{aligned}\n", "g(0) = g(1) = 0\n", "\\end{aligned}\\label{cond} \\tag{14}\n", "$$" ] }, { "cell_type": "markdown", "id": "99794ce5", "metadata": { "editable": true }, "source": [ "In this example, we consider the case when $f(x) = (3x + x^2)\\exp(x)$.\n", "\n", "For this case, a possible trial solution satisfying the conditions could be" ] }, { "cell_type": "markdown", "id": "3f2f40d9", "metadata": { "editable": true }, "source": [ "$$\n", "g_t(x) = x \\cdot (1-x) \\cdot N(P,x)\n", "$$" ] }, { "cell_type": "markdown", "id": "d5590dfb", "metadata": { "editable": true }, "source": [ "The analytical solution for this problem is" ] }, { "cell_type": "markdown", "id": "75ab0f4f", "metadata": { "editable": true }, "source": [ "$$\n", "g(x) = x(1 - x)\\exp(x)\n", "$$" ] }, { "cell_type": "markdown", "id": "d83e4c37", "metadata": { "editable": true }, "source": [ "## Solving the equation using Autograd" ] }, { "cell_type": "code", "execution_count": 47, "id": "95731560", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad, elementwise_grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import pyplot as plt\n", "\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "def deep_neural_network(deep_params, x):\n", " # N_hidden is the number of hidden layers\n", " N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", "\n", " # Assumes input x being an one-dimensional array\n", " num_values = np.size(x)\n", " x = x.reshape(-1, num_values)\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", "\n", " # Due to multiple hidden layers, define a variable referencing to the\n", " # output of the previous layer:\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = deep_params[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = deep_params[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output\n", "\n", "def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):\n", " # num_hidden_neurons is now a list of number of neurons within each hidden layer\n", "\n", " # Find the number of hidden layers:\n", " N_hidden = np.size(num_neurons)\n", "\n", " ## Set up initial weigths and biases\n", "\n", " # Initialize the list of parameters:\n", " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", "\n", " P[0] = npr.randn(num_neurons[0], 2 )\n", " for l in range(1,N_hidden):\n", " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias\n", "\n", " # For the output layer\n", " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", "\n", " print('Initial cost: %g'%cost_function_deep(P, x))\n", "\n", " ## Start finding the optimal weigths using gradient descent\n", "\n", " # Find the Python function that represents the gradient of the cost function\n", " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", " cost_function_deep_grad = grad(cost_function_deep,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " # Evaluate the gradient at the current weights and biases in P.\n", " # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases\n", " # in the hidden layers and output layers evaluated at x.\n", " cost_deep_grad = cost_function_deep_grad(P, x)\n", "\n", " for l in range(N_hidden+1):\n", " P[l] = P[l] - lmb * cost_deep_grad[l]\n", "\n", " print('Final cost: %g'%cost_function_deep(P, x))\n", "\n", " return P\n", "\n", "## Set up the cost function specified for this Poisson equation:\n", "\n", "# The right side of the ODE\n", "def f(x):\n", " return (3*x + x**2)*np.exp(x)\n", "\n", "def cost_function_deep(P, x):\n", "\n", " # Evaluate the trial function with the current parameters P\n", " g_t = g_trial_deep(x,P)\n", "\n", " # Find the derivative w.r.t x of the trial function\n", " d2_g_t = elementwise_grad(elementwise_grad(g_trial_deep,0))(x,P)\n", "\n", " right_side = f(x)\n", "\n", " err_sqr = (-d2_g_t - right_side)**2\n", " cost_sum = np.sum(err_sqr)\n", "\n", " return cost_sum/np.size(err_sqr)\n", "\n", "# The trial solution:\n", "def g_trial_deep(x,P):\n", " return x*(1-x)*deep_neural_network(P,x)\n", "\n", "# The analytic solution;\n", "def g_analytic(x):\n", " return x*(1-x)*np.exp(x)\n", "\n", "if __name__ == '__main__':\n", " npr.seed(4155)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " Nx = 10\n", " x = np.linspace(0,1, Nx)\n", "\n", " ## Set up the initial parameters\n", " num_hidden_neurons = [200,100]\n", " num_iter = 1000\n", " lmb = 1e-3\n", "\n", " P = solve_ode_deep_neural_network(x, num_hidden_neurons, num_iter, lmb)\n", "\n", " g_dnn_ag = g_trial_deep(x,P)\n", " g_analytical = g_analytic(x)\n", "\n", " # Find the maximum absolute difference between the solutons:\n", " max_diff = np.max(np.abs(g_dnn_ag - g_analytical))\n", " print(\"The max absolute difference between the solutions is: %g\"%max_diff)\n", "\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.title('Performance of neural network solving an ODE compared to the analytical solution')\n", " plt.plot(x, g_analytical)\n", " plt.plot(x, g_dnn_ag[0,:])\n", " plt.legend(['analytical','nn'])\n", " plt.xlabel('x')\n", " plt.ylabel('g(x)')\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "559cb774", "metadata": { "editable": true }, "source": [ "## Comparing with a numerical scheme\n", "\n", "The Poisson equation is possible to solve using Taylor series to approximate the second derivative.\n", "\n", "Using Taylor series, the second derivative can be expressed as\n", "\n", "$$\n", "g''(x) = \\frac{g(x + \\Delta x) - 2g(x) + g(x-\\Delta x)}{\\Delta x^2} + E_{\\Delta x}(x)\n", "$$\n", "\n", "where $\\Delta x$ is a small step size and $E_{\\Delta x}(x)$ being the error term.\n", "\n", "Looking away from the error terms gives an approximation to the second derivative:" ] }, { "cell_type": "markdown", "id": "72b9c88d", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{approx} \\tag{15}\n", "g''(x) \\approx \\frac{g(x + \\Delta x) - 2g(x) + g(x-\\Delta x)}{\\Delta x^2}\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "1c7a99a5", "metadata": { "editable": true }, "source": [ "If $x_i = i \\Delta x = x_{i-1} + \\Delta x$ and $g_i = g(x_i)$ for $i = 1,\\dots N_x - 2$ with $N_x$ being the number of values for $x$, ([15](#approx)) becomes" ] }, { "cell_type": "markdown", "id": "b46248f9", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "g''(x_i) &\\approx \\frac{g(x_i + \\Delta x) - 2g(x_i) + g(x_i -\\Delta x)}{\\Delta x^2} \\\\\n", "&= \\frac{g_{i+1} - 2g_i + g_{i-1}}{\\Delta x^2}\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "fa1d6871", "metadata": { "editable": true }, "source": [ "Since we know from our problem that" ] }, { "cell_type": "markdown", "id": "caa3216a", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "-g''(x) &= f(x) \\\\\n", "&= (3x + x^2)\\exp(x)\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "936f3b89", "metadata": { "editable": true }, "source": [ "along with the conditions $g(0) = g(1) = 0$,\n", "the following scheme can be used to find an approximate solution for $g(x)$ numerically:" ] }, { "cell_type": "markdown", "id": "1e7fa7e3", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation}\n", " \\begin{aligned}\n", " -\\Big( \\frac{g_{i+1} - 2g_i + g_{i-1}}{\\Delta x^2} \\Big) &= f(x_i) \\\\\n", " -g_{i+1} + 2g_i - g_{i-1} &= \\Delta x^2 f(x_i)\n", " \\end{aligned}\n", "\\end{equation} \\label{odesys} \\tag{16}\n", "$$" ] }, { "cell_type": "markdown", "id": "9c46ad0b", "metadata": { "editable": true }, "source": [ "for $i = 1, \\dots, N_x - 2$ where $g_0 = g_{N_x - 1} = 0$ and $f(x_i) = (3x_i + x_i^2)\\exp(x_i)$, which is given for our specific problem.\n", "\n", "The equation can be rewritten into a matrix equation:" ] }, { "cell_type": "markdown", "id": "ccf16979", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{aligned}\n", "\\begin{pmatrix}\n", "2 & -1 & 0 & \\dots & 0 \\\\\n", "-1 & 2 & -1 & \\dots & 0 \\\\\n", "\\vdots & & \\ddots & & \\vdots \\\\\n", "0 & \\dots & -1 & 2 & -1 \\\\\n", "0 & \\dots & 0 & -1 & 2\\\\\n", "\\end{pmatrix}\n", "\\begin{pmatrix}\n", "g_1 \\\\\n", "g_2 \\\\\n", "\\vdots \\\\\n", "g_{N_x - 3} \\\\\n", "g_{N_x - 2}\n", "\\end{pmatrix}\n", "&=\n", "\\Delta x^2\n", "\\begin{pmatrix}\n", "f(x_1) \\\\\n", "f(x_2) \\\\\n", "\\vdots \\\\\n", "f(x_{N_x - 3}) \\\\\n", "f(x_{N_x - 2})\n", "\\end{pmatrix} \\\\\n", "\\boldsymbol{A}\\boldsymbol{g} &= \\boldsymbol{f},\n", "\\end{aligned}\n", "$$" ] }, { "cell_type": "markdown", "id": "e940bd60", "metadata": { "editable": true }, "source": [ "which makes it possible to solve for the vector $\\boldsymbol{g}$." ] }, { "cell_type": "markdown", "id": "ee51ae7f", "metadata": { "editable": true }, "source": [ "## Setting up the code\n", "\n", "We can then compare the result from this numerical scheme with the output from our network using Autograd:" ] }, { "cell_type": "code", "execution_count": 48, "id": "c538fed6", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad, elementwise_grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import pyplot as plt\n", "\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "def deep_neural_network(deep_params, x):\n", " # N_hidden is the number of hidden layers\n", " N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", "\n", " # Assumes input x being an one-dimensional array\n", " num_values = np.size(x)\n", " x = x.reshape(-1, num_values)\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", "\n", " # Due to multiple hidden layers, define a variable referencing to the\n", " # output of the previous layer:\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = deep_params[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = deep_params[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_values)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output\n", "\n", "def solve_ode_deep_neural_network(x, num_neurons, num_iter, lmb):\n", " # num_hidden_neurons is now a list of number of neurons within each hidden layer\n", "\n", " # Find the number of hidden layers:\n", " N_hidden = np.size(num_neurons)\n", "\n", " ## Set up initial weigths and biases\n", "\n", " # Initialize the list of parameters:\n", " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", "\n", " P[0] = npr.randn(num_neurons[0], 2 )\n", " for l in range(1,N_hidden):\n", " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias\n", "\n", " # For the output layer\n", " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", "\n", " print('Initial cost: %g'%cost_function_deep(P, x))\n", "\n", " ## Start finding the optimal weigths using gradient descent\n", "\n", " # Find the Python function that represents the gradient of the cost function\n", " # w.r.t the 0-th input argument -- that is the weights and biases in the hidden and output layer\n", " cost_function_deep_grad = grad(cost_function_deep,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " # Evaluate the gradient at the current weights and biases in P.\n", " # The cost_grad consist now of N_hidden + 1 arrays; the gradient w.r.t the weights and biases\n", " # in the hidden layers and output layers evaluated at x.\n", " cost_deep_grad = cost_function_deep_grad(P, x)\n", "\n", " for l in range(N_hidden+1):\n", " P[l] = P[l] - lmb * cost_deep_grad[l]\n", "\n", " print('Final cost: %g'%cost_function_deep(P, x))\n", "\n", " return P\n", "\n", "## Set up the cost function specified for this Poisson equation:\n", "\n", "# The right side of the ODE\n", "def f(x):\n", " return (3*x + x**2)*np.exp(x)\n", "\n", "def cost_function_deep(P, x):\n", "\n", " # Evaluate the trial function with the current parameters P\n", " g_t = g_trial_deep(x,P)\n", "\n", " # Find the derivative w.r.t x of the trial function\n", " d2_g_t = elementwise_grad(elementwise_grad(g_trial_deep,0))(x,P)\n", "\n", " right_side = f(x)\n", "\n", " err_sqr = (-d2_g_t - right_side)**2\n", " cost_sum = np.sum(err_sqr)\n", "\n", " return cost_sum/np.size(err_sqr)\n", "\n", "# The trial solution:\n", "def g_trial_deep(x,P):\n", " return x*(1-x)*deep_neural_network(P,x)\n", "\n", "# The analytic solution;\n", "def g_analytic(x):\n", " return x*(1-x)*np.exp(x)\n", "\n", "if __name__ == '__main__':\n", " npr.seed(4155)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " Nx = 10\n", " x = np.linspace(0,1, Nx)\n", "\n", " ## Set up the initial parameters\n", " num_hidden_neurons = [200,100]\n", " num_iter = 1000\n", " lmb = 1e-3\n", "\n", " P = solve_ode_deep_neural_network(x, num_hidden_neurons, num_iter, lmb)\n", "\n", " g_dnn_ag = g_trial_deep(x,P)\n", " g_analytical = g_analytic(x)\n", "\n", " # Find the maximum absolute difference between the solutons:\n", "\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.title('Performance of neural network solving an ODE compared to the analytical solution')\n", " plt.plot(x, g_analytical)\n", " plt.plot(x, g_dnn_ag[0,:])\n", " plt.legend(['analytical','nn'])\n", " plt.xlabel('x')\n", " plt.ylabel('g(x)')\n", "\n", " ## Perform the computation using the numerical scheme\n", "\n", " dx = 1/(Nx - 1)\n", "\n", " # Set up the matrix A\n", " A = np.zeros((Nx-2,Nx-2))\n", "\n", " A[0,0] = 2\n", " A[0,1] = -1\n", "\n", " for i in range(1,Nx-3):\n", " A[i,i-1] = -1\n", " A[i,i] = 2\n", " A[i,i+1] = -1\n", "\n", " A[Nx - 3, Nx - 4] = -1\n", " A[Nx - 3, Nx - 3] = 2\n", "\n", " # Set up the vector f\n", " f_vec = dx**2 * f(x[1:-1])\n", "\n", " # Solve the equation\n", " g_res = np.linalg.solve(A,f_vec)\n", "\n", " g_vec = np.zeros(Nx)\n", " g_vec[1:-1] = g_res\n", "\n", " # Print the differences between each method\n", " max_diff1 = np.max(np.abs(g_dnn_ag - g_analytical))\n", " max_diff2 = np.max(np.abs(g_vec - g_analytical))\n", " print(\"The max absolute difference between the analytical solution and DNN Autograd: %g\"%max_diff1)\n", " print(\"The max absolute difference between the analytical solution and numerical scheme: %g\"%max_diff2)\n", "\n", " # Plot the results\n", " plt.figure(figsize=(10,10))\n", "\n", " plt.plot(x,g_vec)\n", " plt.plot(x,g_analytical)\n", " plt.plot(x,g_dnn_ag[0,:])\n", "\n", " plt.legend(['numerical scheme','analytical','dnn'])\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "9855543b", "metadata": { "editable": true }, "source": [ "## Partial Differential Equations\n", "\n", "A partial differential equation (PDE) has a solution here the function\n", "is defined by multiple variables. The equation may involve all kinds\n", "of combinations of which variables the function is differentiated with\n", "respect to.\n", "\n", "In general, a partial differential equation for a function $g(x_1,\\dots,x_N)$ with $N$ variables may be expressed as" ] }, { "cell_type": "markdown", "id": "a1b8cd5d", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{PDE} \\tag{17}\n", " f\\left(x_1, \\, \\dots \\, , x_N, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1}, \\dots , \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_N}, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(x_1,\\dots,x_N) }{\\partial x_N^n} \\right) = 0\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "75d6ae90", "metadata": { "editable": true }, "source": [ "where $f$ is an expression involving all kinds of possible mixed derivatives of $g(x_1,\\dots,x_N)$ up to an order $n$. In order for the solution to be unique, some additional conditions must also be given." ] }, { "cell_type": "markdown", "id": "83d6d123", "metadata": { "editable": true }, "source": [ "## Type of problem\n", "\n", "The problem our network must solve for, is similar to the ODE case.\n", "We must have a trial solution $g_t$ at hand.\n", "\n", "For instance, the trial solution could be expressed as" ] }, { "cell_type": "markdown", "id": "7d9f4892", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", " g_t(x_1,\\dots,x_N) = h_1(x_1,\\dots,x_N) + h_2(x_1,\\dots,x_N,N(x_1,\\dots,x_N,P))\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "id": "bc1c8c1b", "metadata": { "editable": true }, "source": [ "where $h_1(x_1,\\dots,x_N)$ is a function that ensures $g_t(x_1,\\dots,x_N)$ satisfies some given conditions.\n", "The neural network $N(x_1,\\dots,x_N,P)$ has weights and biases described by $P$ and $h_2(x_1,\\dots,x_N,N(x_1,\\dots,x_N,P))$ is an expression using the output from the neural network in some way.\n", "\n", "The role of the function $h_2(x_1,\\dots,x_N,N(x_1,\\dots,x_N,P))$, is to ensure that the output of $N(x_1,\\dots,x_N,P)$ is zero when $g_t(x_1,\\dots,x_N)$ is evaluated at the values of $x_1,\\dots,x_N$ where the given conditions must be satisfied. The function $h_1(x_1,\\dots,x_N)$ should alone make $g_t(x_1,\\dots,x_N)$ satisfy the conditions." ] }, { "cell_type": "markdown", "id": "6fdbc09b", "metadata": { "editable": true }, "source": [ "## Network requirements\n", "\n", "The network tries then the minimize the cost function following the\n", "same ideas as described for the ODE case, but now with more than one\n", "variables to consider. The concept still remains the same; find a set\n", "of parameters $P$ such that the expression $f$ in ([17](#PDE)) is as\n", "close to zero as possible.\n", "\n", "As for the ODE case, the cost function is the mean squared error that\n", "the network must try to minimize. The cost function for the network to\n", "minimize is" ] }, { "cell_type": "markdown", "id": "15f911ae", "metadata": { "editable": true }, "source": [ "$$\n", "C\\left(x_1, \\dots, x_N, P\\right) = \\left( f\\left(x_1, \\, \\dots \\, , x_N, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1}, \\dots , \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_N}, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(x_1,\\dots,x_N) }{\\partial x_N^n} \\right) \\right)^2\n", "$$" ] }, { "cell_type": "markdown", "id": "73d815fc", "metadata": { "editable": true }, "source": [ "## More details\n", "\n", "If we let $\\boldsymbol{x} = \\big( x_1, \\dots, x_N \\big)$ be an array containing the values for $x_1, \\dots, x_N$ respectively, the cost function can be reformulated into the following:" ] }, { "cell_type": "markdown", "id": "a8b12b31", "metadata": { "editable": true }, "source": [ "$$\n", "C\\left(\\boldsymbol{x}, P\\right) = f\\left( \\left( \\boldsymbol{x}, \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x}) }{\\partial x_N^n} \\right) \\right)^2\n", "$$" ] }, { "cell_type": "markdown", "id": "0bca5395", "metadata": { "editable": true }, "source": [ "If we also have $M$ different sets of values for $x_1, \\dots, x_N$, that is $\\boldsymbol{x}_i = \\big(x_1^{(i)}, \\dots, x_N^{(i)}\\big)$ for $i = 1,\\dots,M$ being the rows in matrix $X$, the cost function can be generalized into" ] }, { "cell_type": "markdown", "id": "189bac4d", "metadata": { "editable": true }, "source": [ "$$\n", "C\\left(X, P \\right) = \\sum_{i=1}^M f\\left( \\left( \\boldsymbol{x}_i, \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x}_i) }{\\partial x_N^n} \\right) \\right)^2.\n", "$$" ] }, { "cell_type": "markdown", "id": "9b596e36", "metadata": { "editable": true }, "source": [ "## Example: The diffusion equation\n", "\n", "In one spatial dimension, the equation reads" ] }, { "cell_type": "markdown", "id": "3dc31dd6", "metadata": { "editable": true }, "source": [ "$$\n", "\\frac{\\partial g(x,t)}{\\partial t} = \\frac{\\partial^2 g(x,t)}{\\partial x^2}\n", "$$" ] }, { "cell_type": "markdown", "id": "263f026e", "metadata": { "editable": true }, "source": [ "where a possible choice of conditions are" ] }, { "cell_type": "markdown", "id": "717dae65", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", "g(0,t) &= 0 ,\\qquad t \\geq 0 \\\\\n", "g(1,t) &= 0, \\qquad t \\geq 0 \\\\\n", "g(x,0) &= u(x),\\qquad x\\in [0,1]\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "id": "0f898a32", "metadata": { "editable": true }, "source": [ "with $u(x)$ being some given function." ] }, { "cell_type": "markdown", "id": "47634dd9", "metadata": { "editable": true }, "source": [ "## Defining the problem\n", "\n", "For this case, we want to find $g(x,t)$ such that" ] }, { "cell_type": "markdown", "id": "6af0489e", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation}\n", " \\frac{\\partial g(x,t)}{\\partial t} = \\frac{\\partial^2 g(x,t)}{\\partial x^2}\n", "\\end{equation} \\label{diffonedim} \\tag{18}\n", "$$" ] }, { "cell_type": "markdown", "id": "03dabab2", "metadata": { "editable": true }, "source": [ "and" ] }, { "cell_type": "markdown", "id": "796edf7a", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", "g(0,t) &= 0 ,\\qquad t \\geq 0 \\\\\n", "g(1,t) &= 0, \\qquad t \\geq 0 \\\\\n", "g(x,0) &= u(x),\\qquad x\\in [0,1]\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "id": "78d4195f", "metadata": { "editable": true }, "source": [ "with $u(x) = \\sin(\\pi x)$.\n", "\n", "First, let us set up the deep neural network.\n", "The deep neural network will follow the same structure as discussed in the examples solving the ODEs.\n", "First, we will look into how Autograd could be used in a network tailored to solve for bivariate functions." ] }, { "cell_type": "markdown", "id": "13f63286", "metadata": { "editable": true }, "source": [ "## Setting up the network using Autograd\n", "\n", "The only change to do here, is to extend our network such that\n", "functions of multiple parameters are correctly handled. In this case\n", "we have two variables in our function to solve for, that is time $t$\n", "and position $x$. The variables will be represented by a\n", "one-dimensional array in the program. The program will evaluate the\n", "network at each possible pair $(x,t)$, given an array for the desired\n", "$x$-values and $t$-values to approximate the solution at." ] }, { "cell_type": "code", "execution_count": 49, "id": "e05767f4", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "def deep_neural_network(deep_params, x):\n", " # x is now a point and a 1D numpy array; make it a column vector\n", " num_coordinates = np.size(x,0)\n", " x = x.reshape(num_coordinates,-1)\n", "\n", " num_points = np.size(x,1)\n", "\n", " # N_hidden is the number of hidden layers\n", " N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = deep_params[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_points)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = deep_params[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_points)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output[0][0]" ] }, { "cell_type": "markdown", "id": "d4ebc584", "metadata": { "editable": true }, "source": [ "## Setting up the network using Autograd; The trial solution\n", "\n", "The cost function must then iterate through the given arrays\n", "containing values for $x$ and $t$, defines a point $(x,t)$ the deep\n", "neural network and the trial solution is evaluated at, and then finds\n", "the Jacobian of the trial solution.\n", "\n", "A possible trial solution for this PDE is\n", "\n", "$$\n", "g_t(x,t) = h_1(x,t) + x(1-x)tN(x,t,P)\n", "$$\n", "\n", "with $A(x,t)$ being a function ensuring that $g_t(x,t)$ satisfies our given conditions, and $N(x,t,P)$ being the output from the deep neural network using weights and biases for each layer from $P$.\n", "\n", "To fulfill the conditions, $A(x,t)$ could be:\n", "\n", "$$\n", "h_1(x,t) = (1-t)\\Big(u(x) - \\big((1-x)u(0) + x u(1)\\big)\\Big) = (1-t)u(x) = (1-t)\\sin(\\pi x)\n", "$$\n", "since $(0) = u(1) = 0$ and $u(x) = \\sin(\\pi x)$." ] }, { "cell_type": "markdown", "id": "124b550e", "metadata": { "editable": true }, "source": [ "## Why the jacobian?\n", "\n", "The Jacobian is used because the program must find the derivative of\n", "the trial solution with respect to $x$ and $t$.\n", "\n", "This gives the necessity of computing the Jacobian matrix, as we want\n", "to evaluate the gradient with respect to $x$ and $t$ (note that the\n", "Jacobian of a scalar-valued multivariate function is simply its\n", "gradient).\n", "\n", "In Autograd, the differentiation is by default done with respect to\n", "the first input argument of your Python function. Since the points is\n", "an array representing $x$ and $t$, the Jacobian is calculated using\n", "the values of $x$ and $t$.\n", "\n", "To find the second derivative with respect to $x$ and $t$, the\n", "Jacobian can be found for the second time. The result is a Hessian\n", "matrix, which is the matrix containing all the possible second order\n", "mixed derivatives of $g(x,t)$." ] }, { "cell_type": "code", "execution_count": 50, "id": "53c4c879", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "# Set up the trial function:\n", "def u(x):\n", " return np.sin(np.pi*x)\n", "\n", "def g_trial(point,P):\n", " x,t = point\n", " return (1-t)*u(x) + x*(1-x)*t*deep_neural_network(P,point)\n", "\n", "# The right side of the ODE:\n", "def f(point):\n", " return 0.\n", "\n", "# The cost function:\n", "def cost_function(P, x, t):\n", " cost_sum = 0\n", "\n", " g_t_jacobian_func = jacobian(g_trial)\n", " g_t_hessian_func = hessian(g_trial)\n", "\n", " for x_ in x:\n", " for t_ in t:\n", " point = np.array([x_,t_])\n", "\n", " g_t = g_trial(point,P)\n", " g_t_jacobian = g_t_jacobian_func(point,P)\n", " g_t_hessian = g_t_hessian_func(point,P)\n", "\n", " g_t_dt = g_t_jacobian[1]\n", " g_t_d2x = g_t_hessian[0][0]\n", "\n", " func = f(point)\n", "\n", " err_sqr = ( (g_t_dt - g_t_d2x) - func)**2\n", " cost_sum += err_sqr\n", "\n", " return cost_sum" ] }, { "cell_type": "markdown", "id": "b11a26b8", "metadata": { "editable": true }, "source": [ "## Setting up the network using Autograd; The full program\n", "\n", "Having set up the network, along with the trial solution and cost function, we can now see how the deep neural network performs by comparing the results to the analytical solution.\n", "\n", "The analytical solution of our problem is\n", "\n", "$$\n", "g(x,t) = \\exp(-\\pi^2 t)\\sin(\\pi x)\n", "$$\n", "\n", "A possible way to implement a neural network solving the PDE, is given below.\n", "Be aware, though, that it is fairly slow for the parameters used.\n", "A better result is possible, but requires more iterations, and thus longer time to complete.\n", "\n", "Indeed, the program below is not optimal in its implementation, but rather serves as an example on how to implement and use a neural network to solve a PDE.\n", "Using TensorFlow results in a much better execution time. Try it!" ] }, { "cell_type": "code", "execution_count": 51, "id": "fa2d8508", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import jacobian,hessian,grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import cm\n", "from matplotlib import pyplot as plt\n", "from mpl_toolkits.mplot3d import axes3d\n", "\n", "## Set up the network\n", "\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "def deep_neural_network(deep_params, x):\n", " # x is now a point and a 1D numpy array; make it a column vector\n", " num_coordinates = np.size(x,0)\n", " x = x.reshape(num_coordinates,-1)\n", "\n", " num_points = np.size(x,1)\n", "\n", " # N_hidden is the number of hidden layers\n", " N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = deep_params[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_points)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = deep_params[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_points)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output[0][0]\n", "\n", "## Define the trial solution and cost function\n", "def u(x):\n", " return np.sin(np.pi*x)\n", "\n", "def g_trial(point,P):\n", " x,t = point\n", " return (1-t)*u(x) + x*(1-x)*t*deep_neural_network(P,point)\n", "\n", "# The right side of the ODE:\n", "def f(point):\n", " return 0.\n", "\n", "# The cost function:\n", "def cost_function(P, x, t):\n", " cost_sum = 0\n", "\n", " g_t_jacobian_func = jacobian(g_trial)\n", " g_t_hessian_func = hessian(g_trial)\n", "\n", " for x_ in x:\n", " for t_ in t:\n", " point = np.array([x_,t_])\n", "\n", " g_t = g_trial(point,P)\n", " g_t_jacobian = g_t_jacobian_func(point,P)\n", " g_t_hessian = g_t_hessian_func(point,P)\n", "\n", " g_t_dt = g_t_jacobian[1]\n", " g_t_d2x = g_t_hessian[0][0]\n", "\n", " func = f(point)\n", "\n", " err_sqr = ( (g_t_dt - g_t_d2x) - func)**2\n", " cost_sum += err_sqr\n", "\n", " return cost_sum /( np.size(x)*np.size(t) )\n", "\n", "## For comparison, define the analytical solution\n", "def g_analytic(point):\n", " x,t = point\n", " return np.exp(-np.pi**2*t)*np.sin(np.pi*x)\n", "\n", "## Set up a function for training the network to solve for the equation\n", "def solve_pde_deep_neural_network(x,t, num_neurons, num_iter, lmb):\n", " ## Set up initial weigths and biases\n", " N_hidden = np.size(num_neurons)\n", "\n", " ## Set up initial weigths and biases\n", "\n", " # Initialize the list of parameters:\n", " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", "\n", " P[0] = npr.randn(num_neurons[0], 2 + 1 ) # 2 since we have two points, +1 to include bias\n", " for l in range(1,N_hidden):\n", " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias\n", "\n", " # For the output layer\n", " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", "\n", " print('Initial cost: ',cost_function(P, x, t))\n", "\n", " cost_function_grad = grad(cost_function,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " cost_grad = cost_function_grad(P, x , t)\n", "\n", " for l in range(N_hidden+1):\n", " P[l] = P[l] - lmb * cost_grad[l]\n", "\n", " print('Final cost: ',cost_function(P, x, t))\n", "\n", " return P\n", "\n", "if __name__ == '__main__':\n", " ### Use the neural network:\n", " npr.seed(15)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " Nx = 10; Nt = 10\n", " x = np.linspace(0, 1, Nx)\n", " t = np.linspace(0,1,Nt)\n", "\n", " ## Set up the parameters for the network\n", " num_hidden_neurons = [100, 25]\n", " num_iter = 250\n", " lmb = 0.01\n", "\n", " P = solve_pde_deep_neural_network(x,t, num_hidden_neurons, num_iter, lmb)\n", "\n", " ## Store the results\n", " g_dnn_ag = np.zeros((Nx, Nt))\n", " G_analytical = np.zeros((Nx, Nt))\n", " for i,x_ in enumerate(x):\n", " for j, t_ in enumerate(t):\n", " point = np.array([x_, t_])\n", " g_dnn_ag[i,j] = g_trial(point,P)\n", "\n", " G_analytical[i,j] = g_analytic(point)\n", "\n", " # Find the map difference between the analytical and the computed solution\n", " diff_ag = np.abs(g_dnn_ag - G_analytical)\n", " print('Max absolute difference between the analytical solution and the network: %g'%np.max(diff_ag))\n", "\n", " ## Plot the solutions in two dimensions, that being in position and time\n", "\n", " T,X = np.meshgrid(t,x)\n", "\n", " fig = plt.figure(figsize=(10,10))\n", " ax = fig.gca(projection='3d')\n", " ax.set_title('Solution from the deep neural network w/ %d layer'%len(num_hidden_neurons))\n", " s = ax.plot_surface(T,X,g_dnn_ag,linewidth=0,antialiased=False,cmap=cm.viridis)\n", " ax.set_xlabel('Time $t$')\n", " ax.set_ylabel('Position $x$');\n", "\n", "\n", " fig = plt.figure(figsize=(10,10))\n", " ax = fig.gca(projection='3d')\n", " ax.set_title('Analytical solution')\n", " s = ax.plot_surface(T,X,G_analytical,linewidth=0,antialiased=False,cmap=cm.viridis)\n", " ax.set_xlabel('Time $t$')\n", " ax.set_ylabel('Position $x$');\n", "\n", " fig = plt.figure(figsize=(10,10))\n", " ax = fig.gca(projection='3d')\n", " ax.set_title('Difference')\n", " s = ax.plot_surface(T,X,diff_ag,linewidth=0,antialiased=False,cmap=cm.viridis)\n", " ax.set_xlabel('Time $t$')\n", " ax.set_ylabel('Position $x$');\n", "\n", " ## Take some slices of the 3D plots just to see the solutions at particular times\n", " indx1 = 0\n", " indx2 = int(Nt/2)\n", " indx3 = Nt-1\n", "\n", " t1 = t[indx1]\n", " t2 = t[indx2]\n", " t3 = t[indx3]\n", "\n", " # Slice the results from the DNN\n", " res1 = g_dnn_ag[:,indx1]\n", " res2 = g_dnn_ag[:,indx2]\n", " res3 = g_dnn_ag[:,indx3]\n", "\n", " # Slice the analytical results\n", " res_analytical1 = G_analytical[:,indx1]\n", " res_analytical2 = G_analytical[:,indx2]\n", " res_analytical3 = G_analytical[:,indx3]\n", "\n", " # Plot the slices\n", " plt.figure(figsize=(10,10))\n", " plt.title(\"Computed solutions at time = %g\"%t1)\n", " plt.plot(x, res1)\n", " plt.plot(x,res_analytical1)\n", " plt.legend(['dnn','analytical'])\n", "\n", " plt.figure(figsize=(10,10))\n", " plt.title(\"Computed solutions at time = %g\"%t2)\n", " plt.plot(x, res2)\n", " plt.plot(x,res_analytical2)\n", " plt.legend(['dnn','analytical'])\n", "\n", " plt.figure(figsize=(10,10))\n", " plt.title(\"Computed solutions at time = %g\"%t3)\n", " plt.plot(x, res3)\n", " plt.plot(x,res_analytical3)\n", " plt.legend(['dnn','analytical'])\n", "\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "4dfabc5e", "metadata": { "editable": true }, "source": [ "## Example: Solving the wave equation with Neural Networks\n", "\n", "The wave equation is" ] }, { "cell_type": "markdown", "id": "8f6f58e6", "metadata": { "editable": true }, "source": [ "$$\n", "\\frac{\\partial^2 g(x,t)}{\\partial t^2} = c^2\\frac{\\partial^2 g(x,t)}{\\partial x^2}\n", "$$" ] }, { "cell_type": "markdown", "id": "8a4abe82", "metadata": { "editable": true }, "source": [ "with $c$ being the specified wave speed.\n", "\n", "Here, the chosen conditions are" ] }, { "cell_type": "markdown", "id": "57c34190", "metadata": { "editable": true }, "source": [ "$$\n", "\\begin{align*}\n", "\tg(0,t) &= 0 \\\\\n", "\tg(1,t) &= 0 \\\\\n", "\tg(x,0) &= u(x) \\\\\n", "\t\\frac{\\partial g(x,t)}{\\partial t} \\Big |_{t = 0} &= v(x)\n", "\\end{align*}\n", "$$" ] }, { "cell_type": "markdown", "id": "8c5b8137", "metadata": { "editable": true }, "source": [ "where $\\frac{\\partial g(x,t)}{\\partial t} \\Big |_{t = 0}$ means the derivative of $g(x,t)$ with respect to $t$ is evaluated at $t = 0$, and $u(x)$ and $v(x)$ being given functions." ] }, { "cell_type": "markdown", "id": "08cb8e62", "metadata": { "editable": true }, "source": [ "## The problem to solve for\n", "\n", "The wave equation to solve for, is" ] }, { "cell_type": "markdown", "id": "f7b1bf8c", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{equation} \\label{wave} \\tag{19}\n", "\\frac{\\partial^2 g(x,t)}{\\partial t^2} = c^2 \\frac{\\partial^2 g(x,t)}{\\partial x^2}\n", "\\end{equation}\n", "$$" ] }, { "cell_type": "markdown", "id": "1127e6e8", "metadata": { "editable": true }, "source": [ "where $c$ is the given wave speed.\n", "The chosen conditions for this equation are" ] }, { "cell_type": "markdown", "id": "7a099129", "metadata": { "editable": true }, "source": [ "\n", "
\n", "\n", "$$\n", "\\begin{aligned}\n", "g(0,t) &= 0, &t \\geq 0 \\\\\n", "g(1,t) &= 0, &t \\geq 0 \\\\\n", "g(x,0) &= u(x), &x\\in[0,1] \\\\\n", "\\frac{\\partial g(x,t)}{\\partial t}\\Big |_{t = 0} &= v(x), &x \\in [0,1]\n", "\\end{aligned} \\label{condwave} \\tag{20}\n", "$$" ] }, { "cell_type": "markdown", "id": "772c7eec", "metadata": { "editable": true }, "source": [ "In this example, let $c = 1$ and $u(x) = \\sin(\\pi x)$ and $v(x) = -\\pi\\sin(\\pi x)$." ] }, { "cell_type": "markdown", "id": "be8649c8", "metadata": { "editable": true }, "source": [ "## The trial solution\n", "Setting up the network is done in similar matter as for the example of solving the diffusion equation.\n", "The only things we have to change, is the trial solution such that it satisfies the conditions from ([20](#condwave)) and the cost function.\n", "\n", "The trial solution becomes slightly different since we have other conditions than in the example of solving the diffusion equation. Here, a possible trial solution $g_t(x,t)$ is\n", "\n", "$$\n", "g_t(x,t) = h_1(x,t) + x(1-x)t^2N(x,t,P)\n", "$$\n", "\n", "where\n", "\n", "$$\n", "h_1(x,t) = (1-t^2)u(x) + tv(x)\n", "$$\n", "\n", "Note that this trial solution satisfies the conditions only if $u(0) = v(0) = u(1) = v(1) = 0$, which is the case in this example." ] }, { "cell_type": "markdown", "id": "27484fe1", "metadata": { "editable": true }, "source": [ "## The analytical solution\n", "\n", "The analytical solution for our specific problem, is\n", "\n", "$$\n", "g(x,t) = \\sin(\\pi x)\\cos(\\pi t) - \\sin(\\pi x)\\sin(\\pi t)\n", "$$" ] }, { "cell_type": "markdown", "id": "16f8dd35", "metadata": { "editable": true }, "source": [ "## Solving the wave equation - the full program using Autograd" ] }, { "cell_type": "code", "execution_count": 52, "id": "51011a0f", "metadata": { "collapsed": false, "editable": true }, "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import hessian,grad\n", "import autograd.numpy.random as npr\n", "from matplotlib import cm\n", "from matplotlib import pyplot as plt\n", "from mpl_toolkits.mplot3d import axes3d\n", "\n", "## Set up the trial function:\n", "def u(x):\n", " return np.sin(np.pi*x)\n", "\n", "def v(x):\n", " return -np.pi*np.sin(np.pi*x)\n", "\n", "def h1(point):\n", " x,t = point\n", " return (1 - t**2)*u(x) + t*v(x)\n", "\n", "def g_trial(point,P):\n", " x,t = point\n", " return h1(point) + x*(1-x)*t**2*deep_neural_network(P,point)\n", "\n", "## Define the cost function\n", "def cost_function(P, x, t):\n", " cost_sum = 0\n", "\n", " g_t_hessian_func = hessian(g_trial)\n", "\n", " for x_ in x:\n", " for t_ in t:\n", " point = np.array([x_,t_])\n", "\n", " g_t_hessian = g_t_hessian_func(point,P)\n", "\n", " g_t_d2x = g_t_hessian[0][0]\n", " g_t_d2t = g_t_hessian[1][1]\n", "\n", " err_sqr = ( (g_t_d2t - g_t_d2x) )**2\n", " cost_sum += err_sqr\n", "\n", " return cost_sum / (np.size(t) * np.size(x))\n", "\n", "## The neural network\n", "def sigmoid(z):\n", " return 1/(1 + np.exp(-z))\n", "\n", "def deep_neural_network(deep_params, x):\n", " # x is now a point and a 1D numpy array; make it a column vector\n", " num_coordinates = np.size(x,0)\n", " x = x.reshape(num_coordinates,-1)\n", "\n", " num_points = np.size(x,1)\n", "\n", " # N_hidden is the number of hidden layers\n", " N_hidden = np.size(deep_params) - 1 # -1 since params consist of parameters to all the hidden layers AND the output layer\n", "\n", " # Assume that the input layer does nothing to the input x\n", " x_input = x\n", " x_prev = x_input\n", "\n", " ## Hidden layers:\n", "\n", " for l in range(N_hidden):\n", " # From the list of parameters P; find the correct weigths and bias for this layer\n", " w_hidden = deep_params[l]\n", "\n", " # Add a row of ones to include bias\n", " x_prev = np.concatenate((np.ones((1,num_points)), x_prev ), axis = 0)\n", "\n", " z_hidden = np.matmul(w_hidden, x_prev)\n", " x_hidden = sigmoid(z_hidden)\n", "\n", " # Update x_prev such that next layer can use the output from this layer\n", " x_prev = x_hidden\n", "\n", " ## Output layer:\n", "\n", " # Get the weights and bias for this layer\n", " w_output = deep_params[-1]\n", "\n", " # Include bias:\n", " x_prev = np.concatenate((np.ones((1,num_points)), x_prev), axis = 0)\n", "\n", " z_output = np.matmul(w_output, x_prev)\n", " x_output = z_output\n", "\n", " return x_output[0][0]\n", "\n", "## The analytical solution\n", "def g_analytic(point):\n", " x,t = point\n", " return np.sin(np.pi*x)*np.cos(np.pi*t) - np.sin(np.pi*x)*np.sin(np.pi*t)\n", "\n", "def solve_pde_deep_neural_network(x,t, num_neurons, num_iter, lmb):\n", " ## Set up initial weigths and biases\n", " N_hidden = np.size(num_neurons)\n", "\n", " ## Set up initial weigths and biases\n", "\n", " # Initialize the list of parameters:\n", " P = [None]*(N_hidden + 1) # + 1 to include the output layer\n", "\n", " P[0] = npr.randn(num_neurons[0], 2 + 1 ) # 2 since we have two points, +1 to include bias\n", " for l in range(1,N_hidden):\n", " P[l] = npr.randn(num_neurons[l], num_neurons[l-1] + 1) # +1 to include bias\n", "\n", " # For the output layer\n", " P[-1] = npr.randn(1, num_neurons[-1] + 1 ) # +1 since bias is included\n", "\n", " print('Initial cost: ',cost_function(P, x, t))\n", "\n", " cost_function_grad = grad(cost_function,0)\n", "\n", " # Let the update be done num_iter times\n", " for i in range(num_iter):\n", " cost_grad = cost_function_grad(P, x , t)\n", "\n", " for l in range(N_hidden+1):\n", " P[l] = P[l] - lmb * cost_grad[l]\n", "\n", "\n", " print('Final cost: ',cost_function(P, x, t))\n", "\n", " return P\n", "\n", "if __name__ == '__main__':\n", " ### Use the neural network:\n", " npr.seed(15)\n", "\n", " ## Decide the vales of arguments to the function to solve\n", " Nx = 10; Nt = 10\n", " x = np.linspace(0, 1, Nx)\n", " t = np.linspace(0,1,Nt)\n", "\n", " ## Set up the parameters for the network\n", " num_hidden_neurons = [50,20]\n", " num_iter = 1000\n", " lmb = 0.01\n", "\n", " P = solve_pde_deep_neural_network(x,t, num_hidden_neurons, num_iter, lmb)\n", "\n", " ## Store the results\n", " res = np.zeros((Nx, Nt))\n", " res_analytical = np.zeros((Nx, Nt))\n", " for i,x_ in enumerate(x):\n", " for j, t_ in enumerate(t):\n", " point = np.array([x_, t_])\n", " res[i,j] = g_trial(point,P)\n", "\n", " res_analytical[i,j] = g_analytic(point)\n", "\n", " diff = np.abs(res - res_analytical)\n", " print(\"Max difference between analytical and solution from nn: %g\"%np.max(diff))\n", "\n", " ## Plot the solutions in two dimensions, that being in position and time\n", "\n", " T,X = np.meshgrid(t,x)\n", "\n", " fig = plt.figure(figsize=(10,10))\n", " ax = fig.gca(projection='3d')\n", " ax.set_title('Solution from the deep neural network w/ %d layer'%len(num_hidden_neurons))\n", " s = ax.plot_surface(T,X,res,linewidth=0,antialiased=False,cmap=cm.viridis)\n", " ax.set_xlabel('Time $t$')\n", " ax.set_ylabel('Position $x$');\n", "\n", "\n", " fig = plt.figure(figsize=(10,10))\n", " ax = fig.gca(projection='3d')\n", " ax.set_title('Analytical solution')\n", " s = ax.plot_surface(T,X,res_analytical,linewidth=0,antialiased=False,cmap=cm.viridis)\n", " ax.set_xlabel('Time $t$')\n", " ax.set_ylabel('Position $x$');\n", "\n", "\n", " fig = plt.figure(figsize=(10,10))\n", " ax = fig.gca(projection='3d')\n", " ax.set_title('Difference')\n", " s = ax.plot_surface(T,X,diff,linewidth=0,antialiased=False,cmap=cm.viridis)\n", " ax.set_xlabel('Time $t$')\n", " ax.set_ylabel('Position $x$');\n", "\n", " ## Take some slices of the 3D plots just to see the solutions at particular times\n", " indx1 = 0\n", " indx2 = int(Nt/2)\n", " indx3 = Nt-1\n", "\n", " t1 = t[indx1]\n", " t2 = t[indx2]\n", " t3 = t[indx3]\n", "\n", " # Slice the results from the DNN\n", " res1 = res[:,indx1]\n", " res2 = res[:,indx2]\n", " res3 = res[:,indx3]\n", "\n", " # Slice the analytical results\n", " res_analytical1 = res_analytical[:,indx1]\n", " res_analytical2 = res_analytical[:,indx2]\n", " res_analytical3 = res_analytical[:,indx3]\n", "\n", " # Plot the slices\n", " plt.figure(figsize=(10,10))\n", " plt.title(\"Computed solutions at time = %g\"%t1)\n", " plt.plot(x, res1)\n", " plt.plot(x,res_analytical1)\n", " plt.legend(['dnn','analytical'])\n", "\n", " plt.figure(figsize=(10,10))\n", " plt.title(\"Computed solutions at time = %g\"%t2)\n", " plt.plot(x, res2)\n", " plt.plot(x,res_analytical2)\n", " plt.legend(['dnn','analytical'])\n", "\n", " plt.figure(figsize=(10,10))\n", " plt.title(\"Computed solutions at time = %g\"%t3)\n", " plt.plot(x, res3)\n", " plt.plot(x,res_analytical3)\n", " plt.legend(['dnn','analytical'])\n", "\n", " plt.show()" ] }, { "cell_type": "markdown", "id": "fff9e254", "metadata": { "editable": true }, "source": [ "## Resources on differential equations and deep learning\n", "\n", "1. [Artificial neural networks for solving ordinary and partial differential equations by I.E. Lagaris et al](https://pdfs.semanticscholar.org/d061/df393e0e8fbfd0ea24976458b7d42419040d.pdf)\n", "\n", "2. [Neural networks for solving differential equations by A. Honchar](https://becominghuman.ai/neural-networks-for-solving-differential-equations-fa230ac5e04c)\n", "\n", "3. [Solving differential equations using neural networks by M.M Chiaramonte and M. Kiener](http://cs229.stanford.edu/proj2013/ChiaramonteKiener-SolvingDifferentialEquationsUsingNeuralNetworks.pdf)\n", "\n", "4. [Introduction to Partial Differential Equations by A. Tveito, R. Winther](https://www.springer.com/us/book/9783540225515)" ] } ], "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.10" } }, "nbformat": 4, "nbformat_minor": 5 }