diff --git a/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb b/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb new file mode 100644 index 000000000..6d6019289 --- /dev/null +++ b/doc/LectureNotes/.ipynb_checkpoints/exercisesweek43-checkpoint.ipynb @@ -0,0 +1,1515 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b2937d10", + "metadata": {}, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "3dd00d19", + "metadata": {}, + "source": [ + "# Exercises weeks 43 and 44 \n", + "**October 23-27, 2023**\n", + "\n", + "Date: **Deadline is Sunday November 5 at midnight**\n", + "\n", + "You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points." + ] + }, + { + "cell_type": "markdown", + "id": "82a19a1d", + "metadata": {}, + "source": [ + "# Overarching aims of the exercises weeks 43 and 44\n", + "\n", + "The aim of the exercises this week and next week is to get started with writing a neural network code\n", + "of relevance for project 2. \n", + "\n", + "During week 41 we discussed three different types of gates, the\n", + "so-called XOR, the OR and the AND gates. In order to develop a code\n", + "for neural networks, it can be useful to set up a simpler system with\n", + "only two inputs and one output. This can make it easier to debug and\n", + "study the feed forward pass and the back propagation part. In the\n", + "exercise this and next week, we propose to study this system with just\n", + "one hidden layer and two hidden nodes. There is only one output node\n", + "and we can choose to use either a simple regression case (fitting a\n", + "line) or just a binary classification case with the cross-entropy as\n", + "cost function.\n", + "\n", + "Their inputs and outputs can be\n", + "summarized using the following tables, first for the OR gate with\n", + "inputs $x_1$ and $x_2$ and outputs $y$:\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
$x_1$ $x_2$ $y$
0 0 0
0 1 1
1 0 1
1 1 1
" + ] + }, + { + "cell_type": "markdown", + "id": "f74f69af", + "metadata": {}, + "source": [ + "## The AND and XOR Gates\n", + "\n", + "The AND gate is defined as\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
$x_1$ $x_2$ $y$
0 0 0
0 1 0
1 0 0
1 1 1
\n", + "\n", + "And finally we have the XOR gate\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
$x_1$ $x_2$ $y$
0 0 0
0 1 1
1 0 1
1 1 0
" + ] + }, + { + "cell_type": "markdown", + "id": "1b52d47a", + "metadata": {}, + "source": [ + "## Representing the Data Sets\n", + "\n", + "Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads" + ] + }, + { + "cell_type": "markdown", + "id": "3e6910cb", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n", + " 0 & 1 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 1 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "90a3b78a", + "metadata": {}, + "source": [ + "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n", + "\n", + "Your tasks here are\n", + "\n", + "1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n", + "\n", + "2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n", + "\n", + "3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n", + "\n", + "4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n", + "\n", + "5. Set up the cost function (cross entropy for classification of binary cases).\n", + "\n", + "6. Calculate the gradients needed for the back propagation part.\n", + "\n", + "7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n", + "\n", + "8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n", + "\n", + "Everything you develop here can be used directly into the code for the project." + ] + }, + { + "cell_type": "markdown", + "id": "d6a3ab1e", + "metadata": {}, + "source": [ + "## Setting up the Neural Network\n", + "\n", + "We define first our design matrix and the various output vectors for the different gates." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "152123b0", + "metadata": {}, + "outputs": [], + "source": [ + "%matplotlib inline\n", + "\n", + "\"\"\"\n", + "Simple code that tests XOR, OR and AND gates with linear regression\n", + "\"\"\"\n", + "\n", + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn import datasets\n", + "\n", + "def sigmoid(x):\n", + " return 1/(1 + np.exp(-x))\n", + "\n", + "def feed_forward(X):\n", + " # weighted sum of inputs to the hidden layer\n", + " z_h = np.matmul(X, hidden_weights) + hidden_bias\n", + " # activation in the hidden layer\n", + " a_h = sigmoid(z_h)\n", + " \n", + " # weighted sum of inputs to the output layer\n", + " z_o = np.matmul(a_h, output_weights) + output_bias\n", + " # softmax output\n", + " # axis 0 holds each input and axis 1 the probabilities of each category\n", + " probabilities = sigmoid(z_o)\n", + " return probabilities\n", + "\n", + "# we obtain a prediction by taking the class with the highest likelihood\n", + "def predict(X):\n", + " probabilities = feed_forward(X)\n", + " return np.argmax(probabilities, axis=1)\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# Design matrix\n", + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [ 0, 1 ,1, 0])\n", + "# The OR gate\n", + "yOR = np.array( [ 0, 1 ,1, 1])\n", + "# The AND gate\n", + "yAND = np.array( [ 0, 0 ,0, 1])\n", + "\n", + "# Defining the neural network\n", + "n_inputs, n_features = X.shape\n", + "n_hidden_neurons = 2\n", + "n_categories = 2\n", + "n_features = 2\n", + "\n", + "# we make the weights normally distributed using numpy.random.randn\n", + "\n", + "# weights and bias in the hidden layer\n", + "hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n", + "hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n", + "\n", + "# weights and bias in the output layer\n", + "output_weights = np.random.randn(n_hidden_neurons, n_categories)\n", + "output_bias = np.zeros(n_categories) + 0.01\n", + "\n", + "probabilities = feed_forward(X)\n", + "print(probabilities)\n", + "\n", + "\n", + "predictions = predict(X)\n", + "print(predictions)" + ] + }, + { + "cell_type": "markdown", + "id": "73319f0a", + "metadata": {}, + "source": [ + "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above." + ] + }, + { + "cell_type": "markdown", + "id": "a7e0c47a", + "metadata": {}, + "source": [ + "## The Code using Scikit-Learn" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "dbbacc67", + "metadata": {}, + "outputs": [], + "source": [ + "# import necessary packages\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.neural_network import MLPClassifier\n", + "from sklearn.metrics import accuracy_score\n", + "import seaborn as sns\n", + "\n", + "# ensure the same random numbers appear every time\n", + "np.random.seed(0)\n", + "\n", + "# Design matrix\n", + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [ 0, 1 ,1, 0])\n", + "# The OR gate\n", + "yOR = np.array( [ 0, 1 ,1, 1])\n", + "# The AND gate\n", + "yAND = np.array( [ 0, 0 ,0, 1])\n", + "\n", + "# Defining the neural network\n", + "n_inputs, n_features = X.shape\n", + "n_hidden_neurons = 2\n", + "n_categories = 2\n", + "n_features = 2\n", + "\n", + "eta_vals = np.logspace(-5, 1, 7)\n", + "lmbd_vals = np.logspace(-5, 1, 7)\n", + "# store models for later use\n", + "DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", + "epochs = 100\n", + "\n", + "for i, eta in enumerate(eta_vals):\n", + " for j, lmbd in enumerate(lmbd_vals):\n", + " dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n", + " alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n", + " dnn.fit(X, yXOR)\n", + " DNN_scikit[i][j] = dnn\n", + " print(\"Learning rate = \", eta)\n", + " print(\"Lambda = \", lmbd)\n", + " print(\"Accuracy score on data set: \", dnn.score(X, yXOR))\n", + " print()\n", + "\n", + "sns.set()\n", + "test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n", + "for i in range(len(eta_vals)):\n", + " for j in range(len(lmbd_vals)):\n", + " dnn = DNN_scikit[i][j]\n", + " test_pred = dnn.predict(X)\n", + " test_accuracy[i][j] = accuracy_score(yXOR, test_pred)\n", + "\n", + "fig, ax = plt.subplots(figsize = (10, 10))\n", + "sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n", + "ax.set_title(\"Test Accuracy\")\n", + "ax.set_ylabel(\"$\\eta$\")\n", + "ax.set_xlabel(\"$\\lambda$\")\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "1cac501a", + "metadata": {}, + "source": [ + "## Building a neural network code\n", + "\n", + "Here we present a flexible object oriented codebase\n", + "for a feed forward neural network, along with a demonstration of how\n", + "to use it. Before we get into the details of the neural network, we\n", + "will first present some implementations of various schedulers, cost\n", + "functions and activation functions that can be used together with the\n", + "neural network.\n", + "\n", + "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023." + ] + }, + { + "cell_type": "markdown", + "id": "dd153528", + "metadata": {}, + "source": [ + "### Learning rate methods\n", + "\n", + "The code below shows object oriented implementations of the Constant,\n", + "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n", + "of the classes belong to the shared abstract Scheduler class, and\n", + "share the update_change() and reset() methods allowing for any of the\n", + "schedulers to be seamlessly used during the training stage, as will\n", + "later be shown in the fit() method of the neural\n", + "network. Update_change() only has one parameter, the gradient\n", + "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n", + "from the weights. The reset() function takes no parameters, and resets\n", + "the desired variables. For Constant and Momentum, reset does nothing." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "f55eea63", + "metadata": {}, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "class Scheduler:\n", + " \"\"\"\n", + " Abstract class for Schedulers\n", + " \"\"\"\n", + "\n", + " def __init__(self, eta):\n", + " self.eta = eta\n", + "\n", + " # should be overwritten\n", + " def update_change(self, gradient):\n", + " raise NotImplementedError\n", + "\n", + " # overwritten if needed\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Constant(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + "\n", + " def update_change(self, gradient):\n", + " return self.eta * gradient\n", + " \n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Momentum(Scheduler):\n", + " def __init__(self, eta: float, momentum: float):\n", + " super().__init__(eta)\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " self.change = self.momentum * self.change + self.eta * gradient\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Adagrad(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " return self.eta * gradient * G_t_inverse\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class AdagradMomentum(Scheduler):\n", + " def __init__(self, eta, momentum):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class RMS_prop(Scheduler):\n", + " def __init__(self, eta, rho):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.second = 0.0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n", + " return self.eta * gradient / (np.sqrt(self.second + delta))\n", + "\n", + " def reset(self):\n", + " self.second = 0.0\n", + "\n", + "\n", + "class Adam(Scheduler):\n", + " def __init__(self, eta, rho, rho2):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.rho2 = rho2\n", + " self.moment = 0\n", + " self.second = 0\n", + " self.n_epochs = 1\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n", + " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n", + "\n", + " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n", + " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n", + "\n", + " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n", + "\n", + " def reset(self):\n", + " self.n_epochs += 1\n", + " self.moment = 0\n", + " self.second = 0" + ] + }, + { + "cell_type": "markdown", + "id": "1a9bcb3e", + "metadata": {}, + "source": [ + "### Usage of the above learning rate schedulers\n", + "\n", + "To initalize a scheduler, simply create the object and pass in the\n", + "necessary parameters such as the learning rate and the momentum as\n", + "shown below. As the Scheduler class is an abstract class it should not\n", + "called directly, and will raise an error upon usage." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "86013cb4", + "metadata": {}, + "outputs": [], + "source": [ + "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", + "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)" + ] + }, + { + "cell_type": "markdown", + "id": "535331f6", + "metadata": {}, + "source": [ + "Here is a small example for how a segment of code using schedulers\n", + "could look. Switching out the schedulers is simple." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "7e0f6b5a", + "metadata": {}, + "outputs": [], + "source": [ + "weights = np.ones((3,3))\n", + "print(f\"Before scheduler:\\n{weights=}\")\n", + "\n", + "epochs = 10\n", + "for e in range(epochs):\n", + " gradient = np.random.rand(3, 3)\n", + " change = adam_scheduler.update_change(gradient)\n", + " weights = weights - change\n", + " adam_scheduler.reset()\n", + "\n", + "print(f\"\\nAfter scheduler:\\n{weights=}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f018ae57", + "metadata": {}, + "source": [ + "### Cost functions\n", + "\n", + "Here we discuss cost functions that can be used when creating the\n", + "neural network. Every cost function takes the target vector as its\n", + "parameter, and returns a function valued only at $x$ such that it may\n", + "easily be differentiated." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "c13507bf", + "metadata": {}, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "def CostOLS(target):\n", + " \n", + " def func(X):\n", + " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostLogReg(target):\n", + "\n", + " def func(X):\n", + " \n", + " return -(1.0 / target.shape[0]) * np.sum(\n", + " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n", + " )\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostCrossEntropy(target):\n", + " \n", + " def func(X):\n", + " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n", + "\n", + " return func" + ] + }, + { + "cell_type": "markdown", + "id": "6dab17bc", + "metadata": {}, + "source": [ + "Below we give a short example of how these cost function may be used\n", + "to obtain results if you wish to test them out on your own using\n", + "AutoGrad's automatics differentiation." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "a5dbba01", + "metadata": {}, + "outputs": [], + "source": [ + "from autograd import grad\n", + "\n", + "target = np.array([[1, 2, 3]]).T\n", + "a = np.array([[4, 5, 6]]).T\n", + "\n", + "cost_func = CostCrossEntropy\n", + "cost_func_derivative = grad(cost_func(target))\n", + "\n", + "valued_at_a = cost_func_derivative(a)\n", + "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b55d31d4", + "metadata": {}, + "source": [ + "### Activation functions\n", + "\n", + "Finally, before we look at the neural network, we will look at the\n", + "activation functions which can be specified between the hidden layers\n", + "and as the output function. Each function can be valued for any given\n", + "vector or matrix X, and can be differentiated via derivate()." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "b3e045a6", + "metadata": {}, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from autograd import elementwise_grad\n", + "\n", + "def identity(X):\n", + " return X\n", + "\n", + "\n", + "def sigmoid(X):\n", + " try:\n", + " return 1.0 / (1 + np.exp(-X))\n", + " except FloatingPointError:\n", + " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n", + "\n", + "\n", + "def softmax(X):\n", + " X = X - np.max(X, axis=-1, keepdims=True)\n", + " delta = 10e-10\n", + " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n", + "\n", + "\n", + "def RELU(X):\n", + " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n", + "\n", + "\n", + "def LRELU(X):\n", + " delta = 10e-4\n", + " return np.where(X > np.zeros(X.shape), X, delta * X)\n", + "\n", + "\n", + "def derivate(func):\n", + " if func.__name__ == \"RELU\":\n", + "\n", + " def func(X):\n", + " return np.where(X > 0, 1, 0)\n", + "\n", + " return func\n", + "\n", + " elif func.__name__ == \"LRELU\":\n", + "\n", + " def func(X):\n", + " delta = 10e-4\n", + " return np.where(X > 0, 1, delta)\n", + "\n", + " return func\n", + "\n", + " else:\n", + " return elementwise_grad(func)" + ] + }, + { + "cell_type": "markdown", + "id": "c0189342", + "metadata": {}, + "source": [ + "Below follows a short demonstration of how to use an activation\n", + "function. The derivative of the activation function will be important\n", + "when calculating the output delta term during backpropagation. Note\n", + "that derivate() can also be used for cost functions for a more\n", + "generalized approach." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "640aa861", + "metadata": {}, + "outputs": [], + "source": [ + "z = np.array([[4, 5, 6]]).T\n", + "print(f\"Input to activation function:\\n{z}\")\n", + "\n", + "act_func = sigmoid\n", + "a = act_func(z)\n", + "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n", + "\n", + "act_func_derivative = derivate(act_func)\n", + "valued_at_z = act_func_derivative(a)\n", + "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")" + ] + }, + { + "cell_type": "markdown", + "id": "1007ccdd", + "metadata": {}, + "source": [ + "### The Neural Network\n", + "\n", + "Now that we have gotten a good understanding of the implementation of\n", + "some important components, we can take a look at an object oriented\n", + "implementation of a feed forward neural network. The feed forward\n", + "neural network has been implemented as a class named FFNN, which can\n", + "be initiated as a regressor or classifier dependant on the choice of\n", + "cost function. The FFNN can have any number of input nodes, hidden\n", + "layers with any amount of hidden nodes, and any amount of output nodes\n", + "meaning it can perform multiclass classification as well as binary\n", + "classification and regression problems. Although there is a lot of\n", + "code present, it makes for an easy to use and generalizeable interface\n", + "for creating many types of neural networks as will be demonstrated\n", + "below." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "9584a2da", + "metadata": {}, + "outputs": [], + "source": [ + "import math\n", + "import autograd.numpy as np\n", + "import sys\n", + "import warnings\n", + "from autograd import grad, elementwise_grad\n", + "from random import random, seed\n", + "from copy import deepcopy, copy\n", + "from typing import Tuple, Callable\n", + "from sklearn.utils import resample\n", + "\n", + "warnings.simplefilter(\"error\")\n", + "\n", + "\n", + "class FFNN:\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Feed Forward Neural Network with interface enabling flexible design of a\n", + " nerual networks architecture and the specification of activation function\n", + " in the hidden layers and output layer respectively. This model can be used\n", + " for both regression and classification problems, depending on the output function.\n", + "\n", + " Attributes:\n", + " ------------\n", + " I dimensions (tuple[int]): A list of positive integers, which specifies the\n", + " number of nodes in each of the networks layers. The first integer in the array\n", + " defines the number of nodes in the input layer, the second integer defines number\n", + " of nodes in the first hidden layer and so on until the last number, which\n", + " specifies the number of nodes in the output layer.\n", + " II hidden_func (Callable): The activation function for the hidden layers\n", + " III output_func (Callable): The activation function for the output layer\n", + " IV cost_func (Callable): Our cost function\n", + " V seed (int): Sets random seed, makes results reproducible\n", + " \"\"\"\n", + "\n", + " def __init__(\n", + " self,\n", + " dimensions: tuple[int],\n", + " hidden_func: Callable = sigmoid,\n", + " output_func: Callable = lambda x: x,\n", + " cost_func: Callable = CostOLS,\n", + " seed: int = None,\n", + " ):\n", + " self.dimensions = dimensions\n", + " self.hidden_func = hidden_func\n", + " self.output_func = output_func\n", + " self.cost_func = cost_func\n", + " self.seed = seed\n", + " self.weights = list()\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + " self.classification = None\n", + "\n", + " self.reset_weights()\n", + " self._set_classification()\n", + "\n", + " def fit(\n", + " self,\n", + " X: np.ndarray,\n", + " t: np.ndarray,\n", + " scheduler: Scheduler,\n", + " batches: int = 1,\n", + " epochs: int = 100,\n", + " lam: float = 0,\n", + " X_val: np.ndarray = None,\n", + " t_val: np.ndarray = None,\n", + " ):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " This function performs the training the neural network by performing the feedforward and backpropagation\n", + " algorithm to update the networks weights.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray) : training data\n", + " II t (np.ndarray) : target data\n", + " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n", + " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " V batches (int) : number of batches the datasets are split into, default equal to 1\n", + " VI epochs (int) : number of iterations used to train the network, default equal to 100\n", + " VII lam (float) : regularization hyperparameter lambda\n", + " VIII X_val (np.ndarray) : validation set\n", + " IX t_val (np.ndarray) : validation target set\n", + "\n", + " Returns:\n", + " ------------\n", + " I scores (dict) : A dictionary containing the performance metrics of the model.\n", + " The number of the metrics depends on the parameters passed to the fit-function.\n", + "\n", + " \"\"\"\n", + "\n", + " # setup \n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " val_set = False\n", + " if X_val is not None and t_val is not None:\n", + " val_set = True\n", + "\n", + " # creating arrays for score metrics\n", + " train_errors = np.empty(epochs)\n", + " train_errors.fill(np.nan)\n", + " val_errors = np.empty(epochs)\n", + " val_errors.fill(np.nan)\n", + "\n", + " train_accs = np.empty(epochs)\n", + " train_accs.fill(np.nan)\n", + " val_accs = np.empty(epochs)\n", + " val_accs.fill(np.nan)\n", + "\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + "\n", + " batch_size = X.shape[0] // batches\n", + "\n", + " X, t = resample(X, t)\n", + "\n", + " # this function returns a function valued only at X\n", + " cost_function_train = self.cost_func(t)\n", + " if val_set:\n", + " cost_function_val = self.cost_func(t_val)\n", + "\n", + " # create schedulers for each weight matrix\n", + " for i in range(len(self.weights)):\n", + " self.schedulers_weight.append(copy(scheduler))\n", + " self.schedulers_bias.append(copy(scheduler))\n", + "\n", + " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n", + "\n", + " try:\n", + " for e in range(epochs):\n", + " for i in range(batches):\n", + " # allows for minibatch gradient descent\n", + " if i == batches - 1:\n", + " # If the for loop has reached the last batch, take all thats left\n", + " X_batch = X[i * batch_size :, :]\n", + " t_batch = t[i * batch_size :, :]\n", + " else:\n", + " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n", + " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n", + "\n", + " self._feedforward(X_batch)\n", + " self._backpropagate(X_batch, t_batch, lam)\n", + "\n", + " # reset schedulers for each epoch (some schedulers pass in this call)\n", + " for scheduler in self.schedulers_weight:\n", + " scheduler.reset()\n", + "\n", + " for scheduler in self.schedulers_bias:\n", + " scheduler.reset()\n", + "\n", + " # computing performance metrics\n", + " pred_train = self.predict(X)\n", + " train_error = cost_function_train(pred_train)\n", + "\n", + " train_errors[e] = train_error\n", + " if val_set:\n", + " \n", + " pred_val = self.predict(X_val)\n", + " val_error = cost_function_val(pred_val)\n", + " val_errors[e] = val_error\n", + "\n", + " if self.classification:\n", + " train_acc = self._accuracy(self.predict(X), t)\n", + " train_accs[e] = train_acc\n", + " if val_set:\n", + " val_acc = self._accuracy(pred_val, t_val)\n", + " val_accs[e] = val_acc\n", + "\n", + " # printing progress bar\n", + " progression = e / epochs\n", + " print_length = self._progress_bar(\n", + " progression,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " except KeyboardInterrupt:\n", + " # allows for stopping training at any point and seeing the result\n", + " pass\n", + "\n", + " # visualization of training progression (similiar to tensorflow progression bar)\n", + " sys.stdout.write(\"\\r\" + \" \" * print_length)\n", + " sys.stdout.flush()\n", + " self._progress_bar(\n", + " 1,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " sys.stdout.write(\"\")\n", + "\n", + " # return performance metrics for the entire run\n", + " scores = dict()\n", + "\n", + " scores[\"train_errors\"] = train_errors\n", + "\n", + " if val_set:\n", + " scores[\"val_errors\"] = val_errors\n", + "\n", + " if self.classification:\n", + " scores[\"train_accs\"] = train_accs\n", + "\n", + " if val_set:\n", + " scores[\"val_accs\"] = val_accs\n", + "\n", + " return scores\n", + "\n", + " def predict(self, X: np.ndarray, *, threshold=0.5):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs prediction after training of the network has been finished.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n", + " in classification problems\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " This vector is thresholded if regression=False, meaning that classification results\n", + " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n", + "\n", + " \"\"\"\n", + "\n", + " predict = self._feedforward(X)\n", + "\n", + " if self.classification:\n", + " return np.where(predict > threshold, 1, 0)\n", + " else:\n", + " return predict\n", + "\n", + " def reset_weights(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Resets/Reinitializes the weights in order to train the network for a new problem.\n", + "\n", + " \"\"\"\n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " self.weights = list()\n", + " for i in range(len(self.dimensions) - 1):\n", + " weight_array = np.random.randn(\n", + " self.dimensions[i] + 1, self.dimensions[i + 1]\n", + " )\n", + " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n", + "\n", + " self.weights.append(weight_array)\n", + "\n", + " def _feedforward(self, X: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates the activation of each layer starting at the input and ending at the output.\n", + " Each following activation is calculated from a weighted sum of each of the preceeding\n", + " activations (except in the case of the input layer).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " \"\"\"\n", + "\n", + " # reset matrices\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + "\n", + " # if X is just a vector, make it into a matrix\n", + " if len(X.shape) == 1:\n", + " X = X.reshape((1, X.shape[0]))\n", + "\n", + " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n", + " # to add bias to our data\n", + " bias = np.ones((X.shape[0], 1)) * 0.01\n", + " X = np.hstack([bias, X])\n", + "\n", + " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n", + " # exponent indicates layer number).\n", + " a = X\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(a)\n", + "\n", + " # The feed forward algorithm\n", + " for i in range(len(self.weights)):\n", + " if i < len(self.weights) - 1:\n", + " z = a @ self.weights[i]\n", + " self.z_matrices.append(z)\n", + " a = self.hidden_func(z)\n", + " # bias column again added to the data here\n", + " bias = np.ones((a.shape[0], 1)) * 0.01\n", + " a = np.hstack([bias, a])\n", + " self.a_matrices.append(a)\n", + " else:\n", + " try:\n", + " # a^L, the nodes in our output layers\n", + " z = a @ self.weights[i]\n", + " a = self.output_func(z)\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(z)\n", + " except Exception as OverflowError:\n", + " print(\n", + " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n", + " )\n", + "\n", + " # this will be a^L\n", + " return a\n", + "\n", + " def _backpropagate(self, X, t, lam):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs the backpropagation algorithm. In other words, this method\n", + " calculates the gradient of all the layers starting at the\n", + " output layer, and moving from right to left accumulates the gradient until\n", + " the input layer is reached. Each layers respective weights are updated while\n", + " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each.\n", + " II t (np.ndarray): The target vector, with n rows of p targets.\n", + " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n", + "\n", + " Returns:\n", + " ------------\n", + " No return value.\n", + "\n", + " \"\"\"\n", + " out_derivative = derivate(self.output_func)\n", + " hidden_derivative = derivate(self.hidden_func)\n", + "\n", + " for i in range(len(self.weights) - 1, -1, -1):\n", + " # delta terms for output\n", + " if i == len(self.weights) - 1:\n", + " # for multi-class classification\n", + " if (\n", + " self.output_func.__name__ == \"softmax\"\n", + " ):\n", + " delta_matrix = self.a_matrices[i + 1] - t\n", + " # for single class classification\n", + " else:\n", + " cost_func_derivative = grad(self.cost_func(t))\n", + " delta_matrix = out_derivative(\n", + " self.z_matrices[i + 1]\n", + " ) * cost_func_derivative(self.a_matrices[i + 1])\n", + "\n", + " # delta terms for hidden layer\n", + " else:\n", + " delta_matrix = (\n", + " self.weights[i + 1][1:, :] @ delta_matrix.T\n", + " ).T * hidden_derivative(self.z_matrices[i + 1])\n", + "\n", + " # calculate gradient\n", + " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n", + " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n", + " 1, delta_matrix.shape[1]\n", + " )\n", + "\n", + " # regularization term\n", + " gradient_weights += self.weights[i][1:, :] * lam\n", + "\n", + " # use scheduler\n", + " update_matrix = np.vstack(\n", + " [\n", + " self.schedulers_bias[i].update_change(gradient_bias),\n", + " self.schedulers_weight[i].update_change(gradient_weights),\n", + " ]\n", + " )\n", + "\n", + " # update weights and bias\n", + " self.weights[i] -= update_matrix\n", + "\n", + " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates accuracy of given prediction to target\n", + "\n", + " Parameters:\n", + " ------------\n", + " I prediction (np.ndarray): vector of predicitons output network\n", + " (1s and 0s in case of classification, and real numbers in case of regression)\n", + " II target (np.ndarray): vector of true values (What the network ideally should predict)\n", + "\n", + " Returns:\n", + " ------------\n", + " A floating point number representing the percentage of correctly classified instances.\n", + " \"\"\"\n", + " assert prediction.size == target.size\n", + " return np.average((target == prediction))\n", + " def _set_classification(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Decides if FFNN acts as classifier (True) og regressor (False),\n", + " sets self.classification during init()\n", + " \"\"\"\n", + " self.classification = False\n", + " if (\n", + " self.cost_func.__name__ == \"CostLogReg\"\n", + " or self.cost_func.__name__ == \"CostCrossEntropy\"\n", + " ):\n", + " self.classification = True\n", + "\n", + " def _progress_bar(self, progression, **kwargs):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Displays progress of training\n", + " \"\"\"\n", + " print_length = 40\n", + " num_equals = int(progression * print_length)\n", + " num_not = print_length - num_equals\n", + " arrow = \">\" if num_equals > 0 else \"\"\n", + " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n", + " perc_print = self._format(progression * 100, decimals=5)\n", + " line = f\" {bar} {perc_print}% \"\n", + "\n", + " for key in kwargs:\n", + " if not np.isnan(kwargs[key]):\n", + " value = self._format(kwargs[key], decimals=4)\n", + " line += f\"| {key}: {value} \"\n", + " sys.stdout.write(\"\\r\" + line)\n", + " sys.stdout.flush()\n", + " return len(line)\n", + "\n", + " def _format(self, value, decimals=4):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Formats decimal numbers for progress bar\n", + " \"\"\"\n", + " if value > 0:\n", + " v = value\n", + " elif value < 0:\n", + " v = -10 * value\n", + " else:\n", + " v = 1\n", + " n = 1 + math.floor(math.log10(v))\n", + " if n >= decimals - 1:\n", + " return str(round(value))\n", + " return f\"{value:.{decimals-n-1}f}\"" + ] + }, + { + "cell_type": "markdown", + "id": "9ccd1fc1", + "metadata": {}, + "source": [ + "Before we make a model, we will quickly generate a dataset we can use\n", + "for our linear regression problem as shown below" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "7f3a5b31", + "metadata": {}, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "def SkrankeFunction(x, y):\n", + " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n", + "\n", + "def create_X(x, y, n):\n", + " if len(x.shape) > 1:\n", + " x = np.ravel(x)\n", + " y = np.ravel(y)\n", + "\n", + " N = len(x)\n", + " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n", + " X = np.ones((N, l))\n", + "\n", + " for i in range(1, n + 1):\n", + " q = int((i) * (i + 1) / 2)\n", + " for k in range(i + 1):\n", + " X[:, q + k] = (x ** (i - k)) * (y**k)\n", + "\n", + " return X\n", + "\n", + "step=0.5\n", + "x = np.arange(0, 1, step)\n", + "y = np.arange(0, 1, step)\n", + "x, y = np.meshgrid(x, y)\n", + "target = SkrankeFunction(x, y)\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "poly_degree=3\n", + "X = create_X(x, y, poly_degree)\n", + "\n", + "X_train, X_test, t_train, t_test = train_test_split(X, target)" + ] + }, + { + "cell_type": "markdown", + "id": "1ac05bb6", + "metadata": {}, + "source": [ + "Now that we have our dataset ready for the regression, we can create\n", + "our regressor. Note that with the seed parameter, we can make sure our\n", + "results stay the same every time we run the neural network. For\n", + "inititialization, we simply specify the dimensions (we wish the amount\n", + "of input nodes to be equal to the datapoints, and the output to\n", + "predict one value)." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "0f857604", + "metadata": {}, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "eeff4315", + "metadata": {}, + "source": [ + "We then fit our model with our training data using the scheduler of our choice." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "46246810", + "metadata": {}, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Constant(eta=1e-3)\n", + "scores = linear_regression.fit(X_train, t_train, scheduler)" + ] + }, + { + "cell_type": "markdown", + "id": "8c6f9954", + "metadata": {}, + "source": [ + "Due to the progress bar we can see the MSE (train_error) throughout\n", + "the FFNN's training. Note that the fit() function has some optional\n", + "parameters with defualt arguments. For example, the regularization\n", + "hyperparameter can be left ignored if not needed, and equally the FFNN\n", + "will by default run for 100 epochs. These can easily be changed, such\n", + "as for example:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "2661939c", + "metadata": {}, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "74c5624c", + "metadata": {}, + "source": [ + "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", + "\n", + "Let us then switch to a binary classification. We use a binary\n", + "classification dataset, and follow a similar setup to the regression\n", + "case." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "4b8eb115", + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.datasets import load_breast_cancer\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "\n", + "wisconsin = load_breast_cancer()\n", + "X = wisconsin.data\n", + "target = wisconsin.target\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "X_train, X_val, t_train, t_val = train_test_split(X, target)\n", + "\n", + "scaler = MinMaxScaler()\n", + "scaler.fit(X_train)\n", + "X_train = scaler.transform(X_train)\n", + "X_val = scaler.transform(X_val)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "2c0f92bd", + "metadata": {}, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "49201ae4", + "metadata": {}, + "source": [ + "We will now make use of our validation data by passing it into our fit function as a keyword argument" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "55b5e426", + "metadata": {}, + "outputs": [], + "source": [ + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "b51762fb", + "metadata": {}, + "source": [ + "Finally, we will create a neural network with 2 hidden layers with activation functions." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "6b59e27d", + "metadata": {}, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 1\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "72c87921", + "metadata": {}, + "outputs": [], + "source": [ + "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "ed40d7d2", + "metadata": {}, + "source": [ + "### Multiclass classification\n", + "\n", + "Finally, we will demonstrate the use case of multiclass classification\n", + "using our FFNN with the famous MNIST dataset, which contain images of\n", + "digits between the range of 0 to 9." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "315ef3fe", + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.datasets import load_digits\n", + "\n", + "def onehot(target: np.ndarray):\n", + " onehot = np.zeros((target.size, target.max() + 1))\n", + " onehot[np.arange(target.size), target] = 1\n", + " return onehot\n", + "\n", + "digits = load_digits()\n", + "\n", + "X = digits.data\n", + "target = digits.target\n", + "target = onehot(target)\n", + "\n", + "input_nodes = 64\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 10\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n", + "\n", + "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = multiclass.fit(X, target, scheduler, epochs=1000)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/LectureNotes/exercisesweek43.ipynb b/doc/LectureNotes/exercisesweek43.ipynb index 438de41c2..ff4eb9b4f 100644 --- a/doc/LectureNotes/exercisesweek43.ipynb +++ b/doc/LectureNotes/exercisesweek43.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "b2937d10", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "3dd00d19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Exercises weeks 43 and 44 \n", "**October 23-27, 2023**\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "82a19a1d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Overarching aims of the exercises weeks 43 and 44\n", "\n", @@ -70,9 +64,7 @@ { "cell_type": "markdown", "id": "f74f69af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The AND and XOR Gates\n", "\n", @@ -108,9 +100,7 @@ { "cell_type": "markdown", "id": "1b52d47a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Representing the Data Sets\n", "\n", @@ -120,9 +110,7 @@ { "cell_type": "markdown", "id": "3e6910cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n", @@ -135,9 +123,7 @@ { "cell_type": "markdown", "id": "90a3b78a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n", "\n", @@ -165,9 +151,7 @@ { "cell_type": "markdown", "id": "d6a3ab1e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Setting up the Neural Network\n", "\n", @@ -176,13 +160,22 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 3, "id": "152123b0", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[0.80625657 0.36420967]\n", + " [0.90297441 0.30170017]\n", + " [0.89823921 0.28566769]\n", + " [0.93420126 0.25920793]]\n", + "[0 0 0 0]\n" + ] + } + ], "source": [ "%matplotlib inline\n", "\n", @@ -256,9 +249,7 @@ { "cell_type": "markdown", "id": "73319f0a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above." ] @@ -266,9 +257,7 @@ { "cell_type": "markdown", "id": "a7e0c47a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Code using Scikit-Learn" ] @@ -277,11 +266,253 @@ "cell_type": "code", "execution_count": 2, "id": "dbbacc67", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Learning rate = 1e-05\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1e-05\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1e-05\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1e-05\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1e-05\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1e-05\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1e-05\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.0001\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.001\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.25\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.75\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 0.75\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.01\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 1.0\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 1.0\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 0.1\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1.0\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.75\n", + "\n", + "Learning rate = 1.0\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.75\n", + "\n", + "Learning rate = 1.0\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 0.75\n", + "\n", + "Learning rate = 1.0\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 0.5\n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n", + "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", + " warnings.warn(\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Learning rate = 1.0\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1.0\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 1.0\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 1e-05\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 0.0001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 0.001\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 0.01\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 0.1\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 1.0\n", + "Accuracy score on data set: 0.5\n", + "\n", + "Learning rate = 10.0\n", + "Lambda = 10.0\n", + "Accuracy score on data set: 0.5\n", + "\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABrR0lEQVR4nO3dd3xUddbH8e9QJgMpQISQSOgEYpQSDEVpIqDriuIKPsqKC4tEmrCCCzYUUGwLigKCq1J0kQWlWigi7q4U6ShCCE1aME0gCSE9mecPJLvjBBhg7r2Z5PN+XvN6mN/cO/fk7BA5c8691+Z0Op0CAAAAABNUsDoAAAAAAOUHBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAICBuNcrAACuKlkdAACY6emnn9ayZcsuuU2dOnX0zTffXPOxPv30Ux0+fFhPP/30ZbddvHixnnvuOXXq1EkffPDBNR8bAIDSyubk6zkA5cjx48d1+vTp4uczZ85UXFycZsyYUbxmt9sVFRV1zce6/fbb1bZtW7322muX3bZv377KzMzUoUOH9NVXX6lu3brXfHwAAEojOiAAypV69eqpXr16xc+Dg4Nlt9vVqlUry2I6cuSIdu7cqffee09jxozRJ598oieffNKyeAAAMBLngABACQ4cOKDBgwerdevWat26tYYPH64TJ064bPOPf/xDv/vd79S8eXN16tRJEyZMUGZmpqTz3Y+TJ09q2bJlatasmRISEi56rCVLligwMFC33HKLfve732nJkiXKy8tz227Pnj0aNGiQbr75ZrVv316jRo1SYmJi8eunTp3Ss88+q1tvvVXR0dF6+OGHtWPHjuLXmzVrpunTp7u85/Tp09WsWbPi508//bT69++v8ePHKyYmRn/4wx9UUFCg06dPa+LEieratatuuukmtW3bVsOHD3f7ub788kvdf//9atmypW677TZNnjxZeXl5OnjwoJo1a6ZFixa5bJ+cnKwbbrjhsmNxAICygwIEAH7jyJEjeuihh3Tq1Cm99tprevnll3XixAn17dtXp06dknT+H9qvv/66Hn74Yc2ePVvDhw/XihUrNGnSJEnSjBkzVKtWLXXp0kWLFi1SSEhIiccqLCzUihUr9Pvf/152u13333+/Tp06pa+//tplu/j4ePXt21fZ2dl67bXX9OKLLyouLk4DBw5Ufn6+srKy9NBDD2nTpk168sknNWPGDPn7+2vQoEE6fPjwFf3827dv17FjxzR9+nQNHz5cFStW1ODBg7Vx40Y9+eSTmj17toYNG6ZNmzbphRdeKN5v4cKFGj16tG644QbNmDFDgwcP1oIFCzRhwgRFRESoZcuWWrFihcuxVqxYIYfDoTvvvPOKYgQA+C5GsADgN2bMmCGHw6F58+YpICBAknTLLbeoe/fu+uCDD/TUU09py5YtqlOnjh5++GFVqFBBbdu2VdWqVXXmzBlJUlRUlOx2u4KDgy853vXtt98qJSVFvXv3liS1atVKTZo00T//+U/9/ve/L95u5syZqlatmubMmSM/Pz9JUmhoqJ544gnt379fP/zwg06cOKHly5crMjJSkhQTE6P77rtP27ZtU+PGjT3++QsKCjRx4kTVr19f0vkuRZUqVfTUU08pJiZGktSuXTslJCRo4cKFkqSioiJNnz5dPXr00Msvv1z8Xrm5uVq2bJny8vLUu3dvvfDCCzpx4kTxOS7Lly/XXXfdpapVq3ocHwDAt9EBAYDf2Lx5s9q1ayeHw6GCggIVFBQoICBAMTEx2rRpkySpffv2Onr0qO6///7iE9nvuece9e/f/4qOtWTJEtWvX18NGzZURkaGMjIydNddd2nr1q0unYsdO3aoc+fOxcWHJLVo0ULffPONbrrpJm3fvl3h4eHFxYck+fn5adWqVXrooYeuKCaHw+Fynkzt2rX10UcfKSYmRj///LO+++47zZ8/Xzt37lR+fr6k812jX375Rd27d3d5rwEDBmjFihWy2+26++67VaVKleIuyO7du3X48GHdf//9VxQfAMC30QEBgN9IS0vTypUrtXLlSrfXgoODJUm///3vVVRUpAULFmjGjBl6++23VadOHT355JO6++67PTrO6dOn9e9//1v5+flq06aN2+uLFi3Ss88+WxzTddddd8mYL/X6lbjuuutks9lc1j777DO9+eabSkxMVPXq1RUZGSmHw+Fy/Av7XkxAQIB+97vf6bPPPtPjjz+uZcuWqX79+sVdFQBA+UABAgC/ERgYqFtvvVV//vOf3V6rVOm/vzZ79uypnj176uzZs9qwYYPef/99jRkzRjExMapdu/Zlj7NixQrl5+drxowZCgoKcnntnXfe0fLlyzV69Gg5HA4FBga6XD74gv/85z+KjIxUYGBgiSe679q1SwEBAYqIiJB0/pyT/5WVlXXZOLdv366nnnpK/fr106OPPqrQ0FBJ0t/+9rfik9wvxP/bGNPS0rR37161atVK/v7+6t27t5YtW6bdu3drzZo1euSRRy57fABA2cIIFgD8Rtu2bXXo0CHdcMMNat68uZo3b66bbrpJ8+bN09q1ayVJTzzxhB5//HFJ5wuWu+66S8OGDVNhYaFSUlIkSRUqXPpX7NKlS9WqVSv16NFD7dq1c3n07dtX6enpWrVqlaTz53OsX7/e5epY+/fv12OPPaYff/xRMTExOnHihPbv31/8el5enkaMGKFPPvlE0vkORFJSkksMO3fuvGw+du3apaKiIo0cObK4+CgsLCweRysqKlKjRo1Uo0YNrVu3zmXfzz//XLGxscrNzZUktWnTRg0aNNDkyZN15swZ3XfffZc9PgCgbKEAAYDfGDZsmI4fP67Bgwfr66+/1vr16zVixAh9+eWXxedYtG/fXmvXrtXrr7+u7777TmvWrNHbb7+tBg0aFG8TFBSkuLg4bd26VTk5OS7H2L17tw4cOHDRca1u3bqpWrVqxSd5Dxs2TGfOnFFsbKy++eYbrV69Wk888YRuvPFGde7cWffff7/q1q2roUOHasWKFVq/fr1GjhypnJyc4i7Dbbfdpi+//FILFizQd999p7Fjx+rYsWOXzUeLFi0kSS+++KI2b96sr776Sn/+858VHx8v6XwXpWLFihoxYoTWrFmjCRMmaOPGjfr444/11ltvqW/fvsWja5LUu3dvbd26VbfccovCwsKu5H8aAEAZQAECAL8RGRmpjz/+WDabTWPHjtXIkSOVmpqqd955R3fccYck6aGHHtK4ceP07bffasiQIXrhhRfUuHFjzZkzR5UrV5YkDRw4UL/88oseffRR7dmzx+UYS5YsUcWKFV2udPW/7Ha77rrrLn3//ffat2+foqKi9I9//ENFRUUaNWqUXnzxRbVq1Urvv/++7Ha7AgICNH/+fEVHR+vll1/WX/7yF+Xm5uof//hH8QnlzzzzjG6//XZNnjxZI0eOVJUqVTy64WG7du30wgsvaNeuXYqNjdWrr76q66+/vvju8RfGsB5++GG99tpr2r59uwYPHqw5c+Zo4MCBevrpp13e77bbbpMkTj4HgHLK5nQ6nVYHAQAoP95//3198MEHWr9+vex2u9XhAABMxknoAABTLFu2TAcOHNCCBQv02GOPUXwAQDlFAQIAMEV8fLwWLlyo7t27KzY21upwAAAWYQQLAAAAgGbOnKnvvvtO//jHPy66zZkzZzRp0iR9++23kqTf/e53euaZZ1S1alWPj8NJ6AAAAEA5N2/ePE2bNu2y240cOVInTpwo3n7jxo2aOHHiFR2LESwAAACgnEpOTtZzzz2nHTt2qGHDhpfcdteuXdq6datWrlypxo0bSzp/ifZBgwZp9OjRHt2EV6IDAgAAAJRbe/fuVbVq1fTZZ5+pZcuWl9x2+/btqlWrVnHxIZ2/ea/NZiu+JLsn6IAAAAAAPqxbt26XfH3dunUXfe3222/X7bff7tFxkpOT3W4ga7fbVb16dSUmJnr0HlI5K0BaD51qdQgAAAC4hJ2zRlkdwkUVJTW1OoSLqGvKUbKzs0u8hLqfn59yc3M9fp9yVYAAAAAAZc2lOhze5HA4lJeX57aem5vLVbAAAAAAeFdoaKhSUlJc1vLy8pSWlubxCegSBQgAAADgkaJS+n9madOmjZKSknTs2LHitS1btkiSWrdu7fH7UIAAAAAAcFNYWKjU1FTl5ORIklq2bKnWrVtr1KhR2r17tzZv3qzx48frvvvuowMCAAAA4NokJiaqY8eOWrlypSTJZrNpxowZCg8PV//+/fXEE0+oc+fOmjBhwhW9r83pdDoNiLdU4ipYAAAApVtpvgpWbmIjq0MokV/YT1aHcEXogAAAAAAwDQUIAAAAANNwHxAAAADAA0UqN2cuGIoOCAAAAADTUIAAAAAAMA0jWAAAAIAHzLzpX1lGBwQAAACAaShAAAAAAJiGESwAAADAA4Xl5/7dhqIDAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaRjBAgAAADxQyAiWV9ABAQAAAGAaChAAAAAApmEECwAAAPAAV8HyDjogAAAAAExDAQIAAADANIxgAQAAAB4odDKC5Q10QAAAAACYhgIEAAAAgGkYwQIAAAA8UGR1AGUEHRAAAAAApqEAAQAAAGAaRrAAAAAADxRyI0KvoAMCAAAAwDQUIAAAAABMwwgWAAAA4IFCJrC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgAe4EaF30AEBAAAAYBoKEAAAAACmYQQLAAAA8EChbFaHUCbQAQEAAABgGgoQAAAAAKZhBAsAAADwQBE3IvQKOiAAAAAATEMBAgAAAMA0jGBZ5Nao+hp2bwc1DAtW2tlsLV6/W3PXbLvo9g1q19DSCQPc1o8knVbviR8aGKnvIsfGIr/GI8fGI8fGIr/GI8fm4ipY3kEBYoEWjcI0dWgvfbXjgGZ+tlGtmtTR8Hs7qILNptmrt5a4T9O6tSRJsW9+qryCguL1nLyCErcv78ixsciv8cix8cixsciv8cgxfBUFiAUG391e+xNS9fy81ZKkTXHHVKlCBQ24s43mr9uh3PxCt32ahYfo5C/p2nEwwexwfRI5Nhb5NR45Nh45Nhb5NR45hq/iHBCTVa5UUTdHhOubXYdc1r/edVD+Druim9Qpcb9m4bW0PyHVjBB9Hjk2Fvk1Hjk2Hjk2Fvk1Hjm2RqFspfLha+iAmCy8ZjXZK1fS8ZQzLusnUtMkSfVDamjzvuNu+zUNr6UjSac0b8yDalY3RGezcvX55jjN+myTCoqKzAjdZ5BjY5Ff45Fj45FjY5Ff45Fj+DLLC5CCggJ99dVX2r59u37++Wfl5eWpSpUqCg0NVUxMjHr06KFKlSwP02sCq/hJkjJz8lzWs3597l/F7rZPcGBV1azmL6fTqWnLNijxTIbaNqunAXfEqHaNAI2bu9r4wH0IOTYW+TUeOTYeOTYW+TUeOYYvs/Rf9sePH1dsbKySk5MVFRWlkJAQVatWTbm5udq3b5+WLFmi6dOn64MPPtD1119vZaheY7P92iZzlnwnm5K+fDiXk6shby/RseTTSj6TKUnaefCk8gsKNbxXB81etVVHkk4bFbLPIcfGIr/GI8fGI8fGIr/GI8fWKHL63rhTaWTpOSATJ05UeHi41q9frwULFuitt97S66+/rrfeeksLFizQt99+q7CwML344otWhulVZ7NzJUn+DtdvJqr++jwzJ9dtn9z8Qm2NP178y+KC9XuOSJIiwmsaEarPIsfGIr/GI8fGI8fGIr/GI8fwZZYWIDt27NDYsWMVGBhY4utBQUEaM2aMtm27+PWsfU1CapoKCotUN6S6y3rdWuef/5R4ym2f+rVrqHenFm6/ZByVzzew0jJzDInVV5FjY5Ff45Fj45FjY5Ff45Fj+DJLC5CgoCClpKRccpuff/5ZDofDpIiMl1dQqF2HEnR7qyYu692jI5SRlaO9R5Pc9gmpHqDn/thN3aMjXNbviGmqzOxc7TuebGjMvoYcG4v8Go8cG48cG4v8Go8cW8Pqq11xFSwv6NOnj5555hmNHDlS7dq1U1hYmOx2u/Ly8pScnKytW7dqypQp6tOnj5Vhet0Hq7Zq1sjeen3Q3Vrx3V61bBSmP/WI0bTl65WbXyh/h12NwoJ1IjVdaZnZ2nEgQdv2n9DoPp3l8Kuso0mn1al5Qz10W7TeWvqtzma5t1nLO3JsLPJrPHJsPHJsLPJrPHIMX2VzOi9y9pIJnE6n3nnnHc2dO1dZWVlur/v7++vhhx/WX/7yF1WocO3NmtZDp17ze3hL15aNNaTnLapfu4ZS0s/pk39/r/nrdkqSbo4I1/ujH9D4D9fo881xkqQAh12De96i21o2Vs1q/kpITdeCb3Zq2cY9Vv4YpRo5Nhb5NR45Nh45Nhb5NV5ZzPHOWaOsDuGidh6vZ3UIJWpdz/2Sy6WZpQXIBfn5+dq3b5+Sk5OVnZ0th8Oh0NBQRUZGym53v4zc1SpNBQgAAADcleYCZNvxBlaHUKI29Y5aHcIVKRU32KhcubJatGhhdRgAAAAADGbpSegAAAAAypdS0QEBAAAASjtuROgddEAAAAAAmIYCBAAAAIBpGMECAAAAPOCLN/0rjeiAAAAAADANBQgAAAAA0zCCBQAAAHig0Ml3995AFgEAAACYhgIEAAAAgGkYwQIAAAA8UMR3915BFgEAAACYhgIEAAAAgGkYwQIAAAA8wI0IvYMOCAAAAADTUIAAAAAAMA0jWAAAAIAHuBGhd5BFAAAAAKahAAEAAABgGkawAAAAAA8UcRUsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOCBQr679wqyCAAAAMA0FCAAAAAATMMIFgAAAOABbkToHWQRAAAAgGkoQAAAAACYhhEsAAAAwANFfHfvFWQRAAAAgGkoQAAAAACYhhEsAAAAwAOFTpvVIZQJdEAAAAAAmIYCBAAAAIBpKEAAAAAAmIZzQAAAAAAPFPLdvVeQRQAAAKCcKioq0rRp09SpUye1bNlSAwcO1LFjxy66/YkTJzRkyBC1bdtWHTp00KRJk5SdnX1Fx6QAAQAAAMqpmTNnauHChZo0aZIWLVokm82m2NhY5eXluW179uxZ9e3bV+np6frggw/07rvvas+ePRo+fPgVHZMCBAAAAPBAkbNCqXxcrby8PM2ZM0cjRoxQly5dFBkZqalTpyo5OVlr1651237ZsmXKzMzUO++8oxYtWqh58+aaOnWqNm3apO3bt3t8XAoQAAAAoByKj4/XuXPn1L59++K1oKAgRUVFadu2bW7bHzlyRI0aNVJwcHDxWlhYmGrUqKGtW7d6fFxOQgcAAAB8WLdu3S75+rp160pcT0pKknS+iPhfISEhSkxMdNu+Vq1aSk1NVWFhoSpWrChJyszMVHp6uk6dOuVxvHRAAAAAAA8UqkKpfFytCyeP2+12l3U/Pz/l5ua6bX/33XcrPT1dr7zyis6dO6eMjAyNHz9eNputxHNGLoYOCAAAAODDLtbhuByHwyHp/LkgF/4sSbm5uapSpYrb9vXr19f06dP1wgsv6OOPP5bD4dAjjzyim266SQEBAR4flwIEAAAAKIcujF6lpKSoXr16xespKSmKjIwscZ8uXbroP//5j1JTUxUYGCiHw6Fbb71V999/v8fHZQQLAAAA8ECh01YqH1crMjJSAQEB2rJlS/FaRkaG4uLiFBMT47b9jh071K9fP+Xl5alWrVpyOBzaunWrzpw5o1tvvdXj49IBAQAAAMohu92ufv36acqUKQoODladOnU0efJkhYaGqkePHiosLNTp06eLOx2NGzfWwYMH9corr+jRRx/ViRMnNHbsWD300EOqW7eux8elAAEAAADKqZEjR6qgoEDjxo1TTk6O2rRpo9mzZ8tutyshIUHdunXTq6++qvvvv1/Vq1fXe++9p1dffVX33HOPatSooYceekhDhw69omPanE6n06Cfp9RpPXSq1SEAAADgEnbOGmV1CBf14UHPx4zM1D9ik9UhXBHOAQEAAABgGgoQAAAAAKbhHBAAAADAA4VOvrv3BrIIAAAAwDQUIAAAAABMwwgWAAAA4IEiXf1N//BfdEAAAAAAmIYCBAAAAIBpGMECAAAAPMBVsLyDLAIAAAAwDQUIAAAAANMwggUAAAB4oJDv7r2CLAIAAAAwDQUIAAAAANMwggUAAAB4oMjJjQi9gQ4IAAAAANNQgAAAAAAwDSNYAAAAgAe4CpZ3kEUAAAAApqEAAQAAAGAaRrAAAAAADxQ5+e7eG8giAAAAANNQgAAAAAAwDSNYAAAAgAcKxY0IvYEOCAAAAADTUIAAAAAAMA0jWAAAAIAHuAqWd5BFAAAAAKahAAEAAABgGkawAAAAAA9wFSzvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AGuguUdZBEAAACAaShAAAAAAJiGESwAAADAA4WMYHkFWQQAAABgGgoQAAAAAKZhBAsAAADwQBE3IvQKOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BFgEAAACYhgIEAAAAgGkYwQIAAAA8UOTkKljeQAcEAAAAgGkoQAAAAACYhhEsAAAAwAOFfHfvFRQgFrk1qr6G3dtBDcOClXY2W4vX79bcNdsuun2D2jW0dMIAt/UjSafVe+KHBkbqu8ixsciv8cix8cixsciv8cgxfBEFiAVaNArT1KG99NWOA5r52Ua1alJHw+/toAo2m2av3lriPk3r1pIkxb75qfIKCorXc/IKSty+vCPHxiK/xiPHxiPHxiK/xiPH8FUUIBYYfHd77U9I1fPzVkuSNsUdU6UKFTTgzjaav26HcvML3fZpFh6ik7+ka8fBBLPD9Unk2Fjk13jk2Hjk2Fjk13jk2HxcBcs7GGQzWeVKFXVzRLi+2XXIZf3rXQfl77ArukmdEvdrFl5L+xNSzQjR55FjY5Ff45Fj45FjY5Ff45Fj+DI6ICYLr1lN9sqVdDzljMv6idQ0SVL9kBravO+4235Nw2vpSNIpzRvzoJrVDdHZrFx9vjlOsz7bpIKiIjNC9xnk2Fjk13jk2Hjk2Fjk13jkGL6MAsRkgVX8JEmZOXku61m/PvevYnfbJziwqmpW85fT6dS0ZRuUeCZDbZvV04A7YlS7RoDGzV1tfOA+hBwbi/wajxwbjxwbi/wajxxbo4jhIa+gADGZzfbr7KDTWeLrJX35cC4nV0PeXqJjyaeVfCZTkrTz4EnlFxRqeK8Omr1qq44knTYqZJ9Djo1Ffo1Hjo1Hjo1Ffo1HjuHLKONMdjY7V5Lk73D9ZqLqr88zc3Ld9snNL9TW+OPFvywuWL/niCQpIrymEaH6LHJsLPJrPHJsPHJsLPJrPHIMX2Z5B+SRRx75bxV/GR999JHB0RgvITVNBYVFqhtS3WW9bq3zz39KPOW2T/3aNRTTtK5Wb4vXuf9ptToqn/+fLy0zx7B4fRE5Nhb5NR45Nh45Nhb5NR45tkYhV8HyCss7ILfccou2bdumU6dOqU6dOpd8lAV5BYXadShBt7dq4rLePTpCGVk52ns0yW2fkOoBeu6P3dQ9OsJl/Y6YpsrMztW+48mGxuxryLGxyK/xyLHxyLGxyK/xyDF8meUdkGHDhqlq1aqaNm2a/v73vys8PNzqkAz3waqtmjWyt14fdLdWfLdXLRuF6U89YjRt+Xrl5hfK32FXo7BgnUhNV1pmtnYcSNC2/Sc0uk9nOfwq62jSaXVq3lAP3Ratt5Z+q7NZ7m3W8o4cG4v8Go8cG48cG4v8Go8cw1fZnM6LnL1kskGDBql69eqaMmWKYcdoPXSqYe99pbq2bKwhPW9R/do1lJJ+Tp/8+3vNX7dTknRzRLjeH/2Axn+4Rp9vjpMkBTjsGtzzFt3WsrFqVvNXQmq6FnyzU8s27rHyxyjVyLGxyK/xyLHxyLGxyK/xymKOd84aZXUIF/WXXX2tDqFEb0f/0+oQrkipKUCSk5MVFxenrl27GnaM0lSAAAAAwB0FyJXztQLE8hGsC2rXrq3atWtbHQYAAAAAA5WaAgQAAAAozYqcll+/qUwgiwAAAABMQwECAAAAwDSMYAEAAAAeKBQ3IvQGOiAAAAAATEMBAgAAAMA0jGABAAAAHihyMoLlDXRAAAAAAJiGAgQAAACAaRjBAgAAADzAjQi9gywCAAAAMA0FCAAAAADTMIIFAAAAeKCIGxF6BR0QAAAAAKahAAEAAABgGkawAAAAAA8UciNCr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHWQRAAAAgGkoQAAAAACYhhEsAAAAwANFXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHigSIxgeQMdEAAAAACmoQABAAAAYBpGsAAAAAAPcBUs76ADAgAAAJRTRUVFmjZtmjp16qSWLVtq4MCBOnbs2EW3T01N1ejRo9WuXTu1a9dOf/nLX5SUlHRFx6QAAQAAAMqpmTNnauHChZo0aZIWLVokm82m2NhY5eXllbj9qFGjlJiYqLlz52ru3LlKSkrSsGHDruiYFCAAAACAB4qcFUrl42rl5eVpzpw5GjFihLp06aLIyEhNnTpVycnJWrt2rdv2GRkZ2rZtm2JjYxUVFaWoqCg99thj2rt3r86cOePxcSlAAAAAgHIoPj5e586dU/v27YvXgoKCFBUVpW3btrlt7+fnp6pVq2r58uXKzMxUZmamVqxYoQYNGqhatWoeH5eT0AEAAIBy6MK5G2FhYS7rISEhSkxMdNvez89PL7/8sl588UXFxMTIZrOpVq1amj9/vipU8LyvQQECAAAAeKC0XgWrW7dul3x93bp1Ja5nZ2dLkux2u8u6n5+f0tPT3bZ3Op3av3+/oqOjNWjQIBUWFmrq1KkaPny4/vnPfyogIMCjeClAAAAAgHLI4XBIOn8uyIU/S1Jubq6qVKnitv2XX36pBQsW6F//+ldxsfHuu++qa9euWrJkifr37+/RcSlAAAAAAB92sQ7H5VwYvUpJSVG9evWK11NSUhQZGem2/Y4dO9SwYUOXTke1atXUsGFDHT161OPjchI6AAAA4IEi2Url42pFRkYqICBAW7ZsKV7LyMhQXFycYmJi3LYPCwvTsWPHlJubW7yWnZ2thIQE1a9f3+PjUoAAAAAA5ZDdble/fv00ZcoUrVu3TvHx8Ro1apRCQ0PVo0cPFRYWKjU1VTk5OZKk++67T5L0xBNPKD4+vnh7u92u+++/3+PjUoAAAAAA5dTIkSPVp08fjRs3Tn379lXFihU1e/Zs2e12JSYmqmPHjlq5cqWk81fHWrBggZxOp/r3768///nPqly5sv75z38qKCjI42PanE6n06gfqLRpPXSq1SEAAADgEnbOGmV1CBf1h43DrQ6hRMs6vGN1CFeEDggAAAAA01CAAAAAADANl+EFAAAAPFBab0Toa+iAAAAAADANBQgAAAAA0zCCBQAAAHiAESzvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AFGsLyDDggAAAAA01CAAAAAADANI1gAAACAB4rECJY30AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAHmAEyzvogAAAAAAwDQUIAAAAANMwggUAAAB4gBEs76ADAgAAAMA0FCAAAAAATMMIFgAAAOABRrC8gw4IAAAAANNQgAAAAAAwDSNYAAAAgAecjGB5BR0QAAAAAKahAAEAAABgGkawAAAAAA8UiREsb6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaRjBAgAAADzAjQi9gw4IAAAAANNQgAAAAAAwDSNYAAAAgAe4CpZ30AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BBwQAAACAaShAAAAAAJiGESwAAADAA06n1RGUDXRAAAAAAJiGAgQAAACAaRjBAgAAADxQJK6C5Q10QAAAAACYhgIEAAAAgGkYwQIAAAA84ORGhF5BBwQAAACAaShAAAAAAJiGESyL3BpVX8Pu7aCGYcFKO5utxet3a+6abRfdvkHtGlo6YYDb+pGk0+o98UMDI/Vd5NhY5Nd45Nh45NhY5Nd45NhcRYxgeQUFiAVaNArT1KG99NWOA5r52Ua1alJHw+/toAo2m2av3lriPk3r1pIkxb75qfIKCorXc/IKSty+vCPHxiK/xiPHxiPHxiK/xiPH8FUUIBYYfHd77U9I1fPzVkuSNsUdU6UKFTTgzjaav26HcvML3fZpFh6ik7+ka8fBBLPD9Unk2Fjk13jk2Hjk2Fjk13jkGL6Kc0BMVrlSRd0cEa5vdh1yWf9610H5O+yKblKnxP2ahdfS/oRUM0L0eeTYWOTXeOTYeOTYWOTXeOTYGk5n6Xz4GjogJguvWU32ypV0POWMy/qJ1DRJUv2QGtq877jbfk3Da+lI0inNG/OgmtUN0dmsXH2+OU6zPtukgqIiM0L3GeTYWOTXeOTYeOTYWOTXeOQYvszyAuTIkSP64osvlJ6erk6dOqlLly4ur2dmZurll1/Wq6++alGE3hVYxU+SlJmT57Ke9etz/yp2t32CA6uqZjV/OZ1OTVu2QYlnMtS2WT0NuCNGtWsEaNzc1cYH7kPIsbHIr/HIsfHIsbHIr/HIMXyZpQXIjh079Oijj6p27dpyOp36+OOP1b17d73xxhuy28//xcnJydHy5cvLTAFis/169YSL9MtK+vLhXE6uhry9RMeSTyv5TKYkaefBk8ovKNTwXh00e9VWHUk6bVTIPoccG4v8Go8cG48cG4v8Go8cW4MbEXqHpeeAvPHGG+rTp4/WrFmjr776Sm+++aY2btyoIUOGKD8/38rQDHM2O1eS5O9w/Wai6q/PM3Ny3fbJzS/U1vjjxb8sLli/54gkKSK8phGh+ixybCzyazxybDxybCzyazxyDF9maQGyf/9+9evXr/j5XXfdpffff1+7du3S2LFjLYzMOAmpaSooLFLdkOou63VrnX/+U+Ipt33q166h3p1auP2ScVQ+38BKy8wxJFZfRY6NRX6NR46NR46NRX6NR47hyywtQAICAnTmjOvJUzfffLMmT56sNWvWlJmxq/+VV1CoXYcSdHurJi7r3aMjlJGVo71Hk9z2CakeoOf+2E3doyNc1u+IaarM7FztO55saMy+hhwbi/wajxwbjxwbi/wajxxbw+m0lcqHr7H0HJAuXbroxRdf1IQJExQVFaXKlStLkrp3765nn31WkyZNUmJiopUhGuKDVVs1a2RvvT7obq34bq9aNgrTn3rEaNry9crNL5S/w65GYcE6kZqutMxs7TiQoG37T2h0n85y+FXW0aTT6tS8oR66LVpvLf1WZ7Pc26zlHTk2Fvk1Hjk2Hjk2Fvk1HjmGr7I5ndZdPTg9PV2jRo3Sd999p7///e/q3Lmzy+sLFizQK6+8osLCQu3bt++aj9d66NRrfg9v6dqysYb0vEX1a9dQSvo5ffLv7zV/3U5J0s0R4Xp/9AMa/+Eafb45TpIU4LBrcM9bdFvLxqpZzV8Jqela8M1OLdu4x8ofo1Qjx8Yiv8Yjx8Yjx8Yiv8YrizneOWuU1SFcVPPPxlsdQol+vHei1SFcEUsLkAuOHz+uGjVqKDAw0O21I0eO6KuvvtLgwYOv+TilqQABAACAu9JcgNy4YoLVIZRob68JVodwRSy/D4gk1atX76KvNWzY0CvFBwAAAADrWXoSOgAAAIDypVR0QAAAAIDSzvoTF8oGOiAAAAAATEMBAgAAAMA0jGABAAAAHvDFm/6VRnRAAAAAAJiGAgQAAACAaRjBAgAAADzACJZ30AEBAAAAYBoKEAAAAACmYQQLAAAA8AD3IfQOOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BBwQAAACAaShAAAAAAJiGESwAAADAE1wGyyvogAAAAAAwDQUIAAAAANMwggUAAAB4gKtgeQcdEAAAAACmoQABAAAAyqmioiJNmzZNnTp1UsuWLTVw4EAdO3asxG2nT5+uZs2alfh45plnPD4mBQgAAADgAaezdD6uxcyZM7Vw4UJNmjRJixYtks1mU2xsrPLy8ty2HThwoDZs2ODyeOKJJ+RwONS/f3+Pj0kBAgAAAJRDeXl5mjNnjkaMGKEuXbooMjJSU6dOVXJystauXeu2vb+/v2rVqlX8yM7O1t///nc9/fTTioyM9Pi4FCAAAABAORQfH69z586pffv2xWtBQUGKiorStm3bLrv/a6+9poiICD344INXdFyuggUAAAB4oLReBatbt26XfH3dunUlriclJUmSwsLCXNZDQkKUmJh4yff88ccftW7dOn344YeqUOHKehp0QAAAAIByKDs7W5Jkt9td1v38/JSbm3vJfefNm6eWLVu6dE88RQcEAAAA8GEX63BcjsPhkHT+XJALf5ak3NxcValS5aL7ZWVlae3atRo/fvxVHZcOCAAAAOAJp610Pq7ShdGrlJQUl/WUlBSFhoZedL/169erqKhIPXr0uKrjUoAAAAAA5VBkZKQCAgK0ZcuW4rWMjAzFxcUpJibmovvt2LFDN954o4KCgq7quIxgAQAAAOWQ3W5Xv379NGXKFAUHB6tOnTqaPHmyQkND1aNHDxUWFur06dMKDAx0GdGKj49X06ZNr/q4dEAAAAAAD1h9w0EjbkQ4cuRI9enTR+PGjVPfvn1VsWJFzZ49W3a7XYmJierYsaNWrlzpss8vv/yi6tWrX/UxbU7ntYbtO1oPnWp1CAAAALiEnbNGWR3CRTX65ytWh1Cin/o+a3UIV4QOCAAAAADTcA4IAAAA4IlyMzdkLDogAAAAAExDAQIAAADANIxgAQAAAB5wXsNN//BfdEAAAAAAmIYCBAAAAIBpGMECAAAAPMFVsLyCDggAAAAA01CAAAAAADANI1gAAACAB7gKlnfQAQEAAABgGgoQAAAAAKZhBAsAAADwBFfB8go6IAAAAABMU646INf9fZPVIZRp6SsjrA6hzCtYUcvqEMq0Sr1SrQ6hzOMzDAAoVwUIAAAAcPW4CpY3MIIFAAAAwDQUIAAAAABMwwgWAAAA4AmuguUVdEAAAAAAmIYCBAAAAIBpGMECAAAAPMEIllfQAQEAAABgGgoQAAAAAKZhBAsAAADwhJMbEXoDHRAAAAAApqEAAQAAAGAaRrAAAAAADzi5CpZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8AQjWF5BBwQAAACAaShAAAAAAJiGESwAAADAE9yI0CvogAAAAAAwDQUIAAAAANMwggUAAAB4wMZVsLyCDggAAAAA01CAAAAAADANI1gAAACAJxjB8go6IAAAAABMQwECAAAAwDSMYAEAAACe4EaEXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMATXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANI1gAAACAJ5w2qyMoE+iAAAAAADANBQgAAAAA0zCCBQAAAHjAxlWwvIIOCAAAAADTUIAAAAAAMA0jWAAAAIAnGMHyCjogAAAAAExDB8QkMXe20p9fekj1osKVnpqhL/7+lRa+tvyi21eqXEl9nuypHn+6TbXqXqdfEk7rmwXrtfC15SrILyjebtHP7ys4tLrb/v8XFqszyWne/0FKqfY1IzQ0oocaBoToTN45LT2xVR/+9J8St727TmuNb97nou81cfen+vLnXZKkVV2f0XV+gW7b3PXNKzqVl+md4H3UrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt/AZ9gafI6NRX6NR47hiyhATBB1S1O9uOIp/WfRJs19fqFu6hipP0/qqwoVKmjBK0tL3GfoWwPU409d9PGkJTqw7ZCatG6kR8Y/oJD6tfTmoFmSpOoh1RQcWl2zRs/Tvu8OuOyfceqs4T9XadG8ej290foRrU38UbMOrlWrGg00NKKHKsimuT/92237jSn7NfC7WS5rNpv07I33y7+Snzamns9lsD1A1/kFauq+L/Vj2nGX7dPyswz7eXxBi0Zhmjq0l77acUAzP9uoVk3qaPi9HVTBZtPs1VtL3Kdp3VqSpNg3P1VewX+L6Jy8ghK3L0/4DFuDz7GxyK/xyDF8FQWICR554QEd/v6oXu8/XZK0fc33qlS5oh586j4tfvML5eXkuWwfWCNAPQf30AdPf6xPp3wmSdr1zR5J0mN/e0Szn/5Y6b9kqEl0Q0nShqVblHL8FxN/otIltsntOpCRqAk/fipJ2vzLQVWyVdCfGnXRgqMblFvk+ks1Lf+c0tLPuaw9WP9WNQiopUGb31Va/vnXmgaFSZL+lbxXSTlpxv8gPmTw3e21PyFVz89bLUnaFHdMlSpU0IA722j+uh3KzS9026dZeIhO/pKuHQcTzA631OMzbA0+x8Yiv8Yjx/BVV3QOSGpqqiZMmKBHH31UTz31lObNm6ft27crOzvbqPh8XmV7JbW47UZtWLbFZf3bxZtVNbCKmneKdNvHv1pVffH3tfrus+0u6wkHfpYkhTUKkSQ1btVAZ89kluvio7KtoloHN9K/kve6rK9L2iP/Sn5qVaPBZd/jOnuAhkT00JLjW7Q3/b+/kJsGhikjP5t/uP1G5UoVdXNEuL7Zdchl/etdB+XvsCu6SZ0S92sWXkv7E1LNCNGn8Bm2Bp9jY5Ff45Fj+LIr6oA8++yz2rBhgyIiIpSQkKDPP/9cTqdTFSpUUKNGjXTTTTepefPmat68uSIjI1W5cmWj4vYZYY1qy+5XWSd/LR4u+PlQkiQpvOn12rF2t8trSUdTNH34B27v1fEP7ZSfV6CEA4mSpMYtGyjzzDmNX/xXRXdrrgoVK2jzFzv07uh5Op2UZswPVMrUqRose4VKOp51ymU94dfn9fxrasupQyXtWuyxiO4qchbp3YNrXdabBl2vs/nZer3Vw2pzXWNVsNm0IXW/psZ/qVO55WfE7bfCa1aTvXIlHU8547J+IjVNklQ/pIY27zvutl/T8Fo6knRK88Y8qGZ1Q3Q2K1efb47TrM82qaCoyIzQSyU+w9bgc2ws8ms8cmwNbkToHVdUgOzatUtjxozRwIEDJUlZWVnau3evfvzxR/3444/atm2bli1bJkmy2+3avXv3pd5OkpSbm6uDBw+qSZMmcjgc2rdvn+bPn6/k5GRFRESof//+Cg0NvYofrXTwr+4vSTqX4dolyjp7/nnVoCoevU/H+9up+yOdtWzaSmWmnR+vaNyqgWqGX6eVH3ytJW99oXo3hKv/xAf1xr8namjrscrJyvXiT1I6BVZ2SJLOFeS4rGcVnh9r86/kd8n9a9j99fvro/Xx0Q3K/M17NA0MU4gjSMsTtumfxzaqgX8tDY7orr+3jVW/TdOVU5jvxZ/EdwRWOZ/TzN+MDmb9+ty/it1tn+DAqqpZzV9Op1PTlm1Q4pkMtW1WTwPuiFHtGgEaN3e18YGXUnyGrcHn2Fjk13jkGL7sigoQPz8/RUVFFT+vWrWq2rRpozZt2hSvpaWlaffu3dqzZ89l3+/w4cMaMGCAUlNTdf3112vSpEkaNmyYwsPD1bhxY3399ddaunSpFixYoMaNG19JqKVGhQq2839wllwyFxVdvpTu1Lu9np4/Urv/E6fZT39cvD5l4Ezl5eTp8PdHJUl7NsTr2N4TemvDJHX/Uxd98e5X1xx/aWfThfyW/HrRRfJ+wX3hbVTBZtPCo5vcXntpz2LlFhbowNnzHafvzxzVT5nJ+qD9EN19fWstObHFbZ/ywGa73Gfafe1cTq6GvL1Ex5JPK/nM+Ssv7Tx4UvkFhRreq4Nmr9qqI0mnjQq5VOMzbA0+x8Yiv8Yjx/BlV3QOSPfu3RUXF3fJbapXr67OnTtr2LBhl32/v/3tb4qOjtby5ct18803a+jQobrnnnv0+eef6+2339aqVavUoUMHvfrqq1cSZqlyoVtRNaiqy3rVwPOdj3Ppl74STe9RPfXcwlHauyFez9/7uvL/5yoV+zYfKC4+Lti7ab8y086pccv6Xoi+9Lvwje9vvyWuWvH8Nz/nCi7dBbo99CZt+eVQ8Um7/+vHtBPF/3C7YHfacZ3Nz1ZEoO925a7V2ezzOfV3uH67VvXX55k57jnPzS/U1vjjxf/Bu2D9niOSpIjwmkaE6hP4DFuDz7GxyK/xyLFFnLbS+fAxV1SA9O7dW6tWrdKhQ5eeR/bU1q1b9cQTTygyMlJPPfWUcnNz1bdv3+KqvlKlShoyZIh27NjhleNZ4efDySosKNT1TVz/Y3/h+fG4i1+FYvi0gRryRn+tX7xZz939inLO/Xe8wr9aVd35566qd0O4236V7JWU/kv5mO9OyDqtgqJChVe9zmX9wvOfMlMuum+IX5CaBV2vr5N+dHstoJJD99S5WQ39Q9xeq1yhYrm+hGlCapoKCotUN6S6y3rdWuef/5R4ym2f+rVrqHenFm7/oXRUPt+ETcvMcdunvOAzbA0+x8Yiv8Yjx/BlV1SA/N///Z/27NmjBx54QM8884xWrlypY8eOXfXBHQ6HcnLOf9hr1qyp//u//5Ofn+u3gBkZGQoMdL+Jlq/Iz83X7m/3qeMf2rmsd+7TXmfPZCp+a8nF3MBX/qj7Hr9Li6d+oZf7TnXpfEhSQV6BRr4zSA89dZ/L+q292shR1U8//Nv1ijplVV5Rgb4/c1RdQ290We8WepMy8rMVl37iovtGVa8rSfrhjPtnOL+oUGOj7lX/Rp1d1ruE3CBHRbt2nP7JC9H7pryCQu06lKDbWzVxWe8eHaGMrBztPZrktk9I9QA998du6h4d4bJ+R0xTZWbnat/xZENjLs34DFuDz7GxyK/xyDF82RWdAzJp0iTt27dPe/fu1apVq7Rs2TLZbDb5+/srKipKN910k8aOHevx+3Xs2FEvvfSSJk2apMaNG+vFF18sfs3pdGrr1q2aOHGiunfvfiVhljoLXl6i19c+r+cXjdbqud8o6tZmeuCv9+qDpz9WXk6eqgZWUf2ocP18OFnpv2SoccsGenBsL+3fdkjffrJJN7Rz/UVxLC5BWWez9cnkz9Tv+T46k5Ku7at3qWGL+vrT+P/T5i92aNc6929Ey6o5h/+lGW0G6tVWffVZwg61qF5P/Rp20oz9a5RbVCD/in5qGBCihKzTLmMqTQJqK7cwXyez3eddc4vy9Y8j6zWoye06lZepzakH1SQwVLFNumlDSry2nTps5o9Y6nywaqtmjeyt1wfdrRXf7VXLRmH6U48YTVu+Xrn5hfJ32NUoLFgnUtOVlpmtHQcStG3/CY3u01kOv8o6mnRanZo31EO3Reutpd/qbDm4YMKl8Bm2Bp9jY5Ff45FjC3AVLK+wOZ2XOcPxIoqKinT48GHt3btXe/bsUVxcnOLj47Vz506P3+P06dMaMmSI6tatqzfeeMPltS+//FJPPvmkOnXqpKlTpyogIOBqwnTRo8ID1/weV6vDfW31pwn/p/Bm1+vUydP6bOZqLX7zC0lSiy5ReuNfEzX5z+/oqw//rf4TH1S/5/tc9L2e7Dpeu/8TJ5vNpnuG3qGeQ+7Q9Y1DlXHqrL755wZ9NP4Tt5sbmiF9ZcTlNzLIbSFRio3opvr+tZSak6FPj2/WgqMbJEmtgxvq3baxmvjjYn158r+fz7FR96pr7Rt1179KPsfIJpt612un3nXbqU7VYKXnZ+mrn3/Qe4e+drsxnFkKVtSy5Lgl6dqysYb0vEX1a9dQSvo5ffLv7zV/3fn83hwRrvdHP6DxH67R55vPnzcW4LBrcM9bdFvLxqpZzV8Jqela8M1OLdt4+QtWmKVSL+uujc9n2Bpl8XNcmpBf45XFHO+cNcrqEC6q0VtvWh1CiX56YrTVIVyRqy5ASuJ0Ov97VYYrkJaWpurVq7usnT59WikpKYqMdL9R39WysgApD6wsQMqL0vaPt7LGygKkvOAzDOByKECunK8VIFc0gnU5V1N8SHIrPiQpODhYwcHB1xgRAAAA4CWMYHnFFZ2EDgAAAADXggIEAAAAgGm8OoIFAAAAlFU2RrC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4gBsRegcdEAAAAACmoQABAAAAYBoKEAAAAMATTlvpfFyDoqIiTZs2TZ06dVLLli01cOBAHTt27KLb5+fn64033lCnTp3UqlUr9evXT/v27buiY1KAAAAAAOXUzJkztXDhQk2aNEmLFi2SzWZTbGys8vLyStx+woQJWrx4sV566SUtWbJE1atXV2xsrM6ePevxMSlAAAAAgHIoLy9Pc+bM0YgRI9SlSxdFRkZq6tSpSk5O1tq1a922P3HihBYvXqxXX31Vt912mxo3bqxXXnlFdrtde/bs8fi4FCAAAACAJ5yl9HGV4uPjde7cObVv3754LSgoSFFRUdq2bZvb9hs2bFBQUJA6d+7ssv0333yjW265xePjchleAAAAwId169btkq+vW7euxPWkpCRJUlhYmMt6SEiIEhMT3bY/evSo6tatq6+++krvvfeekpOTFRUVpaefflqNGzf2OF46IAAAAEA5lJ2dLUmy2+0u635+fsrNzXXbPjMzU8ePH9fMmTM1evRozZo1S5UqVdIf//hHnTp1yuPj0gEBAAAAPFBab0R4sQ7H5TgcDknnzwW58GdJys3NVZUqVdy2r1y5ss6ePaupU6cWdzymTp2qLl26aNmyZRo0aJBHx6UDAgAAAJRDF0avUlJSXNZTUlIUGhrqtn1oaKgqVarkMm7lcDhUt25dJSQkeHxcChAAAACgHIqMjFRAQIC2bNlSvJaRkaG4uDjFxMS4bR8TE6OCggL9+OOPxWs5OTk6ceKE6tev7/FxGcECAAAAPFFKR7Cult1uV79+/TRlyhQFBwerTp06mjx5skJDQ9WjRw8VFhbq9OnTCgwMlMPhUExMjG699VY99dRTevHFF1W9enVNmzZNFStWVK9evTw+Lh0QAAAAoJwaOXKk+vTpo3Hjxqlv376qWLGiZs+eLbvdrsTERHXs2FErV64s3n769Olq27atHn/8cfXp00eZmZn66KOPFBwc7PExbU6ns4zVchfXo8IDVodQpqWvjLA6hDKvYEUtq0Mo0yr1SrU6hDKPzzCAy9k5a5TVIVxUxKtTrQ6hRAefKb05KwkjWAAAAIAHSutVsHwNI1gAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaShAAAAAAJiGAgQAAACAaShAAAAAAJiGq2ABAAAAnuAqWF5BBwQAAACAaShAAAAAAJiGESwAAADAA9yI0DvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsryhXBcipwbdaHULZtsLqAMq+7eNnWR0CcE1iVgy1OgQAgMUYwQIAAABgmnLVAQEAAACuGiNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMAD3IjQO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AGuguUddEAAAAAAmIYCBAAAAIBpGMECAAAAPMEIllfQAQEAAABgGgoQAAAAAKZhBAsAAADwBCNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMAD3IjQO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANI1gAAACAB7gRoXfQAQEAAABgGgoQAAAAAKZhBAsAAADwBCNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMADXAXLO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANBQgAAAAA03AOCAAAAOABm9UBlBF0QAAAAACYhgIEAAAAgGkYwbLIrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt9Fjq2XmCL1+rM0Y5LUNtrqaMomcnxt+D1hLPJrPHJsMi7D6xUUIBZo0ShMU4f20lc7DmjmZxvVqkkdDb+3gyrYbJq9emuJ+zStW0uSFPvmp8orKChez8krKHH78o4cW+9kkhQ7RjqbaRO/sY1Bjq8NvyeMRX6NR47hqyhALDD47vban5Cq5+etliRtijumShUqaMCdbTR/3Q7l5he67dMsPEQnf0nXjoMJZofrk8ixdYqKpOWrpb/NsjqSsoscewe/J4xFfo1HjuGrOAfEZJUrVdTNEeH6Ztchl/Wvdx2Uv8Ou6CZ1StyvWXgt7U9INSNEn0eOrbX/sDRxqnTfndLrz1kdTdlEjq8dvyeMRX6NR46tYXOWzoevoQNisvCa1WSvXEnHU864rJ9ITZMk1Q+poc37jrvt1zS8lo4kndK8MQ+qWd0Qnc3K1eeb4zTrs00qKCoyI3SfQY6tFVZbWvOxFBoibd1ldTRlEzm+dvyeMBb5NR45hi8rtQXIPffco/fee09hYWFWh+JVgVX8JEmZOXku61m/PvevYnfbJziwqmpW85fT6dS0ZRuUeCZDbZvV04A7YlS7RoDGzV1tfOA+hBxbq3qQpCCroyjbyPG14/eEsciv8cgxfJmlBcjy5csv+tqxY8e0atUqBQcHS5Luu+8+c4IymM326y1snCX3y0r68uFcTq6GvL1Ex5JPK/lMpiRp58GTyi8o1PBeHTR71VYdSTptVMg+hxwDuBx+TxiL/BqPHFvEB8edSiNLC5CJEycqJydHkuQs4S/Q3/72N0nn/5KVlQLkbHauJMnf4frNRNVfn2fm5Lrtk5tfqK3x7m3U9XuOaHivDooIr8kvjP9BjgFcDr8njEV+jUeO4cssPQl96dKlioqKUrt27fSf//xH8fHxxY8qVapo7dq1io+P1759+6wM06sSUtNUUFikuiHVXdbr1jr//KfEU2771K9dQ707tXD7JeOofL5+TMvMMSRWX0WOAVwOvyeMRX6NR47hyywtQBo2bKhFixapRYsW6tWrl1auXGllOKbIKyjUrkMJur1VE5f17tERysjK0d6jSW77hFQP0HN/7Kbu0REu63fENFVmdq72HU82NGZfQ44BXA6/J4xFfo1Hji3iLKUPH2P5SeiVKlXS6NGj1alTJz311FNat26dJkyYYHVYhvpg1VbNGtlbrw+6Wyu+26uWjcL0px4xmrZ8vXLzC+XvsKtRWLBOpKYrLTNbOw4kaNv+Exrdp7McfpV1NOm0OjVvqIdui9ZbS7/V2Sz3Nmt5R44BXA6/J4xFfo1HjuGrbM6STr6wSEZGhiZOnKjt27fr1KlTWrVqlerWreu19289dKrX3utadW3ZWEN63qL6tWsoJf2cPvn395q/bqck6eaIcL0/+gGN/3CNPt8cJ0kKcNg1uOctuq1lY9Ws5q+E1HQt+Ganlm3cY+WPUaqVxRxvH+9bd57bukvq/4RNH77lVNtoq6Mpm3wtxzETh1odgouy+HuiNCG/xiuLOd45a5TVIVxUy5Gl59+S/+uHaaU3ZyUpVQXIBcuXL9fSpUs1ZcoUhYSEeO19S1MBAlwNXytAgN8qbQUIgNKnNBcgrUaUzn9Lfj+99OasJJaPYJXkvvvuKzNXvQIAAADwX5aehA4AAACgfCmVHRAAAACg1Cl1Jy74JjogAAAAAExDAQIAAADANIxgAQAAAB6wMYLlFXRAAAAAAJiGAgQAAAAop4qKijRt2jR16tRJLVu21MCBA3Xs2LGLbr9s2TI1a9bM7XGpfX6LESwAAADAE2VwBGvmzJlauHChXn31VdWuXVuTJ09WbGysvvjiC9ntdrft9+/fr7Zt2+rNN990WQ8ODvb4mHRAAAAAgHIoLy9Pc+bM0YgRI9SlSxdFRkZq6tSpSk5O1tq1a0vc58CBA4qMjFStWrVcHhUrVvT4uBQgAAAAQDkUHx+vc+fOqX379sVrQUFBioqK0rZt20rcZ//+/WrSpMk1HZcCBAAAAPCAzVk6H1crKSlJkhQWFuayHhISosTERLftT58+rV9++UXbtm1Tz5491bFjRw0fPlxHjhy5ouNyDggAAADgw7p163bJ19etW1fienZ2tiS5nevh5+en9PR0t+0PHDggSapYsaJef/11ZWVlaebMmfrjH/+ozz//XDVr1vQoXgoQAAAAoBxyOBySzp8LcuHPkpSbm6sqVaq4bd++fXtt3bpV1apVK15755131LVrVy1dulSPPfaYR8elAAEAAAA8UUqvgnWxDsflXBi9SklJUb169YrXU1JSFBkZWeI+/1t8SFLVqlUVHh6u5ORkj4/LOSAAAABAORQZGamAgABt2bKleC0jI0NxcXGKiYlx237BggVq166dcnJyitcyMzN19OjRKzoxnQIEAAAAKIfsdrv69eunKVOmaN26dYqPj9eoUaMUGhqqHj16qLCwUKmpqcUFR9euXeV0OjV27FgdPHhQP/74o0aMGKHg4GD94Q9/8Pi4FCAAAACAJ5yl9HENRo4cqT59+mjcuHHq27evKlasqNmzZ8tutysxMVEdO3bUypUrJZ0f2frwww917tw59e3bVwMGDFBgYKA++ugjl3NILodzQAAAAIByqmLFihozZozGjBnj9lp4eLj279/vsnbDDTdo9uzZ13RMOiAAAAAATEMHBAAAAPDAtdz0D/9FBwQAAACAaShAAAAAAJiGESwAAADAE4xgeQUdEAAAAACmoQABAAAAYBpGsAAAAAAP2JzMYHkDHRAAAAAApqEAAQAAAGAaRrAAAAAATzCB5RV0QAAAAACYhgIEAAAAgGkYwQIAAAA8YGMEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABroLlHXRAAAAAAJiGAgQAAACAaRjBAgAAADzBCJZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwTbnqgFTqlWp1CMA1af99H6tDKNMKVtSyOgQAAMq8clWAAAAAAFeLq2B5ByNYAAAAAExDAQIAAADANIxgAQAAAJ5wMoPlDXRAAAAAAJiGAgQAAACAaRjBAgAAADzAVbC8gw4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAHrAVWR1B2UAHBAAAAIBpKEAAAAAAmIYRLAAAAMATXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHjAxgiWV9ABAQAAAGAaChAAAAAApmEECwAAAPCEkxksb6ADAgAAAMA0FCAAAAAATMMIFgAAAOABroLlHXRAAAAAAJiGAgQAAACAaRjBAgAAADzBCJZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAnnAyg+UNdEAAAAAAmIYCBAAAAIBpGMECAAAAPMBVsLyDDggAAAAA01CAAAAAADANI1gAAACAJxjB8go6IAAAAABMQwECAAAAwDSMYAEAAAAe4CpY3kEHBAAAAIBpKEAAAAAAmIYRLAAAAMATRcxgeQMdEAAAAACmoQNikvY1IzQ0oocaBoToTN45LT2xVR/+9J8St727TmuNb97nou81cfen+vLnXZKkVV2f0XV+gW7b3PXNKzqVl+md4H0A+TUeOTbfrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt9Fjo1Ffo1HjuGLKEBM0Lx6Pb3R+hGtTfxRsw6uVasaDTQ0oocqyKa5P/3bbfuNKfs18LtZLms2m/TsjffLv5KfNqYekCQF2wN0nV+gpu77Uj+mHXfZPi0/y7Cfp7Qhv8Yjx+Zr0ShMU4f20lc7DmjmZxvVqkkdDb+3gyrYbJq9emuJ+zStW0uSFPvmp8orKChez8krKHH78o4cG4v8Go8cW4AJLK+gADFBbJPbdSAjURN+/FSStPmXg6pkq6A/NeqiBUc3KLfI9S99Wv45paWfc1l7sP6tahBQS4M2v6u0/POvNQ0KkyT9K3mvknLSjP9BSinyazxybL7Bd7fX/oRUPT9vtSRpU9wxVapQQQPubKP563YoN7/QbZ9m4SE6+Uu6dhxMMDtcn0SOjUV+jUeO4as4B8RglW0V1Tq4kf6VvNdlfV3SHvlX8lOrGg0u+x7X2QM0JKKHlhzfor3p//2F0TQwTBn52eX6H27k13jk2HyVK1XUzRHh+mbXIZf1r3cdlL/DrugmdUrcr1l4Le1PSDUjRJ9Hjo1Ffo1HjuHL6IAYrE7VYNkrVNLxrFMu6wm/Pq/nX1NbTh0qaddij0V0V5GzSO8eXOuy3jToep3Nz9brrR5Wm+saq4LNpg2p+zU1/kudyj3r3R+klCK/xiPH5guvWU32ypV0POWMy/qJ1DRJUv2QGtq877jbfk3Da+lI0inNG/OgmtUN0dmsXH2+OU6zPtukgqIiM0L3GeTYWOTXeOTYGtyI0DssLUAWL16se++9V3a7vXht8+bNmjNnjpKSkhQREaGhQ4eqSZMmFkZ5bQIrOyRJ5wpyXNazCvMkSf6V/C65fw27v35/fbQ+PrpBmb95j6aBYQpxBGl5wjb989hGNfCvpcER3fX3trHqt2m6cgrzvfiTlE7k13jk2HyBVc7nNDMnz2U969fn/lXsbvsEB1ZVzWr+cjqdmrZsgxLPZKhts3oacEeMatcI0Li5q40P3IeQY2ORX+ORY/gySwuQ559/Xl27dtV1110nSdqwYYNiY2PVoUMHdezYUXv27FHv3r01d+5ctW7d2spQr5pNtvN/uEjFXOS8dCl9X3gbVbDZtPDoJrfXXtqzWLmFBTpwNlGS9P2Zo/opM1kftB+iu69vrSUntlxT7L6A/BqPHJvPZruQ85JzW9KXlOdycjXk7SU6lnxayWfOXz1s58GTyi8o1PBeHTR71VYdSTptVMg+hxwbi/wajxzDl1l6DojzN39pZs6cqT/96U/64IMPNHbsWH300Ud66KGHNGXKFIsivHYXvvH97bfEVSue/2biXEHuJfe/PfQmbfnlUPFJu//rx7QTxf9wu2B32nGdzc9WRGDotYTtM8iv8cix+c5mn8+pv8P1G8yqvz7PzHHPeW5+obbGHy/+R8UF6/cckSRFhNc0IlSfRY6NRX6NR44t4nSWzoePKVUnoR87dky9evVyWXvwwQcVFxdnUUTXLiHrtAqKChVe9TqX9QvPf8pMuei+IX5BahZ0vb5O+tHttYBKDt1T52Y19A9xe61yhYrl5hKm5Nd45Nh8CalpKigsUt2Q6i7rdWudf/5T4im3ferXrqHenVq4/WPEUfl8ozstM8dtn/KMHBuL/BqPHMOXWVqAFLcPf9WgQQNlZbn+o+PMmTMKDHS/SZmvyCsq0Pdnjqpr6I0u691Cb1JGfrbi0k9cdN+o6nUlST+cOeb2Wn5RocZG3av+jTq7rHcJuUGOinbtOP2TF6Iv/civ8cix+fIKCrXrUIJub+V6/lv36AhlZOVo79Ekt31CqgfouT92U/foCJf1O2KaKjM7V/uOJxsas68hx8Yiv8Yjx/Bllp4D4nQ61a1bNzVs2FCNGzeW3W7X5MmTNX/+fFWuXFk7d+7UxIkT1aVLFyvDvGZzDv9LM9oM1Kut+uqzhB1qUb2e+jXspBn71yi3qED+Ff3UMCBECVmnXcZUmgTUVm5hvk5mu89j5hbl6x9H1mtQk9t1Ki9Tm1MPqklgqGKbdNOGlHhtO3XYzB/RUuTXeOTYfB+s2qpZI3vr9UF3a8V3e9WyUZj+1CNG05avV25+ofwddjUKC9aJ1HSlZWZrx4EEbdt/QqP7dJbDr7KOJp1Wp+YN9dBt0Xpr6bc6m3XpUbnyiBwbi/wajxybj6tgeYfN+dsTMUz0888/a//+/Tpw4EDx/z969Ki2b98uh8Oh6OhoNWvWTLNmzVKNGjWu+XhtVz/rhaivzm0hUYqN6Kb6/rWUmpOhT49v1oKjGyRJrYMb6t22sZr442J9eXJn8T5jo+5V19o36q5/vVrie9pkU+967dS7bjvVqRqs9PwsffXzD3rv0NduN4Yr68iv8cpDjgtW1DL9mJfStWVjDel5i+rXrqGU9HP65N/fa/668/m9OSJc749+QOM/XKPPN58fUw1w2DW45y26rWVj1azmr4TUdC34ZqeWbdxj5Y9RqpFjY5Ff45XFHO+cNcrqEC6q652vWx1Cif615imrQ7gilhYgJcnPz1flypUlSfv371fTpk3dRrWulpUFCIDSr7QVIABQHlGAXDlfK0BK1UnokoqLD0lq1qyZ14oPAAAA4Jo4S+njGhQVFWnatGnq1KmTWrZsqYEDB+rYMfdzN0vy+eefq1mzZkpISLiiY5a6AgQAAACAOWbOnKmFCxdq0qRJWrRokWw2m2JjY5WXl3fJ/U6ePKmJEyde1TEpQAAAAIByKC8vT3PmzNGIESPUpUsXRUZGaurUqUpOTtbatWsvul9RUZHGjBmjG2+88aLbXAoFCAAAAOABm9NZKh9XKz4+XufOnVP79u2L14KCghQVFaVt27ZddL93331X+fn5Gjx48FUd19LL8AIAAAC4Nt26dbvk6+vWrStxPSnp/P1iwsLCXNZDQkKUmJhY4j67d+/WnDlztHjxYiUnX929Y+iAAAAAAOVQdna2JMlut7us+/n5KTfX/b4wWVlZ+utf/6q//vWvatCgwVUflw4IAAAA4IkiqwMo2cU6HJfjcDgknT8X5MKfJSk3N1dVqlRx237SpElq0KCBHnrooasL9FcUIAAAAEA5dGH0KiUlRfXq1SteT0lJUWRkpNv2S5Yskd1uV3R0tCSpsLBQktSzZ0/de++9evHFFz06LgUIAAAAUA5FRkYqICBAW7ZsKS5AMjIyFBcXp379+rlt/9VXX7k8/+GHHzRmzBi99957aty4scfHpQABAAAAPHAtV5wqjex2u/r166cpU6YoODhYderU0eTJkxUaGqoePXqosLBQp0+fVmBgoBwOh+rXr++y/4WT2K+//npdd911Hh+Xk9ABAACAcmrkyJHq06ePxo0bp759+6pixYqaPXu27Ha7EhMT1bFjR61cudKrx7Q5nWWslLuEtquftToEAKVYwYpaVocAAOXezlmjrA7horrd/qrVIZRo3TfPWB3CFWEECwAAAPBEufna3liMYAEAAAAwDQUIAAAAANMwggUAAAB4ovycOm0oOiAAAAAATEMBAgAAAMA0jGABAAAAHrAxgeUVdEAAAAAAmIYCBAAAAIBpGMECAAAAPMFVsLyCDggAAAAA01CAAAAAADANI1gAAACAB2xFVkdQNtABAQAAAGAaChAAAAAApmEECwAAAPAEV8HyCjogAAAAAExDAQIAAADANIxgAQAAAJ5gAssr6IAAAAAAMA0FCAAAAADTMIIFAAAAeMDGVbC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAniiyOoCygQ4IAAAAANNQgAAAAAAwDSNYAAAAgAe4EaF30AEBAAAAYBoKEAAAAACmYQQLAAAA8AQjWF5BBwQAAACAaShAAAAAAJiGESwAAADAE4xgeUW5KkAKVtSyOgQAAACgXGMECwAAAIBpylUHBAAAALhqRVYHUDbQAQEAAABgGgoQAAAAAKZhBAsAAADwgI2rYHkFHRAAAAAApqEAAQAAAGAaRrAAAAAATzCC5RV0QAAAAACYhgIEAAAAgGkYwQIAAAA8wQiWV9ABAQAAAGAaChAAAAAApmEECwAAAPAEI1heQQcEAAAAgGkoQAAAAACYhhEsAAAAwBNFVgdQNtABAQAAAGAaChAAAAAApmEECwAAAPCAjatgeQUdEAAAAACmoQABAAAAYBpGsAAAAABPMILlFXRAAAAAAJiGAgQAAACAaRjBAgAAADxRxAiWN9ABAQAAAGAaChAAAAAApmEECwAAAPAEV8HyCjogAAAAAExDAQIAAADANIxgAQAAAJ5gBMsr6IAAAAAAMA0FCAAAAADTMIIFAAAAeIIRLK+gAwIAAADANBQgAAAAAEzDCBYAAADgiSJGsLyBDggAAAAA01CAAAAAADANI1gAAACAJ5xFVkdQJtABAQAAAGAaChAAAAAApmEECwAAAPAENyL0CjogAAAAAExDAQIAAADANIxgAQAAAJ7gRoReQQFikVuj6mvYvR3UMCxYaWeztXj9bs1ds+2i2zeoXUNLJwxwWz+SdFq9J35oYKS+ixwbi/wajxwbjxwbi/wajxzDF1GAWKBFozBNHdpLX+04oJmfbVSrJnU0/N4OqmCzafbqrSXu07RuLUlS7JufKq+goHg9J6+gxO3LO3JsLPJrPHJsPHJsLPJrPHIMX0UBYoHBd7fX/oRUPT9vtSRpU9wxVapQQQPubKP563YoN7/QbZ9m4SE6+Uu6dhxMMDtcn0SOjUV+jUeOjUeOjUV+jUeOLcBVsLyCk9BNVrlSRd0cEa5vdh1yWf9610H5O+yKblKnxP2ahdfS/oRUM0L0eeTYWOTXeOTYeOTYWOTXeOQYvowOiMnCa1aTvXIlHU8547J+IjVNklQ/pIY27zvutl/T8Fo6knRK88Y8qGZ1Q3Q2K1efb47TrM82qaCoyIzQfQY5Nhb5NR45Nh45Nhb5NR45hi+zvAD54YcftGXLFj322GOSpM2bN2vevHlKSEhQvXr1NHDgQMXExFgcpfcEVvGTJGXm5LmsZ/363L+K3W2f4MCqqlnNX06nU9OWbVDimQy1bVZPA+6IUe0aARo3d7XxgfsQcmws8ms8cmw8cmws8ms8cmwRRrC8wtICZPXq1Ro9erRuvfVWPfbYY/rXv/6lYcOGqXPnzurSpYsOHDig/v37a8aMGeratauVoXqNzWY7/4eLfIBL+vLhXE6uhry9RMeSTyv5TKYkaefBk8ovKNTwXh00e9VWHUk6bVTIPoccG4v8Go8cG48cG4v8Go8cw1uKioo0Y8YMffrpp8rIyNDNN9+s8ePHq379+iVuv2fPHk2ePFm7d++Wn5+f7rjjDv31r39VUFCQx8e09ByQGTNm6PHHH9cHH3wgSZo1a5aGDBmiv//97xozZozef/99Pf7445o2bZqVYXrV2excSZK/w/Wbiaq/Ps/MyXXbJze/UFvjjxf/srhg/Z4jkqSI8JpGhOqzyLGxyK/xyLHxyLGxyK/xyDG8ZebMmVq4cKEmTZqkRYsWyWazKTY2Vnl5eW7bpqSk6M9//rPq1aunZcuWaebMmdq5c6eeeuqpKzqmpQXI8ePHdc899xQ/T0hI0J133umyTc+ePXX48GGzQzNMQmqaCgqLVDekust63Vrnn/+UeMptn/q1a6h3pxZuv2Qclc83sNIycwyJ1VeRY2ORX+ORY+ORY2ORX+ORY4s4naXzcZXy8vI0Z84cjRgxQl26dFFkZKSmTp2q5ORkrV271m37kydPqlOnTho/frwaNGig1q1b64EHHtB33313Rce1tACpW7eu/vOf/xQ/v+GGGxQfH++yze7du1W7dm2zQzNMXkGhdh1K0O2tmrisd4+OUEZWjvYeTXLbJ6R6gJ77Yzd1j45wWb8jpqkys3O173iyoTH7GnJsLPJrPHJsPHJsLPJrPHIMb4iPj9e5c+fUvn374rWgoCBFRUVp2zb3G1pGR0frzTffVKVK54vWQ4cOadmyZerQocMVHdfSc0BiY2P13HPPKSkpST179tSwYcP09NNPKzc3VxEREfrhhx/0zjvv6PHHH7cyTK/7YNVWzRrZW68Pulsrvturlo3C9KceMZq2fL1y8wvl77CrUViwTqSmKy0zWzsOJGjb/hMa3aezHH6VdTTptDo1b6iHbovWW0u/1dks9zZreUeOjUV+jUeOjUeOjUV+jUeOcUG3bt0u+fq6detKXE9KOl+ohoWFuayHhIQoMTHxku9555136ujRo6pTp45mzpx5BdFKNqfT2tP5V6xYoWnTpunkyZOy2Wz633D8/f01aNAgDR061CvHaj10qlfexxu6tmysIT1vUf3aNZSSfk6f/Pt7zV+3U5J0c0S43h/9gMZ/uEafb46TJAU47Brc8xbd1rKxalbzV0JquhZ8s1PLNu6x8sco1cixsciv8cix8cixsciv8cpijnfOGmV1CBd1V9hwq0MoUV5U/CVfv1gBsmLFCo0dO1b79u1ThQr/HYwaO3asUlJSNG/evIu+548//qicnBxNmTJFp06d0ooVK+Tv7+9RvJYXIBf89NNPOnr0qDIzM1W5cmWFhoYqKipKfn5+XjtGaSpAAAAA4I4C5MqtSnznqvZbs2aNRo4cqR9++EEOh6N4/S9/+Yvy8vI0a9asy75HSkqKunTpoldffVX33XefR8e1/D4gFzRq1EiNGjWyOgwAAACgXLgwepWSkqJ69eoVr6ekpCgyMtJt+8OHDyshIUFdunQpXgsJCVG1atWUnOz5OUSWnoQOAAAA+Ayrr3bl5atgRUZGKiAgQFu2bCley8jIUFxcXIk3Al+/fr3+8pe/KDPzv5dyPn78uM6cOaPGjRt7fFwKEAAAAKAcstvt6tevn6ZMmaJ169YpPj5eo0aNUmhoqHr06KHCwkKlpqYqJ+f8JZp79eqlwMBAjRkzRgcPHtT27ds1cuRItWjR4opuGk4BAgAAAJRTI0eOVJ8+fTRu3Dj17dtXFStW1OzZs2W325WYmKiOHTtq5cqVkqQaNWroo48+UlFRkfr27avhw4crKipKs2fPVsWKFT0+Zqk5Cd0MnIQOAABQupXqk9Bre+fKrN62KvnyJ4uXJnRAAAAAAJiGAgQAAACAaUrNZXgBAACAUq2o3Jy5YCg6IAAAAABMQwECAAAAwDSMYAEAAAAecDqLrA6hTKADAgAAAMA0FCAAAAAATMMIFgAAAOAJroLlFXRAAAAAAJiGAgQAAACAaRjBAgAAADzhZATLG+iAAAAAADANBQgAAAAA0zCCBQAAAHiiiBsRegMdEAAAAACmoQABAAAAYBpGsAAAAABPcBUsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABJ1fB8go6IAAAAABMQwECAAAAwDSMYAEAAACe4CpYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMATRYxgeQMdEAAAAACmoQABAAAAYBpGsAAAAABPOLkRoTfQAQEAAABgGgoQAAAAAKZhBAsAAADwgJOrYHkFHRAAAAAApqEAAQAAAGAaRrAAAAAAT3AVLK+gAwIAAADANBQgAAAAAEzDCBYAAADgAa6C5R10QAAAAACYhgIEAAAAgGkYwQIAAAA8wVWwvIIOCAAAAADTUIAAAAAAMI3N6XRyOj8AAAAAU9ABAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaCpBSpqioSNOmTVOnTp3UsmVLDRw4UMeOHbM6rDJr5syZeuSRR6wOo0xJS0vTCy+8oM6dO6t169bq27evtm/fbnVYZcqpU6c0ZswYtW/fXtHR0Xrsscd06NAhq8Mqk44cOaLo6GgtXbrU6lDKlJMnT6pZs2Zuj08//dTq0MqU5cuX6/e//72aN2+uu+++W6tWrbI6JEASBUipM3PmTC1cuFCTJk3SokWLZLPZFBsbq7y8PKtDK3PmzZunadOmWR1GmTN69Gj98MMPevPNN7V48WLdeOONevTRR3X48GGrQyszhg4dqhMnTuj999/X4sWL5XA4NGDAAGVnZ1sdWpmSn5+vv/71r8rKyrI6lDJn//798vPz0/r167Vhw4bixz333GN1aGXGihUr9Oyzz+rBBx/UF198od///vcaPXq0du3aZXVoAAVIaZKXl6c5c+ZoxIgR6tKliyIjIzV16lQlJydr7dq1VodXZiQnJ2vQoEF6++231bBhQ6vDKVOOHTumjRs3avz48YqJiVGjRo303HPPqXbt2vriiy+sDq9MOHPmjMLDw/XSSy+pefPmaty4sYYNG6bU1FQdPHjQ6vDKlOnTp8vf39/qMMqkAwcOqGHDhgoJCVGtWrWKHw6Hw+rQygSn06m3335b/fv3V//+/VW/fn0NHz5ct956q7Zu3Wp1eAAFSGkSHx+vc+fOqX379sVrQUFBioqK0rZt2yyMrGzZu3evqlWrps8++0wtW7a0OpwypUaNGnrvvfd00003Fa/ZbDY5nU6lp6dbGFnZUaNGDb355puKiIiQJP3yyy+aPXu2QkND1aRJE4ujKzu2bdumRYsW6fXXX7c6lDJp//79fF4N9NNPP+nkyZNuHaXZs2dr8ODBFkUF/FclqwPAfyUlJUmSwsLCXNZDQkKUmJhoRUhl0u23367bb7/d6jDKpKCgIHXp0sVlbdWqVTp+/Lg6duxoUVRl1/PPP69PPvlEdrtds2bNUtWqVa0OqUzIyMjQ2LFjNW7cOLffx/COAwcOqFatWvrjH/+oo0ePqn79+ho2bJg6depkdWhlwtGjRyVJWVlZevTRRxUXF6fw8HANHTqU//6hVKADUopcmN+22+0u635+fsrNzbUiJOCa7NixQ88++6y6devGf/QM0L9/fy1ZskT33nuvhg8frr1791odUpkwYcIEtWrVivMRDJKXl6ejR48qMzNTTzzxhN577z01b95csbGx+u6776wOr0zIzMyUJD311FPq2bOn5syZow4dOmjYsGHkGKUCHZBS5MLsa15ensscbG5urqpUqWJVWMBV+frrr/XXv/5VLVu21Jtvvml1OGXShRGWl156Sd9//73mz5+vV1991eKofNvy5cu1fft2ff7551aHUmbZ7XZt27ZNlSpVKv7C7aabbtLhw4c1e/Zs3XLLLRZH6PsqV64sSXr00Uf1hz/8QZJ0ww03KC4uTnPnziXHsBwdkFLkQqs/JSXFZT0lJUWhoaFWhARclfnz52vEiBHq3Lmz3n//fU4s9aJTp07piy++UGFhYfFahQoV1LhxY7ffHbhyS5Ys0alTp3TbbbcpOjpa0dHRkqTx48fr7rvvtji6sqNq1apu3f6mTZsqOTnZoojKlgv/ZmjatKnLepMmTZSQkGBFSIALCpBSJDIyUgEBAdqyZUvxWkZGhuLi4hQTE2NhZIDnFixYoJdeekkPP/yw3nrrLbd/ZODapKSk6Mknn3S5kk1+fr7i4uLUuHFjCyMrG6ZMmaKVK1dq+fLlxQ9JGjlypN577z1rgysj4uPjFR0d7XZ/oD179nBiupdERUXJ399fP/zwg8v6gQMHVK9ePYuiAv6LEaxSxG63q1+/fpoyZYqCg4NVp04dTZ48WaGhoerRo4fV4QGXdeTIEb3yyivq0aOHBg8erFOnThW/5nA4FBgYaGF0ZUNkZKQ6duyoiRMnatKkSQoKCtK7776rjIwMDRgwwOrwfF7t2rVLXL/uuutUp04dk6Mpm5o2baqIiAhNnDhR48ePV40aNfTJJ5/o+++/1+LFi60Or0xwOBwaNGiQ3nnnHdWuXVstWrTQl19+qY0bN2revHlWhwdQgJQ2I0eOVEFBgcaNG6ecnBy1adNGs2fP5ltk+IQ1a9YoPz9fa9eudbt3zR/+8Ae99tprFkVWdthsNr311lt644039MQTT+js2bOKiYnRxx9/rOuvv97q8IDLqlChgt59911NmTJFTzzxhDIyMhQVFaW5c+eqWbNmVodXZgwbNkxVqlQpvp9Y48aNNX36dLVr187q0ADZnE6n0+ogAAAAAJQPnAMCAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAD4mNzdXUVFRio6O1ksvvWR1OAAAXBEKEADwMTabTR9++KFatGih+fPn68iRI1aHBACAxyhAAMDH2O12tWnTRoMGDZIk7d271+KIAADwHAUIAPioRo0aSZL27dtncSQAAHiOAgQAfNT7778vSYqPj7c4EgAAPEcBAgA+aMOGDfrnP/+patWqKS4uzupwAADwGAUIAPiYjIwMPfvss+rWrZv69u2r06dPKzk52eqwAADwCAUIAPiYiRMnqqCgQJMmTVJUVJQkxrAAAL6DAgQAfMjq1av1xRdf6OWXX1ZwcHBxAcKJ6AAAX0EBAgA+IjU1VePHj9eDDz6orl27SpLq1q2roKAgzgMBAPgMChAA8BHPP/+8qlWrpqefftpl/YYbbmAECwDgMyhAAMAHfPrpp/r222/1t7/9TVWrVnV5LSoqSsePH1dmZqZF0QEA4Dmb0+l0Wh0EAAAAgPKBDggAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADDN/wOpUE87FNLINAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "# import necessary packages\n", "import numpy as np\n", @@ -345,9 +576,7 @@ { "cell_type": "markdown", "id": "1cac501a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Building a neural network code\n", "\n", @@ -364,9 +593,7 @@ { "cell_type": "markdown", "id": "dd153528", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Learning rate methods\n", "\n", @@ -386,10 +613,7 @@ "cell_type": "code", "execution_count": 3, "id": "f55eea63", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -527,9 +751,7 @@ { "cell_type": "markdown", "id": "1a9bcb3e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Usage of the above learning rate schedulers\n", "\n", @@ -543,10 +765,7 @@ "cell_type": "code", "execution_count": 4, "id": "86013cb4", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", @@ -556,9 +775,7 @@ { "cell_type": "markdown", "id": "535331f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here is a small example for how a segment of code using schedulers\n", "could look. Switching out the schedulers is simple." @@ -568,10 +785,7 @@ "cell_type": "code", "execution_count": 5, "id": "7e0f6b5a", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "weights = np.ones((3,3))\n", @@ -590,9 +804,7 @@ { "cell_type": "markdown", "id": "f018ae57", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Cost functions\n", "\n", @@ -606,10 +818,7 @@ "cell_type": "code", "execution_count": 6, "id": "c13507bf", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -644,9 +853,7 @@ { "cell_type": "markdown", "id": "6dab17bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Below we give a short example of how these cost function may be used\n", "to obtain results if you wish to test them out on your own using\n", @@ -657,10 +864,7 @@ "cell_type": "code", "execution_count": 7, "id": "a5dbba01", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from autograd import grad\n", @@ -678,9 +882,7 @@ { "cell_type": "markdown", "id": "b55d31d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Activation functions\n", "\n", @@ -694,10 +896,7 @@ "cell_type": "code", "execution_count": 8, "id": "b3e045a6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -752,9 +951,7 @@ { "cell_type": "markdown", "id": "c0189342", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Below follows a short demonstration of how to use an activation\n", "function. The derivative of the activation function will be important\n", @@ -767,10 +964,7 @@ "cell_type": "code", "execution_count": 9, "id": "640aa861", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "z = np.array([[4, 5, 6]]).T\n", @@ -788,9 +982,7 @@ { "cell_type": "markdown", "id": "1007ccdd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### The Neural Network\n", "\n", @@ -812,10 +1004,7 @@ "cell_type": "code", "execution_count": 10, "id": "9584a2da", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import math\n", @@ -1284,9 +1473,7 @@ { "cell_type": "markdown", "id": "9ccd1fc1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Before we make a model, we will quickly generate a dataset we can use\n", "for our linear regression problem as shown below" @@ -1296,10 +1483,7 @@ "cell_type": "code", "execution_count": 11, "id": "7f3a5b31", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -1340,9 +1524,7 @@ { "cell_type": "markdown", "id": "1ac05bb6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Now that we have our dataset ready for the regression, we can create\n", "our regressor. Note that with the seed parameter, we can make sure our\n", @@ -1356,10 +1538,7 @@ "cell_type": "code", "execution_count": 12, "id": "0f857604", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -1371,9 +1550,7 @@ { "cell_type": "markdown", "id": "eeff4315", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We then fit our model with our training data using the scheduler of our choice." ] @@ -1382,10 +1559,7 @@ "cell_type": "code", "execution_count": 13, "id": "46246810", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1397,9 +1571,7 @@ { "cell_type": "markdown", "id": "8c6f9954", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Due to the progress bar we can see the MSE (train_error) throughout\n", "the FFNN's training. Note that the fit() function has some optional\n", @@ -1413,10 +1585,7 @@ "cell_type": "code", "execution_count": 14, "id": "2661939c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1427,9 +1596,7 @@ { "cell_type": "markdown", "id": "74c5624c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", "\n", @@ -1442,10 +1609,7 @@ "cell_type": "code", "execution_count": 15, "id": "4b8eb115", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", @@ -1468,10 +1632,7 @@ "cell_type": "code", "execution_count": 16, "id": "2c0f92bd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -1483,9 +1644,7 @@ { "cell_type": "markdown", "id": "49201ae4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will now make use of our validation data by passing it into our fit function as a keyword argument" ] @@ -1494,10 +1653,7 @@ "cell_type": "code", "execution_count": 17, "id": "55b5e426", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1509,9 +1665,7 @@ { "cell_type": "markdown", "id": "b51762fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Finally, we will create a neural network with 2 hidden layers with activation functions." ] @@ -1520,10 +1674,7 @@ "cell_type": "code", "execution_count": 18, "id": "6b59e27d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -1540,10 +1691,7 @@ "cell_type": "code", "execution_count": 19, "id": "72c87921", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1555,9 +1703,7 @@ { "cell_type": "markdown", "id": "ed40d7d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "### Multiclass classification\n", "\n", @@ -1570,10 +1716,7 @@ "cell_type": "code", "execution_count": 20, "id": "315ef3fe", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_digits\n", @@ -1605,7 +1748,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, "nbformat": 4, "nbformat_minor": 5 }