Morten Hjorth-Jensen
1353 lines
74 KiB
Plaintext
1353 lines
74 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "bd3b4df9",
|
||
"metadata": {},
|
||
"source": [
|
||
"# Neural Networks in Python\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "07fd4fb5",
|
||
"metadata": {},
|
||
"source": [
|
||
"In this notebook we will present a flexible object oriented codebase for a feed forward neural network, along with a demonstration of how to use it. Before we get into the details of the neural network, we will first present some implementations of various schedulers, cost functions and activation functions that can be used together with the neural network."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "e0eaf00e",
|
||
"metadata": {},
|
||
"source": [
|
||
"## I. Schedulers\n",
|
||
"The code below shows object oriented implementations of the Constant, Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All of the classes belong to the shared abstract Scheduler class, and share the update_change() and reset() methods allowing for any of the schedulers to be seamlessly used during the training stage, as will later be shown in the fit() method of the neural network. Update_change() only has one parameter, the gradient ($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted from the weights. The reset() function takes no parameters, and resets the desired variables. For Constant and Momentum, reset does nothing."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "bbb1c207",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"\n",
|
||
"class Scheduler:\n",
|
||
" \"\"\"\n",
|
||
" Abstract class for Schedulers\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" def __init__(self, eta):\n",
|
||
" self.eta = eta\n",
|
||
"\n",
|
||
" # should be overwritten\n",
|
||
" def update_change(self, gradient):\n",
|
||
" raise NotImplementedError\n",
|
||
"\n",
|
||
" # overwritten if needed\n",
|
||
" def reset(self):\n",
|
||
" pass\n",
|
||
"\n",
|
||
"\n",
|
||
"class Constant(Scheduler):\n",
|
||
" def __init__(self, eta):\n",
|
||
" super().__init__(eta)\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" return self.eta * gradient\n",
|
||
" \n",
|
||
" def reset(self):\n",
|
||
" pass\n",
|
||
"\n",
|
||
"\n",
|
||
"class Momentum(Scheduler):\n",
|
||
" def __init__(self, eta: float, momentum: float):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.momentum = momentum\n",
|
||
" self.change = 0\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" self.change = self.momentum * self.change + self.eta * gradient\n",
|
||
" return self.change\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" pass\n",
|
||
"\n",
|
||
"\n",
|
||
"class Adagrad(Scheduler):\n",
|
||
" def __init__(self, eta):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.G_t = None\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
"\n",
|
||
" if self.G_t is None:\n",
|
||
" self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n",
|
||
"\n",
|
||
" self.G_t += gradient @ gradient.T\n",
|
||
"\n",
|
||
" G_t_inverse = 1 / (\n",
|
||
" delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n",
|
||
" )\n",
|
||
" return self.eta * gradient * G_t_inverse\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.G_t = None\n",
|
||
"\n",
|
||
"\n",
|
||
"class AdagradMomentum(Scheduler):\n",
|
||
" def __init__(self, eta, momentum):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.G_t = None\n",
|
||
" self.momentum = momentum\n",
|
||
" self.change = 0\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
"\n",
|
||
" if self.G_t is None:\n",
|
||
" self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n",
|
||
"\n",
|
||
" self.G_t += gradient @ gradient.T\n",
|
||
"\n",
|
||
" G_t_inverse = 1 / (\n",
|
||
" delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n",
|
||
" )\n",
|
||
" self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n",
|
||
" return self.change\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.G_t = None\n",
|
||
"\n",
|
||
"\n",
|
||
"class RMS_prop(Scheduler):\n",
|
||
" def __init__(self, eta, rho):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.rho = rho\n",
|
||
" self.second = 0.0\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
" self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n",
|
||
" return self.eta * gradient / (np.sqrt(self.second + delta))\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.second = 0.0\n",
|
||
"\n",
|
||
"\n",
|
||
"class Adam(Scheduler):\n",
|
||
" def __init__(self, eta, rho, rho2):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.rho = rho\n",
|
||
" self.rho2 = rho2\n",
|
||
" self.moment = 0\n",
|
||
" self.second = 0\n",
|
||
" self.n_epochs = 1\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
"\n",
|
||
" self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n",
|
||
" self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n",
|
||
"\n",
|
||
" moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n",
|
||
" second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n",
|
||
"\n",
|
||
" return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.n_epochs += 1\n",
|
||
" self.moment = 0\n",
|
||
" self.second = 0"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b043fddf",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Usage of schedulers\n",
|
||
"To initalize a scheduler, simply create the object and pass in the necessary parameters such as the learning rate and the momentum as shown below. As the Scheduler class is an abstract class it should not called directly, and will raise an error upon usage."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "d0b0fb8f",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
|
||
"adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1d8cc996",
|
||
"metadata": {},
|
||
"source": [
|
||
"Here is a small example for how a segment of code using schedulers could look. This code works for any scheduler."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "5895e189",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Before scheduler:\n",
|
||
"weights=array([[1., 1., 1.],\n",
|
||
" [1., 1., 1.],\n",
|
||
" [1., 1., 1.]])\n",
|
||
"\n",
|
||
"After scheduler:\n",
|
||
"weights=array([[0.993993 , 0.99399302, 0.993993 ],\n",
|
||
" [0.993993 , 0.99399303, 0.99399303],\n",
|
||
" [0.993993 , 0.993993 , 0.99399334]])\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"weights = np.ones((3,3))\n",
|
||
"print(f\"Before scheduler:\\n{weights=}\")\n",
|
||
"\n",
|
||
"epochs = 10\n",
|
||
"for e in range(epochs):\n",
|
||
" gradient = np.random.rand(3, 3)\n",
|
||
" change = adam_scheduler.update_change(gradient)\n",
|
||
" weights = weights - change\n",
|
||
" adam_scheduler.reset()\n",
|
||
"\n",
|
||
"print(f\"\\nAfter scheduler:\\n{weights=}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "0c7fa586",
|
||
"metadata": {},
|
||
"source": [
|
||
"## II. Cost functions\n",
|
||
"In this section we will quickly look at the cost functions that can be used when creating the neural network. Every cost function takes the target vector as its parameter, and returns a function valued only at X such that it may easily be differentiated."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "1ab9e722",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"\n",
|
||
"def CostOLS(target):\n",
|
||
" \n",
|
||
" def func(X):\n",
|
||
" return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
"\n",
|
||
"def CostLogReg(target):\n",
|
||
"\n",
|
||
" def func(X):\n",
|
||
" \n",
|
||
" return -(1.0 / target.shape[0]) * np.sum(\n",
|
||
" (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n",
|
||
" )\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
"\n",
|
||
"def CostCrossEntropy(target):\n",
|
||
" \n",
|
||
" def func(X):\n",
|
||
" return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n",
|
||
"\n",
|
||
" return func"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "7eed4b4d",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Usage of cost functions\n",
|
||
"Below we will provide a short example of how these cost function may be used to obtain results if you wish to test them out on your own using AutoGrad's automatics differentiation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "c9b9d18a",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Derivative of cost function CostCrossEntropy valued at a:\n",
|
||
"[[-0.08333333]\n",
|
||
" [-0.13333333]\n",
|
||
" [-0.16666667]]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"from autograd import grad\n",
|
||
"\n",
|
||
"target = np.array([[1, 2, 3]]).T\n",
|
||
"a = np.array([[4, 5, 6]]).T\n",
|
||
"\n",
|
||
"cost_func = CostCrossEntropy\n",
|
||
"cost_func_derivative = grad(cost_func(target))\n",
|
||
"\n",
|
||
"valued_at_a = cost_func_derivative(a)\n",
|
||
"print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "02ba305e",
|
||
"metadata": {},
|
||
"source": [
|
||
"## III. Activation functions\n",
|
||
"Finally, before we look at the neural network, we will look at the activation functions which can be specified between the hidden layers and as the output function. Each function can be valued for any given vector or matrix X, and can be differentiated via derivate()."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "e282d670",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"from autograd import elementwise_grad\n",
|
||
"\n",
|
||
"def identity(X):\n",
|
||
" return X\n",
|
||
"\n",
|
||
"\n",
|
||
"def sigmoid(X):\n",
|
||
" try:\n",
|
||
" return 1.0 / (1 + np.exp(-X))\n",
|
||
" except FloatingPointError:\n",
|
||
" return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n",
|
||
"\n",
|
||
"\n",
|
||
"def softmax(X):\n",
|
||
" X = X - np.max(X, axis=-1, keepdims=True)\n",
|
||
" delta = 10e-10\n",
|
||
" return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n",
|
||
"\n",
|
||
"\n",
|
||
"def RELU(X):\n",
|
||
" return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n",
|
||
"\n",
|
||
"\n",
|
||
"def LRELU(X):\n",
|
||
" delta = 10e-4\n",
|
||
" return np.where(X > np.zeros(X.shape), X, delta * X)\n",
|
||
"\n",
|
||
"\n",
|
||
"def derivate(func):\n",
|
||
" if func.__name__ == \"RELU\":\n",
|
||
"\n",
|
||
" def func(X):\n",
|
||
" return np.where(X > 0, 1, 0)\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
" elif func.__name__ == \"LRELU\":\n",
|
||
"\n",
|
||
" def func(X):\n",
|
||
" delta = 10e-4\n",
|
||
" return np.where(X > 0, 1, delta)\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
" else:\n",
|
||
" return elementwise_grad(func)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1145f141",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Usage of activation functions\n",
|
||
"Below is a short demonstration of how to use an activation function. The derivative of the activation function will be important when calculating the output delta term during backpropagation. Note that derivate() can also be used for cost functions for a more generalized approach."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "a7f46ecc",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Input to activation function:\n",
|
||
"[[4]\n",
|
||
" [5]\n",
|
||
" [6]]\n",
|
||
"\n",
|
||
"Output from sigmoid activation function:\n",
|
||
"[[0.98201379]\n",
|
||
" [0.99330715]\n",
|
||
" [0.99752738]]\n",
|
||
"\n",
|
||
"Derivative of sigmoid activation function valued at z:\n",
|
||
"[[0.19824029]\n",
|
||
" [0.19721923]\n",
|
||
" [0.19683648]]\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"z = np.array([[4, 5, 6]]).T\n",
|
||
"print(f\"Input to activation function:\\n{z}\")\n",
|
||
"\n",
|
||
"act_func = sigmoid\n",
|
||
"a = act_func(z)\n",
|
||
"print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n",
|
||
"\n",
|
||
"act_func_derivative = derivate(act_func)\n",
|
||
"valued_at_z = act_func_derivative(a)\n",
|
||
"print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "026e16e3",
|
||
"metadata": {},
|
||
"source": [
|
||
"## IV. The Neural Network\n",
|
||
"Now that we have gotten a good understanding of the implementation of some important components, we can take a look at an object oriented implementation of a feed forward neural network. The feed forward neural network has been implemented as a class named FFNN, which can be initiated as a regressor or classifier dependant on the choice of cost function. The FFNN can have any number of input nodes, hidden layers with any amount of hidden nodes, and any amount of output nodes meaning it can perform multiclass classification as well as binary classification and regression problems. Although there is a lot of code present, it makes for an easy to use and generalizeable interface for creating many types of neural networks as will be demonstrated below."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "d4196a44",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import math\n",
|
||
"import autograd.numpy as np\n",
|
||
"import sys\n",
|
||
"import warnings\n",
|
||
"from autograd import grad, elementwise_grad\n",
|
||
"from random import random, seed\n",
|
||
"from copy import deepcopy, copy\n",
|
||
"from typing import Tuple, Callable\n",
|
||
"from sklearn.utils import resample\n",
|
||
"\n",
|
||
"warnings.simplefilter(\"error\")\n",
|
||
"\n",
|
||
"\n",
|
||
"class FFNN:\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Feed Forward Neural Network with interface enabling flexible design of a\n",
|
||
" nerual networks architecture and the specification of activation function\n",
|
||
" in the hidden layers and output layer respectively. This model can be used\n",
|
||
" for both regression and classification problems, depending on the output function.\n",
|
||
"\n",
|
||
" Attributes:\n",
|
||
" ------------\n",
|
||
" I dimensions (tuple[int]): A list of positive integers, which specifies the\n",
|
||
" number of nodes in each of the networks layers. The first integer in the array\n",
|
||
" defines the number of nodes in the input layer, the second integer defines number\n",
|
||
" of nodes in the first hidden layer and so on until the last number, which\n",
|
||
" specifies the number of nodes in the output layer.\n",
|
||
" II hidden_func (Callable): The activation function for the hidden layers\n",
|
||
" III output_func (Callable): The activation function for the output layer\n",
|
||
" IV cost_func (Callable): Our cost function\n",
|
||
" V seed (int): Sets random seed, makes results reproducible\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" def __init__(\n",
|
||
" self,\n",
|
||
" dimensions: tuple[int],\n",
|
||
" hidden_func: Callable = sigmoid,\n",
|
||
" output_func: Callable = lambda x: x,\n",
|
||
" cost_func: Callable = CostOLS,\n",
|
||
" seed: int = None,\n",
|
||
" ):\n",
|
||
" self.dimensions = dimensions\n",
|
||
" self.hidden_func = hidden_func\n",
|
||
" self.output_func = output_func\n",
|
||
" self.cost_func = cost_func\n",
|
||
" self.seed = seed\n",
|
||
" self.weights = list()\n",
|
||
" self.schedulers_weight = list()\n",
|
||
" self.schedulers_bias = list()\n",
|
||
" self.a_matrices = list()\n",
|
||
" self.z_matrices = list()\n",
|
||
" self.classification = None\n",
|
||
"\n",
|
||
" self.reset_weights()\n",
|
||
" self._set_classification()\n",
|
||
"\n",
|
||
" def fit(\n",
|
||
" self,\n",
|
||
" X: np.ndarray,\n",
|
||
" t: np.ndarray,\n",
|
||
" scheduler: Scheduler,\n",
|
||
" batches: int = 1,\n",
|
||
" epochs: int = 100,\n",
|
||
" lam: float = 0,\n",
|
||
" X_val: np.ndarray = None,\n",
|
||
" t_val: np.ndarray = None,\n",
|
||
" ):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" This function performs the training the neural network by performing the feedforward and backpropagation\n",
|
||
" algorithm to update the networks weights.\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray) : training data\n",
|
||
" II t (np.ndarray) : target data\n",
|
||
" III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n",
|
||
" IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n",
|
||
"\n",
|
||
" Optional Parameters:\n",
|
||
" ------------\n",
|
||
" V batches (int) : number of batches the datasets are split into, default equal to 1\n",
|
||
" VI epochs (int) : number of iterations used to train the network, default equal to 100\n",
|
||
" VII lam (float) : regularization hyperparameter lambda\n",
|
||
" VIII X_val (np.ndarray) : validation set\n",
|
||
" IX t_val (np.ndarray) : validation target set\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" I scores (dict) : A dictionary containing the performance metrics of the model.\n",
|
||
" The number of the metrics depends on the parameters passed to the fit-function.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" # setup \n",
|
||
" if self.seed is not None:\n",
|
||
" np.random.seed(self.seed)\n",
|
||
"\n",
|
||
" val_set = False\n",
|
||
" if X_val is not None and t_val is not None:\n",
|
||
" val_set = True\n",
|
||
"\n",
|
||
" # creating arrays for score metrics\n",
|
||
" train_errors = np.empty(epochs)\n",
|
||
" train_errors.fill(np.nan)\n",
|
||
" val_errors = np.empty(epochs)\n",
|
||
" val_errors.fill(np.nan)\n",
|
||
"\n",
|
||
" train_accs = np.empty(epochs)\n",
|
||
" train_accs.fill(np.nan)\n",
|
||
" val_accs = np.empty(epochs)\n",
|
||
" val_accs.fill(np.nan)\n",
|
||
"\n",
|
||
" self.schedulers_weight = list()\n",
|
||
" self.schedulers_bias = list()\n",
|
||
"\n",
|
||
" batch_size = X.shape[0] // batches\n",
|
||
"\n",
|
||
" X, t = resample(X, t)\n",
|
||
"\n",
|
||
" # this function returns a function valued only at X\n",
|
||
" cost_function_train = self.cost_func(t)\n",
|
||
" if val_set:\n",
|
||
" cost_function_val = self.cost_func(t_val)\n",
|
||
"\n",
|
||
" # create schedulers for each weight matrix\n",
|
||
" for i in range(len(self.weights)):\n",
|
||
" self.schedulers_weight.append(copy(scheduler))\n",
|
||
" self.schedulers_bias.append(copy(scheduler))\n",
|
||
"\n",
|
||
" print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n",
|
||
"\n",
|
||
" try:\n",
|
||
" for e in range(epochs):\n",
|
||
" for i in range(batches):\n",
|
||
" # allows for minibatch gradient descent\n",
|
||
" if i == batches - 1:\n",
|
||
" # If the for loop has reached the last batch, take all thats left\n",
|
||
" X_batch = X[i * batch_size :, :]\n",
|
||
" t_batch = t[i * batch_size :, :]\n",
|
||
" else:\n",
|
||
" X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n",
|
||
" t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n",
|
||
"\n",
|
||
" self._feedforward(X_batch)\n",
|
||
" self._backpropagate(X_batch, t_batch, lam)\n",
|
||
"\n",
|
||
" # reset schedulers for each epoch (some schedulers pass in this call)\n",
|
||
" for scheduler in self.schedulers_weight:\n",
|
||
" scheduler.reset()\n",
|
||
"\n",
|
||
" for scheduler in self.schedulers_bias:\n",
|
||
" scheduler.reset()\n",
|
||
"\n",
|
||
" # computing performance metrics\n",
|
||
" pred_train = self.predict(X)\n",
|
||
" train_error = cost_function_train(pred_train)\n",
|
||
"\n",
|
||
" train_errors[e] = train_error\n",
|
||
" if val_set:\n",
|
||
" \n",
|
||
" pred_val = self.predict(X_val)\n",
|
||
" val_error = cost_function_val(pred_val)\n",
|
||
" val_errors[e] = val_error\n",
|
||
"\n",
|
||
" if self.classification:\n",
|
||
" train_acc = self._accuracy(self.predict(X), t)\n",
|
||
" train_accs[e] = train_acc\n",
|
||
" if val_set:\n",
|
||
" val_acc = self._accuracy(pred_val, t_val)\n",
|
||
" val_accs[e] = val_acc\n",
|
||
"\n",
|
||
" # printing progress bar\n",
|
||
" progression = e / epochs\n",
|
||
" print_length = self._progress_bar(\n",
|
||
" progression,\n",
|
||
" train_error=train_errors[e],\n",
|
||
" train_acc=train_accs[e],\n",
|
||
" val_error=val_errors[e],\n",
|
||
" val_acc=val_accs[e],\n",
|
||
" )\n",
|
||
" except KeyboardInterrupt:\n",
|
||
" # allows for stopping training at any point and seeing the result\n",
|
||
" pass\n",
|
||
"\n",
|
||
" # visualization of training progression (similiar to tensorflow progression bar)\n",
|
||
" sys.stdout.write(\"\\r\" + \" \" * print_length)\n",
|
||
" sys.stdout.flush()\n",
|
||
" self._progress_bar(\n",
|
||
" 1,\n",
|
||
" train_error=train_errors[e],\n",
|
||
" train_acc=train_accs[e],\n",
|
||
" val_error=val_errors[e],\n",
|
||
" val_acc=val_accs[e],\n",
|
||
" )\n",
|
||
" sys.stdout.write(\"\")\n",
|
||
"\n",
|
||
" # return performance metrics for the entire run\n",
|
||
" scores = dict()\n",
|
||
"\n",
|
||
" scores[\"train_errors\"] = train_errors\n",
|
||
"\n",
|
||
" if val_set:\n",
|
||
" scores[\"val_errors\"] = val_errors\n",
|
||
"\n",
|
||
" if self.classification:\n",
|
||
" scores[\"train_accs\"] = train_accs\n",
|
||
"\n",
|
||
" if val_set:\n",
|
||
" scores[\"val_accs\"] = val_accs\n",
|
||
"\n",
|
||
" return scores\n",
|
||
"\n",
|
||
" def predict(self, X: np.ndarray, *, threshold=0.5):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Performs prediction after training of the network has been finished.\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray): The design matrix, with n rows of p features each\n",
|
||
"\n",
|
||
" Optional Parameters:\n",
|
||
" ------------\n",
|
||
" II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n",
|
||
" in classification problems\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n",
|
||
" This vector is thresholded if regression=False, meaning that classification results\n",
|
||
" in a vector of 1s and 0s, while regressions in an array of decimal numbers\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" predict = self._feedforward(X)\n",
|
||
"\n",
|
||
" if self.classification:\n",
|
||
" return np.where(predict > threshold, 1, 0)\n",
|
||
" else:\n",
|
||
" return predict\n",
|
||
"\n",
|
||
" def reset_weights(self):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Resets/Reinitializes the weights in order to train the network for a new problem.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" if self.seed is not None:\n",
|
||
" np.random.seed(self.seed)\n",
|
||
"\n",
|
||
" self.weights = list()\n",
|
||
" for i in range(len(self.dimensions) - 1):\n",
|
||
" weight_array = np.random.randn(\n",
|
||
" self.dimensions[i] + 1, self.dimensions[i + 1]\n",
|
||
" )\n",
|
||
" weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n",
|
||
"\n",
|
||
" self.weights.append(weight_array)\n",
|
||
"\n",
|
||
" def _feedforward(self, X: np.ndarray):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Calculates the activation of each layer starting at the input and ending at the output.\n",
|
||
" Each following activation is calculated from a weighted sum of each of the preceeding\n",
|
||
" activations (except in the case of the input layer).\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray): The design matrix, with n rows of p features each\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" # reset matrices\n",
|
||
" self.a_matrices = list()\n",
|
||
" self.z_matrices = list()\n",
|
||
"\n",
|
||
" # if X is just a vector, make it into a matrix\n",
|
||
" if len(X.shape) == 1:\n",
|
||
" X = X.reshape((1, X.shape[0]))\n",
|
||
"\n",
|
||
" # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n",
|
||
" # to add bias to our data\n",
|
||
" bias = np.ones((X.shape[0], 1)) * 0.01\n",
|
||
" X = np.hstack([bias, X])\n",
|
||
"\n",
|
||
" # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n",
|
||
" # exponent indicates layer number).\n",
|
||
" a = X\n",
|
||
" self.a_matrices.append(a)\n",
|
||
" self.z_matrices.append(a)\n",
|
||
"\n",
|
||
" # The feed forward algorithm\n",
|
||
" for i in range(len(self.weights)):\n",
|
||
" if i < len(self.weights) - 1:\n",
|
||
" z = a @ self.weights[i]\n",
|
||
" self.z_matrices.append(z)\n",
|
||
" a = self.hidden_func(z)\n",
|
||
" # bias column again added to the data here\n",
|
||
" bias = np.ones((a.shape[0], 1)) * 0.01\n",
|
||
" a = np.hstack([bias, a])\n",
|
||
" self.a_matrices.append(a)\n",
|
||
" else:\n",
|
||
" try:\n",
|
||
" # a^L, the nodes in our output layers\n",
|
||
" z = a @ self.weights[i]\n",
|
||
" a = self.output_func(z)\n",
|
||
" self.a_matrices.append(a)\n",
|
||
" self.z_matrices.append(z)\n",
|
||
" except Exception as OverflowError:\n",
|
||
" print(\n",
|
||
" \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n",
|
||
" )\n",
|
||
"\n",
|
||
" # this will be a^L\n",
|
||
" return a\n",
|
||
"\n",
|
||
" def _backpropagate(self, X: np.ndarray, t: np.ndarray, lam: float):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Performs the backpropagation algorithm. In other words, this method\n",
|
||
" calculates the gradient of all the layers starting at the\n",
|
||
" output layer, and moving from right to left accumulates the gradient until\n",
|
||
" the input layer is reached. Each layers respective weights are updated while\n",
|
||
" the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray): The design matrix, with n rows of p features each.\n",
|
||
" II t (np.ndarray): The target vector, with n rows of p targets.\n",
|
||
" III lam (float32): regularization parameter used to punish the weights in case of overfitting\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" No return value.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" out_derivative = derivate(self.output_func)\n",
|
||
" hidden_derivative = derivate(self.hidden_func)\n",
|
||
"\n",
|
||
" for i in range(len(self.weights) - 1, -1, -1):\n",
|
||
" # delta terms for output\n",
|
||
" if i == len(self.weights) - 1:\n",
|
||
" # for multi-class classification\n",
|
||
" if (\n",
|
||
" self.output_func.__name__ == \"softmax\"\n",
|
||
" ):\n",
|
||
" delta_matrix = self.a_matrices[i + 1] - t\n",
|
||
" # for binary classification\n",
|
||
" else:\n",
|
||
" cost_func_derivative = grad(self.cost_func(t))\n",
|
||
" delta_matrix = out_derivative(\n",
|
||
" self.z_matrices[i + 1]\n",
|
||
" ) * cost_func_derivative(self.a_matrices[i + 1])\n",
|
||
"\n",
|
||
" # delta terms for hidden layer\n",
|
||
" else:\n",
|
||
" delta_matrix = (\n",
|
||
" self.weights[i + 1][1:, :] @ delta_matrix.T\n",
|
||
" ).T * hidden_derivative(self.z_matrices[i + 1])\n",
|
||
"\n",
|
||
" # calculate gradient\n",
|
||
" gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n",
|
||
" gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n",
|
||
" 1, delta_matrix.shape[1]\n",
|
||
" )\n",
|
||
"\n",
|
||
" # regularization term\n",
|
||
" gradient_weights += self.weights[i][1:, :] * lam\n",
|
||
"\n",
|
||
" # use scheduler\n",
|
||
" update_matrix = np.vstack(\n",
|
||
" [\n",
|
||
" self.schedulers_bias[i].update_change(gradient_bias),\n",
|
||
" self.schedulers_weight[i].update_change(gradient_weights),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
"\n",
|
||
" # update weights and bias\n",
|
||
" self.weights[i] -= update_matrix\n",
|
||
"\n",
|
||
" def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Calculates accuracy of given prediction to target\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I prediction (np.ndarray): vector of predicitons output network\n",
|
||
" (1s and 0s in case of classification, and real numbers in case of regression)\n",
|
||
" II target (np.ndarray): vector of true values (What the network ideally should predict)\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" A floating point number representing the percentage of correctly classified instances.\n",
|
||
" \"\"\"\n",
|
||
" assert prediction.size == target.size\n",
|
||
" return np.average((target == prediction))\n",
|
||
" \n",
|
||
" def _set_classification(self):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Decides if FFNN acts as classifier (True) og regressor (False),\n",
|
||
" sets self.classification during init()\n",
|
||
" \"\"\"\n",
|
||
" self.classification = False\n",
|
||
" if (\n",
|
||
" self.cost_func.__name__ == \"CostLogReg\"\n",
|
||
" or self.cost_func.__name__ == \"CostCrossEntropy\"\n",
|
||
" ):\n",
|
||
" self.classification = True\n",
|
||
"\n",
|
||
" def _progress_bar(self, progression, **kwargs):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Displays progress of training\n",
|
||
" \"\"\"\n",
|
||
" print_length = 40\n",
|
||
" num_equals = int(progression * print_length)\n",
|
||
" num_not = print_length - num_equals\n",
|
||
" arrow = \">\" if num_equals > 0 else \"\"\n",
|
||
" bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n",
|
||
" perc_print = self._format(progression * 100, decimals=5)\n",
|
||
" line = f\" {bar} {perc_print}% \"\n",
|
||
"\n",
|
||
" for key in kwargs:\n",
|
||
" if not np.isnan(kwargs[key]):\n",
|
||
" value = self._format(kwargs[key], decimals=4)\n",
|
||
" line += f\"| {key}: {value} \"\n",
|
||
" sys.stdout.write(\"\\r\" + line)\n",
|
||
" sys.stdout.flush()\n",
|
||
" return len(line)\n",
|
||
"\n",
|
||
" def _format(self, value, decimals=4):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Formats decimal numbers for progress bar\n",
|
||
" \"\"\"\n",
|
||
" if value > 0:\n",
|
||
" v = value\n",
|
||
" elif value < 0:\n",
|
||
" v = -10 * value\n",
|
||
" else:\n",
|
||
" v = 1\n",
|
||
" n = 1 + math.floor(math.log10(v))\n",
|
||
" if n >= decimals - 1:\n",
|
||
" return str(round(value))\n",
|
||
" return f\"{value:.{decimals-n-1}f}\"\n",
|
||
"\n",
|
||
"\n"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "913fee68",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Usage of neural networks\n",
|
||
"This section will show how to initiate and use various different neural networks.\n",
|
||
"\n",
|
||
"### Linear regression\n",
|
||
" Before we make a model, we will quickly generate a dataset we can use for our linear regression problem as shown below:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "87086a73",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"\n",
|
||
"def SkrankeFunction(x, y):\n",
|
||
" return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n",
|
||
"\n",
|
||
"def create_X(x, y, n):\n",
|
||
" if len(x.shape) > 1:\n",
|
||
" x = np.ravel(x)\n",
|
||
" y = np.ravel(y)\n",
|
||
"\n",
|
||
" N = len(x)\n",
|
||
" l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n",
|
||
" X = np.ones((N, l))\n",
|
||
"\n",
|
||
" for i in range(1, n + 1):\n",
|
||
" q = int((i) * (i + 1) / 2)\n",
|
||
" for k in range(i + 1):\n",
|
||
" X[:, q + k] = (x ** (i - k)) * (y**k)\n",
|
||
"\n",
|
||
" return X\n",
|
||
"\n",
|
||
"step=0.2\n",
|
||
"x = np.arange(0, 1, step)\n",
|
||
"y = np.arange(0, 1, step)\n",
|
||
"x, y = np.meshgrid(x, y)\n",
|
||
"target = SkrankeFunction(x, y)\n",
|
||
"target = target.reshape(target.shape[0], 1)\n",
|
||
"\n",
|
||
"poly_degree=3\n",
|
||
"X = create_X(x, y, poly_degree)\n",
|
||
"\n",
|
||
"X_train, X_test, t_train, t_test = train_test_split(X, target)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d731194b",
|
||
"metadata": {},
|
||
"source": [
|
||
"Now that we have our dataset ready for the regression, we can create our regressor. Note that with the seed parameter, we can make sure our results stay the same every time we run the neural network. For inititialization, we simply specify the dimensions (we wish the amount of input nodes to be equal to the datapoints, and the output to predict one value)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "93e202f8",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = X_train.shape[1]\n",
|
||
"output_nodes = 1\n",
|
||
"\n",
|
||
"linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "92473ad5",
|
||
"metadata": {},
|
||
"source": [
|
||
"We then fit our model with our training data using the scheduler of our choice."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "e6b02b3c",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Constant: Eta=0.01, Lambda=0\n",
|
||
" [=======================================>] 100.0% | train_error: 1.34 "
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Constant(eta=1e-2)\n",
|
||
"scores = linear_regression.fit(X_train, t_train, scheduler)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3111a514",
|
||
"metadata": {},
|
||
"source": [
|
||
"Due to the progress bar we can see the MSE (train_error) throughout the FFNN's training. Note that the fit() function has some optional parameters with defualt arguments. For example, the regularization hyperparameter can be left ignored if not needed, and equally the FFNN will by default run for 100 epochs. These can easily be changed, such as for example:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "1cae319f",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Constant: Eta=0.01, Lambda=0.001\n",
|
||
" [=======================================>] 100.0% | train_error: 0.230 "
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-3, epochs=1000)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fec76144",
|
||
"metadata": {},
|
||
"source": [
|
||
"We see that given more epochs to train on, the regressor reaches a lower training MSE. If we want to use our test data, we simply pass it into our predict function."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "59e11120",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Prediction of test data:\n",
|
||
"[[2.66173972]\n",
|
||
" [3.89485327]\n",
|
||
" [0.3144022 ]\n",
|
||
" [1.24160891]\n",
|
||
" [5.13634463]\n",
|
||
" [2.20394971]\n",
|
||
" [3.49810527]]\n",
|
||
"\n",
|
||
"Target values of test data:\n",
|
||
"[[3.12]\n",
|
||
" [4.84]\n",
|
||
" [0. ]\n",
|
||
" [1.08]\n",
|
||
" [5.76]\n",
|
||
" [2.24]\n",
|
||
" [3.8 ]]\n",
|
||
"\n",
|
||
"test MSE:\n",
|
||
"0.24423672695163173\n"
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"pred_test = linear_regression.predict(X_test)\n",
|
||
"\n",
|
||
"print(f\"Prediction of test data:\\n{pred_test}\\n\")\n",
|
||
"print(f\"Target values of test data:\\n{t_test}\\n\")\n",
|
||
"print(f\"test MSE:\\n{CostOLS(t_test)(pred_test)}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "55c2539d",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Binary classification\n",
|
||
"Let us load in a binary classification dataset, here the Wisconsin Cancer data set, and follow a similar setup to the regression case. We will create a logistic regression classifier, and a feed forward neural network with 2 hidden layers."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "5aa72672",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.datasets import load_breast_cancer\n",
|
||
"from sklearn.preprocessing import MinMaxScaler\n",
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"\n",
|
||
"wisconsin = load_breast_cancer()\n",
|
||
"X = wisconsin.data\n",
|
||
"target = wisconsin.target\n",
|
||
"target = target.reshape(target.shape[0], 1)\n",
|
||
"\n",
|
||
"X_train, X_val, t_train, t_val = train_test_split(X, target)\n",
|
||
"\n",
|
||
"scaler = MinMaxScaler()\n",
|
||
"scaler.fit(X_train)\n",
|
||
"X_train = scaler.transform(X_train)\n",
|
||
"X_val = scaler.transform(X_val)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "3a61e226",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = X_train.shape[1]\n",
|
||
"output_nodes = 1\n",
|
||
"\n",
|
||
"logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "fd912453",
|
||
"metadata": {},
|
||
"source": [
|
||
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "634c8fc4",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Adam: Eta=0.001, Lambda=0\n",
|
||
" [=======================================>] 100.0% | train_error: 14.0 | train_acc: 0.324 | val_error: 12.8 | val_acc: 0.385 "
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n",
|
||
"scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=500, X_val=X_val, t_val=t_val)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ad60c785",
|
||
"metadata": {},
|
||
"source": [
|
||
"Finally, we will create a neural network with 2 hidden layers with activation functions. We will also make use of the 'batches' keyword argument to run stochastic gradient descent by specifying a number of batches we wish to split our data into."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "8812fe13",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = X_train.shape[1]\n",
|
||
"hidden_nodes1 = 100\n",
|
||
"hidden_nodes2 = 30\n",
|
||
"output_nodes = 1\n",
|
||
"\n",
|
||
"dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n",
|
||
"\n",
|
||
"neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "32c6e297",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Adam: Eta=0.0001, Lambda=0\n",
|
||
" [=======================================>] 100.0% | train_error: 0.195 | train_acc: 0.991 | val_error: 1.74 | val_acc: 0.916 "
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n",
|
||
"scores = neural_network.fit(X_train, t_train, scheduler, batches=10, epochs=300, X_val=X_val, t_val=t_val)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6bf55a80",
|
||
"metadata": {},
|
||
"source": [
|
||
"To plot the results, simply use the dictionary returned by the fit function"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"id": "eb790ef5",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"data": {
|
||
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQAElEQVR4nO3deXhU5f3+8fdkksm+EBKyQAJhkS2AEJCyuWAFUXEX3KVV+6UuiLhUSq1K/RX3SqWgVnFpqaCCVisVUQHZRKAg+x5IgISQkIUkkGXm/P6YZCAkQBgmOZPJ/bquuTJzljmfOUw7t8/znOdYDMMwEBEREfERfmYXICIiIuJJCjciIiLiUxRuRERExKco3IiIiIhPUbgRERERn6JwIyIiIj5F4UZERER8ir/ZBTQ2h8PBwYMHCQ8Px2KxmF2OiIiI1INhGBw9epTExET8/M7cNtPsws3BgwdJSkoyuwwRERFxQ2ZmJm3atDnjNs0u3ISHhwPOkxMREWFyNSIiIlIfRUVFJCUluX7Hz6TZhZvqrqiIiAiFGxERkSamPkNKNKBYREREfIrCjYiIiPgUhRsRERHxKQo3IiIi4lMUbkRERMSnmBpufvjhB0aOHEliYiIWi4XPP//8rPssWbKEtLQ0goKCaN++PW+++WbDFyoiIiJNhqnhpqSkhF69ejFt2rR6bZ+ens5VV13FkCFDWLduHb///e8ZN24cc+fObeBKRUREpKkwdZ6bESNGMGLEiHpv/+abb5KcnMzrr78OQNeuXVmzZg2vvPIKN910U537lJWVUVZW5npdVFR0XjWLiIiId2tSY25WrlzJsGHDaiwbPnw4a9asoaKios59pkyZQmRkpOuhWy+IiIj4tiYVbrKzs4mLi6uxLC4ujsrKSnJzc+vcZ+LEiRQWFroemZmZjVGqiIiImKTJ3X7h1GmXDcOoc3m1wMBAAgMDG7wuERER8Q5NquUmPj6e7OzsGstycnLw9/enZcuWJlUlIiIi3qRJtdwMGDCAL7/8ssayb775hr59+xIQEGBSVSIiYrZDRcepsDvMLkOqWP0sJEQGm3Z8U8NNcXExu3btcr1OT09n/fr1REdHk5yczMSJEzlw4AAffvghAGPHjmXatGlMmDCB+++/n5UrV/Luu+/y0UcfmfURRETERMfK7Tz80Tq+3XrI7FLkJK3CA/lp0i9NO76p4WbNmjVcdtllrtcTJkwA4J577uH9998nKyuLjIwM1/qUlBTmz5/Po48+yt/+9jcSExP561//etrLwEVEvMmynbn848e9VNgNs0vxGRlHStmVU4zFAjZrkxpp4dMCA8z9t7AY1SNym4mioiIiIyMpLCwkIiLC7HJEpAnZcegoBwqOubVv+uES/t/8rdgdzer/chtFeJA/743pR9920WaXIg3oXH6/m9SYGxERMxiGwfTFu3l5wfbzfq+reyZwSadYD1QlAFhgcMcYEqPMG98h3kfhRkQ8zu4weHTOetZl5ptdikc4HLhabLrEh+NvrXvqiTOxYOHyrq0YN7QTfn7nvr+I1J/CjYh43Bc/H+CLnw+aXYbHTbqqK/df3N7sMkTkLBRuROS8GIZBWeWJS3ArHQavfrMDgN9c3J4rU+PNKs2jWoUH0qZFiNlliEg9KNyIiNtKyyu5btpyduYU11rXKjyQR395AcE2qwmViUhzpuvmRMRtM5el1xls/Cww8aouCjYiYgq13Ij4iMNHyzhUdLzRjldWaeetJXsAeOWWXjW6n/z9LAQFKNiIiDkUbkR8wKdr9/PU3A1UmjCHSteECG7s3VpXAImI11C4EWki/rUqg3n/24/9lHk3HQb8nFkAQExYIAFuXKbsruAAK3+6rruCjYh4FYUbES9kGAYr9+S5upl+zizk/RV7z7jPrwel8IeruypoiEizp3Aj4mUcDoOn/72JWasyaq178LIOXJjUotbyVuGB9GwTicWiYCMionAj4mHHK+xMnLfReZdiN4bA2A2D0nI7FgsM7NASP4sFq5+F6y9szfW9W3u+YBERH6NwI3Keio5XMOenTPJLywH4cU8e/8soOK/3tPn78cotvbi2V6IHKhQRaV4UbkTOQdHxCkrL7K7XpeWVjJu9jk0HimpsFxbozxu39SYlJtSt40SH2YgICjivWkVEmiuFG5F6mr54F699s6POy61bhtq49sJELFgI8Ldwc582dIoLN6FKERFRuBGp4nAYzFyezqr0I7XWFR+vZOWePIBal1p3iA1j2u196NgqrFHqFBGRM1O4EZ/ncBisyyygtLzyjNv9e/1BPl27/4zb6K7QIiLeT+FGfNqxcjsPzFrLou2H67W91c/Cw0M7EhcRVGvdBXHhpLWtfRm2iIh4F4Ub8Ulfb8pi6ne7OHy0jNziMgL9/Wgfe+Zuo+AAPx68rCOXd41rpCpFRKQhKNxIk7J052F2HKp9F+qTHT5axls/7Kb6LgXhgf68O6YfF6VEN0KFIiJiNoUbaTJ+zizgrnd/qvf2t/ZL4ua0NnRqFU5kiC6rFhFpLhRupMl4acE2wHkX6k5nuTKpX0o0d/ZP1u0IRESaIYUb8Vrbsov4aFUG5XYHpeV2lu/Kw2b14+270kiKDjG7PBER8VIKN2I6u8Ng9+FiKu0nJsfLzC/lsY9/pris5uXbd/wiWcFGRETOSOFGTPf0vzfxrzrugA3Qr10LLu4UC0BIoD+3X5TcmKWJiEgTpHAjptpx6Cizf3IGm9jwwBrrLr0glj9dn0pQgNWM0kREpIlSuJFGU3S8gmU7c6mwO1zL5qzOxGHAiNR4ZtyZZmJ1IiLiKxRupFFkHinljndWkXGktNY6Pws8NqyzCVWJiIgvUriRBldYWsGot1aSVXicVuGBdIqreRn3lakJuumkiIh4jMKNNLh//3yArMLjJEUH8+nYgXXet0lERMRT/MwuQHzf5+sOADBmYIqCjYiINDiFG2lQGXml/C+jAD8LjOyZYHY5IiLSDCjcSIP64mdnq83ADjG0UquNiIg0AoUbaVALNh8C4NpeiSZXIiIizYXCjTSY/JJyNh0sBODSzrEmVyMiIs2Fwo00mJV78jAMuCAuTF1SIiLSaBRupMEs25ULwKCOMSZXIiIizYnmuZEGYRgGy6vCzWCFGzlZ3m7I3tC4x/Tzh5SLISiycY8rIqZQuBGPm/rtTqZ+twOHAf5+Fvq3b2l2SeItyo7C3y+D44WNf+zUm+Hmdxv/uCLS6BRuxKOOV9h5Z+keHIbz9VU9EggL1NdMqmz7yhlsgqIgLrVxjllRAgfXQeaqxjmeiJhOvzpy3o4eryDQ34rN349F23I4WlZJYmQQX40bQlRIgNnliTfZMMf59xcPwKW/a5xjHiuAF9tCYabzeXBU4xxXREyjcCPnJT23hBFTf+Cyzq2YcWcan693Tto38sJEWoTaTK5OGkRJLlSWnft+xwtgz2Ln8x43e7KiMwuOgog2ULQfcrZA24GNd2x3FR8Ge7nZVYi4z+IHEebNSq9wI+flv5uyOF7h4L+bstmWXcSibYcBuP7C1iZXJg1i5d9gwe/P7z3a9IOWHTxTT33FdXeGm0ObvT/cLPsLfPus2VWInJ+weHh8u2mHV7iR81J9RRTAfR+sodzuoEt8OF0TIkysShqEYcCqN53P/fyd/2V2rvyDYdAjnq2rPuK6w84FznDjzRx2WPW287lfAFgs5tYj4i6ruS33CjfituMVdlbvzXe93p9/DIBxl3cyqyRpSJmroCADbGHw+E6whZhdUf3FdXf+9fZws285HD3ovGT98Z3gH2h2RSJNkibxE7et2ZtPeaWDlqE2AqzO/8Ls0TqSEanxJlcmDaJ6MHDXkU0r2MCJcJOzBRwOc2s5k+pz3O16BRuR86CWG3Fb9QzEl3ZuhdUPPl93kElXd8WipnTfsu0rWDAJCvc7X/e4xdx63NGyo7OZvLwY/tLNvS61xlDsvNEsPUeZW4dIE6dwI25zzUDcqSUjeyYy6apuROrSb9+zeArkpzufR7eHlEvMrccd1gBoNwR2fwdHs8yu5sxiOkOylw96FvFyCjfilpPv+D2oQwz+Vj8iQ7z0v4bFfTlbIXujc3DrPV9CfCpYm+j/bdw229kthWF2JWfWshP46X9LIufD9P+Xmj59Oi+//DJZWVl0796d119/nSFDhpx2+7/97W9MmzaNvXv3kpyczKRJk7j77rsbsWIBWLFbd/xuFjZ87Pzb6QpoO8DcWs6Xvw0SLzS7ChFpBKaGmzlz5jB+/HimT5/OoEGDeOuttxgxYgRbtmwhOTm51vYzZsxg4sSJ/P3vf6dfv3789NNP3H///bRo0YKRI0ea8AmaL93xu4lw2OH7P8GRdPf2T//B+VdjQESkCbEYhmFaG23//v3p06cPM2bMcC3r2rUr119/PVOmTKm1/cCBAxk0aBAvv/yya9n48eNZs2YNy5Ytq9cxi4qKiIyMpLCwkIgIzcXirotfWkTGkVJmjunL0C5xZpcjp7P1S5hz5/m9R1AUPLYNAoI9UpKIiDvO5ffbtJab8vJy1q5dy1NPPVVj+bBhw1ixYkWd+5SVlREUVLMLJDg4mJ9++omKigoCAmoPZi0rK6Os7MRU8UVFRR6ovnnLyCsl40gp/n4WLkrRHb+9WnW30gUjoMNQ996j7QAFGxFpUkwLN7m5udjtduLiav5Xf1xcHNnZ2XXuM3z4cN555x2uv/56+vTpw9q1a5k5cyYVFRXk5uaSkFD7PhZTpkzhueeea5DP0Fy9u2wPAP3aReuO397sWAHs+Nr5fOgkiO9hajkiIo3F9F+mU+dEMQzjtPOkPP3002RnZ/OLX/wCwzCIi4tjzJgxvPTSS1it1jr3mThxIhMmTHC9LioqIikpyXMfoJnJyCvlXz9lAPDw0I4mV9NEORywaS6UHG7Y4+Rsdt58MbYrxKU27LFERLyIaeEmJiYGq9Vaq5UmJyenVmtOteDgYGbOnMlbb73FoUOHSEhI4O233yY8PJyYmLoHtgYGBhIYqJk+PeWv3++kwm4wpFMMAzWY2D3bv4J59zXe8XreonsUiUizYlq4sdlspKWlsXDhQm644QbX8oULF3Ldddedcd+AgADatGkDwOzZs7nmmmvw07wQDc7uMFiw2RlGHx6q+0e5bf8a59+YzpDQs2GPFRwNF/2mYY8hIuJlTO2WmjBhAnfddRd9+/ZlwIABvP3222RkZDB27FjA2aV04MABPvzwQwB27NjBTz/9RP/+/cnPz+e1115j06ZNfPDBB2Z+jGZj44FCjh6vJDzIn7S2Lcwup+mqvnlj/99Av0ZswRERaSZMDTejR48mLy+PyZMnk5WVRWpqKvPnz6dt27YAZGVlkZGR4drebrfz6quvsn37dgICArjssstYsWIF7dq1M+kTNC/Vt1sY2KElVj91c7itOtxoHIyISIMwdZ4bM2ieG/fd9vaPrNyTx5+u685dA9qZXU7TVHoEXkpxPn8qA4Iiza1HRKSJOJffbw1UkXo5Vm5n7b58QLMSn5ecLc6/UckKNiIiDUThRs7KMAxe/WY75XYHiZFBpMSEml1S01XdJdWqu7l1iIj4MNPnuRHvZncY/OHzTXxUNbfNuMs7nXYeIqkH13gbhRsRkYaicCOnZXcYjJ+zni9/PoifBabc2IPR/Wrf0NQn7PoOPr4byktqr/MPhKtfg9531P/9Pn8A1v+rjhVVQ9wUbkREGoy6peS0vt+Ww5c/HyTAauGN2/r4brABWPEGlBfjDB+nPCqPw7K/QH3H3ufvg/Wz6n4vcN6Isu0gD38AERGpppYbOa01e48AcHNaElf3rH3fLp9xNBvSlzif/2YxRLQ+sa7iGPytP+TthKz1kNj77O+38RPn33ZD4OaZtdcHRTpbg0REpEEo3MhprcssAKB3cpSpdTS4TXPBcECbi+oOL51HwOZ5zjtsny3cGMaJO3H3uhXCWnm+XhEROSOFG6lTpd3Bxv2FAFyYFFW/nSqOwfwnoN1g5w97U1EdRnqOqnt9z1HOcPO/D+HA2jO/l6MScreDNRC6jvRsnSIiUi8KN1KnnTnFHKuwExboT4fYsPrt9PNsWPcP2PwZdL0WbCENW6QnHN7h7G7y84fuN9S9TYfLITwBjmZB5qr6vW+36zSPjYiISRRupE4/V3VJ9WgdWf9bLVSPNSkvhh3/hdSbGqY4T9pY1WrT4XIIPc3khP42uO9bOLiufu/pF+BsvRIREVMo3Eid1leFm1717ZIqyIB9y0+83vCJ94ebk8fHnK5LqlpkG+dDRES8nsKN1PK/jHy+2pgF1GO8jcMBq96E3d85X7dIgfx02LUQFkw6/X5droa2A53P83Y7u7PsFXDBcEi5uH6Fbvy0/q0pdSkrgoJ9YAuDzle5/z4iIuJVFG6khj2Hi7nznVWUlttJa9uCoV3OcrXP1i9gwcQTr4dMgNXvQNbPsHLa6fdbPwse2+Hs8vnPeEj/wbl87Qfw+I6zj9cpPABz763XZzqrriObxvggERGpF4UbqeFfqzIoLbfTJzmKf9x7ETb/s8zzuGGO82/KxdDxCuh1u/OS6g2zwWGve5/1/4LSXGfrTsKFkL7UuTwkxrm8PuN1sjc4/4YnQI9b6v35agkIhr4eCkkiIuIVFG7Exe4w+OLngwCMvaQDIbazfD1Kj8DOhc7nI16CVl2dz1t1gV8+e/r9DIezVWfDx5C3CzAgeSC0HQBLX63feJ1Dm5x/2w2BYX8662cTEZHmQ+FGXH7ck0fO0TIigwO4tPNJ3VEleZCx0hlKTpbxIzgqIL7HiWBTHz1HOcPNjq9P3Eiy5yhIrgo3uxY6J9bzC6i5X2AYpFwCflY4tMW5TPdoEhGRUyjciMvn6w4AcFWPhJrdUbNvh8wfT79jj7NcaXSq+J4Q09k52V3eTmeI6XYdhEQ712VvgE9/Xfe+V70CF91/0t21U8/t2CIi4vMUbsRl2a5cAK45+T5SubucwcZihaSLau8UGgt97j63A1kscOWfYdnrzhl9u9/oDDYAw/8f/PCy88qpkx3Lh8PbnFdV9b7LGYpALTciIlKLwo0AUFhaQVbhcQB6tDlpZt3qSe46Xg53fOK5A3b8pfNxqpSL674UvCQXXu3svApr65fOLrLgFhAe77maRETEJ5zlUhhpLrZlFwHQOiqYiKCqsS6GceJqqHPtevK00BjnLMIA3092/o1LdbYCiYiInEThRgDYln0UgK4J4ScW7l8D+XshIBS6eMEkd9WzCBdkOP+qS0pEROqgcCPAiXDTOf6kcFPdatP1GrCFmlDVKbpc7ZwXx2J1dkl1u97sikRExAtpzI0AsL2qW6pzfIRzgb0CNs9zPje7S6paQDD83xKzqxARES+nlhvB4TDYXt0tVd1ys3sRlOY5r4Zqf6l5xYmIiJwjhRvhQMExSsrt2Kx+tIup6n6q7pJKvQmsauATEZGmQ+FGXONtOrQKI8DqB2VHYdtXzpU9vaRLSkREpJ4UboRtWc7xNq4uqW1fQeUxiO4AiX1MrExEROTcKdwI2w6dcqXUhqqJ+3qO0jwyIiLS5CjciKvlpktCBBTnwJ5FzhU9bjGxKhEREfco3DRzxyvs7M0rBaBLfLjzbtyGA1r3hZYdTK5ORETk3CncNHO7coqxOwyiQgJoFR54UpfUaHMLExERcZPCTTNXfaVUl/hwLHm74eD/nDMAd7/B5MpERETco3DTzFXPTNwlPuLEHcA7DIWwWBOrEhERcZ/CTTPnarmJC1OXlIiI+ASFm2bM7jDYsL8QgDT/3ZCf7j13ABcREXGTwk0ztvlgIYXHKggP9KdD9nznQm+5A7iIiIibFG6asWW7cgEY1D4Sv82fORd6yx3ARURE3KRw04wtrwo3N0anQ2mu7gAuIiI+QeGmmTpeYWf13nwAegcdci5sO1B3ABcRkSZP4aaZWrM3n/JKB/ERQcTYc5wLI5PMLUpERMQDFG6aqQ0HCgDolxKNpTDTuVDhRkREfIDCTTO1/aSZiSnc71wYpXAjIiJNn8JNM7Ut6+RwU91y08bEikRERDxD4aYZKq90sPtwMQBdYgOg5LBzhbqlRETEByjcNEN7coupdBiEB/mTSJ5zYUAoBLcwtzAREREPULhphk7ukrKcPN7GYjGxKhEREc9QuGmGqm+W2VnjbURExAcp3DRD27KLAOgcH3HiSimNtxERER+hcNMMVV8G3jU+HArUciMiIr7F9HAzffp0UlJSCAoKIi0tjaVLl55x+1mzZtGrVy9CQkJISEjgV7/6FXl5eY1UbdNXWFpBVuFxAC44uVsqKtnEqkRERDzH1HAzZ84cxo8fz6RJk1i3bh1DhgxhxIgRZGRk1Ln9smXLuPvuu7n33nvZvHkzn3zyCatXr+a+++5r5MqbruouqdZRwUQEWCBnq3OFwo2IiPgIU8PNa6+9xr333st9991H165def3110lKSmLGjBl1bv/jjz/Srl07xo0bR0pKCoMHD+b//u//WLNmTSNX3nRtP3TS5H3pi513Aw+OhsQ+5hYmIiLiIaaFm/LyctauXcuwYcNqLB82bBgrVqyoc5+BAweyf/9+5s+fj2EYHDp0iE8//ZSrr776tMcpKyujqKioxqM525p10pVSGz5xLux+A/jbTKxKRETEc0wLN7m5udjtduLi4mosj4uLIzs7u859Bg4cyKxZsxg9ejQ2m434+HiioqJ44403TnucKVOmEBkZ6XokJTXvq4K2V3VLdYuxwrb/OBf2HG1iRSIiIp5l+oBiyykTxxmGUWtZtS1btjBu3Dj++Mc/snbtWr7++mvS09MZO3bsad9/4sSJFBYWuh6ZmZkerb8pcTgM15VSvSvXQ3kxRLWFpIvMLUxERMSD/M06cExMDFartVYrTU5OTq3WnGpTpkxh0KBBPPHEEwD07NmT0NBQhgwZwvPPP09CQkKtfQIDAwkMDPT8B2iCDhQco6TcToDVQrwjx7mwdR/NTCwiIj7FtJYbm81GWloaCxcurLF84cKFDBw4sM59SktL8fOrWbLVagWcLT5yZtUzE3eIDcNamutcGNrKxIpEREQ8z9RuqQkTJvDOO+8wc+ZMtm7dyqOPPkpGRoarm2nixIncfffdru1HjhzJvHnzmDFjBnv27GH58uWMGzeOiy66iMTERLM+RpOxM8cZbi6IC4eSqpabsFgTKxIREfE807qlAEaPHk1eXh6TJ08mKyuL1NRU5s+fT9u2bQHIysqqMefNmDFjOHr0KNOmTeOxxx4jKiqKoUOH8uKLL5r1EZqU9MMlALSPDYVDh50L1XIjIiI+xmI0s/6coqIiIiMjKSwsJCIiwuxyGtVNM1awdl8+f72tN9euuh0OroPbZkPnEWaXJiIickbn8vtt+tVS0nj2HC4GoH1MKBSr5UZERHyTwk0zkV9STn5pBQDtY0I05kZERHyWwk0zsSfX2WqTEBlEiFEK9nLnilCFGxER8S0KN83E7pMHE1d3SdnCISDYxKpEREQ8T+GmmdhTHW5iwtQlJSIiPk3hpplIr+qWcrbcVIUbDSYWEREfpHDTTJzolgqDkqpuKbXciIiID1K4aQaOlJSzK8fZctMtIUItNyIi4tMUbpqB5buc95HqEh9ObHjgSS03CjciIuJ7FG6agepwM7hjjHNBdbgJjTGpIhERkYZj6r2lpOEZhsHSnbmkWLK4hR3w03I4tNm5Ut1SIiLigxRufFzGkVIOFBzju8BX6bDmYM2VEbqTuoiI+B6FGx+3bFcuEZTQwVIVbLpdB1gguj0k9jG1NhERkYagcOPjlu/KpbMl0/kiMglGfWhuQSIiIg1MA4p9mN1hsGJ3Hp39qsJNq27mFiQiItIIFG582JaDRRSUVtDTvyrcxHU3tyAREZFGoHDjw5ZVXQLeJ6hqvI3CjYiINANuhZvFixd7uAzxNMMwWLQtBwsO2lbudS5UuBERkWbArXBz5ZVX0qFDB55//nkyMzM9XZOcJ7vD4Km5G/lp7xGSLLkE2EvBaoOWHc0uTUREpMG5FW4OHjzII488wrx580hJSWH48OF8/PHHlJeXe7o+ccN7y9OZsyYTPwtMGWRxLoztDNYAcwsTERFpBG6Fm+joaMaNG8f//vc/1qxZQ+fOnXnwwQdJSEhg3Lhx/Pzzz56uU+qp6HgFf1u0C4Bnr+3OIPtq54rWfU2sSkREpPGc94DiCy+8kKeeeooHH3yQkpISZs6cSVpaGkOGDGHz5s2eqFHOwTs/7CG/tIIOsaHc3qcVbPnCuaLHLeYWJiIi0kjcDjcVFRV8+umnXHXVVbRt25YFCxYwbdo0Dh06RHp6OklJSdxyi35QG9tHq51joB4b1hn/Xd9AWRFEtIHkASZXJiIi0jjcmqH44Ycf5qOPPgLgzjvv5KWXXiI1NdW1PjQ0lBdeeIF27dp5pEipn/JKB4ePlgEwoH1L+PIT54oeN4OfrvoXEZHmwa1ws2XLFt544w1uuukmbDZbndskJiayaNGi8ypOzk1usTPYBFgtRAVbYc9i54rUG80rSkREpJG5FW6+++67s7+xvz+XXHKJO28vbsqparWJDQvEUpAB5cXOS8BbaX4bERFpPtzqq5gyZQozZ86stXzmzJm8+OKL512UuCen6DgAsRFBcKhqMHdsF7Dq/qgiItJ8uBVu3nrrLbp06VJreffu3XnzzTfPuyhxz+GqbqlW4YGQs8W5MC71DHuIiIj4HrfCTXZ2NgkJCbWWx8bGkpWVdd5FiXtyiqq6pcID4dAm58I43QlcRESaF7fCTVJSEsuXL6+1fPny5SQmJp53UeKe6jE3rcIDT3RL6X5SIiLSzLg1GOO+++5j/PjxVFRUMHToUMA5yPjJJ5/kscce82iBUn+HjzrH3CSEGJC327lQ3VIiItLMuBVunnzySY4cOcIDDzzgup9UUFAQv/vd75g4caJHC5T6C8/fTCCRtHNkAgaExEBYK7PLEhERaVRuhRuLxcKLL77I008/zdatWwkODqZTp04EBgZ6uj6pr5/n8JeCR+jrfzmtS52taRpvIyIizdF5XSMcFhZGv379PFWLnAdjzXtYgOutywjYk+9c2P4yU2sSERExg9vhZvXq1XzyySdkZGS4uqaqzZs377wLk3OQvw9L5koAQi1lkP0/53LdLFNERJoht66Wmj17NoMGDWLLli189tlnVFRUsGXLFr7//nsiIyM9XaOczUbnPaQchuXEsraDICrJpIJERETM41a4+fOf/8xf/vIX/vOf/2Cz2Zg6dSpbt25l1KhRJCcne7pGOZtNcwF4y37NiWU9R5lUjIiIiLncCje7d+/m6quvBiAwMJCSkhIsFguPPvoob7/9tkcLlLOwV8DhbQDMrLySZSGXQ3wP6H6DyYWJiIiYw61wEx0dzdGjRwFo3bo1mzY5Z8MtKCigtLTUc9XJ2RUdBMNBpcVGLpF8fcFzMHYZBKl7UEREmie3BhQPGTKEhQsX0qNHD0aNGsUjjzzC999/z8KFC7n88ss9XaOcSWEmAIcsLTHwY2CHGJMLEhERMZdb4WbatGkcP+6cDXfixIkEBASwbNkybrzxRp5++mmPFihnUbgfgPSKaCwWGNC+pckFiYiImOucw01lZSVffvklw4cPB8DPz48nn3ySJ5980uPFST0UOFtuDhoxpCZG0iLUZnJBIiIi5jrnMTf+/v789re/paysrCHqkXNV1S11wIhhcCd1SYmIiLg1oLh///6sW7fO07WIO6q6pQ7SkkEabyMiIuLemJsHHniAxx57jP3795OWlkZoaGiN9T179vRIcXJ2RkEmFpwtN10Sws0uR0RExHRuhZvRo0cDMG7cONcyi8WCYRhYLBbsdrtnqpMzMwyMQme4OUQs0SEabyMiIuJWuElPT/d0HeKO0iP4VR4DoCI0AT8/y1l2EBER8X1uhZu2bdt6ug5xR9Vg4hwjisgIdUmJiIiAm+Hmww8/POP6u+++261i5BwVVl8G3pJW4YEmFyMiIuId3Ao3jzzySI3XFRUVlJaWYrPZCAkJUbhpLFVXSu03YmgVoXAjIiICbl4Knp+fX+NRXFzM9u3bGTx4MB999NE5vdf06dNJSUkhKCiItLQ0li5detptx4wZg8ViqfXo3r27Ox+j6TtpAr/Y8CCTixEREfEOboWbunTq1IkXXnihVqvOmcyZM4fx48czadIk1q1bx5AhQxgxYgQZGRl1bj916lSysrJcj8zMTKKjo7nllls89TGalpMm8ItVt5SIiAjgwXADYLVaOXjwYL23f+2117j33nu577776Nq1K6+//jpJSUnMmDGjzu0jIyOJj493PdasWUN+fj6/+tWvTnuMsrIyioqKajx8RvUEfhpzIyIi4uLWmJsvvviixmvDMMjKymLatGkMGjSoXu9RXl7O2rVreeqpp2osHzZsGCtWrKjXe7z77rv88pe/POPVW1OmTOG5556r1/s1OYUnuqUUbkRERJzcCjfXX399jdcWi4XY2FiGDh3Kq6++Wq/3yM3NxW63ExcXV2N5XFwc2dnZZ90/KyuL//73v/zrX/8643YTJ05kwoQJrtdFRUUkJSXVq0avVnEMSg4D1QOKNeZGREQE3Aw3DofDYwVYLDUnnque5fhs3n//faKiomoFrVMFBgYSGOiDrRqFBwAoMQIpJJSYMM1OLCIiAh4ec3MuYmJisFqttVppcnJyarXmnMowDGbOnMldd92FzdZMf9RPGkwcFWIj0N9qckEiIiLewa1wc/PNN/PCCy/UWv7yyy/X+8olm81GWloaCxcurLF84cKFDBw48Iz7LlmyhF27dnHvvffWv2hfo/E2IiIidXIr3CxZsoSrr7661vIrr7ySH374od7vM2HCBN555x1mzpzJ1q1befTRR8nIyGDs2LGAc7xMXRMCvvvuu/Tv35/U1FR3yvcNVVdKHTBiaKU5bkRERFzcGnNTXFxcZ3dQQEDAOV1qPXr0aPLy8pg8eTJZWVmkpqYyf/5819VPWVlZtea8KSwsZO7cuUydOtWd0n1HQXW3VEvNcSMiInISt8JNamoqc+bM4Y9//GON5bNnz6Zbt27n9F4PPPAADzzwQJ3r3n///VrLIiMjKS0tPadj+KSTxty00mBiERERF7fCzdNPP81NN93E7t27GTp0KADfffcdH330EZ988olHC5TTcE3gF0O3MLXciIiIVHMr3Fx77bV8/vnn/PnPf+bTTz8lODiYnj178u2333LJJZd4ukY51dFsKNgHwF4jjpYKNyIiIi5uhRuAq6++us5BxdIINs0Fw8EWaxcO00Jz3IiIiJzEraulVq9ezapVq2otX7VqFWvWrDnvouQsNnwMwJfGEABi1HIjIiLi4la4efDBB8nMzKy1/MCBAzz44IPnXZScweEdkLUew8+fT473BdDVUiIiIidxK9xs2bKFPn361Freu3dvtmzZct5FyRls/wqAinaXkesIByA6VN1SIiIi1dwKN4GBgRw6dKjW8qysLPz93R7GI/WRvQmAwtg0AKJCAgiwmnYXDREREa/j1q/iFVdcwcSJEyksLHQtKygo4Pe//z1XXHGFx4qTOhzaDMDhkI6AxtuIiIicyq1mlldffZWLL76Ytm3b0rt3bwDWr19PXFwc//jHPzxaoJyksgxydwCQaWsPZNNSXVIiIiI1uBVuWrduzYYNG5g1axY///wzwcHB/OpXv+K2224jICDA0zVKtdwdYNghKJIDlS2AbGI0mFhERKQGtwfIhIaGMnjwYJKTkykvLwfgv//9L+Cc5E8aQFWXFHGp5JY4z3msuqVERERqcCvc7NmzhxtuuIGNGzdisVgwDAOLxeJab7fbPVagnOSQczAxcd3JLS4DULeUiIjIKdwaUPzII4+QkpLCoUOHCAkJYdOmTSxZsoS+ffuyePFiD5coLoeqLrNv1Y28YmfLjbqlREREanIr3KxcuZLJkycTGxuLn58fVquVwYMHM2XKFMaNG+fpGqXayd1SVS03ulpKRESkJrfCjd1uJywsDICYmBgOHjwIQNu2bdm+fbvnqpMTSnKhONv5vFVXcqtablrqvlIiIiI1uDXmJjU1lQ0bNtC+fXv69+/PSy+9hM1m4+2336Z9+/aerlHgRKtNi3YYtlAOV7XcaECxiIhITW6Fmz/84Q+UlJQA8Pzzz3PNNdcwZMgQWrZsyZw5czxaoFTJqRpvE5dKfmkF5ZUOAFpFKNyIiIiczK1wM3z4cNfz9u3bs2XLFo4cOUKLFi1qXDUlHnTSlVLZhccB5z2lAv2tJhYlIiLifTx2I6jo6GhPvZXUpbpbqlU3DhU5w018RJCJBYmIiHgn3XGxKXDYIWeb83lcKtnV4SZS4UZERORUCjdNwZF0qDwG/sEQneLqlopTy42IiEgtCjdNQfV4m1ZdwM/qCjfqlhIREalN4aYpqLoTOK26AZzULaUrpURERE6lcNMUFGQ4/0a1BTgxoDgy2KyKREREvJbCTVNQmOn8G5UEnNRyo24pERGRWhRumoLC/c6/kW04XmGnoLQCULgRERGpi8KNtzOMk8JNkmswcVCAHxHBHpumSERExGco3Hi7klyoPA5YIKJ1jS4pzQYtIiJSm8KNtyusGkwcHg/+tpMGE6tLSkREpC4KN97upPE2AOsyCgBI1JVSIiIidVK48XYFVVdKRSZxoOAY/1rlbMm5rndrE4sSERHxXgo33u6klpvXF+6g3O5gQPuWXNwpxty6REREvJTCjbermuOmIrwNn68/AMDjwztrMLGIiMhpKNx4u6pws68ymgq7QUyYjT7JUebWJCIi4sUUbrxd1ZibjcURAPRqE6VWGxERkTNQuPFm5SVw7AgAP+Y5r47qlRRlYkEiIiLeT+HGm1UPJg6M4McsOwAXKtyIiIickcKNN6sab1MZ3pp9eaWAs1tKRERETk/hxptVjbcpsMUB0D4mlMiQADMrEhER8XoKN96sqltq+WHneJveyS3MrEZERKRJULjxYo4C52zEW0sjSYwM4pHLO5lckYiIiPdTuPFiRw/tBeBIQCs++e1AkluGmFuQiIhIE6Bw48Xs+c6Wm3btu9A6SjfKFBERqQ+FGy9VVl5OREUOAP0v7GVyNSIiIk2Hwo2X+vHnzfjjoBIrvbt1MbscERGRJkPhxkvt2L4VgKO2Vvj5+5tcjYiISNOhcOOlCrP3AOCIaGNyJSIiIk2Lwo0XOl5hJ6DAGW6CYtqaXI2IiEjTonDjhTYfLGKY308AhHQcYnI1IiIiTYvp4Wb69OmkpKQQFBREWloaS5cuPeP2ZWVlTJo0ibZt2xIYGEiHDh2YOXNmI1XbODK2/ERXv0wqCMDS/TqzyxEREWlSTB2pOmfOHMaPH8/06dMZNGgQb731FiNGjGDLli0kJyfXuc+oUaM4dOgQ7777Lh07diQnJ4fKyspGrrxhhe/8DICMloPpEKxbLoiIiJwLi2EYhlkH79+/P3369GHGjBmuZV27duX6669nypQptbb/+uuvufXWW9mzZw/R0dFuHbOoqIjIyEgKCwuJiIhwu/YGYxgcmtyROCOXLUOm0e3yu8yuSERExHTn8vttWrdUeXk5a9euZdiwYTWWDxs2jBUrVtS5zxdffEHfvn156aWXaN26NRdccAGPP/44x44dO+1xysrKKCoqqvHwZseP7CfOyKXS8CM+7VqzyxEREWlyTOuWys3NxW63ExcXV2N5XFwc2dnZde6zZ88eli1bRlBQEJ999hm5ubk88MADHDly5LTjbqZMmcJzzz3n8fobSuHe9QQB6STSMdILW5ZERES8nOkDii0WS43XhmHUWlbN4XBgsViYNWsWF110EVdddRWvvfYa77///mlbbyZOnEhhYaHrkZmZ6fHP4EllBzYAkOGfctrzICIiIqdnWstNTEwMVqu1VitNTk5OrdacagkJCbRu3ZrIyEjXsq5du2IYBvv376dTp0619gkMDCQwMNCzxTcgS85mAHJCOphciYiISNNkWsuNzWYjLS2NhQsX1li+cOFCBg4cWOc+gwYN4uDBgxQXF7uW7dixAz8/P9q08Y2ZfEPytwNwNPICkysRERFpmkztlpowYQLvvPMOM2fOZOvWrTz66KNkZGQwduxYwNmldPfdd7u2v/3222nZsiW/+tWv2LJlCz/88ANPPPEEv/71rwkODjbrY3hOZTlRJekAVMR0M7kYERGRpsnUeW5Gjx5NXl4ekydPJisri9TUVObPn0/bts5bDmRlZZGRkeHaPiwsjIULF/Lwww/Tt29fWrZsyahRo3j++efN+gielbsDK3aKjBBCdNsFERERt5g6z40ZvHqemw0fw7z7+cnRmbxb/s2IHglmVyQiIuIVmsQ8N1KHQ5sA2OZIJj4yyORiREREmiaFGy9iZDuvlNpmKNyIiIi4S+HGiziqWm62G0nEhjWdy9dFRES8icKNtyg9grXYOedPXkhH/K36pxEREXGHfkG9xSFnl1SGI5aIKN0JXERExF0KN97i0EnjbSI03kZERMRdCjfeIqc63CTRpkWIycWIiIg0XQo3XqIk03nDzO1GMrdelGRyNSIiIk2Xwo03cDiw5m4FILnrRVwQF25yQSIiIk2Xwo0XKD20iyCjjONGALdfeanZ5YiIiDRpCjdeoGDvOgB2W5JIilGrjYiIyPlQuPECZQc2AnDA1t7kSkRERJo+hRsvYD28BYD8sI4mVyIiItL0Kdx4gfDC7QAci+5qciUiIiJNn8KN2cqKiTx+AABLXKrJxYiIiDR9CjdmO7wNPwxyjCiiWyWaXY2IiEiTp3BjturbLjiSSIwKNrkYERGRpk/hxmSOQ5sA5z2lWivciIiInDeFG5NVVF0GvpNkWoUHmlyNiIhI06dwYybDcF0GnhfWET8/i8kFiYiINH0KN2Y6moV/eSGVhh/lUZ3MrkZERMQnKNyYqWow8R4jgVYtIk0uRkRExDco3JipajDxdiOJLgm6p5SIiIgnKNyYqHS/czDxVkcy1/TUHDciIiKeoHBjBocdMlZRvm+182Wr7prjRkRExEP8zS6gWVr1FiyYSFTVyy69fmFmNSIiIj5F4cYM+5YDcMBoybeOvlzXr7fJBYmIiPgOhRszVA0kfrxiLBVJg7knVJP3iYiIeIrG3DS2sqOQvxdw3k9qUMcYc+sRERHxMQo3jS1nm/MP0eQTweBOCjciIiKepHDT2Kq6pLba2xBqs3JhUpS59YiIiPgYhZvGVjUr8VYjmf7tWxJg1T+BiIiIJ+mXtbHlOG+Uuc2RTPfECJOLERER8T0KN43JMGrcciE61GZyQSIiIr5H4aYxHfgfHC/kuCWQ3UYiLUIUbkRERDxN4aYxbfwYgJUBv6CcAFqo5UZERMTjFG4ai70SNs0F4AvHYABahASYWZGIiIhPUrjxtKKDsOjPcCy/5vI9i6HkMITE8M3xrgDqlhIREWkACjee9vVTsORF+P75mss3zAGgsusNlFQ6T7u6pURERDxP4caTjhXA9q+dzzfNA3uF83lZMWz7DwBHOt4AQIDVQqjNakKRIiIivk3hxpO2fgH2MufzY0dg13fO59vnQ0UptEghJ7w74OySslgsJhUqIiLiu3RXcE/a4LwaiqAoOF4AK6dBaR6sfc+5vOco8o85W3M03kZERKRhqOXGUwoPwN5lzufX/tX5d+9S+PcDsH+183WPUeSXVoWbUF0pJSIi0hDUcuMpjkrofQeU5EHXa+GSp+DA2hPr2w2GmI7k79gLqOVGRESkoSjceEqLtnDd3068vmxinZvll5Y7N9eVUiIiIg1C3VKNrKC6W0oT+ImIiDQIhZtGdqSkquVG3VIiIiINQuGmkbm6pRRuREREGoTCTSM7MeZG3VIiIiINQeGmkeWXaJ4bERGRhmR6uJk+fTopKSkEBQWRlpbG0qVLT7vt4sWLsVgstR7btm1rxIrPj7qlREREGpap4WbOnDmMHz+eSZMmsW7dOoYMGcKIESPIyMg4437bt28nKyvL9ejUqVMjVXx+VuzOpbTcDuhScBERkYZiarh57bXXuPfee7nvvvvo2rUrr7/+OklJScyYMeOM+7Vq1Yr4+HjXw2o9/Q0oy8rKKCoqqvEww9p9+Yx5zzlT8RXd4ogM1pgbERGRhmBauCkvL2ft2rUMGzasxvJhw4axYsWKM+7bu3dvEhISuPzyy1m0aNEZt50yZQqRkZGuR1JS0nnX7o6PfsqgvNLBxRfE8sZtvU2pQUREpDkwLdzk5uZit9uJi4ursTwuLo7s7Ow690lISODtt99m7ty5zJs3j86dO3P55Zfzww8/nPY4EydOpLCw0PXIzMz06Oeor23Zzhaj2y9KIijg9C1NIiIicn5Mv/2CxWKp8dowjFrLqnXu3JnOnTu7Xg8YMIDMzExeeeUVLr744jr3CQwMJDAw0HMFu6HS7mDnoWIAusRHmFqLiIiIrzMt3MTExGC1Wmu10uTk5NRqzTmTX/ziF/zzn//0dHketTevlLJKB8EBVpKjQ8wuR0TEZzgcDsrLy80uQzzEZrPh53f+nUqmhRubzUZaWhoLFy7khhtucC1fuHAh1113Xb3fZ926dSQkJDREiR6zPfsoABfEheHnV3erlIiInJvy8nLS09NxOBxmlyIe4ufnR0pKCjbb+V1RbGq31IQJE7jrrrvo27cvAwYM4O233yYjI4OxY8cCzvEyBw4c4MMPPwTg9ddfp127dnTv3p3y8nL++c9/MnfuXObOnWvmxzir7VXjbdQlJSLiGYZhkJWVhdVqJSkpySP/tS/mcjgcHDx4kKysLJKTk087RKU+TA03o0ePJi8vj8mTJ5OVlUVqairz58+nbdu2AGRlZdWY86a8vJzHH3+cAwcOEBwcTPfu3fnqq6+46qqrzPoI9bK1quWmc3y4yZWIiPiGyspKSktLSUxMJCRE3f2+IjY2loMHD1JZWUlAgPtTplgMwzA8WJfXKyoqIjIyksLCQiIiGqcl5eKXFpFxpJR/3defgR1jGuWYIiK+7Pjx46Snp9OuXTuCg4PNLkc85NixY+zdu9d154KTncvvt9rxGlhxWSUZR0oBtdyIiHja+XRdiPfx1L+nwk0D27i/EIDEyCBahpl7SbqIiEhzoHDTwNZnFgBwYXKUqXWIiIjvadeuHa+//nq9t6++AXVBQUGD1eQNTJ/Ez9f9XBVuerWJMrUOERHxDpdeeikXXnjhOYWS01m9ejWhoaH13n7gwIFkZWURGRl53sf2Zgo3DczVcpMUZWodIiLSNBiGgd1ux9//7D/RsbGx5/TeNpuN+Ph4d0trMtQt1YCyC4+TXXQcPwuktvbtlCwiYibDMCgtrzTlcS4XHY8ZM4YlS5YwdepULBYLFouF999/H4vFwoIFC+jbty+BgYEsXbqU3bt3c9111xEXF0dYWBj9+vXj22+/rfF+p3ZLWSwW3nnnHW644QZCQkLo1KkTX3zxhWv9qd1S77//PlFRUSxYsICuXbsSFhbGlVdeSVZWlmufyspKxo0bR1RUFC1btuR3v/sd99xzD9dff71b/1aNQS03Dai61eaCuHBCA3WqRUQayrEKO93+uMCUY2+ZPJwQW/3+P37q1Kns2LGD1NRUJk+eDMDmzZsBePLJJ3nllVdo3749UVFR7N+/n6uuuornn3+eoKAgPvjgA0aOHMn27dtJTk4+7TGee+45XnrpJV5++WXeeOMN7rjjDvbt20d0dHSd25eWlvLKK6/wj3/8Az8/P+68804ef/xxZs2aBcCLL77IrFmzeO+99+jatStTp07l888/57LLLjuX09So1HLTgDYeKADUJSUiIk6RkZHYbDZCQkKIj48nPj4eq9UKwOTJk7niiivo0KEDLVu2pFevXvzf//0fPXr0oFOnTjz//PO0b9++RktMXcaMGcNtt91Gx44d+fOf/0xJSQk//fTTabevqKjgzTffpG/fvvTp04eHHnqI7777zrX+jTfeYOLEidxwww106dKFadOmERUV5ZHz0VDUnNCAsgqOA9Aupv6DvURE5NwFB1jZMnm4acf2hL59+9Z4XVJSwnPPPcd//vMf16y9x44dqzFzf1169uzpeh4aGkp4eDg5OTmn3T4kJIQOHTq4XickJLi2Lyws5NChQ1x00UWu9VarlbS0NK++p5fCTQPKK3HeqTY69PxuACYiImdmsVjq3TXkrU696umJJ55gwYIFvPLKK3Ts2JHg4GBuvvnms94F/dTbFlgsljMGkbq2P3Uc0amT63n7zQ3ULdWAjlSFm5YKNyIiUsVms2G328+63dKlSxkzZgw33HADPXr0ID4+nr179zZ8gSeJjIwkLi6uRreW3W5n3bp1jVrHuWraMdfLHVHLjYiInKJdu3asWrWKvXv3EhYWdtpWlY4dOzJv3jxGjhyJxWLh6aefNqUr6OGHH2bKlCl07NiRLl268MYbb5Cfn+/Vt75Qy00DyispA6BlqG67ICIiTo8//jhWq5Vu3boRGxt72jE0f/nLX2jRogUDBw5k5MiRDB8+nD59+jRytfC73/2O2267jbvvvpsBAwYQFhbG8OHDa93Y0pvoruANpLS80nVZ4qbnhhOmS8FFRDym+q7gdd09WhqWw+Gga9eujBo1ij/96U8efe8z/buey++3fnEbSF6xs0vK5u9HqM0zI+lFREQa2759+/jmm2+45JJLKCsrY9q0aaSnp3P77bebXdppqVuqgbjG24TYvLpfUkRE5Ez8/Px4//336devH4MGDWLjxo18++23dO3a1ezSTkstNw1Eg4lFRMQXJCUlsXz5crPLOCdquWkgrsvAwxRuREREGpPCTQNRy42IiIg5FG4aiGYnFhERMYfCTQM54prjRuFGRESkMSncNJAT3VKawE9ERKQxKdw0EHVLiYiImEPhpoHoaikREWkI7dq14/XXX3e9tlgsfP7556fdfu/evVgsFtavX39ex/XU+zQGzXPTQI4Uq+VGREQaXlZWFi1atPDoe44ZM4aCgoIaoSkpKYmsrCxiYmI8eqyGoHDTAMoq7RwtqwQ0oFhERBpWfHx8oxzHarU22rHOl7qlGsDunBIAggOsRAQFmFyNiEgzYBhQXmLO4xzuP/3WW2/RunVrHA5HjeXXXnst99xzD7t37+a6664jLi6OsLAw+vXrx7fffnvG9zy1W+qnn36id+/eBAUF0bdvX9atW1dje7vdzr333ktKSgrBwcF07tyZqVOnutY/++yzfPDBB/z73//GYrFgsVhYvHhxnd1SS5Ys4aKLLiIwMJCEhASeeuopKisrXesvvfRSxo0bx5NPPkl0dDTx8fE8++yz9T5f7lLLjQdV2h3kHC1jxe5cAPq3j8bPT/eVEhFpcBWl8OdEc479+4NgC63Xprfccgvjxo1j0aJFXH755QDk5+ezYMECvvzyS4qLi7nqqqt4/vnnCQoK4oMPPmDkyJFs376d5OTks75/SUkJ11xzDUOHDuWf//wn6enpPPLIIzW2cTgctGnTho8//piYmBhWrFjBb37zGxISEhg1ahSPP/44W7dupaioiPfeew+A6OhoDh48WON9Dhw4wFVXXcWYMWP48MMP2bZtG/fffz9BQUE1AswHH3zAhAkTWLVqFStXrmTMmDEMGjSIK664ol7nzB0KNx6y+WAhD3+0jkB/K7Hhzsu/B3f0/n5JERFpPNHR0Vx55ZX861//coWbTz75hOjoaC6//HKsViu9evVybf/888/z2Wef8cUXX/DQQw+d9f1nzZqF3W5n5syZhISE0L17d/bv389vf/tb1zYBAQE899xzrtcpKSmsWLGCjz/+mFGjRhEWFkZwcDBlZWVn7IaaPn06SUlJTJs2DYvFQpcuXTh48CC/+93v+OMf/4ifn7NzqGfPnjzzzDMAdOrUiWnTpvHdd98p3DQFiZHBHC4q42hZJVuznMsGKdyIiDSOgBBnC4pZxz4Hd9xxB7/5zW+YPn06gYGBzJo1i1tvvRWr1UpJSQnPPfcc//nPfzh48CCVlZUcO3aMjIyMer331q1b6dWrFyEhJ2oaMGBAre3efPNN3nnnHfbt28exY8coLy/nwgsvPKfPsXXrVgYMGIDFcqKHYtCgQRQXF7N//35XS1PPnj1r7JeQkEBOTs45HetcKdx4SItQG7+5uD2vLtwBQEyYjc5x4SZXJSLSTFgs9e4aMtvIkSNxOBx89dVX9OvXj6VLl/Laa68B8MQTT7BgwQJeeeUVOnbsSHBwMDfffDPl5eX1em+jHuN/Pv74Yx599FFeffVVBgwYQHh4OC+//DKrVq06p89hGEaNYHPy8U9eHhBQc+ypxWKpNebI0xRuPOjXg1P4YOU+covLGNghRuNtRESkluDgYG688UZmzZrFrl27uOCCC0hLSwNg6dKljBkzhhtuuAGA4uJi9u7dW+/37tatG//4xz84duwYwcHBAPz44481tlm6dCkDBw7kgQcecC3bvXt3jW1sNht2u/2sx5o7d26NkLNixQrCw8Np3bp1vWtuCLpayoNCA/3503XdadMimLsGtDW7HBER8VJ33HEHX331FTNnzuTOO+90Le/YsSPz5s1j/fr1/Pzzz9x+++3n1Mpx++234+fnx7333suWLVuYP38+r7zySo1tOnbsyJo1a1iwYAE7duzg6aefZvXq1TW2adeuHRs2bGD79u3k5uZSUVFR61gPPPAAmZmZPPzww2zbto1///vfPPPMM0yYMME13sYsCjceNqJHAst+N5R+7aLNLkVERLzU0KFDiY6OZvv27dx+++2u5X/5y19o0aIFAwcOZOTIkQwfPpw+ffrU+33DwsL48ssv2bJlC71792bSpEm8+OKLNbYZO3YsN954I6NHj6Z///7k5eXVaMUBuP/+++ncuTN9+/YlNjaW5cuX1zpW69atmT9/Pj/99BO9evVi7Nix3HvvvfzhD384x7PheRajPh10PqSoqIjIyEgKCwuJiIgwuxwREXHD8ePHSU9PJyUlhaCgILPLEQ8507/rufx+q+VGREREfIrCjYiIiPgUhRsRERHxKQo3IiIi4lMUbkREpMlqZtfE+DxP/Xsq3IiISJNjtVoB6j1zrzQN1f+e1f++7tIMxSIi0uT4+/sTEhLC4cOHCQgIMH3SODl/DoeDw4cPExISgr//+cUThRsREWlyLBYLCQkJpKens2/fPrPLEQ/x8/MjOTm51j2rzpXCjYiINEk2m41OnTqpa8qH2Gw2j7TCKdyIiEiT5efnpxmKpRZ1UoqIiIhPUbgRERERn6JwIyIiIj6l2Y25qZ4gqKioyORKREREpL6qf7frM9Ffsws3R48eBSApKcnkSkRERORcHT16lMjIyDNuYzGa2dzVDoeDgwcPEh4eft7X0Z+qqKiIpKQkMjMziYiI8Oh7+xqdq3Oj81V/OlfnRuer/nSu6q8hzpVhGBw9epTExMSzXi7e7Fpu/Pz8aNOmTYMeIyIiQl/8etK5Ojc6X/Wnc3VudL7qT+eq/jx9rs7WYlNNA4pFRETEpyjciIiIiE9RuPGgwMBAnnnmGQIDA80uxevpXJ0bna/607k6Nzpf9adzVX9mn6tmN6BYREREfJtabkRERMSnKNyIiIiIT1G4EREREZ+icCMiIiI+ReHGQ6ZPn05KSgpBQUGkpaWxdOlSs0vyCs8++ywWi6XGIz4+3rXeMAyeffZZEhMTCQ4O5tJLL2Xz5s0mVtx4fvjhB0aOHEliYiIWi4XPP/+8xvr6nJuysjIefvhhYmJiCA0N5dprr2X//v2N+Ckax9nO1ZgxY2p9z37xi1/U2Ka5nKspU6bQr18/wsPDadWqFddffz3bt2+vsY2+WyfU53zp++U0Y8YMevbs6ZqYb8CAAfz3v/91rfem75XCjQfMmTOH8ePHM2nSJNatW8eQIUMYMWIEGRkZZpfmFbp3705WVpbrsXHjRte6l156iddee41p06axevVq4uPjueKKK1z3APNlJSUl9OrVi2nTptW5vj7nZvz48Xz22WfMnj2bZcuWUVxczDXXXIPdbm+sj9EoznauAK688soa37P58+fXWN9cztWSJUt48MEH+fHHH1m4cCGVlZUMGzaMkpIS1zb6bp1Qn/MF+n4BtGnThhdeeIE1a9awZs0ahg4dynXXXecKMF71vTLkvF100UXG2LFjayzr0qWL8dRTT5lUkfd45plnjF69etW5zuFwGPHx8cYLL7zgWnb8+HEjMjLSePPNNxupQu8AGJ999pnrdX3OTUFBgREQEGDMnj3btc2BAwcMPz8/4+uvv2602hvbqefKMAzjnnvuMa677rrT7tNcz5VhGEZOTo4BGEuWLDEMQ9+tszn1fBmGvl9n0qJFC+Odd97xuu+VWm7OU3l5OWvXrmXYsGE1lg8bNowVK1aYVJV32blzJ4mJiaSkpHDrrbeyZ88eANLT08nOzq5x7gIDA7nkkkua/bmrz7lZu3YtFRUVNbZJTEwkNTW1WZ6/xYsX06pVKy644ALuv/9+cnJyXOua87kqLCwEIDo6GtB362xOPV/V9P2qyW63M3v2bEpKShgwYIDXfa8Ubs5Tbm4udruduLi4Gsvj4uLIzs42qSrv0b9/fz788EMWLFjA3//+d7Kzsxk4cCB5eXmu86NzV1t9zk12djY2m40WLVqcdpvmYsSIEcyaNYvvv/+eV199ldWrVzN06FDKysqA5nuuDMNgwoQJDB48mNTUVEDfrTOp63yBvl8n27hxI2FhYQQGBjJ27Fg+++wzunXr5nXfq2Z3V/CGYrFYarw2DKPWsuZoxIgRruc9evRgwIABdOjQgQ8++MA1IE/n7vTcOTfN8fyNHj3a9Tw1NZW+ffvStm1bvvrqK2688cbT7ufr5+qhhx5iw4YNLFu2rNY6fbdqO9350vfrhM6dO7N+/XoKCgqYO3cu99xzD0uWLHGt95bvlVpuzlNMTAxWq7VW6szJyamVYAVCQ0Pp0aMHO3fudF01pXNXW33OTXx8POXl5eTn5592m+YqISGBtm3bsnPnTqB5nquHH36YL774gkWLFtGmTRvXcn236na681WX5vz9stlsdOzYkb59+zJlyhR69erF1KlTve57pXBznmw2G2lpaSxcuLDG8oULFzJw4ECTqvJeZWVlbN26lYSEBFJSUoiPj69x7srLy1myZEmzP3f1OTdpaWkEBATU2CYrK4tNmzY1+/OXl5dHZmYmCQkJQPM6V4Zh8NBDDzFv3jy+//57UlJSaqzXd6ums52vujTn79epDMOgrKzM+75XHh2e3EzNnj3bCAgIMN59911jy5Ytxvjx443Q0FBj7969Zpdmuscee8xYvHixsWfPHuPHH380rrnmGiM8PNx1bl544QUjMjLSmDdvnrFx40bjtttuMxISEoyioiKTK294R48eNdatW2esW7fOAIzXXnvNWLdunbFv3z7DMOp3bsaOHWu0adPG+Pbbb43//e9/xtChQ41evXoZlZWVZn2sBnGmc3X06FHjscceM1asWGGkp6cbixYtMgYMGGC0bt26WZ6r3/72t0ZkZKSxePFiIysry/UoLS11baPv1glnO1/6fp0wceJE44cffjDS09ONDRs2GL///e8NPz8/45tvvjEMw7u+Vwo3HvK3v/3NaNu2rWGz2Yw+ffrUuIywORs9erSRkJBgBAQEGImJicaNN95obN682bXe4XAYzzzzjBEfH28EBgYaF198sbFx40YTK248ixYtMoBaj3vuuccwjPqdm2PHjhkPPfSQER0dbQQHBxvXXHONkZGRYcKnaVhnOlelpaXGsGHDjNjYWCMgIMBITk427rnnnlrnobmcq7rOE2C89957rm303TrhbOdL368Tfv3rX7t+52JjY43LL7/cFWwMw7u+VxbDMAzPtgWJiIiImEdjbkRERMSnKNyIiIiIT1G4EREREZ+icCMiIiI+ReFGREREfIrCjYiIiPgUhRsRERHxKQo3IiIi4lMUbkSk2Vm8eDEWi4WCggKzSxGRBqBwIyIiIj5F4UZERER8isKNiDQ6wzB46aWXaN++PcHBwfTq1YtPP/0UONFl9NVXX9GrVy+CgoLo378/GzdurPEec+fOpXv37gQGBtKuXTteffXVGuvLysp48sknSUpKIjAwkE6dOvHuu+/W2Gbt2rX07duXkJAQBg4cyPbt213rfv75Zy677DLCw8OJiIggLS2NNWvWNNAZERFP8je7ABFpfv7whz8wb948ZsyYQadOnfjhhx+48847iY2NdW3zxBNPMHXqVOLj4/n973/Ptddey44dOwgICGDt2rWMGjWKZ599ltGjR7NixQoeeOABWrZsyZgxYwC4++67WblyJX/961/p1asX6enp5Obm1qhj0qRJvPrqq8TGxjJ27Fh+/etfs3z5cgDuuOMOevfuzYwZM7Baraxfv56AgIBGO0cich48fp9xEZEzKC4uNoKCgowVK1bUWH7vvfcat912m7Fo0SIDMGbPnu1al5eXZwQHBxtz5swxDMMwbr/9duOKK66osf8TTzxhdOvWzTAMw9i+fbsBGAsXLqyzhupjfPvtt65lX331lQEYx44dMwzDMMLDw43333///D+wiDQ6dUuJSKPasmULx48f54orriAsLMz1+PDDD9m9e7druwEDBrieR0dH07lzZ7Zu3QrA1q1bGTRoUI33HTRoEDt37sRut7N+/XqsViuXXHLJGWvp2bOn63lCQgIAOTk5AEyYMIH77ruPX/7yl7zwwgs1ahMR76ZwIyKNyuFwAPDVV1+xfv1612PLli2ucTenY7FYAOeYnern1QzDcD0PDg6uVy0ndzNVv191fc8++yybN2/m6quv5vvvv6dbt2589tln9XpfETGXwo2INKpu3boRGBhIRkYGHTt2rPFISkpybffjjz+6nufn57Njxw66dOnieo9ly5bVeN8VK1ZwwQUXYLVa6dGjBw6HgyVLlpxXrRdccAGPPvoo33zzDTfeeCPvvffeeb2fiDQODSgWkUYVHh7O448/zqOPPorD4WDw4MEUFRWxYsUKwsLCaNu2LQCTJ0+mZcuWxMXFMWnSJGJiYrj++usBeOyxx+jXrx9/+tOfGD16NCtXrmTatGlMnz4dgHbt2nHPPffw61//2jWgeN++feTk5DBq1Kiz1njs2DGeeOIJbr75ZlJSUti/fz+rV6/mpptuarDzIiIeZPagHxFpfhwOhzF16lSjc+fORkBAgBEbG2sMHz7cWLJkiWuw75dffml0797dsNlsRr9+/Yz169fXeI9PP/3U6NatmxEQEGAkJycbL7/8co31x44dMx599FEjISHBsNlsRseOHY2ZM2cahnFiQHF+fr5r+3Xr1hmAkZ6ebpSVlRm33nqrkZSUZNhsNiMxMdF46KGHXIONRcS7WQzjpI5qERGTLV68mMsuu4z8/HyioqLMLkdEmiCNuRERERGfonAjIiIiPkXdUiIiIuJT1HIjIiIiPkXhRkRERHyKwo2IiIj4FIUbERER8SkKNyIiIuJTFG5ERETEpyjciIiIiE9RuBERERGf8v8BuYRXSLcNyusAAAAASUVORK5CYII=\n",
|
||
"text/plain": [
|
||
"<Figure size 640x480 with 1 Axes>"
|
||
]
|
||
},
|
||
"metadata": {},
|
||
"output_type": "display_data"
|
||
}
|
||
],
|
||
"source": [
|
||
"import matplotlib.pyplot as plt\n",
|
||
"\n",
|
||
"plt.plot(scores[\"train_accs\"], label=\"training\")\n",
|
||
"plt.plot(scores[\"val_accs\"], label=\"validation\")\n",
|
||
"plt.xlabel(\"epochs\")\n",
|
||
"plt.ylabel(\"accuracy\")\n",
|
||
"plt.legend()\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "807224cb",
|
||
"metadata": {},
|
||
"source": [
|
||
"### Multiclass classification\n",
|
||
"Finally, we will demonstrate the use case of multiclass classification using our FFNN with the famous MNIST dataset, which contain images of digits between the range of 0 to 9. The MNIST dataset has 64 features."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"id": "70ed6a7e",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.datasets import load_digits\n",
|
||
"\n",
|
||
"def onehot(target: np.ndarray):\n",
|
||
" onehot = np.zeros((target.size, target.max() + 1))\n",
|
||
" onehot[np.arange(target.size), target] = 1\n",
|
||
" return onehot\n",
|
||
"\n",
|
||
"digits = load_digits()\n",
|
||
"\n",
|
||
"X = digits.data\n",
|
||
"target = digits.target\n",
|
||
"target = onehot(target)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 21,
|
||
"id": "fed4ae3c",
|
||
"metadata": {},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = 64\n",
|
||
"hidden_nodes1 = 100\n",
|
||
"hidden_nodes2 = 30\n",
|
||
"output_nodes = 10\n",
|
||
"\n",
|
||
"dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n",
|
||
"\n",
|
||
"multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 22,
|
||
"id": "15587373",
|
||
"metadata": {},
|
||
"outputs": [
|
||
{
|
||
"name": "stdout",
|
||
"output_type": "stream",
|
||
"text": [
|
||
"Adam: Eta=0.0001, Lambda=0\n",
|
||
" [=======================================>] 100.0% | train_error: 0.175 | train_acc: 0.983 "
|
||
]
|
||
}
|
||
],
|
||
"source": [
|
||
"multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n",
|
||
"scores = multiclass.fit(X, target, scheduler, epochs=1000)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "5e5f6e3e",
|
||
"metadata": {},
|
||
"source": [
|
||
"A quick note to finish on is that the different demonstrations that were made such as sending in the test data using predict for linear regression can as easily be done for a binary or multi class classification problem, and the same goes for vice versa for plotting different scores over epochs."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {
|
||
"kernelspec": {
|
||
"display_name": "Python 3 (ipykernel)",
|
||
"language": "python",
|
||
"name": "python3"
|
||
},
|
||
"language_info": {
|
||
"codemirror_mode": {
|
||
"name": "ipython",
|
||
"version": 3
|
||
},
|
||
"file_extension": ".py",
|
||
"mimetype": "text/x-python",
|
||
"name": "python",
|
||
"nbconvert_exporter": "python",
|
||
"pygments_lexer": "ipython3",
|
||
"version": "3.9.10"
|
||
}
|
||
},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|