Morten Hjorth-Jensen
1612 lines
56 KiB
Plaintext
1612 lines
56 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b2937d10",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
||
"doconce format html exercisesweek43.do.txt -->\n",
|
||
"<!-- dom:TITLE: Exercises weeks 43 and 44 -->"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3dd00d19",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"# Exercises weeks 43 and 44 \n",
|
||
"**October 23-27, 2023**\n",
|
||
"\n",
|
||
"Date: **Deadline is Sunday November 5 at midnight**\n",
|
||
"\n",
|
||
"You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "82a19a1d",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"# Overarching aims of the exercises weeks 43 and 44\n",
|
||
"\n",
|
||
"The aim of the exercises this week and next week is to get started with writing a neural network code\n",
|
||
"of relevance for project 2. \n",
|
||
"\n",
|
||
"During week 41 we discussed three different types of gates, the\n",
|
||
"so-called XOR, the OR and the AND gates. In order to develop a code\n",
|
||
"for neural networks, it can be useful to set up a simpler system with\n",
|
||
"only two inputs and one output. This can make it easier to debug and\n",
|
||
"study the feed forward pass and the back propagation part. In the\n",
|
||
"exercise this and next week, we propose to study this system with just\n",
|
||
"one hidden layer and two hidden nodes. There is only one output node\n",
|
||
"and we can choose to use either a simple regression case (fitting a\n",
|
||
"line) or just a binary classification case with the cross-entropy as\n",
|
||
"cost function.\n",
|
||
"\n",
|
||
"Their inputs and outputs can be\n",
|
||
"summarized using the following tables, first for the OR gate with\n",
|
||
"inputs $x_1$ and $x_2$ and outputs $y$:\n",
|
||
"\n",
|
||
"<table class=\"dotable\" border=\"1\">\n",
|
||
"<thead>\n",
|
||
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
|
||
"</thead>\n",
|
||
"<tbody>\n",
|
||
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
||
"</tbody>\n",
|
||
"</table>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f74f69af",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"## The AND and XOR Gates\n",
|
||
"\n",
|
||
"The AND gate is defined as\n",
|
||
"\n",
|
||
"<table class=\"dotable\" border=\"1\">\n",
|
||
"<thead>\n",
|
||
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
|
||
"</thead>\n",
|
||
"<tbody>\n",
|
||
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
||
"</tbody>\n",
|
||
"</table>\n",
|
||
"\n",
|
||
"And finally we have the XOR gate\n",
|
||
"\n",
|
||
"<table class=\"dotable\" border=\"1\">\n",
|
||
"<thead>\n",
|
||
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
|
||
"</thead>\n",
|
||
"<tbody>\n",
|
||
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
||
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
||
"</tbody>\n",
|
||
"</table>"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1b52d47a",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"## Representing the Data Sets\n",
|
||
"\n",
|
||
"Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "3e6910cb",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"$$\n",
|
||
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
|
||
" 0 & 1 \\\\\n",
|
||
"\t\t 1 & 0 \\\\\n",
|
||
"\t\t 1 & 1 \\end{bmatrix},\n",
|
||
"$$"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "90a3b78a",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
|
||
"\n",
|
||
"Your tasks here are\n",
|
||
"\n",
|
||
"1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n",
|
||
"\n",
|
||
"2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n",
|
||
"\n",
|
||
"3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n",
|
||
"\n",
|
||
"4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n",
|
||
"\n",
|
||
"5. Set up the cost function (cross entropy for classification of binary cases).\n",
|
||
"\n",
|
||
"6. Calculate the gradients needed for the back propagation part.\n",
|
||
"\n",
|
||
"7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n",
|
||
"\n",
|
||
"8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n",
|
||
"\n",
|
||
"Everything you develop here can be used directly into the code for the project."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "d6a3ab1e",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"## Setting up the Neural Network\n",
|
||
"\n",
|
||
"We define first our design matrix and the various output vectors for the different gates."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"id": "152123b0",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"%matplotlib inline\n",
|
||
"\n",
|
||
"\"\"\"\n",
|
||
"Simple code that tests XOR, OR and AND gates with linear regression\n",
|
||
"\"\"\"\n",
|
||
"\n",
|
||
"# import necessary packages\n",
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"from sklearn import datasets\n",
|
||
"\n",
|
||
"def sigmoid(x):\n",
|
||
" return 1/(1 + np.exp(-x))\n",
|
||
"\n",
|
||
"def feed_forward(X):\n",
|
||
" # weighted sum of inputs to the hidden layer\n",
|
||
" z_h = np.matmul(X, hidden_weights) + hidden_bias\n",
|
||
" # activation in the hidden layer\n",
|
||
" a_h = sigmoid(z_h)\n",
|
||
" \n",
|
||
" # weighted sum of inputs to the output layer\n",
|
||
" z_o = np.matmul(a_h, output_weights) + output_bias\n",
|
||
" # softmax output\n",
|
||
" # axis 0 holds each input and axis 1 the probabilities of each category\n",
|
||
" probabilities = sigmoid(z_o)\n",
|
||
" return probabilities\n",
|
||
"\n",
|
||
"# we obtain a prediction by taking the class with the highest likelihood\n",
|
||
"def predict(X):\n",
|
||
" probabilities = feed_forward(X)\n",
|
||
" return np.argmax(probabilities, axis=1)\n",
|
||
"\n",
|
||
"# ensure the same random numbers appear every time\n",
|
||
"np.random.seed(0)\n",
|
||
"\n",
|
||
"# Design matrix\n",
|
||
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
|
||
"\n",
|
||
"# The XOR gate\n",
|
||
"yXOR = np.array( [ 0, 1 ,1, 0])\n",
|
||
"# The OR gate\n",
|
||
"yOR = np.array( [ 0, 1 ,1, 1])\n",
|
||
"# The AND gate\n",
|
||
"yAND = np.array( [ 0, 0 ,0, 1])\n",
|
||
"\n",
|
||
"# Defining the neural network\n",
|
||
"n_inputs, n_features = X.shape\n",
|
||
"n_hidden_neurons = 2\n",
|
||
"n_categories = 2\n",
|
||
"n_features = 2\n",
|
||
"\n",
|
||
"# we make the weights normally distributed using numpy.random.randn\n",
|
||
"\n",
|
||
"# weights and bias in the hidden layer\n",
|
||
"hidden_weights = np.random.randn(n_features, n_hidden_neurons)\n",
|
||
"hidden_bias = np.zeros(n_hidden_neurons) + 0.01\n",
|
||
"\n",
|
||
"# weights and bias in the output layer\n",
|
||
"output_weights = np.random.randn(n_hidden_neurons, n_categories)\n",
|
||
"output_bias = np.zeros(n_categories) + 0.01\n",
|
||
"\n",
|
||
"probabilities = feed_forward(X)\n",
|
||
"print(probabilities)\n",
|
||
"\n",
|
||
"\n",
|
||
"predictions = predict(X)\n",
|
||
"print(predictions)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "73319f0a",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "a7e0c47a",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"## The Code using Scikit-Learn"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"id": "dbbacc67",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"# import necessary packages\n",
|
||
"import numpy as np\n",
|
||
"import matplotlib.pyplot as plt\n",
|
||
"from sklearn.neural_network import MLPClassifier\n",
|
||
"from sklearn.metrics import accuracy_score\n",
|
||
"import seaborn as sns\n",
|
||
"\n",
|
||
"# ensure the same random numbers appear every time\n",
|
||
"np.random.seed(0)\n",
|
||
"\n",
|
||
"# Design matrix\n",
|
||
"X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n",
|
||
"\n",
|
||
"# The XOR gate\n",
|
||
"yXOR = np.array( [ 0, 1 ,1, 0])\n",
|
||
"# The OR gate\n",
|
||
"yOR = np.array( [ 0, 1 ,1, 1])\n",
|
||
"# The AND gate\n",
|
||
"yAND = np.array( [ 0, 0 ,0, 1])\n",
|
||
"\n",
|
||
"# Defining the neural network\n",
|
||
"n_inputs, n_features = X.shape\n",
|
||
"n_hidden_neurons = 2\n",
|
||
"n_categories = 2\n",
|
||
"n_features = 2\n",
|
||
"\n",
|
||
"eta_vals = np.logspace(-5, 1, 7)\n",
|
||
"lmbd_vals = np.logspace(-5, 1, 7)\n",
|
||
"# store models for later use\n",
|
||
"DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n",
|
||
"epochs = 100\n",
|
||
"\n",
|
||
"for i, eta in enumerate(eta_vals):\n",
|
||
" for j, lmbd in enumerate(lmbd_vals):\n",
|
||
" dnn = MLPClassifier(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',\n",
|
||
" alpha=lmbd, learning_rate_init=eta, max_iter=epochs)\n",
|
||
" dnn.fit(X, yXOR)\n",
|
||
" DNN_scikit[i][j] = dnn\n",
|
||
" print(\"Learning rate = \", eta)\n",
|
||
" print(\"Lambda = \", lmbd)\n",
|
||
" print(\"Accuracy score on data set: \", dnn.score(X, yXOR))\n",
|
||
" print()\n",
|
||
"\n",
|
||
"sns.set()\n",
|
||
"test_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))\n",
|
||
"for i in range(len(eta_vals)):\n",
|
||
" for j in range(len(lmbd_vals)):\n",
|
||
" dnn = DNN_scikit[i][j]\n",
|
||
" test_pred = dnn.predict(X)\n",
|
||
" test_accuracy[i][j] = accuracy_score(yXOR, test_pred)\n",
|
||
"\n",
|
||
"fig, ax = plt.subplots(figsize = (10, 10))\n",
|
||
"sns.heatmap(test_accuracy, annot=True, ax=ax, cmap=\"viridis\")\n",
|
||
"ax.set_title(\"Test Accuracy\")\n",
|
||
"ax.set_ylabel(\"$\\eta$\")\n",
|
||
"ax.set_xlabel(\"$\\lambda$\")\n",
|
||
"plt.show()"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1cac501a",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"## Building a neural network code\n",
|
||
"\n",
|
||
"Here we present a flexible object oriented codebase\n",
|
||
"for a feed forward neural network, along with a demonstration of how\n",
|
||
"to use it. Before we get into the details of the neural network, we\n",
|
||
"will first present some implementations of various schedulers, cost\n",
|
||
"functions and activation functions that can be used together with the\n",
|
||
"neural network.\n",
|
||
"\n",
|
||
"The codes here were developed by Eric Reber and Gregor Kajda during spring 2023."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "dd153528",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"### Learning rate methods\n",
|
||
"\n",
|
||
"The code below shows object oriented implementations of the Constant,\n",
|
||
"Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n",
|
||
"of the classes belong to the shared abstract Scheduler class, and\n",
|
||
"share the update_change() and reset() methods allowing for any of the\n",
|
||
"schedulers to be seamlessly used during the training stage, as will\n",
|
||
"later be shown in the fit() method of the neural\n",
|
||
"network. Update_change() only has one parameter, the gradient\n",
|
||
"($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n",
|
||
"from the weights. The reset() function takes no parameters, and resets\n",
|
||
"the desired variables. For Constant and Momentum, reset does nothing."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"id": "f55eea63",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"\n",
|
||
"class Scheduler:\n",
|
||
" \"\"\"\n",
|
||
" Abstract class for Schedulers\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" def __init__(self, eta):\n",
|
||
" self.eta = eta\n",
|
||
"\n",
|
||
" # should be overwritten\n",
|
||
" def update_change(self, gradient):\n",
|
||
" raise NotImplementedError\n",
|
||
"\n",
|
||
" # overwritten if needed\n",
|
||
" def reset(self):\n",
|
||
" pass\n",
|
||
"\n",
|
||
"\n",
|
||
"class Constant(Scheduler):\n",
|
||
" def __init__(self, eta):\n",
|
||
" super().__init__(eta)\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" return self.eta * gradient\n",
|
||
" \n",
|
||
" def reset(self):\n",
|
||
" pass\n",
|
||
"\n",
|
||
"\n",
|
||
"class Momentum(Scheduler):\n",
|
||
" def __init__(self, eta: float, momentum: float):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.momentum = momentum\n",
|
||
" self.change = 0\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" self.change = self.momentum * self.change + self.eta * gradient\n",
|
||
" return self.change\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" pass\n",
|
||
"\n",
|
||
"\n",
|
||
"class Adagrad(Scheduler):\n",
|
||
" def __init__(self, eta):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.G_t = None\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
"\n",
|
||
" if self.G_t is None:\n",
|
||
" self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n",
|
||
"\n",
|
||
" self.G_t += gradient @ gradient.T\n",
|
||
"\n",
|
||
" G_t_inverse = 1 / (\n",
|
||
" delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n",
|
||
" )\n",
|
||
" return self.eta * gradient * G_t_inverse\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.G_t = None\n",
|
||
"\n",
|
||
"\n",
|
||
"class AdagradMomentum(Scheduler):\n",
|
||
" def __init__(self, eta, momentum):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.G_t = None\n",
|
||
" self.momentum = momentum\n",
|
||
" self.change = 0\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
"\n",
|
||
" if self.G_t is None:\n",
|
||
" self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n",
|
||
"\n",
|
||
" self.G_t += gradient @ gradient.T\n",
|
||
"\n",
|
||
" G_t_inverse = 1 / (\n",
|
||
" delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n",
|
||
" )\n",
|
||
" self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n",
|
||
" return self.change\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.G_t = None\n",
|
||
"\n",
|
||
"\n",
|
||
"class RMS_prop(Scheduler):\n",
|
||
" def __init__(self, eta, rho):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.rho = rho\n",
|
||
" self.second = 0.0\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
" self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n",
|
||
" return self.eta * gradient / (np.sqrt(self.second + delta))\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.second = 0.0\n",
|
||
"\n",
|
||
"\n",
|
||
"class Adam(Scheduler):\n",
|
||
" def __init__(self, eta, rho, rho2):\n",
|
||
" super().__init__(eta)\n",
|
||
" self.rho = rho\n",
|
||
" self.rho2 = rho2\n",
|
||
" self.moment = 0\n",
|
||
" self.second = 0\n",
|
||
" self.n_epochs = 1\n",
|
||
"\n",
|
||
" def update_change(self, gradient):\n",
|
||
" delta = 1e-8 # avoid division ny zero\n",
|
||
"\n",
|
||
" self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n",
|
||
" self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n",
|
||
"\n",
|
||
" moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n",
|
||
" second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n",
|
||
"\n",
|
||
" return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n",
|
||
"\n",
|
||
" def reset(self):\n",
|
||
" self.n_epochs += 1\n",
|
||
" self.moment = 0\n",
|
||
" self.second = 0"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1a9bcb3e",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"### Usage of the above learning rate schedulers\n",
|
||
"\n",
|
||
"To initalize a scheduler, simply create the object and pass in the\n",
|
||
"necessary parameters such as the learning rate and the momentum as\n",
|
||
"shown below. As the Scheduler class is an abstract class it should not\n",
|
||
"called directly, and will raise an error upon usage."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"id": "86013cb4",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n",
|
||
"adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "535331f6",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Here is a small example for how a segment of code using schedulers\n",
|
||
"could look. Switching out the schedulers is simple."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"id": "7e0f6b5a",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"weights = np.ones((3,3))\n",
|
||
"print(f\"Before scheduler:\\n{weights=}\")\n",
|
||
"\n",
|
||
"epochs = 10\n",
|
||
"for e in range(epochs):\n",
|
||
" gradient = np.random.rand(3, 3)\n",
|
||
" change = adam_scheduler.update_change(gradient)\n",
|
||
" weights = weights - change\n",
|
||
" adam_scheduler.reset()\n",
|
||
"\n",
|
||
"print(f\"\\nAfter scheduler:\\n{weights=}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "f018ae57",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"### Cost functions\n",
|
||
"\n",
|
||
"Here we discuss cost functions that can be used when creating the\n",
|
||
"neural network. Every cost function takes the target vector as its\n",
|
||
"parameter, and returns a function valued only at $x$ such that it may\n",
|
||
"easily be differentiated."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"id": "c13507bf",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"\n",
|
||
"def CostOLS(target):\n",
|
||
" \n",
|
||
" def func(X):\n",
|
||
" return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
"\n",
|
||
"def CostLogReg(target):\n",
|
||
"\n",
|
||
" def func(X):\n",
|
||
" \n",
|
||
" return -(1.0 / target.shape[0]) * np.sum(\n",
|
||
" (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n",
|
||
" )\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
"\n",
|
||
"def CostCrossEntropy(target):\n",
|
||
" \n",
|
||
" def func(X):\n",
|
||
" return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n",
|
||
"\n",
|
||
" return func"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "6dab17bc",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Below we give a short example of how these cost function may be used\n",
|
||
"to obtain results if you wish to test them out on your own using\n",
|
||
"AutoGrad's automatics differentiation."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"id": "a5dbba01",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from autograd import grad\n",
|
||
"\n",
|
||
"target = np.array([[1, 2, 3]]).T\n",
|
||
"a = np.array([[4, 5, 6]]).T\n",
|
||
"\n",
|
||
"cost_func = CostCrossEntropy\n",
|
||
"cost_func_derivative = grad(cost_func(target))\n",
|
||
"\n",
|
||
"valued_at_a = cost_func_derivative(a)\n",
|
||
"print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b55d31d4",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"### Activation functions\n",
|
||
"\n",
|
||
"Finally, before we look at the neural network, we will look at the\n",
|
||
"activation functions which can be specified between the hidden layers\n",
|
||
"and as the output function. Each function can be valued for any given\n",
|
||
"vector or matrix X, and can be differentiated via derivate()."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 8,
|
||
"id": "b3e045a6",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"from autograd import elementwise_grad\n",
|
||
"\n",
|
||
"def identity(X):\n",
|
||
" return X\n",
|
||
"\n",
|
||
"\n",
|
||
"def sigmoid(X):\n",
|
||
" try:\n",
|
||
" return 1.0 / (1 + np.exp(-X))\n",
|
||
" except FloatingPointError:\n",
|
||
" return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n",
|
||
"\n",
|
||
"\n",
|
||
"def softmax(X):\n",
|
||
" X = X - np.max(X, axis=-1, keepdims=True)\n",
|
||
" delta = 10e-10\n",
|
||
" return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n",
|
||
"\n",
|
||
"\n",
|
||
"def RELU(X):\n",
|
||
" return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n",
|
||
"\n",
|
||
"\n",
|
||
"def LRELU(X):\n",
|
||
" delta = 10e-4\n",
|
||
" return np.where(X > np.zeros(X.shape), X, delta * X)\n",
|
||
"\n",
|
||
"\n",
|
||
"def derivate(func):\n",
|
||
" if func.__name__ == \"RELU\":\n",
|
||
"\n",
|
||
" def func(X):\n",
|
||
" return np.where(X > 0, 1, 0)\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
" elif func.__name__ == \"LRELU\":\n",
|
||
"\n",
|
||
" def func(X):\n",
|
||
" delta = 10e-4\n",
|
||
" return np.where(X > 0, 1, delta)\n",
|
||
"\n",
|
||
" return func\n",
|
||
"\n",
|
||
" else:\n",
|
||
" return elementwise_grad(func)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "c0189342",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Below follows a short demonstration of how to use an activation\n",
|
||
"function. The derivative of the activation function will be important\n",
|
||
"when calculating the output delta term during backpropagation. Note\n",
|
||
"that derivate() can also be used for cost functions for a more\n",
|
||
"generalized approach."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 9,
|
||
"id": "640aa861",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"z = np.array([[4, 5, 6]]).T\n",
|
||
"print(f\"Input to activation function:\\n{z}\")\n",
|
||
"\n",
|
||
"act_func = sigmoid\n",
|
||
"a = act_func(z)\n",
|
||
"print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n",
|
||
"\n",
|
||
"act_func_derivative = derivate(act_func)\n",
|
||
"valued_at_z = act_func_derivative(a)\n",
|
||
"print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1007ccdd",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"### The Neural Network\n",
|
||
"\n",
|
||
"Now that we have gotten a good understanding of the implementation of\n",
|
||
"some important components, we can take a look at an object oriented\n",
|
||
"implementation of a feed forward neural network. The feed forward\n",
|
||
"neural network has been implemented as a class named FFNN, which can\n",
|
||
"be initiated as a regressor or classifier dependant on the choice of\n",
|
||
"cost function. The FFNN can have any number of input nodes, hidden\n",
|
||
"layers with any amount of hidden nodes, and any amount of output nodes\n",
|
||
"meaning it can perform multiclass classification as well as binary\n",
|
||
"classification and regression problems. Although there is a lot of\n",
|
||
"code present, it makes for an easy to use and generalizeable interface\n",
|
||
"for creating many types of neural networks as will be demonstrated\n",
|
||
"below."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 10,
|
||
"id": "9584a2da",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import math\n",
|
||
"import autograd.numpy as np\n",
|
||
"import sys\n",
|
||
"import warnings\n",
|
||
"from autograd import grad, elementwise_grad\n",
|
||
"from random import random, seed\n",
|
||
"from copy import deepcopy, copy\n",
|
||
"from typing import Tuple, Callable\n",
|
||
"from sklearn.utils import resample\n",
|
||
"\n",
|
||
"warnings.simplefilter(\"error\")\n",
|
||
"\n",
|
||
"\n",
|
||
"class FFNN:\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Feed Forward Neural Network with interface enabling flexible design of a\n",
|
||
" nerual networks architecture and the specification of activation function\n",
|
||
" in the hidden layers and output layer respectively. This model can be used\n",
|
||
" for both regression and classification problems, depending on the output function.\n",
|
||
"\n",
|
||
" Attributes:\n",
|
||
" ------------\n",
|
||
" I dimensions (tuple[int]): A list of positive integers, which specifies the\n",
|
||
" number of nodes in each of the networks layers. The first integer in the array\n",
|
||
" defines the number of nodes in the input layer, the second integer defines number\n",
|
||
" of nodes in the first hidden layer and so on until the last number, which\n",
|
||
" specifies the number of nodes in the output layer.\n",
|
||
" II hidden_func (Callable): The activation function for the hidden layers\n",
|
||
" III output_func (Callable): The activation function for the output layer\n",
|
||
" IV cost_func (Callable): Our cost function\n",
|
||
" V seed (int): Sets random seed, makes results reproducible\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" def __init__(\n",
|
||
" self,\n",
|
||
" dimensions: tuple[int],\n",
|
||
" hidden_func: Callable = sigmoid,\n",
|
||
" output_func: Callable = lambda x: x,\n",
|
||
" cost_func: Callable = CostOLS,\n",
|
||
" seed: int = None,\n",
|
||
" ):\n",
|
||
" self.dimensions = dimensions\n",
|
||
" self.hidden_func = hidden_func\n",
|
||
" self.output_func = output_func\n",
|
||
" self.cost_func = cost_func\n",
|
||
" self.seed = seed\n",
|
||
" self.weights = list()\n",
|
||
" self.schedulers_weight = list()\n",
|
||
" self.schedulers_bias = list()\n",
|
||
" self.a_matrices = list()\n",
|
||
" self.z_matrices = list()\n",
|
||
" self.classification = None\n",
|
||
"\n",
|
||
" self.reset_weights()\n",
|
||
" self._set_classification()\n",
|
||
"\n",
|
||
" def fit(\n",
|
||
" self,\n",
|
||
" X: np.ndarray,\n",
|
||
" t: np.ndarray,\n",
|
||
" scheduler: Scheduler,\n",
|
||
" batches: int = 1,\n",
|
||
" epochs: int = 100,\n",
|
||
" lam: float = 0,\n",
|
||
" X_val: np.ndarray = None,\n",
|
||
" t_val: np.ndarray = None,\n",
|
||
" ):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" This function performs the training the neural network by performing the feedforward and backpropagation\n",
|
||
" algorithm to update the networks weights.\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray) : training data\n",
|
||
" II t (np.ndarray) : target data\n",
|
||
" III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n",
|
||
" IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n",
|
||
"\n",
|
||
" Optional Parameters:\n",
|
||
" ------------\n",
|
||
" V batches (int) : number of batches the datasets are split into, default equal to 1\n",
|
||
" VI epochs (int) : number of iterations used to train the network, default equal to 100\n",
|
||
" VII lam (float) : regularization hyperparameter lambda\n",
|
||
" VIII X_val (np.ndarray) : validation set\n",
|
||
" IX t_val (np.ndarray) : validation target set\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" I scores (dict) : A dictionary containing the performance metrics of the model.\n",
|
||
" The number of the metrics depends on the parameters passed to the fit-function.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" # setup \n",
|
||
" if self.seed is not None:\n",
|
||
" np.random.seed(self.seed)\n",
|
||
"\n",
|
||
" val_set = False\n",
|
||
" if X_val is not None and t_val is not None:\n",
|
||
" val_set = True\n",
|
||
"\n",
|
||
" # creating arrays for score metrics\n",
|
||
" train_errors = np.empty(epochs)\n",
|
||
" train_errors.fill(np.nan)\n",
|
||
" val_errors = np.empty(epochs)\n",
|
||
" val_errors.fill(np.nan)\n",
|
||
"\n",
|
||
" train_accs = np.empty(epochs)\n",
|
||
" train_accs.fill(np.nan)\n",
|
||
" val_accs = np.empty(epochs)\n",
|
||
" val_accs.fill(np.nan)\n",
|
||
"\n",
|
||
" self.schedulers_weight = list()\n",
|
||
" self.schedulers_bias = list()\n",
|
||
"\n",
|
||
" batch_size = X.shape[0] // batches\n",
|
||
"\n",
|
||
" X, t = resample(X, t)\n",
|
||
"\n",
|
||
" # this function returns a function valued only at X\n",
|
||
" cost_function_train = self.cost_func(t)\n",
|
||
" if val_set:\n",
|
||
" cost_function_val = self.cost_func(t_val)\n",
|
||
"\n",
|
||
" # create schedulers for each weight matrix\n",
|
||
" for i in range(len(self.weights)):\n",
|
||
" self.schedulers_weight.append(copy(scheduler))\n",
|
||
" self.schedulers_bias.append(copy(scheduler))\n",
|
||
"\n",
|
||
" print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n",
|
||
"\n",
|
||
" try:\n",
|
||
" for e in range(epochs):\n",
|
||
" for i in range(batches):\n",
|
||
" # allows for minibatch gradient descent\n",
|
||
" if i == batches - 1:\n",
|
||
" # If the for loop has reached the last batch, take all thats left\n",
|
||
" X_batch = X[i * batch_size :, :]\n",
|
||
" t_batch = t[i * batch_size :, :]\n",
|
||
" else:\n",
|
||
" X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n",
|
||
" t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n",
|
||
"\n",
|
||
" self._feedforward(X_batch)\n",
|
||
" self._backpropagate(X_batch, t_batch, lam)\n",
|
||
"\n",
|
||
" # reset schedulers for each epoch (some schedulers pass in this call)\n",
|
||
" for scheduler in self.schedulers_weight:\n",
|
||
" scheduler.reset()\n",
|
||
"\n",
|
||
" for scheduler in self.schedulers_bias:\n",
|
||
" scheduler.reset()\n",
|
||
"\n",
|
||
" # computing performance metrics\n",
|
||
" pred_train = self.predict(X)\n",
|
||
" train_error = cost_function_train(pred_train)\n",
|
||
"\n",
|
||
" train_errors[e] = train_error\n",
|
||
" if val_set:\n",
|
||
" \n",
|
||
" pred_val = self.predict(X_val)\n",
|
||
" val_error = cost_function_val(pred_val)\n",
|
||
" val_errors[e] = val_error\n",
|
||
"\n",
|
||
" if self.classification:\n",
|
||
" train_acc = self._accuracy(self.predict(X), t)\n",
|
||
" train_accs[e] = train_acc\n",
|
||
" if val_set:\n",
|
||
" val_acc = self._accuracy(pred_val, t_val)\n",
|
||
" val_accs[e] = val_acc\n",
|
||
"\n",
|
||
" # printing progress bar\n",
|
||
" progression = e / epochs\n",
|
||
" print_length = self._progress_bar(\n",
|
||
" progression,\n",
|
||
" train_error=train_errors[e],\n",
|
||
" train_acc=train_accs[e],\n",
|
||
" val_error=val_errors[e],\n",
|
||
" val_acc=val_accs[e],\n",
|
||
" )\n",
|
||
" except KeyboardInterrupt:\n",
|
||
" # allows for stopping training at any point and seeing the result\n",
|
||
" pass\n",
|
||
"\n",
|
||
" # visualization of training progression (similiar to tensorflow progression bar)\n",
|
||
" sys.stdout.write(\"\\r\" + \" \" * print_length)\n",
|
||
" sys.stdout.flush()\n",
|
||
" self._progress_bar(\n",
|
||
" 1,\n",
|
||
" train_error=train_errors[e],\n",
|
||
" train_acc=train_accs[e],\n",
|
||
" val_error=val_errors[e],\n",
|
||
" val_acc=val_accs[e],\n",
|
||
" )\n",
|
||
" sys.stdout.write(\"\")\n",
|
||
"\n",
|
||
" # return performance metrics for the entire run\n",
|
||
" scores = dict()\n",
|
||
"\n",
|
||
" scores[\"train_errors\"] = train_errors\n",
|
||
"\n",
|
||
" if val_set:\n",
|
||
" scores[\"val_errors\"] = val_errors\n",
|
||
"\n",
|
||
" if self.classification:\n",
|
||
" scores[\"train_accs\"] = train_accs\n",
|
||
"\n",
|
||
" if val_set:\n",
|
||
" scores[\"val_accs\"] = val_accs\n",
|
||
"\n",
|
||
" return scores\n",
|
||
"\n",
|
||
" def predict(self, X: np.ndarray, *, threshold=0.5):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Performs prediction after training of the network has been finished.\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray): The design matrix, with n rows of p features each\n",
|
||
"\n",
|
||
" Optional Parameters:\n",
|
||
" ------------\n",
|
||
" II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n",
|
||
" in classification problems\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n",
|
||
" This vector is thresholded if regression=False, meaning that classification results\n",
|
||
" in a vector of 1s and 0s, while regressions in an array of decimal numbers\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" predict = self._feedforward(X)\n",
|
||
"\n",
|
||
" if self.classification:\n",
|
||
" return np.where(predict > threshold, 1, 0)\n",
|
||
" else:\n",
|
||
" return predict\n",
|
||
"\n",
|
||
" def reset_weights(self):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Resets/Reinitializes the weights in order to train the network for a new problem.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" if self.seed is not None:\n",
|
||
" np.random.seed(self.seed)\n",
|
||
"\n",
|
||
" self.weights = list()\n",
|
||
" for i in range(len(self.dimensions) - 1):\n",
|
||
" weight_array = np.random.randn(\n",
|
||
" self.dimensions[i] + 1, self.dimensions[i + 1]\n",
|
||
" )\n",
|
||
" weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n",
|
||
"\n",
|
||
" self.weights.append(weight_array)\n",
|
||
"\n",
|
||
" def _feedforward(self, X: np.ndarray):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Calculates the activation of each layer starting at the input and ending at the output.\n",
|
||
" Each following activation is calculated from a weighted sum of each of the preceeding\n",
|
||
" activations (except in the case of the input layer).\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray): The design matrix, with n rows of p features each\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n",
|
||
" \"\"\"\n",
|
||
"\n",
|
||
" # reset matrices\n",
|
||
" self.a_matrices = list()\n",
|
||
" self.z_matrices = list()\n",
|
||
"\n",
|
||
" # if X is just a vector, make it into a matrix\n",
|
||
" if len(X.shape) == 1:\n",
|
||
" X = X.reshape((1, X.shape[0]))\n",
|
||
"\n",
|
||
" # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n",
|
||
" # to add bias to our data\n",
|
||
" bias = np.ones((X.shape[0], 1)) * 0.01\n",
|
||
" X = np.hstack([bias, X])\n",
|
||
"\n",
|
||
" # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n",
|
||
" # exponent indicates layer number).\n",
|
||
" a = X\n",
|
||
" self.a_matrices.append(a)\n",
|
||
" self.z_matrices.append(a)\n",
|
||
"\n",
|
||
" # The feed forward algorithm\n",
|
||
" for i in range(len(self.weights)):\n",
|
||
" if i < len(self.weights) - 1:\n",
|
||
" z = a @ self.weights[i]\n",
|
||
" self.z_matrices.append(z)\n",
|
||
" a = self.hidden_func(z)\n",
|
||
" # bias column again added to the data here\n",
|
||
" bias = np.ones((a.shape[0], 1)) * 0.01\n",
|
||
" a = np.hstack([bias, a])\n",
|
||
" self.a_matrices.append(a)\n",
|
||
" else:\n",
|
||
" try:\n",
|
||
" # a^L, the nodes in our output layers\n",
|
||
" z = a @ self.weights[i]\n",
|
||
" a = self.output_func(z)\n",
|
||
" self.a_matrices.append(a)\n",
|
||
" self.z_matrices.append(z)\n",
|
||
" except Exception as OverflowError:\n",
|
||
" print(\n",
|
||
" \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n",
|
||
" )\n",
|
||
"\n",
|
||
" # this will be a^L\n",
|
||
" return a\n",
|
||
"\n",
|
||
" def _backpropagate(self, X, t, lam):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Performs the backpropagation algorithm. In other words, this method\n",
|
||
" calculates the gradient of all the layers starting at the\n",
|
||
" output layer, and moving from right to left accumulates the gradient until\n",
|
||
" the input layer is reached. Each layers respective weights are updated while\n",
|
||
" the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I X (np.ndarray): The design matrix, with n rows of p features each.\n",
|
||
" II t (np.ndarray): The target vector, with n rows of p targets.\n",
|
||
" III lam (float32): regularization parameter used to punish the weights in case of overfitting\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" No return value.\n",
|
||
"\n",
|
||
" \"\"\"\n",
|
||
" out_derivative = derivate(self.output_func)\n",
|
||
" hidden_derivative = derivate(self.hidden_func)\n",
|
||
"\n",
|
||
" for i in range(len(self.weights) - 1, -1, -1):\n",
|
||
" # delta terms for output\n",
|
||
" if i == len(self.weights) - 1:\n",
|
||
" # for multi-class classification\n",
|
||
" if (\n",
|
||
" self.output_func.__name__ == \"softmax\"\n",
|
||
" ):\n",
|
||
" delta_matrix = self.a_matrices[i + 1] - t\n",
|
||
" # for single class classification\n",
|
||
" else:\n",
|
||
" cost_func_derivative = grad(self.cost_func(t))\n",
|
||
" delta_matrix = out_derivative(\n",
|
||
" self.z_matrices[i + 1]\n",
|
||
" ) * cost_func_derivative(self.a_matrices[i + 1])\n",
|
||
"\n",
|
||
" # delta terms for hidden layer\n",
|
||
" else:\n",
|
||
" delta_matrix = (\n",
|
||
" self.weights[i + 1][1:, :] @ delta_matrix.T\n",
|
||
" ).T * hidden_derivative(self.z_matrices[i + 1])\n",
|
||
"\n",
|
||
" # calculate gradient\n",
|
||
" gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n",
|
||
" gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n",
|
||
" 1, delta_matrix.shape[1]\n",
|
||
" )\n",
|
||
"\n",
|
||
" # regularization term\n",
|
||
" gradient_weights += self.weights[i][1:, :] * lam\n",
|
||
"\n",
|
||
" # use scheduler\n",
|
||
" update_matrix = np.vstack(\n",
|
||
" [\n",
|
||
" self.schedulers_bias[i].update_change(gradient_bias),\n",
|
||
" self.schedulers_weight[i].update_change(gradient_weights),\n",
|
||
" ]\n",
|
||
" )\n",
|
||
"\n",
|
||
" # update weights and bias\n",
|
||
" self.weights[i] -= update_matrix\n",
|
||
"\n",
|
||
" def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Calculates accuracy of given prediction to target\n",
|
||
"\n",
|
||
" Parameters:\n",
|
||
" ------------\n",
|
||
" I prediction (np.ndarray): vector of predicitons output network\n",
|
||
" (1s and 0s in case of classification, and real numbers in case of regression)\n",
|
||
" II target (np.ndarray): vector of true values (What the network ideally should predict)\n",
|
||
"\n",
|
||
" Returns:\n",
|
||
" ------------\n",
|
||
" A floating point number representing the percentage of correctly classified instances.\n",
|
||
" \"\"\"\n",
|
||
" assert prediction.size == target.size\n",
|
||
" return np.average((target == prediction))\n",
|
||
" def _set_classification(self):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Decides if FFNN acts as classifier (True) og regressor (False),\n",
|
||
" sets self.classification during init()\n",
|
||
" \"\"\"\n",
|
||
" self.classification = False\n",
|
||
" if (\n",
|
||
" self.cost_func.__name__ == \"CostLogReg\"\n",
|
||
" or self.cost_func.__name__ == \"CostCrossEntropy\"\n",
|
||
" ):\n",
|
||
" self.classification = True\n",
|
||
"\n",
|
||
" def _progress_bar(self, progression, **kwargs):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Displays progress of training\n",
|
||
" \"\"\"\n",
|
||
" print_length = 40\n",
|
||
" num_equals = int(progression * print_length)\n",
|
||
" num_not = print_length - num_equals\n",
|
||
" arrow = \">\" if num_equals > 0 else \"\"\n",
|
||
" bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n",
|
||
" perc_print = self._format(progression * 100, decimals=5)\n",
|
||
" line = f\" {bar} {perc_print}% \"\n",
|
||
"\n",
|
||
" for key in kwargs:\n",
|
||
" if not np.isnan(kwargs[key]):\n",
|
||
" value = self._format(kwargs[key], decimals=4)\n",
|
||
" line += f\"| {key}: {value} \"\n",
|
||
" sys.stdout.write(\"\\r\" + line)\n",
|
||
" sys.stdout.flush()\n",
|
||
" return len(line)\n",
|
||
"\n",
|
||
" def _format(self, value, decimals=4):\n",
|
||
" \"\"\"\n",
|
||
" Description:\n",
|
||
" ------------\n",
|
||
" Formats decimal numbers for progress bar\n",
|
||
" \"\"\"\n",
|
||
" if value > 0:\n",
|
||
" v = value\n",
|
||
" elif value < 0:\n",
|
||
" v = -10 * value\n",
|
||
" else:\n",
|
||
" v = 1\n",
|
||
" n = 1 + math.floor(math.log10(v))\n",
|
||
" if n >= decimals - 1:\n",
|
||
" return str(round(value))\n",
|
||
" return f\"{value:.{decimals-n-1}f}\""
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "9ccd1fc1",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Before we make a model, we will quickly generate a dataset we can use\n",
|
||
"for our linear regression problem as shown below"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 11,
|
||
"id": "7f3a5b31",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"import autograd.numpy as np\n",
|
||
"from sklearn.model_selection import train_test_split\n",
|
||
"\n",
|
||
"def SkrankeFunction(x, y):\n",
|
||
" return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n",
|
||
"\n",
|
||
"def create_X(x, y, n):\n",
|
||
" if len(x.shape) > 1:\n",
|
||
" x = np.ravel(x)\n",
|
||
" y = np.ravel(y)\n",
|
||
"\n",
|
||
" N = len(x)\n",
|
||
" l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n",
|
||
" X = np.ones((N, l))\n",
|
||
"\n",
|
||
" for i in range(1, n + 1):\n",
|
||
" q = int((i) * (i + 1) / 2)\n",
|
||
" for k in range(i + 1):\n",
|
||
" X[:, q + k] = (x ** (i - k)) * (y**k)\n",
|
||
"\n",
|
||
" return X\n",
|
||
"\n",
|
||
"step=0.5\n",
|
||
"x = np.arange(0, 1, step)\n",
|
||
"y = np.arange(0, 1, step)\n",
|
||
"x, y = np.meshgrid(x, y)\n",
|
||
"target = SkrankeFunction(x, y)\n",
|
||
"target = target.reshape(target.shape[0], 1)\n",
|
||
"\n",
|
||
"poly_degree=3\n",
|
||
"X = create_X(x, y, poly_degree)\n",
|
||
"\n",
|
||
"X_train, X_test, t_train, t_test = train_test_split(X, target)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "1ac05bb6",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Now that we have our dataset ready for the regression, we can create\n",
|
||
"our regressor. Note that with the seed parameter, we can make sure our\n",
|
||
"results stay the same every time we run the neural network. For\n",
|
||
"inititialization, we simply specify the dimensions (we wish the amount\n",
|
||
"of input nodes to be equal to the datapoints, and the output to\n",
|
||
"predict one value)."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 12,
|
||
"id": "0f857604",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = X_train.shape[1]\n",
|
||
"output_nodes = 1\n",
|
||
"\n",
|
||
"linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "eeff4315",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"We then fit our model with our training data using the scheduler of our choice."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 13,
|
||
"id": "46246810",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Constant(eta=1e-3)\n",
|
||
"scores = linear_regression.fit(X_train, t_train, scheduler)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "8c6f9954",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Due to the progress bar we can see the MSE (train_error) throughout\n",
|
||
"the FFNN's training. Note that the fit() function has some optional\n",
|
||
"parameters with defualt arguments. For example, the regularization\n",
|
||
"hyperparameter can be left ignored if not needed, and equally the FFNN\n",
|
||
"will by default run for 100 epochs. These can easily be changed, such\n",
|
||
"as for example:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 14,
|
||
"id": "2661939c",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "74c5624c",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"We see that given more epochs to train on, the regressor reaches a lower MSE.\n",
|
||
"\n",
|
||
"Let us then switch to a binary classification. We use a binary\n",
|
||
"classification dataset, and follow a similar setup to the regression\n",
|
||
"case."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 15,
|
||
"id": "4b8eb115",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.datasets import load_breast_cancer\n",
|
||
"from sklearn.preprocessing import MinMaxScaler\n",
|
||
"\n",
|
||
"wisconsin = load_breast_cancer()\n",
|
||
"X = wisconsin.data\n",
|
||
"target = wisconsin.target\n",
|
||
"target = target.reshape(target.shape[0], 1)\n",
|
||
"\n",
|
||
"X_train, X_val, t_train, t_val = train_test_split(X, target)\n",
|
||
"\n",
|
||
"scaler = MinMaxScaler()\n",
|
||
"scaler.fit(X_train)\n",
|
||
"X_train = scaler.transform(X_train)\n",
|
||
"X_val = scaler.transform(X_val)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 16,
|
||
"id": "2c0f92bd",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = X_train.shape[1]\n",
|
||
"output_nodes = 1\n",
|
||
"\n",
|
||
"logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "49201ae4",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"We will now make use of our validation data by passing it into our fit function as a keyword argument"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 17,
|
||
"id": "55b5e426",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n",
|
||
"scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "b51762fb",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"Finally, we will create a neural network with 2 hidden layers with activation functions."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 18,
|
||
"id": "6b59e27d",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"input_nodes = X_train.shape[1]\n",
|
||
"hidden_nodes1 = 100\n",
|
||
"hidden_nodes2 = 30\n",
|
||
"output_nodes = 1\n",
|
||
"\n",
|
||
"dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n",
|
||
"\n",
|
||
"neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 19,
|
||
"id": "72c87921",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n",
|
||
"scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"id": "ed40d7d2",
|
||
"metadata": {
|
||
"editable": true
|
||
},
|
||
"source": [
|
||
"### Multiclass classification\n",
|
||
"\n",
|
||
"Finally, we will demonstrate the use case of multiclass classification\n",
|
||
"using our FFNN with the famous MNIST dataset, which contain images of\n",
|
||
"digits between the range of 0 to 9."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 20,
|
||
"id": "315ef3fe",
|
||
"metadata": {
|
||
"collapsed": false,
|
||
"editable": true
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.datasets import load_digits\n",
|
||
"\n",
|
||
"def onehot(target: np.ndarray):\n",
|
||
" onehot = np.zeros((target.size, target.max() + 1))\n",
|
||
" onehot[np.arange(target.size), target] = 1\n",
|
||
" return onehot\n",
|
||
"\n",
|
||
"digits = load_digits()\n",
|
||
"\n",
|
||
"X = digits.data\n",
|
||
"target = digits.target\n",
|
||
"target = onehot(target)\n",
|
||
"\n",
|
||
"input_nodes = 64\n",
|
||
"hidden_nodes1 = 100\n",
|
||
"hidden_nodes2 = 30\n",
|
||
"output_nodes = 10\n",
|
||
"\n",
|
||
"dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n",
|
||
"\n",
|
||
"multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n",
|
||
"\n",
|
||
"multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n",
|
||
"\n",
|
||
"scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n",
|
||
"scores = multiclass.fit(X, target, scheduler, epochs=1000)"
|
||
]
|
||
}
|
||
],
|
||
"metadata": {},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 5
|
||
}
|