Morten Hjorth-Jensen
468 lines
12 KiB
Plaintext
468 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4b4c06bc",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html exercisesweek41.do.txt -->\n",
|
|
"<!-- dom:TITLE: Exercises week 41 -->\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bcb25e64",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Exercises week 42\n",
|
|
"\n",
|
|
"**October 14-18, 2024**\n",
|
|
"\n",
|
|
"Date: **Deadline is Friday October 18 at midnight**\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bb01f126",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Overarching aims of the exercises this week\n",
|
|
"\n",
|
|
"The aim of the exercises this week is to get started with implementing a neural network. There are a lot of technical and finicky parts of implementing a neutal network, so take your time.\n",
|
|
"\n",
|
|
"This week, you will implement only the feed-forward pass. Next week, you will implement backpropagation. We recommend that you do the exercises this week by editing and running this notebook file, as it includes several checks along the way that you have implemented the pieces of the feed-forward pass correctly. If you have trouble running a notebook, or importing pytorch, you can run this notebook in google colab instead: (LINK TO COLAB), though we recommend that you set up VSCode and your python environment to run code like this locally.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c6f61b09",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import autograd.numpy as np\n",
|
|
"from autograd import grad"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6248ec53",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Exercise 1\n",
|
|
"\n",
|
|
"Complete the following parts to compute the activation of the first layer.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "37f30740",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"np.random.seed(2024)\n",
|
|
"\n",
|
|
"\n",
|
|
"def ReLU(z):\n",
|
|
" return np.where(z > 0, z, 0)\n",
|
|
"\n",
|
|
"\n",
|
|
"x = np.random.randn(2) # network input\n",
|
|
"W1 = np.random.randn(4, 2) # first layer weights"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "edf7217b",
|
|
"metadata": {},
|
|
"source": [
|
|
"**a)** Define the bias of the first layer, `b1`with the correct shape\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2129c19f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"b1 = np.random.randn(4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "09e8d453",
|
|
"metadata": {},
|
|
"source": [
|
|
"**b)** Compute the intermediary `z1` for the first layer\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6837119b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"z1 = W1 @ x + b1"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6f71374e",
|
|
"metadata": {},
|
|
"source": [
|
|
"**c)** Compute the activation `a1` for the first layer using the ReLU activation function defined earlier.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8d41ed19",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"a1 = ReLU(z1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "088710c0",
|
|
"metadata": {},
|
|
"source": [
|
|
"Confirm that you got the correct activation with the test below.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4d2f54b4",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"sol1 = np.array([0.60610368, 4.0076268, 0.0, 0.56469864])\n",
|
|
"\n",
|
|
"print(np.allclose(a1, sol1))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7fb0cf46",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Exercise 2\n",
|
|
"\n",
|
|
"Compute the activation of the second layer with an output of length 8 and ReLU activation.\n",
|
|
"\n",
|
|
"**a)** Define the weight and bias of the second layer with the right shapes.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "00063acf",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"W2 = np.random.randn(8, 4)\n",
|
|
"b2 = np.random.randn(8)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5bd7d84b",
|
|
"metadata": {},
|
|
"source": [
|
|
"**b)** Compute intermediary `z2` and activation `a2` for the second layer.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2fd0383d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"z2 = W2 @ a1\n",
|
|
"a2 = ReLU(z2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1b5daae5",
|
|
"metadata": {},
|
|
"source": [
|
|
"Confirm that you got the correct activation shape with the test below.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "f7f2f8a1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(a2.shape == (8,))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3759620d",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Exercise 3\n",
|
|
"\n",
|
|
"We often want our neural networks to have many layers of varying sizes. To avoid writing very long and error-prone code where we explicitly define and evaluate each layer we should keep all our layers in a single variable which is easy to create and use.\n",
|
|
"\n",
|
|
"**a)** Complete the function below so that it returns a list `layers` of weight and bias tuples `(W, b)` for each layer, in order, with the correct shapes that we can use later as our network parameters.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "c58f10f9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def create_layers(network_input_size, output_sizes):\n",
|
|
" layers = []\n",
|
|
"\n",
|
|
" i_size = network_input_size\n",
|
|
" for output_size in output_sizes:\n",
|
|
" W = np.random.rand(output_size, i_size)\n",
|
|
" b = np.random.rand(output_size)\n",
|
|
" layers.append((W, b))\n",
|
|
"\n",
|
|
" i_size = output_size\n",
|
|
" return layers"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bdc0cda2",
|
|
"metadata": {},
|
|
"source": [
|
|
"**b)** Comple the function below so that it evaluates the intermediate `z` and activation `a` for each layer, and returns the final activation `a`. This is the complete feed-forward pass, a full neural network!\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "5262df05",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def feed_forward(layers, input):\n",
|
|
" a = input\n",
|
|
" for W, b in layers:\n",
|
|
" z = W @ a + b\n",
|
|
" a = ReLU(z)\n",
|
|
" return a"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "245adbcb",
|
|
"metadata": {},
|
|
"source": [
|
|
"**c)** Create a network with input size 8 and layers with output sizes 10, 16, 6, 2. Evaluate it and make sure that you get the correct size vectors along the way.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "89a8f70d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "306d8b7c",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Exercise 4\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "221c7b6c",
|
|
"metadata": {},
|
|
"source": [
|
|
"So far, every layer has used the same activation, ReLU. We often want to use other types of activation however, so we need to update our code to support multiple types of activation. Make sure that you have completed every previous exercise before trying this one.\n",
|
|
"\n",
|
|
"**a)** Make the `create_layers` function also accept a list of activation functions, which is used to add activation functions to each of the tuples in `layers`. Make new functions to not mess with the old ones.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9df82312",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def create_layers_4(network_input_size, output_sizes, activation_funcs):\n",
|
|
" layers = []\n",
|
|
"\n",
|
|
" i_size = network_input_size\n",
|
|
" for output_size, activation in zip(output_sizes, activation_funcs):\n",
|
|
" W = np.random.rand(output_size, i_size)\n",
|
|
" b = np.random.rand(output_size)\n",
|
|
" layers.append((W, b, activation))\n",
|
|
"\n",
|
|
" i_size = output_size\n",
|
|
" return layers"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "10896d06",
|
|
"metadata": {},
|
|
"source": [
|
|
"**b)** Update the `feed_forward` function to support this change.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "de062369",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def feed_forward_4(layers, input):\n",
|
|
" a = input\n",
|
|
" for W, b, activation in layers:\n",
|
|
" z = W @ a + b\n",
|
|
" a = activation(z)\n",
|
|
" return a"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "efd07b4e",
|
|
"metadata": {},
|
|
"source": [
|
|
"**c)** Create and evaluate a neural network with 4 inputs and layers with output sizes 12, 10, 3 and activations ReLU, ReLU, softmax.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "ce6fcc2f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from scipy.special import softmax\n",
|
|
"\n",
|
|
"network_input_size = 4\n",
|
|
"output_sizes = [12, 10, 3]\n",
|
|
"activation_funcs = [ReLU, ReLU, softmax]\n",
|
|
"layers = create_layers_4(network_input_size, output_sizes, activation_funcs)\n",
|
|
"\n",
|
|
"x = np.random.randn(network_input_size)\n",
|
|
"predict = feed_forward_4(layers, x)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "54d5fde7",
|
|
"metadata": {},
|
|
"source": [
|
|
"The final exercise will hopefully be very simple if everything has worked so far. You will evaluate your neural network on the iris data set (https://scikit-learn.org/1.5/auto_examples/datasets/plot_iris_dataset.html).\n",
|
|
"\n",
|
|
"This dataset contains data on 150 flowers of 3 different types which can be separated pretty well using the four features given for each flower, which includes the width and length of their leaves. You are not expected to do any training of the network or actual classification, unless you feel like it, in that case you can do exercise 5.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6bd4c148",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Loading and plotting iris dataset\n",
|
|
"from sklearn import datasets\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"iris = datasets.load_iris()\n",
|
|
"\n",
|
|
"_, ax = plt.subplots()\n",
|
|
"scatter = ax.scatter(iris.data[:, 0], iris.data[:, 1], c=iris.target)\n",
|
|
"ax.set(xlabel=iris.feature_names[0], ylabel=iris.feature_names[1])\n",
|
|
"_ = ax.legend(\n",
|
|
" scatter.legend_elements()[0], iris.target_names, loc=\"lower right\", title=\"Classes\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c528846f",
|
|
"metadata": {},
|
|
"source": [
|
|
"**c)** Loop over the iris dataset(`iris.data`) and evaluate the network for each data point.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2efc507d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# No need to change this cell! Just make sure it works!\n",
|
|
"for x in iris.data:\n",
|
|
" prediction = feed_forward_4(layers, x)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "334560b6",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Exercise 5 (Very optional and very hard :)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ea0a8fe0",
|
|
"metadata": {},
|
|
"source": [
|
|
"**a)** Make the iris target values into one-hot vectors.\n",
|
|
"\n",
|
|
"**b)** Define the cross-entropy loss function to evaluate the performance of your network on the data set.\n",
|
|
"\n",
|
|
"**c)** Use the autograd package to take the gradient of the cross entropy wrt. the weights and biases of the network.\n",
|
|
"\n",
|
|
"**d)** Use gradient descent of some sort to optimize the parameters.\n",
|
|
"\n",
|
|
"**e)** Evaluate the accuracy of the network.\n",
|
|
"\n",
|
|
"**e)** Show off how you did in a group session!\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.18"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|