Morten Hjorth-Jensen
170 lines
6.7 KiB
Plaintext
170 lines
6.7 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "20ae416e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html exercisesweek43.do.txt -->\n",
|
|
"<!-- dom:TITLE: Exercises weeks 43 and 44 -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8bfcd25c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Exercises weeks 43 and 44 \n",
|
|
"**October 9-13, 2023**\n",
|
|
"\n",
|
|
"Date: **Deadline is Sunday November 5 at midnight**\n",
|
|
"\n",
|
|
"You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fed7dbe4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Overarching aims of the exercises weeks 43 and 44\n",
|
|
"\n",
|
|
"The aim of the exercises this week and next week is to get started with writing a neural network code\n",
|
|
"of relevance for project 2. \n",
|
|
"\n",
|
|
"During week 41 we discussed three different types of gates, the\n",
|
|
"so-called XOR, the OR and the AND gates. In order to develop a code\n",
|
|
"for neural networks, it can be useful to set up a simpler system with\n",
|
|
"only two inputs and one output. This can make it easier to debug and\n",
|
|
"study the feed forward pass and the back propagation part. In the\n",
|
|
"exercise this and next week, we propose to study this system with just\n",
|
|
"one hidden layer and two hidden nodes. There is only one output node\n",
|
|
"and we can choose to use either a simple regression case (fitting a\n",
|
|
"line) or just a binary classification case with the cross-entropy as\n",
|
|
"cost function.\n",
|
|
"\n",
|
|
"Their inputs and outputs can be\n",
|
|
"summarized using the following tables, first for the OR gate with\n",
|
|
"inputs $x_1$ and $x_2$ and outputs $y$:\n",
|
|
"\n",
|
|
"<table class=\"dotable\" border=\"1\">\n",
|
|
"<thead>\n",
|
|
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
|
|
"</thead>\n",
|
|
"<tbody>\n",
|
|
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
|
"</tbody>\n",
|
|
"</table>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f108a242",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## The AND and XOR Gates\n",
|
|
"\n",
|
|
"The AND gate is defined as\n",
|
|
"\n",
|
|
"<table class=\"dotable\" border=\"1\">\n",
|
|
"<thead>\n",
|
|
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
|
|
"</thead>\n",
|
|
"<tbody>\n",
|
|
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
|
"</tbody>\n",
|
|
"</table>\n",
|
|
"\n",
|
|
"And finally we have the XOR gate\n",
|
|
"\n",
|
|
"<table class=\"dotable\" border=\"1\">\n",
|
|
"<thead>\n",
|
|
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
|
|
"</thead>\n",
|
|
"<tbody>\n",
|
|
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
|
|
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
|
|
"</tbody>\n",
|
|
"</table>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aa6993a7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Representing the Data Sets\n",
|
|
"\n",
|
|
"Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "90bd0efa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
|
|
" 0 & 1 \\\\\n",
|
|
"\t\t 1 & 0 \\\\\n",
|
|
"\t\t 1 & 1 \\end{bmatrix},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "23ba74e1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
|
|
"\n",
|
|
"Your tasks here are\n",
|
|
"\n",
|
|
"1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n",
|
|
"\n",
|
|
"2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n",
|
|
"\n",
|
|
"3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n",
|
|
"\n",
|
|
"4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n",
|
|
"\n",
|
|
"5. Set up the cost function (cross entropy for classification of binary cases).\n",
|
|
"\n",
|
|
"6. Calculate the gradients needed for the back propagation part.\n",
|
|
"\n",
|
|
"7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n",
|
|
"\n",
|
|
"8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n",
|
|
"\n",
|
|
"Everything you develop here can be used directly into the code for the project."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|