diff --git a/doc/src/week43/exercisesweek43.do.txt b/doc/src/week43/exercisesweek43.do.txt index 19348228d..3f00420af 100644 --- a/doc/src/week43/exercisesweek43.do.txt +++ b/doc/src/week43/exercisesweek43.do.txt @@ -1,35 +1,91 @@ -TITLE: Exercises week 42 +TITLE: Exercises weeks 43 and 44 AUTHOR: October 9-13, 2023 -DATE: Deadline is Sunday October 22 at midnight +DATE: Deadline is Sunday November 5 at midnight -You can hand in the exercises from week 41 and week 42 as one exercise and get a total score of two additional points. +You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points. -======= Overarching aims of the exercises this week ======= +======= Overarching aims of the exercises weeks 43 and 44 ======= -The aim of the exercises this week is to get started with implementing -gradient methods of relevance for project 2. The exercise this week is a simple -continuation from the previous week with the addition of automatic differentation. -Everything you develop here will be used in project 2. - -In order to get started, we will now replace in our standard ordinary -least squares (OLS) and Ridge regression codes (from project 1) the -matrix inversion algorithm with our own gradient descent (GD) and SGD -codes. You can use the Franke function or the terrain data from -project 1. _However, we recommend using a simpler function like_ -$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials. -You can obviously test your final codes against for example the Franke -function. Automatic differentiation will be discussed next week. - -You should include in your analysis of the GD and SGD codes the following elements -o A plain gradient descent with a fixed learning rate (you will need to tune it) using automatic differentiation. Compare this with the analytical expression of the gradients you obtained last week. Feel free to use _Autograd_ as Python package or _JAX_. You can use the examples form last week. -o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Compare this with the analytical expression of the gradients you obtained last week. -o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc) -o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD using automatic differentiation.. -o Add RMSprop and Adam to your library of methods for tuning the learning rate. Again using automatic differentiation. - -The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples. +The aim of the exercises this week and next week is to get started with writing a neural network code +of relevance for project 2. -We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning. +During week 41 we discussed three different types of gates, the +so-called XOR, the OR and the AND gates. In order to develop a code +for neural networks, it can be useful to set up a simpler system with +only two inputs and one output. This can make it easier to debug and +study the feed forward pass and the back propagation part. In the +exercise this and next week, we propose to study this system with just +one hidden layer and two hidden nodes. There is only one output node +and we can choose to use either a simple regression case (fitting a +line) or just a binary classification case with the cross-entropy as +cost function. +Their inputs and outputs can be +summarized using the following tables, first for the OR gate with +inputs $x_1$ and $x_2$ and outputs $y$: + +|---------------------| +| $x_1$ | $x_2$ | $y$ | +|---------------------| +| 0 | 0 | 0 | +| 0 | 1 | 1 | +| 1 | 0 | 1 | +| 1 | 1 | 1 | +|---------------------| + +!split +===== The AND and XOR Gates ===== + +The AND gate is defined as + +|---------------------| +| $x_1$ | $x_2$ | $y$ | +|---------------------| +| 0 | 0 | 0 | +| 0 | 1 | 0 | +| 1 | 0 | 0 | +| 1 | 1 | 1 | +|---------------------| + +And finally we have the XOR gate + +|---------------------| +| $x_1$ | $x_2$ | $y$ | +|---------------------| +| 0 | 0 | 0 | +| 0 | 1 | 1 | +| 1 | 0 | 1 | +| 1 | 1 | 0 | +|---------------------| + +!split +===== Representing the Data Sets ===== + +Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads + +!bt +\bm{X}=\begin{bmatrix} 0 & 0 \\ + 0 & 1 \\ + 1 & 0 \\ + 1 & 1 \end{bmatrix}, +!et + +while the vector of outputs is $\bm{y}^T=[0,1,1,0]$ for the XOR gate, $\bm{y}^T=[0,0,0,1]$ for the AND gate and $\bm{y}^T=[0,1,1,1]$ for the OR gate. + + + +Your tasks here are + +o Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs. +o Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function. +o Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output. +o Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets. +o Set up the cost function (cross entropy for classification of binary cases). +o Calculate the gradients needed for the back propagation part. +o Use the gradients to train the network in the back propagation part. Think of using automatic differentiation. +o Train the network and study your results and compare with results obtained either with _scikit-learn_ or _TensorFlow_. + +Everything you develop here can be used directly into the code for the project. + diff --git a/doc/src/week43/exercisesweek43.ipynb b/doc/src/week43/exercisesweek43.ipynb new file mode 100644 index 000000000..31ec7f4c6 --- /dev/null +++ b/doc/src/week43/exercisesweek43.ipynb @@ -0,0 +1,169 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "40653618", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "b661b3a0", + "metadata": { + "editable": true + }, + "source": [ + "# Exercises weeks 43 and 44 \n", + "**October 9-13, 2023**\n", + "\n", + "Date: **Deadline is Sunday November 5 at midnight**\n", + "\n", + "You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points." + ] + }, + { + "cell_type": "markdown", + "id": "01487b34", + "metadata": { + "editable": true + }, + "source": [ + "# Overarching aims of the exercises weeks 43 and 44\n", + "\n", + "The aim of the exercises this week and next week is to get started with writing a neural network code\n", + "of relevance for project 2. \n", + "\n", + "During week 41 we discussed three different types of gates, the\n", + "so-called XOR, the OR and the AND gates. In order to develop a code\n", + "for neural networks, it can be useful to set up a simpler system with\n", + "only two inputs and one output. This can make it easier to debug and\n", + "study the feed forward pass and the back propagation part. In the\n", + "exercise this and next week, we propose to study this system with just\n", + "one hidden layer and two hidden nodes. There is only one output node\n", + "and we can choose to use either a simple regression case (fitting a\n", + "line) or just a binary classification case with the cross-entropy as\n", + "cost function.\n", + "\n", + "Their inputs and outputs can be\n", + "summarized using the following tables, first for the OR gate with\n", + "inputs $x_1$ and $x_2$ and outputs $y$:\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
$x_1$ $x_2$ $y$
0 0 0
0 1 1
1 0 1
1 1 1
" + ] + }, + { + "cell_type": "markdown", + "id": "e8f2df30", + "metadata": { + "editable": true + }, + "source": [ + "## The AND and XOR Gates\n", + "\n", + "The AND gate is defined as\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
$x_1$ $x_2$ $y$
0 0 0
0 1 0
1 0 0
1 1 1
\n", + "\n", + "And finally we have the XOR gate\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
$x_1$ $x_2$ $y$
0 0 0
0 1 1
1 0 1
1 1 0
" + ] + }, + { + "cell_type": "markdown", + "id": "a3d25110", + "metadata": { + "editable": true + }, + "source": [ + "## Representing the Data Sets\n", + "\n", + "Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads" + ] + }, + { + "cell_type": "markdown", + "id": "abdf765d", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n", + " 0 & 1 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 1 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "684ff136", + "metadata": { + "editable": true + }, + "source": [ + "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n", + "\n", + "Your tasks here are\n", + "\n", + "1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n", + "\n", + "2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n", + "\n", + "3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n", + "\n", + "4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n", + "\n", + "5. Set up the cost function (cross entropy for classification of binary cases).\n", + "\n", + "6. Calculate the gradients needed for the back propagation part.\n", + "\n", + "7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n", + "\n", + "8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n", + "\n", + "Everything you develop here can be used directly into the code for the project." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +}