lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

adding exercises

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-23 07:22:24 +02:00
parent 197a4dc38d
commit 3fbf55bdc9
2 changed files with 252 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week43/exercisesweek43.do.txt
+83 -27
View File
@@ -1,35 +1,91 @@
TITLE: Exercises week 42
TITLE: Exercises weeks 43 and 44
AUTHOR: October 9-13, 2023
DATE: Deadline is Sunday October 22 at midnight
DATE: Deadline is Sunday November 5 at midnight
You can hand in the exercises from week 41 and week 42 as one exercise and get a total score of two additional points.
You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points.
======= Overarching aims of the exercises this week =======
======= Overarching aims of the exercises weeks 43 and 44 =======
The aim of the exercises this week is to get started with implementing
gradient methods of relevance for project 2. The exercise this week is a simple
continuation from the previous week with the addition of automatic differentation.
Everything you develop here will be used in project 2.
In order to get started, we will now replace in our standard ordinary
least squares (OLS) and Ridge regression codes (from project 1) the
matrix inversion algorithm with our own gradient descent (GD) and SGD
codes. You can use the Franke function or the terrain data from
project 1. _However, we recommend using a simpler function like_
$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.
You can obviously test your final codes against for example the Franke
function. Automatic differentiation will be discussed next week.
You should include in your analysis of the GD and SGD codes the following elements
o A plain gradient descent with a fixed learning rate (you will need to tune it) using automatic differentiation. Compare this with the analytical expression of the gradients you obtained last week. Feel free to use _Autograd_ as Python package or _JAX_. You can use the examples form last week.
o Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate). Compare this with the analytical expression of the gradients you obtained last week.
o Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)
o Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD using automatic differentiation..
o Add RMSprop and Adam to your library of methods for tuning the learning rate. Again using automatic differentiation.
The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples.
The aim of the exercises this week and next week is to get started with writing a neural network code
of relevance for project 2.
We recommend reading chapter 8 on optimization from the textbook of "Goodfellow, Bengio and Courville":"https://www.deeplearningbook.org/". This chapter contains many useful insights and discussions on the optimization part of machine learning.
During week 41 we discussed three different types of gates, the
so-called XOR, the OR and the AND gates. In order to develop a code
for neural networks, it can be useful to set up a simpler system with
only two inputs and one output. This can make it easier to debug and
study the feed forward pass and the back propagation part. In the
exercise this and next week, we propose to study this system with just
one hidden layer and two hidden nodes. There is only one output node
and we can choose to use either a simple regression case (fitting a
line) or just a binary classification case with the cross-entropy as
cost function.
Their inputs and outputs can be
summarized using the following tables, first for the OR gate with
inputs $x_1$ and $x_2$ and outputs $y$:
|---------------------|
| $x_1$ | $x_2$ | $y$ |
|---------------------|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 1 |
|---------------------|
!split
===== The AND and XOR Gates =====
The AND gate is defined as
|---------------------|
| $x_1$ | $x_2$ | $y$ |
|---------------------|
| 0 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 1 | 1 | 1 |
|---------------------|
And finally we have the XOR gate
|---------------------|
| $x_1$ | $x_2$ | $y$ |
|---------------------|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
|---------------------|
!split
===== Representing the Data Sets =====
Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads
!bt
\bm{X}=\begin{bmatrix} 0 & 0 \\
0 & 1 \\
1 & 0 \\
1 & 1 \end{bmatrix},
!et
while the vector of outputs is $\bm{y}^T=[0,1,1,0]$ for the XOR gate, $\bm{y}^T=[0,0,0,1]$ for the AND gate and $\bm{y}^T=[0,1,1,1]$ for the OR gate.
Your tasks here are
o Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.
o Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.
o Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.
o Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.
o Set up the cost function (cross entropy for classification of binary cases).
o Calculate the gradients needed for the back propagation part.
o Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.
o Train the network and study your results and compare with results obtained either with _scikit-learn_ or _TensorFlow_.
Everything you develop here can be used directly into the code for the project.
doc/src/week43/exercisesweek43.ipynb
+169
View File
@@ -0,0 +1,169 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "40653618",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek43.do.txt -->\n",
"<!-- dom:TITLE: Exercises weeks 43 and 44 -->"
]
},
{
"cell_type": "markdown",
"id": "b661b3a0",
"metadata": {
"editable": true
},
"source": [
"# Exercises weeks 43 and 44 \n",
"**October 9-13, 2023**\n",
"\n",
"Date: **Deadline is Sunday November 5 at midnight**\n",
"\n",
"You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points."
]
},
{
"cell_type": "markdown",
"id": "01487b34",
"metadata": {
"editable": true
},
"source": [
"# Overarching aims of the exercises weeks 43 and 44\n",
"\n",
"The aim of the exercises this week and next week is to get started with writing a neural network code\n",
"of relevance for project 2. \n",
"\n",
"During week 41 we discussed three different types of gates, the\n",
"so-called XOR, the OR and the AND gates. In order to develop a code\n",
"for neural networks, it can be useful to set up a simpler system with\n",
"only two inputs and one output. This can make it easier to debug and\n",
"study the feed forward pass and the back propagation part. In the\n",
"exercise this and next week, we propose to study this system with just\n",
"one hidden layer and two hidden nodes. There is only one output node\n",
"and we can choose to use either a simple regression case (fitting a\n",
"line) or just a binary classification case with the cross-entropy as\n",
"cost function.\n",
"\n",
"Their inputs and outputs can be\n",
"summarized using the following tables, first for the OR gate with\n",
"inputs $x_1$ and $x_2$ and outputs $y$:\n",
"\n",
"<table class=\"dotable\" border=\"1\">\n",
"<thead>\n",
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"</tbody>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"id": "e8f2df30",
"metadata": {
"editable": true
},
"source": [
"## The AND and XOR Gates\n",
"\n",
"The AND gate is defined as\n",
"\n",
"<table class=\"dotable\" border=\"1\">\n",
"<thead>\n",
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"</tbody>\n",
"</table>\n",
"\n",
"And finally we have the XOR gate\n",
"\n",
"<table class=\"dotable\" border=\"1\">\n",
"<thead>\n",
"<tr><th align=\"center\">$x_1$</th> <th align=\"center\">$x_2$</th> <th align=\"center\">$y$</th> </tr>\n",
"</thead>\n",
"<tbody>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 0 </td> </tr>\n",
"<tr><td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> <td align=\"center\"> 1 </td> </tr>\n",
"<tr><td align=\"center\"> 1 </td> <td align=\"center\"> 1 </td> <td align=\"center\"> 0 </td> </tr>\n",
"</tbody>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"id": "a3d25110",
"metadata": {
"editable": true
},
"source": [
"## Representing the Data Sets\n",
"\n",
"Our design matrix is defined by the input values $x_1$ and $x_2$. Since we have four possible outputs, our design matrix reads"
]
},
{
"cell_type": "markdown",
"id": "abdf765d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n",
" 0 & 1 \\\\\n",
"\t\t 1 & 0 \\\\\n",
"\t\t 1 & 1 \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "684ff136",
"metadata": {
"editable": true
},
"source": [
"while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n",
"\n",
"Your tasks here are\n",
"\n",
"1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n",
"\n",
"2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n",
"\n",
"3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n",
"\n",
"4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n",
"\n",
"5. Set up the cost function (cross entropy for classification of binary cases).\n",
"\n",
"6. Calculate the gradients needed for the back propagation part.\n",
"\n",
"7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n",
"\n",
"8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n",
"\n",
"Everything you develop here can be used directly into the code for the project."
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}