FYS-STK4155/doc/LectureNotes/_build/jupyter_execute/exercisesweek43.ipynb

Exercises weeks 43 and 44

October 9-13, 2023

Date: Deadline is Sunday November 5 at midnight

You can hand in the exercises from week 43 and week 44 as one exercise and get a total score of two additional points.

Overarching aims of the exercises weeks 43 and 44

The aim of the exercises this week and next week is to get started with writing a neural network code of relevance for project 2.

During week 41 we discussed three different types of gates, the so-called XOR, the OR and the AND gates. In order to develop a code for neural networks, it can be useful to set up a simpler system with only two inputs and one output. This can make it easier to debug and study the feed forward pass and the back propagation part. In the exercise this and next week, we propose to study this system with just one hidden layer and two hidden nodes. There is only one output node and we can choose to use either a simple regression case (fitting a line) or just a binary classification case with the cross-entropy as cost function.

Their inputs and outputs can be summarized using the following tables, first for the OR gate with inputs x_1 and x_2 and outputs y:

$x_1$	$x_2$	$y$
0	0	0
0	1	1
1	0	1
1	1	1

The AND and XOR Gates

The AND gate is defined as

$x_1$	$x_2$	$y$
0	0	0
0	1	0
1	0	0
1	1	1

And finally we have the XOR gate

$x_1$	$x_2$	$y$
0	0	0
0	1	1
1	0	1
1	1	0

Representing the Data Sets

Our design matrix is defined by the input values x_1 and x_2. Since we have four possible outputs, our design matrix reads

\boldsymbol{X}=\begin{bmatrix} 0 & 0 \\
                       0 & 1 \\
		       1 & 0 \\
		       1 & 1 \end{bmatrix},

while the vector of outputs is \boldsymbol{y}^T=[0,1,1,0] for the XOR gate, \boldsymbol{y}^T=[0,0,0,1] for the AND gate and \boldsymbol{y}^T=[0,1,1,1] for the OR gate.

Your tasks here are

1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.

2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.

3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.

4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.

5. Set up the cost function (cross entropy for classification of binary cases).

6. Calculate the gradients needed for the back propagation part.

7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.

8. Train the network and study your results and compare with results obtained either with scikit-learn or TensorFlow.

Everything you develop here can be used directly into the code for the project.