FYS-STK4155/doc/LectureNotes/_build/jupyter_execute/week42.ipynb

Week 42 Constructing a Neural Network code with examples

Morten Hjorth-Jensen, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University

Date: October 14-18, 2024

Lecture October 14, 2024

1. Building our own Feed-forward Neural Network and discussion of project 2

Readings and videos.

1. These lecture notes

2. For a more in depth discussion on neural networks we recommend Goodfellow et al chapters 6 and 7.

3. Neural Networks demystified at https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs

4. Building Neural Networks from scratch at https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex

5. Video on Neural Networks at https://www.youtube.com/watch?v=CqOfi41LfDw

6. Video on the back propagation algorithm at https://www.youtube.com/watch?v=Ilg3gGewQ5U

I also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at http://neuralnetworksanddeeplearning.com/chap4.html.

Material for the active learning sessions on Tuesday and Wednesday

• Exercise on starting to write a code for neural networks, feed forward part. We will also continue ur discussions of gradient descent methods from last week. If you have time, start considering the back-propagation part as well (exercises for next week)

• Discussion of project 2

Note: some of the codes will also be discussed next week in connection with the solution of differential equations.

Writing a code which implements a feed-forward neural network

Last week we discussed the basics of neural networks and deep learning and the basics of automatic differentiation. We looked also at examples on how compute the parameters of a simple network with scalar inputs and ouputs and no or just one hidden layers.

We ended our discussions with the derivation of the equations for a neural network with one hidden layers and two input variables and two hidden nodes but only one output node.

Mathematics of deep learning

Two recent books online.

1. The Modern Mathematics of Deep Learning, by Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen, published as Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022

2. Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory, Arnulf Jentzen, Benno Kuckuck, Philippe von Wurstemberger

Reminder on books with hands-on material and codes

• Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch

Reading recommendations

1. Rashkca et al., chapter 11, jupyter-notebook sent separately, from GitHub

2. Goodfellow et al, chapter 6 and 7 contain most of the neural network background.

First network example, simple percepetron with one input

As yet another example we define now a simple perceptron model with all quantities given by scalars. We consider only one input variable x and one target value y. We define an activation function \sigma_1 which takes as input

z_1 = w_1x+b_1,

where w_1 is the weight and b_1 is the bias. These are the parameters we want to optimize. The output is a_1=\sigma(z_1) (see graph from whiteboard notes). This output is then fed into the cost/loss function, which we here for the sake of simplicity just define as the squared error

C(x;w_1,b_1)=\frac{1}{2}(a_1-y)^2.

Layout of a simple neural network with no hidden layer

Figure 1:

Optimizing the parameters

In setting up the feed forward and back propagation parts of the algorithm, we need now the derivative of the various variables we want to train.

We need

\frac{\partial C}{\partial w_1} \hspace{0.1cm}\mathrm{and}\hspace{0.1cm}\frac{\partial C}{\partial b_1}.

Using the chain rule we find

\frac{\partial C}{\partial w_1}=\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial w_1}=(a_1-y)\sigma_1'x,

and

\frac{\partial C}{\partial b_1}=\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial b_1}=(a_1-y)\sigma_1',

which we later will just define as

\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}=\delta_1.

Adding a hidden layer

We change our simple model to (see graph) a network with just one hidden layer but with scalar variables only.

Our output variable changes to a_2 and a_1 is now the output from the hidden node and a_0=x. We have then

z_1 = w_1a_0+b_1 \hspace{0.1cm} \wedge a_1 = \sigma_1(z_1),

z_2 = w_2a_1+b_2 \hspace{0.1cm} \wedge a_2 = \sigma_2(z_2),

and the cost function

C(x;\boldsymbol{\Theta})=\frac{1}{2}(a_2-y)^2,

with \boldsymbol{\Theta}=[w_1,w_2,b_1,b_2].

Layout of a simple neural network with one hidden layer

Figure 1:

The derivatives

The derivatives are now, using the chain rule again

\frac{\partial C}{\partial w_2}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial w_2}=(a_2-y)\sigma_2'a_1=\delta_2a_1,

\frac{\partial C}{\partial b_2}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial b_2}=(a_2-y)\sigma_2'=\delta_2,

\frac{\partial C}{\partial w_1}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial w_1}=(a_2-y)\sigma_2'a_1\sigma_1'a_0,

\frac{\partial C}{\partial b_1}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial b_1}=(a_2-y)\sigma_2'\sigma_1'=\delta_1.

Can you generalize this to more than one hidden layer?

Important observations

From the above equations we see that the derivatives of the activation functions play a central role. If they vanish, the training may stop. This is called the vanishing gradient problem, see discussions below. If they become large, the parameters w_i and b_i may simply go to infinity. This is referenced as the exploding gradient problem.

The training

The training of the parameters is done through various gradient descent approximations with

w_{i}\leftarrow w_{i}- \eta \delta_i a_{i-1},

and

b_i \leftarrow b_i-\eta \delta_i,

with \eta is the learning rate.

One iteration consists of one feed forward step and one back-propagation step. Each back-propagation step does one update of the parameters \boldsymbol{\Theta}.

For the first hidden layer a_{i-1}=a_0=x for this simple model.

Code example

The code here implements the above model with one hidden layer and scalar variables for the same function we studied in the previous example. The code is however set up so that we can add multiple inputs x and target values y. Note also that we have the possibility of defining a feature matrix \boldsymbol{X} with more than just one column for the input values. This will turn useful in our next example. We have also defined matrices and vectors for all of our operations although it is not necessary here.

In [1]:

import numpy as np
# We use the Sigmoid function as activation function
def sigmoid(z):
    return 1.0/(1.0+np.exp(-z))

def forwardpropagation(x):
    # weighted sum of inputs to the hidden layer
    z_1 = np.matmul(x, w_1) + b_1
    # activation in the hidden layer
    a_1 = sigmoid(z_1)
    # weighted sum of inputs to the output layer
    z_2 = np.matmul(a_1, w_2) + b_2
    a_2 = z_2
    return a_1, a_2

def backpropagation(x, y):
    a_1, a_2 = forwardpropagation(x)
    # parameter delta for the output layer, note that a_2=z_2 and its derivative wrt z_2 is just 1
    delta_2 = a_2 - y
    print(0.5*((a_2-y)**2))
    # delta for  the hidden layer
    delta_1 = np.matmul(delta_2, w_2.T) * a_1 * (1 - a_1)
    # gradients for the output layer
    output_weights_gradient = np.matmul(a_1.T, delta_2)
    output_bias_gradient = np.sum(delta_2, axis=0)
    # gradient for the hidden layer
    hidden_weights_gradient = np.matmul(x.T, delta_1)
    hidden_bias_gradient = np.sum(delta_1, axis=0)
    return output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient


# ensure the same random numbers appear every time
np.random.seed(0)
# Input variable
x = np.array([4.0],dtype=np.float64)
# Target values
y = 2*x+1.0 

# Defining the neural network, only scalars here
n_inputs = x.shape
n_features = 1
n_hidden_neurons = 1
n_outputs = 1

# Initialize the network
# weights and bias in the hidden layer
w_1 = np.random.randn(n_features, n_hidden_neurons)
b_1 = np.zeros(n_hidden_neurons) + 0.01

# weights and bias in the output layer
w_2 = np.random.randn(n_hidden_neurons, n_outputs)
b_2 = np.zeros(n_outputs) + 0.01

eta = 0.1
for i in range(50):
    # calculate gradients
    derivW2, derivB2, derivW1, derivB1 = backpropagation(x, y)
    # update weights and biases
    w_2 -= eta * derivW2
    b_2 -= eta * derivB2
    w_1 -= eta * derivW1
    b_1 -= eta * derivB1

[36.89563074]
[23.62323175]
[15.1251681]
[9.68402334]
[6.20020163]
[3.96963458]
[2.54150298]
[1.62714703]
[1.0417409]
[0.66694492]
[0.42699034]
[0.27336592]
[0.17501258]
[0.11204514]
[0.07173249]
[0.04592382]
[0.02940083]
[0.01882264]
[0.01205039]
[0.00771475]
[0.00493903]
[0.003162]
[0.00202433]
[0.00129599]
[0.0008297]
[0.00053118]
[0.00034006]
[0.00021771]
[0.00013938]
[8.92313548e-05]
[5.71263851e-05]
[3.6572612e-05]
[2.34139775e-05]
[1.49897504e-05]
[9.5965161e-06]
[6.14373934e-06]
[3.93325371e-06]
[2.51808934e-06]
[1.61209378e-06]
[1.03207075e-06]
[6.60737006e-07]
[4.23007231e-07]
[2.70811405e-07]
[1.73374853e-07]
[1.10995472e-07]
[7.10598715e-08]
[4.54928947e-08]
[2.91247848e-08]
[1.86458368e-08]
[1.19371605e-08]

We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small.

Simple neural network and the back propagation equations

Let us now try to increase our level of ambition and attempt at setting up the equations for a neural network with two input nodes, one hidden layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..

We need to define the following parameters and variables with the input layer (layer (0)) where we label the nodes x_1 and x_2

x_1 = a_1^{(0)} \wedge x_2 = a_2^{(0)}.

The hidden layer (layer (1)) has nodes which yield the outputs a_1^{(1)} and a_2^{(1)}) with weight \boldsymbol{w} and bias \boldsymbol{b} parameters

w_{ij}^{(1)}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)}\right\} \wedge b^{(1)}=\left\{b_1^{(1)},b_2^{(1)}\right\}.

Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node

Figure 1:

The ouput layer

We have the ouput layer given by layer label (2) with output a^{(2)} and weights and biases to be determined given by the variables

w_{i}^{(2)}=\left\{w_{1}^{(2)},w_{2}^{(2)}\right\} \wedge b^{(2)}.

Our output is \tilde{y}=a^{(2)} and we define a generic cost function C(a^{(2)},y;\boldsymbol{\Theta}) where y is the target value (a scalar here). The parameters we need to optimize are given by

\boldsymbol{\Theta}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\right\}.

Compact expressions

We can define the inputs to the activation functions for the various layers in terms of various matrix-vector multiplications and vector additions. The inputs to the first hidden layer are

\begin{bmatrix}z_1^{(1)} \\ z_2^{(1)} \end{bmatrix}=\left(\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\ w_{21}^{(1)} &w_{22}^{(1)} \end{bmatrix}\right)^{T}\begin{bmatrix}a_1^{(0)} \\ a_2^{(0)} \end{bmatrix}+\begin{bmatrix}b_1^{(1)} \\ b_2^{(1)} \end{bmatrix},

with outputs

\begin{bmatrix}a_1^{(1)} \\ a_2^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_1^{(1)}) \\ \sigma^{(1)}(z_2^{(1)}) \end{bmatrix}.

Output layer

For the final output layer we have the inputs to the final activation function

z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},

resulting in the output

a^{(2)}=\sigma^{(2)}(z^{(2)}).

Explicit derivatives

In total we have nine parameters which we need to train. Using the chain rule (or just the back-propagation algorithm) we can find all derivatives. Since we will use automatic differentiation in reverse mode, we start with the derivatives of the cost function with respect to the parameters of the output layer, namely

\frac{\partial C}{\partial w_{i}^{(2)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial w_{i}^{(2)}}=\delta^{(2)}a_i^{(1)},

with

\delta^{(2)}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}

and finally

\frac{\partial C}{\partial b^{(2)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial b^{(2)}}=\delta^{(2)}.

Derivatives of the hidden layer

Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)

\frac{\partial C}{\partial w_{11}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}=    \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}},

which, noting that

z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},

allows us to rewrite

\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}a_1^{(1)}.

Final expression

Defining

\delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)},

we have

\frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)}.

Similarly, we obtain

\frac{\partial C}{\partial w_{12}^{(1)}}=\delta_1^{(1)}a_2^{(1)}.

Completing the list

Similarly, we find

\frac{\partial C}{\partial w_{21}^{(1)}}=\delta_2^{(1)}a_1^{(1)},

and

\frac{\partial C}{\partial w_{22}^{(1)}}=\delta_2^{(1)}a_2^{(1)},

where we have defined

\delta_2^{(1)}=w_2^{(2)}\frac{\partial a_2^{(1)}}{\partial z_2^{(1)}}\delta^{(2)}.

Final expressions for the biases of the hidden layer

For the sake of completeness, we list the derivatives of the biases, which are

\frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)},

and

\frac{\partial C}{\partial b_{2}^{(1)}}=\delta_2^{(1)}.

As we will see below, these expressions can be generalized in a more compact form.

Gradient expressions

For this specific model, with just one output node and two hidden nodes, the gradient descent equations take the following form for output layer

w_{i}^{(2)}\leftarrow w_{i}^{(2)}- \eta \delta^{(2)} a_{i}^{(1)},

and

b^{(2)} \leftarrow b^{(2)}-\eta \delta^{(2)},

and

w_{ij}^{(1)}\leftarrow w_{ij}^{(1)}- \eta \delta_{i}^{(1)} a_{j}^{(0)},

and

b_{i}^{(1)} \leftarrow b_{i}^{(1)}-\eta \delta_{i}^{(1)},

where \eta is the learning rate.

Setting up the equations for a neural network

The questions we want to ask are how do changes in the biases and the weights in our network change the cost function and how can we use the final output to modify the weights and biases?

To derive these equations let us start with a plain regression problem and define our cost function as

{\cal C}(\boldsymbol{\Theta})  =  \frac{1}{2}\sum_{i=1}^n\left(y_i - \tilde{y}_i\right)^2,

Warning:

Output truncated. This notebook contains too many cells to display efficiently.