FYS-STK4155/doc/Programs/ANN/Ann1.ipynb

Nicolas Dronchi

Day 22 Pre-Class assignment: Introduction to Artificial Neural Networks

This entire Artificial Neural Networks module is from Neural Networks Demystified by @stephencwelch. We have streamlined the content to better fit the format of the class. However, if you have questions or are just curious I highly recommend downloading everything from the following git repository. It is a great reference to have:

git clone https://github.com/stephencwelch/Neural-Networks-Demystified

Goals for today's pre-class assignment

1. Think about the architecture of Artificial Neural Networks
2. Model data flow by performing forward propagation
3. Explore A Neural Network (through a visualization)

Assignment instructions

This assignment is due by 11:59 p.m. the day before class and should be uploaded into the appropriate "Pre-class assignments" dropbox folder in the Desire2Learn website.

1. The architecture of Artificial Neural Networks

Watch the following video:

In [1]:

from IPython.display import YouTubeVideo
YouTubeVideo('bxe2T-V8XRs',width=640,height=360)

Out [1]:

We will use the data from the video above:

X = \left[\begin{matrix} 3 & 5 \\ 5 & 1 \\ 10 & 2 \end{matrix}\right] \hspace{1cm} , \hspace{1cm}y = \left[ \begin{matrix} 75 \\ 82 \\ 93 \end{matrix}\right]

✅ Step 1: Initialize your inputs

Create two numpy arrays to store the values of the variables X and y, as well as their normalized counterparts X_{norm} and y_{norm}.

In [18]:

# your code here (Note include needed libraries):
import numpy as np
import matplotlib.pyplot as plt

# input data (hours of sleep, hours of study)
X = np.array([[3,5],[5,1],[10,2]])

# normalized X
X_norm = X/np.amax(X)

# output data (test score)
y = np.array([[75],[82],[93]])

# normalized y
y_norm = y/100

2. Data flow: forward propagation

Data in a neural network flows via a process called forward propagation. Watch the following video:

In [8]:

from IPython.display import YouTubeVideo
YouTubeVideo('UJwK6jAStmg',width=640,height=360, align='Center')

Out [8]:

Question 1: How many input layers, hidden layers and output layers are there in the neural network shown in the video?

In [11]:

# Put your answer here
inputLayerSize = 2
outputLayerSize = 1
hiddenLayerSize = 3

✅ Step 2: Initialize random weights

Randomly initialize two numpy arrays W1 and W2, of the right dimensions, to store the weights (zero-one) in the synapses between input layer --> hidden layer, and hidden layer --> output layer

In [14]:

# your code here:

W1 = np.random.randn(inputLayerSize, hiddenLayerSize)


W2 = np.random.randn(hiddenLayerSize, outputLayerSize)

✅ Step 3: Multiply the normalized input matrix by W^{(1)}

Z^{(2)} = X W^{(1)}

Here is the code using the NumPy dot function. If you get an error you may have initilized the size of your variables incorrectly. Make sure the second dimension of X_norm matches the first dimension of W1:

In [15]:

Z2 = np.dot(X_norm, W1)
Z2

Out [15]:

array([[-0.85206517, -0.46315777,  0.47725383],
       [-0.96041205, -1.14843537,  0.98384797],
       [-1.92082411, -2.29687075,  1.96769593]])

✅ Do This: Implement and test the sigmoid function

a(z) = \frac{1}{1 + e^{-z}}

The implemented sigmoid function should take as input a numpy array and return a numpy array of the same dimension, with the function f applied to each entry.

In [19]:

# your code here:
def sigmoid(z):
    # apply sigmoid activation function
    return 1/(1+np.exp(-z))

Test your sigmoid function using the following testing code:

In [23]:

testInput = np.arange(-6,6,0.01)
plt.plot(testInput, sigmoid(testInput), linewidth= 2)
plt.grid(1)
plt.show()

✅ Step 4: Apply the sigmoid function to Z^{(2)}

a^{(2)} = f({Z^{(2)}})

Here is the code to apply the sigmoid function to Z^{(2)} and display the results

In [24]:

a2 = sigmoid(Z2)
a2

Out [24]:

array([[0.29899982, 0.38623698, 0.6170992 ],
       [0.2767957 , 0.24077498, 0.72787107],
       [0.1277697 , 0.09138246, 0.87736342]])

✅ Step 5: multiply A^{(2)} by W^{(2)} to get Z^{(3)}

Z^{(3)} = A^{(2)} W^{(2)}

In [25]:

Z3 = np.dot(a2, W2)
Z3

Out [25]:

array([[1.72574334],
       [1.70159205],
       [1.58224372]])

✅ Step 6: Apply the sigmoid function again to Z^{(3)} to produce \hat{y}

\hat{y} = f({Z^{(3)}})

In [27]:

# your code here:
yHat = sigmoid(Z3)

Final Comparison

Now compare the estimation output (\hat{y}) to the actual output y_norm.

In [28]:

y_norm

Out [28]:

array([[0.75],
       [0.82],
       [0.93]])

In [29]:

yHat

Out [29]:

array([[0.84886714],
       [0.84574255],
       [0.82952205]])

Of course the results from forward propagation are terrible; no surprises here! The weights have not been properly chosen. That's what training a network does: the goal is to find a combination of weights so that the result of forward propagation fits the intended output data as best as possible.

We will be covering this topic in class.

3. Exploring A Neural Network

Please go to the following website : http://playground.tensorflow.org/

There, you'll have the opportunity to play with an actual neural network (e.g., choosing its architecture and the type of activation function) for classification purpose.

---

Assignment wrap-up

Please fill out the form that appears when you run the code below. You must completely fill this out in order to receive credit for the assignment!

In [30]:

from IPython.display import HTML
HTML(
"""
<iframe 
	src="https://goo.gl/forms/XqTdYAtXYDSc1R7V2" 
	width="80%" 
	height="1200px" 
	frameborder="0" 
	marginheight="0" 
	marginwidth="0">
	Loading...
</iframe>
"""
)

Out [30]:

---

Congratulations, you're done with your pre-class assignment!

Now, you just need to submit this assignment by uploading it to the course Desire2Learn web page for today's dropbox (Don't forget to add your name in the first cell).

© Copyright 2017, Michigan State University Board of Trustees