lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

small update on neural networks

Browse Source
This commit is contained in:
mhjensen
2018-10-16 13:54:54 +02:00
parent ce0c90eccd
commit 9bdb28895e
75 changed files with 6318 additions and 5703 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/NeuralNet-reveal.html
+60 -32
View File
@@ -1383,7 +1383,35 @@ In case we use another activation function than the logistic one, we need to eva
<section>
<h2 id="___sec33">Developing a code for doing neural networks with back propagation </h2>
<h2 id="___sec33">The Softmax function </h2>
In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation \( z_i^l \), that is we need
<p>&nbsp;<br>
$$
\frac{\partial f(z_i^l)}{\partial w_{jk}^l} =
\frac{\partial f(z_i^l)}{\partial z_j^l} \frac{\partial z_j^l}{\partial w_{jk}^l}= \frac{\partial f(z_i^l)}{\partial z_j^l}a_k^{-1}.
$$
<p>&nbsp;<br>
For the Softmax function we have
<p>&nbsp;<br>
$$
f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{k=1}^K\exp{(z_k^l}}.
$$
<p>&nbsp;<br>
Its derivative with respect to \( z_j^l \) gives
<p>&nbsp;<br>
$$
\frac{\partial f(z_i^l)}{\partial z_j^l}= f(z_i^l)\left(\delta_{ij}-f(z_i^l)\right),
$$
<p>&nbsp;<br>
which in case of the simply binary model reduces to having \( i=j \).
</section>
<section>
<h2 id="___sec34">Developing a code for doing neural networks with back propagation </h2>
<p>
One can identify a set of key steps when using neural networks to solve supervised learning problems:
@@ -1405,7 +1433,7 @@ One can identify a set of key steps when using neural networks to solve supervis
<section>
<h2 id="___sec34">Collect and pre-process data </h2>
<h2 id="___sec35">Collect and pre-process data </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
@@ -1505,7 +1533,7 @@ plt.show()
<section>
<h2 id="___sec35">Train and test datasets </h2>
<h2 id="___sec36">Train and test datasets </h2>
<p>
Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.
@@ -1555,7 +1583,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<section>
<h2 id="___sec36">Define model and architecture </h2>
<h2 id="___sec37">Define model and architecture </h2>
<p>
Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation \( y \) of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have
@@ -1609,7 +1637,7 @@ which is inspired by probability theory (see logistic regression) and was most c
<section>
<h2 id="___sec37">Layers </h2>
<h2 id="___sec38">Layers </h2>
<ul>
<p><li> Input</li>
@@ -1661,7 +1689,7 @@ weights to the output layer.
<section>
<h2 id="___sec38">Weights and biases </h2>
<h2 id="___sec39">Weights and biases </h2>
<p>
Typically weights are initialized with small values distributed around zero, drawn from a uniform
@@ -1700,7 +1728,7 @@ output_bias = np.zeros(n_categories) + <span style="color: #B452CD">0.01</span>
<section>
<h2 id="___sec39">Feed-forward pass </h2>
<h2 id="___sec40">Feed-forward pass </h2>
<p>
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
@@ -1735,7 +1763,7 @@ $$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
<section>
<h2 id="___sec40">Matrix multiplications </h2>
<h2 id="___sec41">Matrix multiplications </h2>
<p>
Since our data has the dimensions \( X = (n_{inputs}, n_{features}) \) and our weights to the hidden
@@ -1821,7 +1849,7 @@ predictions = predict(X_train)
<section>
<h2 id="___sec41">Choose cost function and optimizer </h2>
<h2 id="___sec42">Choose cost function and optimizer </h2>
<p>
To measure how well our neural network is doing we need to introduce a cost function.
@@ -1856,7 +1884,7 @@ you got the correct label. The probability of category \( c \) is given by the s
<section>
<h2 id="___sec42">Optimizing the cost function </h2>
<h2 id="___sec43">Optimizing the cost function </h2>
<p>
The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
@@ -1902,7 +1930,7 @@ The various optmization methods, with codes and algorithms, are discussed in o
<section>
<h2 id="___sec43">Regularization </h2>
<h2 id="___sec44">Regularization </h2>
<p>
It is common to add an extra term to the cost function, proportional
@@ -1937,7 +1965,7 @@ calculate the gradient efficently.
<section>
<h2 id="___sec44">Matrix multiplication </h2>
<h2 id="___sec45">Matrix multiplication </h2>
<p>
To more efficently train our network these equations are implemented using matrix operations.
@@ -2064,7 +2092,7 @@ lmbd = <span style="color: #B452CD">0.01</span>
<section>
<h2 id="___sec45">Improving performance </h2>
<h2 id="___sec46">Improving performance </h2>
<p>
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
@@ -2084,7 +2112,7 @@ Andrew Ng goes through some of these considerations in this <a href="https://you
<section>
<h2 id="___sec46">Full object-oriented implementation </h2>
<h2 id="___sec47">Full object-oriented implementation </h2>
<p>
It is very natural to think of the network as an object, with specific instances of the network
@@ -2198,7 +2226,7 @@ being realizations of this object with different hyperparameters. An implementat
<section>
<h2 id="___sec47">Evaluate model performance on test data </h2>
<h2 id="___sec48">Evaluate model performance on test data </h2>
<p>
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
@@ -2236,7 +2264,7 @@ test_predict = dnn.predict(X_test)
<section>
<h2 id="___sec48">Adjust hyperparameters </h2>
<h2 id="___sec49">Adjust hyperparameters </h2>
<p>
We now perform a grid search to find the optimal hyperparameters for the network.
@@ -2270,7 +2298,7 @@ DNN_numpy = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span s
<section>
<h2 id="___sec49">Visualization </h2>
<h2 id="___sec50">Visualization </h2>
<p>
@@ -2313,7 +2341,7 @@ plt.show()
<section>
<h2 id="___sec50">scikit-learn implementation </h2>
<h2 id="___sec51">scikit-learn implementation </h2>
<p>
<b>scikit-learn</b> focuses more
@@ -2353,7 +2381,7 @@ DNN_scikit = np.zeros((<span style="color: #658b00">len</span>(eta_vals), <span
<section>
<h2 id="___sec51">Visualization </h2>
<h2 id="___sec52">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -2396,7 +2424,7 @@ plt.show()
<section>
<h2 id="___sec52">Building neural networks in Tensorflow and Keras </h2>
<h2 id="___sec53">Building neural networks in Tensorflow and Keras </h2>
<p>
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -2411,7 +2439,7 @@ NumPy arrays.
<section>
<h2 id="___sec53">Tensorflow </h2>
<h2 id="___sec54">Tensorflow </h2>
<p>
Tensorflow is an open source library machine learning library
@@ -2457,7 +2485,7 @@ and/or if you use <b>anaconda</b>, just write (or install from the graphical use
<section>
<h2 id="___sec54">Collect and pre-process data </h2>
<h2 id="___sec55">Collect and pre-process data </h2>
<p>
@@ -2524,7 +2552,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<section>
<h2 id="___sec55">Using TensorFlow backend </h2>
<h2 id="___sec56">Using TensorFlow backend </h2>
<ol>
<p><li> Define model and architecture</li>
@@ -2669,7 +2697,7 @@ X_train, X_test, Y_train, Y_test = train_test_split(inputs, labels, train_size=t
<section>
<h2 id="___sec56">Optimizing and using gradient descent </h2>
<h2 id="___sec57">Optimizing and using gradient descent </h2>
<p>
@@ -2748,7 +2776,7 @@ writer.add_graph(tf.get_default_graph())
<section>
<h2 id="___sec57">Using Keras </h2>
<h2 id="___sec58">Using Keras </h2>
<p>
Keras is a high level <a href="https://en.wikipedia.org/wiki/Application_programming_interface" target="_blank">neural network</a>
@@ -2848,7 +2876,7 @@ plt.show()
<section>
<h2 id="___sec58">Which activation function should I use? </h2>
<h2 id="___sec59">Which activation function should I use? </h2>
<p>
The Back propagation algorithm we derived above works by going from
@@ -2877,7 +2905,7 @@ learn at widely different speeds
<section>
<h2 id="___sec59">Is the Logistic activation function (Sigmoid) our choice? </h2>
<h2 id="___sec60">Is the Logistic activation function (Sigmoid) our choice? </h2>
<p>
Although this unfortunate behavior has been empirically observed for
@@ -2907,7 +2935,7 @@ better than the logistic function in deep networks).
<section>
<h2 id="___sec60">The derivative of the Logistic funtion </h2>
<h2 id="___sec61">The derivative of the Logistic funtion </h2>
<p>
Looking at the logistic activation function, when inputs become large
@@ -2943,7 +2971,7 @@ fast to compute).
<section>
<h2 id="___sec61">The RELU function family </h2>
<h2 id="___sec62">The RELU function family </h2>
<p>
The ReLU activation function suffers from a problem known as the dying
@@ -2972,7 +3000,7 @@ $$
<section>
<h2 id="___sec62">Which activation function should we use? </h2>
<h2 id="___sec63">Which activation function should we use? </h2>
<p>
In general it seems that the ELU activation function is better than
@@ -2992,7 +3020,7 @@ bootstrap to evaluate other activation functions.
<section>
<h2 id="___sec63">A top-down perspective on Neural networks </h2>
<h2 id="___sec64">A top-down perspective on Neural networks </h2>
<p>
The first thing we would like to do is divide the data into two or three
@@ -3035,7 +3063,7 @@ supervised learning.
<section>
<h2 id="___sec64">Limitations of supervised learning with deep networks </h2>
<h2 id="___sec65">Limitations of supervised learning with deep networks </h2>
<p>
Like all statistical methods, supervised learning using neural