lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

small update on neural networks

Browse Source
This commit is contained in:
mhjensen
2018-10-16 13:54:54 +02:00
parent ce0c90eccd
commit 9bdb28895e
75 changed files with 6318 additions and 5703 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/NeuralNet.html
+87 -64
View File
@@ -117,54 +117,55 @@ div { text-align: justify; text-justify: inter-word; }
'___sec30'),
('Defining the cost function', 2, None, '___sec31'),
('Example: binary classification problem', 2, None, '___sec32'),
('The Softmax function', 2, None, '___sec33'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec33'),
('Collect and pre-process data', 2, None, '___sec34'),
('Train and test datasets', 2, None, '___sec35'),
('Define model and architecture', 2, None, '___sec36'),
('Layers', 2, None, '___sec37'),
('Weights and biases', 2, None, '___sec38'),
('Feed-forward pass', 2, None, '___sec39'),
('Matrix multiplications', 2, None, '___sec40'),
('Choose cost function and optimizer', 2, None, '___sec41'),
('Optimizing the cost function', 2, None, '___sec42'),
('Regularization', 2, None, '___sec43'),
('Matrix multiplication', 2, None, '___sec44'),
('Improving performance', 2, None, '___sec45'),
('Full object-oriented implementation', 2, None, '___sec46'),
('Evaluate model performance on test data', 2, None, '___sec47'),
('Adjust hyperparameters', 2, None, '___sec48'),
('Visualization', 2, None, '___sec49'),
('scikit-learn implementation', 2, None, '___sec50'),
('Visualization', 2, None, '___sec51'),
'___sec34'),
('Collect and pre-process data', 2, None, '___sec35'),
('Train and test datasets', 2, None, '___sec36'),
('Define model and architecture', 2, None, '___sec37'),
('Layers', 2, None, '___sec38'),
('Weights and biases', 2, None, '___sec39'),
('Feed-forward pass', 2, None, '___sec40'),
('Matrix multiplications', 2, None, '___sec41'),
('Choose cost function and optimizer', 2, None, '___sec42'),
('Optimizing the cost function', 2, None, '___sec43'),
('Regularization', 2, None, '___sec44'),
('Matrix multiplication', 2, None, '___sec45'),
('Improving performance', 2, None, '___sec46'),
('Full object-oriented implementation', 2, None, '___sec47'),
('Evaluate model performance on test data', 2, None, '___sec48'),
('Adjust hyperparameters', 2, None, '___sec49'),
('Visualization', 2, None, '___sec50'),
('scikit-learn implementation', 2, None, '___sec51'),
('Visualization', 2, None, '___sec52'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec52'),
('Tensorflow', 2, None, '___sec53'),
('Collect and pre-process data', 2, None, '___sec54'),
('Using TensorFlow backend', 2, None, '___sec55'),
('Optimizing and using gradient descent', 2, None, '___sec56'),
('Using Keras', 2, None, '___sec57'),
('Which activation function should I use?', 2, None, '___sec58'),
'___sec53'),
('Tensorflow', 2, None, '___sec54'),
('Collect and pre-process data', 2, None, '___sec55'),
('Using TensorFlow backend', 2, None, '___sec56'),
('Optimizing and using gradient descent', 2, None, '___sec57'),
('Using Keras', 2, None, '___sec58'),
('Which activation function should I use?', 2, None, '___sec59'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec59'),
('The derivative of the Logistic funtion', 2, None, '___sec60'),
('The RELU function family', 2, None, '___sec61'),
('Which activation function should we use?', 2, None, '___sec62'),
'___sec60'),
('The derivative of the Logistic funtion', 2, None, '___sec61'),
('The RELU function family', 2, None, '___sec62'),
('Which activation function should we use?', 2, None, '___sec63'),
('A top-down perspective on Neural networks',
2,
None,
'___sec63'),
'___sec64'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec64')]}
'___sec65')]}
end of tocinfo -->
<body>
@@ -1318,10 +1319,32 @@ $$
In case we use another activation function than the logistic one, we need to evaluate other derivatives.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">The Softmax function </h2>
In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation \( z_i^l \), that is we need
$$
\frac{\partial f(z_i^l)}{\partial w_{jk}^l} =
\frac{\partial f(z_i^l)}{\partial z_j^l} \frac{\partial z_j^l}{\partial w_{jk}^l}= \frac{\partial f(z_i^l)}{\partial z_j^l}a_k^{-1}.
$$
For the Softmax function we have
$$
f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{k=1}^K\exp{(z_k^l}}.
$$
Its derivative with respect to \( z_j^l \) gives
$$
\frac{\partial f(z_i^l)}{\partial z_j^l}= f(z_i^l)\left(\delta_{ij}-f(z_i^l)\right),
$$
which in case of the simply binary model reduces to having \( i=j \).
<p>
<!-- !split -->
<h2 id="___sec33">Developing a code for doing neural networks with back propagation </h2>
<h2 id="___sec34">Developing a code for doing neural networks with back propagation </h2>
<p>
One can identify a set of key steps when using neural networks to solve supervised learning problems:
@@ -1337,7 +1360,7 @@ One can identify a set of key steps when using neural networks to solve supervis
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">Collect and pre-process data </h2>
<h2 id="___sec35">Collect and pre-process data </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
@@ -1432,7 +1455,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec35">Train and test datasets </h2>
<h2 id="___sec36">Train and test datasets </h2>
<p>
Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.
@@ -1481,7 +1504,7 @@ X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Define model and architecture </h2>
<h2 id="___sec37">Define model and architecture </h2>
<p>
Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation \( y \) of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have
@@ -1527,7 +1550,7 @@ which is inspired by probability theory (see logistic regression) and was most c
<p>
<!-- !split -->
<h2 id="___sec37">Layers </h2>
<h2 id="___sec38">Layers </h2>
<ul>
<li> Input</li>
@@ -1572,7 +1595,7 @@ weights to the output layer.
<p>
<!-- !split -->
<h2 id="___sec38">Weights and biases </h2>
<h2 id="___sec39">Weights and biases </h2>
<p>
Typically weights are initialized with small values distributed around zero, drawn from a uniform
@@ -1608,7 +1631,7 @@ output_bias <span style="color: #666666">=</span> np<span style="color: #666666"
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">Feed-forward pass </h2>
<h2 id="___sec40">Feed-forward pass </h2>
<p>
Denote \( F \) the number of features, \( H \) the number of hidden neurons and \( C \) the number of categories.
@@ -1635,7 +1658,7 @@ $$ a_{j}^{L} = \frac{\exp{(z_j^{L})}}
<p>
<!-- !split -->
<h2 id="___sec40">Matrix multiplications </h2>
<h2 id="___sec41">Matrix multiplications </h2>
<p>
Since our data has the dimensions \( X = (n_{inputs}, n_{features}) \) and our weights to the hidden
@@ -1710,7 +1733,7 @@ predictions <span style="color: #666666">=</span> predict(X_train)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">Choose cost function and optimizer </h2>
<h2 id="___sec42">Choose cost function and optimizer </h2>
<p>
To measure how well our neural network is doing we need to introduce a cost function.
@@ -1741,7 +1764,7 @@ you got the correct label. The probability of category \( c \) is given by the s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec42">Optimizing the cost function </h2>
<h2 id="___sec43">Optimizing the cost function </h2>
<p>
The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
@@ -1781,7 +1804,7 @@ The various optmization methods, with codes and algorithms, are discussed in o
<p>
<!-- !split -->
<h2 id="___sec43">Regularization </h2>
<h2 id="___sec44">Regularization </h2>
<p>
It is common to add an extra term to the cost function, proportional
@@ -1814,7 +1837,7 @@ calculate the gradient efficently.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Matrix multiplication </h2>
<h2 id="___sec45">Matrix multiplication </h2>
<p>
To more efficently train our network these equations are implemented using matrix operations.
@@ -1928,7 +1951,7 @@ lmbd <span style="color: #666666">=</span> <span style="color: #666666">0.01</sp
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Improving performance </h2>
<h2 id="___sec46">Improving performance </h2>
<p>
As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.
@@ -1948,7 +1971,7 @@ Andrew Ng goes through some of these considerations in this <a href="https://you
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec46">Full object-oriented implementation </h2>
<h2 id="___sec47">Full object-oriented implementation </h2>
<p>
It is very natural to think of the network as an object, with specific instances of the network
@@ -2061,7 +2084,7 @@ being realizations of this object with different hyperparameters. An implementat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">Evaluate model performance on test data </h2>
<h2 id="___sec48">Evaluate model performance on test data </h2>
<p>
To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.
@@ -2096,7 +2119,7 @@ test_predict <span style="color: #666666">=</span> dnn<span style="color: #66666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">Adjust hyperparameters </h2>
<h2 id="___sec49">Adjust hyperparameters </h2>
<p>
We now perform a grid search to find the optimal hyperparameters for the network.
@@ -2129,7 +2152,7 @@ DNN_numpy <span style="color: #666666">=</span> np<span style="color: #666666">.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">Visualization </h2>
<h2 id="___sec50">Visualization </h2>
<p>
@@ -2171,7 +2194,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">scikit-learn implementation </h2>
<h2 id="___sec51">scikit-learn implementation </h2>
<p>
<b>scikit-learn</b> focuses more
@@ -2210,7 +2233,7 @@ DNN_scikit <span style="color: #666666">=</span> np<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Visualization </h2>
<h2 id="___sec52">Visualization </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -2252,7 +2275,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">Building neural networks in Tensorflow and Keras </h2>
<h2 id="___sec53">Building neural networks in Tensorflow and Keras </h2>
<p>
Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
@@ -2267,7 +2290,7 @@ NumPy arrays.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Tensorflow </h2>
<h2 id="___sec54">Tensorflow </h2>
<p>
Tensorflow is an open source library machine learning library
@@ -2312,7 +2335,7 @@ and/or if you use <b>anaconda</b>, just write (or install from the graphical use
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec54">Collect and pre-process data </h2>
<h2 id="___sec55">Collect and pre-process data </h2>
<p>
@@ -2378,7 +2401,7 @@ X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Using TensorFlow backend </h2>
<h2 id="___sec56">Using TensorFlow backend </h2>
<ol>
<li> Define model and architecture</li>
@@ -2523,7 +2546,7 @@ X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_tes
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Optimizing and using gradient descent </h2>
<h2 id="___sec57">Optimizing and using gradient descent </h2>
<p>
@@ -2601,7 +2624,7 @@ writer<span style="color: #666666">.</span>add_graph(tf<span style="color: #6666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Using Keras </h2>
<h2 id="___sec58">Using Keras </h2>
<p>
Keras is a high level <a href="https://en.wikipedia.org/wiki/Application_programming_interface" target="_blank">neural network</a>
@@ -2700,7 +2723,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec58">Which activation function should I use? </h2>
<h2 id="___sec59">Which activation function should I use? </h2>
<p>
The Back propagation algorithm we derived above works by going from
@@ -2729,7 +2752,7 @@ learn at widely different speeds
<p>
<!-- !split -->
<h2 id="___sec59">Is the Logistic activation function (Sigmoid) our choice? </h2>
<h2 id="___sec60">Is the Logistic activation function (Sigmoid) our choice? </h2>
<p>
Although this unfortunate behavior has been empirically observed for
@@ -2759,7 +2782,7 @@ better than the logistic function in deep networks).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec60">The derivative of the Logistic funtion </h2>
<h2 id="___sec61">The derivative of the Logistic funtion </h2>
<p>
Looking at the logistic activation function, when inputs become large
@@ -2795,7 +2818,7 @@ fast to compute).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec61">The RELU function family </h2>
<h2 id="___sec62">The RELU function family </h2>
<p>
The ReLU activation function suffers from a problem known as the dying
@@ -2822,7 +2845,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec62">Which activation function should we use? </h2>
<h2 id="___sec63">Which activation function should we use? </h2>
<p>
In general it seems that the ELU activation function is better than
@@ -2842,7 +2865,7 @@ bootstrap to evaluate other activation functions.
<p>
<!-- !split -->
<h2 id="___sec63">A top-down perspective on Neural networks </h2>
<h2 id="___sec64">A top-down perspective on Neural networks </h2>
<p>
The first thing we would like to do is divide the data into two or three
@@ -2884,7 +2907,7 @@ supervised learning.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec64">Limitations of supervised learning with deep networks </h2>
<h2 id="___sec65">Limitations of supervised learning with deep networks </h2>
<p>
Like all statistical methods, supervised learning using neural