lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2024-10-14 06:03:15 +02:00
parent 7467219361
commit dd67b6769a
324 changed files with 104185 additions and 6250 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week42/html/week42.html
+106 -152
View File
@@ -185,10 +185,10 @@ div.toc p,a {
None,
'simple-neural-network-and-the-back-propagation-equations'),
('Layout of a simple neural network with two input nodes, one '
'hidden layer and one output node',
'hidden layer with two hidden noeds and one output node',
2,
None,
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node'),
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node'),
('The ouput layer', 2, None, 'the-ouput-layer'),
('Compact expressions', 2, None, 'compact-expressions'),
('Output layer', 2, None, 'output-layer'),
@@ -253,10 +253,15 @@ div.toc p,a {
2,
None,
'using-the-chain-rule-and-summing-over-all-k-entries'),
('Setting up the back propagation algorithm',
('Setting up the back propagation algorithm and algorithm for a '
'feed forward NN, initalizations',
2,
None,
'setting-up-the-back-propagation-algorithm'),
'setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations'),
('Setting up the back propagation algorithm, part 1',
2,
None,
'setting-up-the-back-propagation-algorithm-part-1'),
('Setting up the back propagation algorithm, part 2',
2,
None,
@@ -266,17 +271,12 @@ div.toc p,a {
None,
'setting-up-the-back-propagation-algorithm-part-3'),
('Updating the gradients', 2, None, 'updating-the-gradients'),
('Activation functions', 3, None, 'activation-functions'),
('Activation functions', 2, None, 'activation-functions'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'activation-functions-logistic-and-hyperbolic-ones'),
('Relevance', 3, None, 'relevance'),
('Fine-tuning neural network hyperparameters',
2,
None,
'fine-tuning-neural-network-hyperparameters'),
('Hidden layers', 2, None, 'hidden-layers'),
('Relevance', 2, None, 'relevance'),
('Vanishing gradients', 2, None, 'vanishing-gradients'),
('Exploding gradients', 2, None, 'exploding-gradients'),
('Is the Logistic activation function (Sigmoid) our choice?',
@@ -305,6 +305,11 @@ div.toc p,a {
2,
None,
'more-on-activation-functions-output-layers'),
('Fine-tuning neural network hyperparameters',
2,
None,
'fine-tuning-neural-network-hyperparameters'),
('Hidden layers', 2, None, 'hidden-layers'),
('Batch Normalization', 2, None, 'batch-normalization'),
('Dropout', 2, None, 'dropout'),
('Gradient Clipping', 2, None, 'gradient-clipping'),
@@ -323,10 +328,6 @@ div.toc p,a {
('Limitations of NNs', 2, None, 'limitations-of-nns'),
('Homogeneous data', 2, None, 'homogeneous-data'),
('More limitations', 2, None, 'more-limitations'),
('Setting up the back-propagation algorithm',
2,
None,
'setting-up-the-back-propagation-algorithm'),
('Setting up a Multi-layer perceptron model for classification',
2,
None,
@@ -819,7 +820,7 @@ $$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node">Layout of a simple neural network with two input nodes, one hidden layer and one output node </h2>
<h2 id="layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-with-two-hidden-noeds-and-one-output-node">Layout of a simple neural network with two input nodes, one hidden layer with two hidden noeds and one output node </h2>
<br/><br/>
<center>
@@ -830,16 +831,16 @@ $$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-ouput-layer">The ouput layer </h2>
<p>Finally, we have the ouput layer given by layer label \( (2) \) with output \( a^{(2)} \) and weights and biases to be determined given by the variables</p>
<p>We have the ouput layer given by layer label \( (2) \) with output \( a^{(2)} \) and weights and biases to be determined given by the variables</p>
$$
w_{i}^{(2)}=\left\{w_{0}^{(2)},w_{1}^{(2)}\right\} \wedge b^{(2)}.
w_{i}^{(2)}=\left\{w_{1}^{(2)},w_{2}^{(2)}\right\} \wedge b^{(2)}.
$$
<p>Our output is \( \tilde{y}=a^{(2)} \) and we define a generic cost function \( C(a^{(2)},y;\boldsymbol{\Theta}) \) where \( y \) is the target value (a scalar here).
The parameters we need to optimize are given by
</p>
$$
\boldsymbol{\Theta}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\right\}.
\boldsymbol{\Theta}=\left\{w_{11}^{(1)},w_{12}^{(1)},w_{21}^{(1)},w_{22}^{(1)},w_{1}^{(2)},w_{2}^{(2)},b_1^{(1)},b_2^{(1)},b^{(2)}\right\}.
$$
@@ -850,12 +851,12 @@ $$
The inputs to the first hidden layer are
</p>
$$
\begin{bmatrix}z_0^{(1)} \\ z_1^{(1)} \end{bmatrix}=\left(\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\ w_{10}^{(1)} &w_{11}^{(1)} \end{bmatrix}\right)^{T}\begin{bmatrix}a_0^{(0)} \\ a_1^{(0)} \end{bmatrix}+\begin{bmatrix}b_0^{(1)} \\ b_1^{(1)} \end{bmatrix},
\begin{bmatrix}z_1^{(1)} \\ z_2^{(1)} \end{bmatrix}=\left(\begin{bmatrix}w_{11}^{(1)} & w_{12}^{(1)}\\ w_{21}^{(1)} &w_{22}^{(1)} \end{bmatrix}\right)^{T}\begin{bmatrix}a_1^{(0)} \\ a_2^{(0)} \end{bmatrix}+\begin{bmatrix}b_1^{(1)} \\ b_2^{(1)} \end{bmatrix},
$$
<p>with outputs</p>
$$
\begin{bmatrix}a_0^{(1)} \\ a_1^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_0^{(1)}) \\ \sigma^{(1)}(z_1^{(1)}) \end{bmatrix}.
\begin{bmatrix}a_1^{(1)} \\ a_2^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_1^{(1)}) \\ \sigma^{(1)}(z_2^{(1)}) \end{bmatrix}.
$$
@@ -864,7 +865,7 @@ $$
<p>For the final output layer we have the inputs to the final activation function </p>
$$
z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},
z^{(2)} = w_{1}^{(2)}a_1^{(1)} +w_{2}^{(2)}a_2^{(1)}+b^{(2)},
$$
<p>resulting in the output</p>
@@ -903,18 +904,18 @@ $$
<p>Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)</p>
$$
\frac{\partial C}{\partial w_{00}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}}= \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}},
\frac{\partial C}{\partial w_{11}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}= \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}},
$$
<p>which, noting that</p>
$$
z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},
z^{(2)} =w_1^{(2)}a_1^{(1)}+w_2^{(2)}a_2^{(1)}+b^{(2)},
$$
<p>allows us to rewrite </p>
$$
\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}}=w_0^{(2)}\frac{\partial a_0^{(1)}}{\partial z_0^{(1)}}a_0^{(1)}.
\frac{\partial z^{(2)}}{\partial z_1^{(1)}}\frac{\partial z_1^{(1)}}{\partial w_{11}^{(1)}}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}a_1^{(1)}.
$$
@@ -922,17 +923,17 @@ $$
<h2 id="final-expression">Final expression </h2>
<p>Defining</p>
$$
\delta_0^{(1)}=w_0^{(2)}\frac{\partial a_0^{(1)}}{\partial z_0^{(1)}}\delta^{(2)},
\delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)},
$$
<p>we have </p>
$$
\frac{\partial C}{\partial w_{00}^{(1)}}=\delta_0^{(1)}a_0^{(1)}.
\frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)}.
$$
<p>Similarly, we obtain</p>
$$
\frac{\partial C}{\partial w_{01}^{(1)}}=\delta_0^{(1)}a_1^{(1)}.
\frac{\partial C}{\partial w_{12}^{(1)}}=\delta_1^{(1)}a_2^{(1)}.
$$
@@ -941,17 +942,17 @@ $$
<p>Similarly, we find</p>
$$
\frac{\partial C}{\partial w_{10}^{(1)}}=\delta_1^{(1)}a_0^{(1)},
\frac{\partial C}{\partial w_{21}^{(1)}}=\delta_2^{(1)}a_1^{(1)},
$$
<p>and </p>
$$
\frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)},
\frac{\partial C}{\partial w_{22}^{(1)}}=\delta_2^{(1)}a_2^{(1)},
$$
<p>where we have defined </p>
$$
\delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)}.
\delta_2^{(1)}=w_2^{(2)}\frac{\partial a_2^{(1)}}{\partial z_2^{(1)}}\delta^{(2)}.
$$
@@ -960,12 +961,12 @@ $$
<p>For the sake of completeness, we list the derivatives of the biases, which are</p>
$$
\frac{\partial C}{\partial b_{0}^{(1)}}=\delta_0^{(1)},
\frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)},
$$
<p>and</p>
$$
\frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)}.
\frac{\partial C}{\partial b_{2}^{(1)}}=\delta_2^{(1)}.
$$
<p>As we will see below, these expressions can be generalized in a more compact form. </p>
@@ -1250,9 +1251,27 @@ $$
<p>We are now ready to set up the algorithm for back propagation and learning the weights and biases.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the back propagation algorithm </h2>
<h2 id="setting-up-the-back-propagation-algorithm-and-algorithm-for-a-feed-forward-nn-initalizations">Setting up the back propagation algorithm and algorithm for a feed forward NN, initalizations </h2>
<p>The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.</p>
<div class="alert alert-block alert-block alert-text-normal">
<b>The architecture (our model)</b>
<p>
<ol>
<li> Set up your inputs and outputs (scalars, vectors, matrices or higher-order arrays)</li>
<li> Define the number of hidden layers and hidden nodes</li>
<li> Define activation functions for hidden layers and output layers</li>
<li> Define optimizer (plan learning rate, momentum, ADAgrad, RMSprop, ADAM etc) and array of initial learning rates</li>
<li> Define cost function and possible regularization terms with hyperparameters</li>
<li> Initialize weights and biases</li>
<li> Fix number of iterations for the feed forward part and back propagation part</li>
</ol>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-back-propagation-algorithm-part-1">Setting up the back propagation algorithm, part 1 </h2>
<p>The four equations provide us with a way of computing the gradients of the cost function. Let us write this out in the form of an algorithm.</p>
<p><b>First</b>, we set up the input data \( \boldsymbol{x} \) and the activations
\( \boldsymbol{z}_1 \) of the input layer and compute the activation function and
@@ -1320,12 +1339,12 @@ $$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="activation-functions">Activation functions </h3>
<h2 id="activation-functions">Activation functions </h2>
<p>A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
connectivity, is the choice of activation function(s). The following
restrictions are imposed on an activation function for an FFNN to
fulfill the universal approximation theorem
</p>
<ul>
@@ -1359,7 +1378,7 @@ $$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="relevance">Relevance </h3>
<h2 id="relevance">Relevance </h2>
<p>The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
@@ -1462,52 +1481,6 @@ plt<span style="color: #666666">.</span>show()
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="fine-tuning-neural-network-hyperparameters">Fine-tuning neural network hyperparameters </h2>
<p>The flexibility of neural networks is also one of their main
drawbacks: there are many hyperparameters to tweak. Not only can you
use any imaginable network topology (how neurons/nodes are
interconnected), but even in a simple FFNN you can change the number
of layers, the number of neurons per layer, the type of activation
function to use in each layer, the weight initialization logic, the
stochastic gradient optmized and much more. How do you know what
combination of hyperparameters is the best for your task?
</p>
<ul>
<li> You can use grid search with cross-validation to find the right hyperparameters.</li>
</ul>
<p>However,since there are many hyperparameters to tune, and since
training a neural network on a large dataset takes a lot of time, you
will only be able to explore a tiny part of the hyperparameter space.
</p>
<ul>
<li> You can use randomized search.</li>
<li> Or use tools like <a href="http://oscar.calldesk.ai/" target="_blank">Oscar</a>, which implements more complex algorithms to help you find a good set of hyperparameters quickly.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="hidden-layers">Hidden layers </h2>
<p>For many problems you can start with just one or two hidden layers and
it will work just fine. For the MNIST data set you ca easily get a
high accuracy using just one hidden layer with a few hundred neurons.
You can reach for this data set above 98% accuracy using two hidden
layers with the same total amount of neurons, in roughly the same
amount of training time.
</p>
<p>For more complex problems, you can gradually ramp up the number of
hidden layers, until you start overfitting the training set. Very
complex tasks, such as large image classification or speech
recognition, typically require networks with dozens of layers and they
need a huge amount of training data. However, you will rarely have to
train such networks from scratch: it is much more common to reuse
parts of a pretrained state-of-the-art network that performs a similar
task.
</p>
<!-- !split -->
<h2 id="vanishing-gradients">Vanishing gradients </h2>
@@ -1668,6 +1641,52 @@ gradient descent optimization does in general not get stuck.
<li> For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).</li>
<li> For regression tasks, you can simply use no activation function at all.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="fine-tuning-neural-network-hyperparameters">Fine-tuning neural network hyperparameters </h2>
<p>The flexibility of neural networks is also one of their main
drawbacks: there are many hyperparameters to tweak. Not only can you
use any imaginable network topology (how neurons/nodes are
interconnected), but even in a simple FFNN you can change the number
of layers, the number of neurons per layer, the type of activation
function to use in each layer, the weight initialization logic, the
stochastic gradient optmized and much more. How do you know what
combination of hyperparameters is the best for your task?
</p>
<ul>
<li> You can use grid search with cross-validation to find the right hyperparameters.</li>
</ul>
<p>However,since there are many hyperparameters to tune, and since
training a neural network on a large dataset takes a lot of time, you
will only be able to explore a tiny part of the hyperparameter space.
</p>
<ul>
<li> You can use randomized search.</li>
<li> Or use tools like <a href="http://oscar.calldesk.ai/" target="_blank">Oscar</a>, which implements more complex algorithms to help you find a good set of hyperparameters quickly.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="hidden-layers">Hidden layers </h2>
<p>For many problems you can start with just one or two hidden layers and
it will work just fine. For the MNIST data set discussed below you can easily get a
high accuracy using just one hidden layer with a few hundred neurons.
You can reach for this data set above 98% accuracy using two hidden
layers with the same total amount of neurons, in roughly the same
amount of training time.
</p>
<p>For more complex problems, you can gradually ramp up the number of
hidden layers, until you start overfitting the training set. Very
complex tasks, such as large image classification or speech
recognition, typically require networks with dozens of layers and they
need a huge amount of training data. However, you will rarely have to
train such networks from scratch: it is much more common to reuse
parts of a pretrained state-of-the-art network that performs a similar
task.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="batch-normalization">Batch Normalization </h2>
@@ -1797,71 +1816,6 @@ features).
</ul>
<p>Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the back-propagation algorithm </h2>
<p>Let us write this out in the form of an algorithm.</p>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>First, we set up the input data \( \boldsymbol{x} \) and the activations
\( \boldsymbol{z}_1 \) of the input layer and compute the activation function and
the pertinent outputs \( \boldsymbol{a}^1 \).
</p>
</div>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>Secondly, we perform then the feed forward till we reach the output
layer and compute all \( \boldsymbol{z}_l \) of the input layer and compute the
activation function and the pertinent outputs \( \boldsymbol{a}^l \) for
\( l=2,3,\dots,L \).
</p>
</div>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>Thereafter we compute the ouput error \( \boldsymbol{\delta}^L \) by computing all</p>
$$
\delta_j^L = f'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}.
$$
</div>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>Then we compute the back propagate error for each \( l=L-1,L-2,\dots,2 \) as</p>
$$
\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l).
$$
</div>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>Finally, we update the weights and the biases using gradient descent for each \( l=L-1,L-2,\dots,2 \) and update the weights and biases according to the rules</p>
$$
w_{ij}^l\leftarrow = w_{ij}^l- \eta \delta_j^la_i^{l-1},
$$
$$
b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l,
$$
</div>
<p>The parameter \( \eta \) is the learning parameter discussed in connection with the gradient descent methods.
Here it is convenient to use stochastic gradient descent (see the examples below) with mini-batches with an outer loop that steps through multiple epochs of training.
</p>
<!-- !split -->
<h2 id="setting-up-a-multi-layer-perceptron-model-for-classification">Setting up a Multi-layer perceptron model for classification </h2>