lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first neural net attempt for slides

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2017-10-25 08:11:03 +02:00
parent 2e5b3ce390
commit ce43780dba
2 changed files with 1940 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/NeuralNet/NeuralNet.do.txt
+896 -4
View File
@@ -3,9 +3,901 @@ AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of
DATE: today
!split
===== Introduction =====
!bblock
!eblock
Machine learning is the science of giving computers the ability to learn without being explicitly programmed.
The idea is that there exist generic algorithms which can be used to find patterns in a broad class of data sets without
having to write code specifically for each problem. The algorithm will build its own logic based on the data.
Machine learning is a subfield of computer science, and is closely related to computational statistics.
It evolved from the study of pattern recognition in artificial intelligence (AI) research, and has made contributions to
AI tasks like computer vision, natural language processing
and speech recognition. It has also, especially in later years,
found applications in a wide variety of other areas, including bioinformatics, economy, physics, finance and marketing.
The approaches to machine learning are many, but are often split into two main categories.
In *supervised learning* we know the answer to a problem,
and let the computer deduce the logic behind it. On the other hand, *unsupervised learning*
is a method for finding patterns and relationship in data sets without any prior knowledge of the system.
Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm
of learning inspired by behavioural psychology, where learning is achieved by trial-and-error,
solely from rewards and punishment.
Another way to categorize machine learning tasks is to consider the desired output of a system.
Some of the most common tasks are:
* Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning.
* Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values.
* Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning.
======= Artificial neurons =======
The field of artificial neural networks has a long history of development, and is closely connected with
the advancement of computer science and computers in general. A model of artificial neurons
was first developed by McCulloch and Pitts in 1943 to study signal processing in the brain and
has later been refined by others. The general idea is to mimic neural networks in the human brain, which
is composed of billions of neurons that communicate with each other by sending electrical signals.
Each neuron accumulates its incoming signals,
which must exceed an activation threshold to yield an output. If the threshold is not overcome, the neuron
remains inactive, i.e. has zero output.
This behaviour has inspired a simple mathematical model for an artificial neuron.
!bt
\begin{equation}
y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)
label{artificialNeuron}
\end{equation}
!et
Here, the output $y$ of the neuron is the value of its activation function, which have as input
a weighted sum of signals $x_i, \dots ,x_n$ received by $n$ other neurons.
======= Neural network types =======
An artificial neural network (NN), is a computational model that consists of layers of connected neurons, or *nodes*.
It is supposed to mimic a biological nervous system by letting each neuron interact with other neurons
by sending signals in the form of mathematical functions between layers.
A wide variety of different NNs have
been developed, but most of them consist of an input layer, an output layer and eventual layers in-between, called
*hidden layers*. All layers can contain an arbitrary number of nodes, and each connection between two nodes
is associated with a weight variable.
The main factor that separates the different types are how the neurons are *connected*.
This section contains a short presentation of some of the most common types of ANNs, before we move on to
a more detailed description of the NN architecture used in this thesis.
===== Feed-forward neural networks =====
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
the information moves in only one direction: forward through the layers.
Nodes are represented by circles, while the arrows display the connections between the nodes, including the
direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here.
We observe that each node in a layer is connected to *all* nodes in the subsequent layer,
making this a so-called *fully-connected* FFNN.
A different variant of FFNNs are *convolutional neural networks* (CNNs), which have a connectivity pattern
inspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli from
small sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strong
spatially local correlation present in natural images. The response of each neuron can be approximated mathematically
as a convolution operation.
CNNs emulate the behaviour of neurons in the visual cortex by enforcing a *local* connectivity pattern
between nodes of adjacent layers: Each node
in a convolutional layer is connected only to a subset of the nodes in the previous layer,
in contrast to the fully-connected FFNN.
Often, CNNs
consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end,
which gathers all the local data and produces the outputs. They have wide applications in image and video recognition
===== Recurrent neural networks =====
So far we have only mentioned NNs where information flows in one direction: forward. *Recurrent neural networks* on
the other hand, have connections between nodes that form directed *cycles*. This creates a form of
internal memory which are able to capture information on what has been calculated before; the output is dependent
on the previous computations. Recurrent NNs make use of sequential information by performing the same task for
every element in a sequence, where each element depends on previous elements. An example of such information is
sentences, making recurrent NNs especially well-suited for handwriting and speech recognition.
===== Other types of networks =====
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
in multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers:
an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' here
means that each node in the output layer has a linear activation function). The layers are normally fully-connected and
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description
of how a fully-connected FFNN works, and how it can be used to interpolate data sets.
======= Multilayer perceptro =======
In this thesis we use fully-connected feed-forward neural networks with three
or more layers (an input layer, one or more hidden layers and an output layer), mainly
consisting of neurons that have non-linear activation functions.
Such networks are often called *multilayer perceptrons* (MLPs)
===== Why multilayer perceptrons =====
We have chosen to use MLPs to interpolate data sets for the construction of interatomic potentials.
Other NN types could also have been used, but our choice is well justified. According to the
*Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a ''non-constant, bounded and monotonically-increasing continuous function''.
Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always
assumed to be linear, so as to not restrict the range of output values.
The only requirement on the NN information flow is that it must be feed-forward, which is not satisfied by recurrent NNs.
Anyhow, computing energies and forces in MD simulations at a given time step do not require any knowledge of
the earlier states of the system, i.e.\ no memory or recurrency is involved.
Further, it is not obvious how local connectivity should be helpful in a MD context, thus
we might as well use the simplest multilayer FFNN available, the MLP.
We note that this theorem is only applicable to a NN with *one* hidden layer.
Therefore, we can easily construct a NN
that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer
with activation functions that *do*. Furthermore, although the universal approximation theorem
lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice:
A NN can still be able to approximate a given function reasonably well without having the flexibility to fit *all other*
functions.
===== Mathematical model =====
!bt
\begin{equation}
y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(u)
label{artificialNeuron2}
\end{equation}
!et
In a FFNN of such neurons, the *inputs* $x_i$
are the *outputs* of the neurons in the preceding layer. Furthermore, a MLP is fully-connected,
which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer.
First, for each node $i$ in the first hidden layer, we calculate a weighted sum $u_i^1$ of the input coordinates $x_j$,
!bt
\begin{equation}
u_i^1 = \sum_{j=1}^2 w_{ij}^1 x_j + b_i^1
\end{equation}
!et
This value is the argument to the activation function $f_1$ of each neuron $i$,
producing the output $y_i^1$ of all neurons in layer 1,
!bt
\begin{equation}
y_i^1 = f_1(u_i^1) = f_1\left(\sum_{j=1}^2 w_{ij}^1 x_j + b_i^1\right)
label{outputLayer1}
\end{equation}
!et
where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$
!bt
\begin{equation}
y_i^l = f_l(u_i^l) = f_l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
label{generalLayer}
\end{equation}
!et
where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed,
the values of the subsequent layer can be calculated and so forth until the output is obtained.
The output of neuron $i$ in layer 2 is thus,
!bt
\begin{align}
y_i^2 &= f_2\left(\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\right) \\
&= f_2\left[\sum_{j=1}^3 w_{ij}^2f_1\left(\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
label{outputLayer2}
\end{align}
!et
where we have substituted $y_m^1$ with. Finally, the NN output yields,
!bt
\begin{align}
y_1^3 &= f_3\left(\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\right) \\
&= f_3\left[\sum_{j=1}^3 w_{1j}^3 f_2\left(\sum_{k=1}^3 w_{jk}^2 f_1\left(\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
+ b_1^3\right]
\end{align}
!et
We can generalize this expression to a MLP with $l$ hidden layers. The complete functional form
is,
\begin{flalign}
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
\dots \!f_1\!\left(\!\sum_{n=1}^{N_0} \!w_{mn}^1 x_n\! + \!b_m^1\!\right)
\!\dots \!\right) \!+ \!b_k^2\!\right)
\!+ \!b_1^3\!\right] &&
label{completeNN}
\end{flalign}
which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$.
This confirms that a MLP,
despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m$.
In our example, $n=2$ and $m=1$. Consequentially,
the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP.
Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that
the expression is essentially a nested sum of scaled activation functions of the form
!bt
\begin{equation}
h(x) = c_1 f(c_2 x + c_3) + c_4
\end{equation}
!et
where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions
can be shifted up and down or left and right, change slope or be rescaled
which is the key to the flexibility of a NN.
!bt
\begin{equation}
f_o = f(u_o) = u_o
label{outputActivation}
\end{equation}
!et
=== Matrix-vector notation ===
We can introduce a more convenient notation for the activations in a NN.
Additionally, we can represent the biases and activations
as layer-wise column vectors $\vec{b}_l$ and $\vec{y}_l$, so that the $i$-th element of each vector
is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively.
We have that $\mathrm{W}_l$ is a $N_{l-1} \times N_l$ matrix, while $\vec{b}_l$ and $\vec{y}_l$ are $N_l \times 1$ column vectors.
With this notation, the sum in becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 in
!bt
\begin{equation}
\vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) =
f_2\left(\left[\begin{array}{ccc}
w^2_{11} &w^2_{12} &w^2_{13} \\
w^2_{21} &w^2_{22} &w^2_{23} \\
w^2_{31} &w^2_{32} &w^2_{33} \\
\end{array} \right] \cdot
\left[\begin{array}{c}
y^1_1 \\
y^1_2 \\
y^1_3 \\
\end{array}\right] +
\left[\begin{array}{c}
b^2_1 \\
b^2_2 \\
b^2_3 \\
\end{array}\right]\right)
\end{equation}
!et
and we see that the activation of node $i$ in layer 2 is
!bt
\begin{equation}
y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right)
\end{equation}
!et
which is in accordance with. Note that
This is not just a convenient and compact notation, but also
a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications
and vector additions that are used as input to the activation functions. For each operation
$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer.
======= Activation functions =======
A property that characterizes a NN, other than its connectivity, is the choice of activation function(s).
As described in, the following restrictions are imposed on an activation function for a FFNN
to fulfill the universal approximation theorem:
* Non-constant
* Bounded
* Monotonically-increasing
* Continous
We realize that the second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each
layer simply performs a linear transformation of its inputs. Consequentially, regardless of the number of layers,
the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of
non-linearity to the NN to be able to fit non-linear functions.
!bt
\begin{equation}
f(x) = \frac{1}{1 + e^{-x}}
label{sigmoidActivationFunction}
\end{equation}
!et
and the hyperbolic tangent
!bt
\begin{equation}
f(x) = \tanh(x)
label{tanhActivationFunction}
\end{equation}
!et
The logistic functon is often referred to as ''the sigmoid'', which is the naming convention we will use
in the following. Both functions are depicted in
are bounded above by 1, while the lower bound is -1 and 0 for the hyperbolic tangent and sigmoid respectively. This property
keeps the activations from diverging.
The sigmoid are more biologically plausible because
the output of inactive neurons are zero. Such activation function are called *one-sided*. However,
it has been shown that the hyperbolic tangent
performs better than the sigmoid for training MLPs. This will be further discussed in
Nevertheless, one should assess the problem
at hand when deciding what activation function to use; the performance can vary from problem to problem.
!bt
\begin{equation}
f(x) = \max(0,x)
label{reluActivationFunction}
\end{equation}
!et
has become the most popular for *deep neural networks*
======= Training =======
In the beginning of this chapter we defined machine learning as the ''science of giving computers the ability to learn without
being explicitly programmed.'' In the neural network case, this learning is achieved by iteratively feeding the network with data.
With the help of certain learning algorithms, the network will then automatically adjust its parameters, i.e.\ the weights and biases,
Thus, the input values to the function that we want to fit are atomic configurations,
while the output values are the total energies of these configurations.
Together they form what we refer to as a *data set* or a *reference set*.
The output values are also known as the *target values*.
The input data $\mathrm{X}$ and output data $\mathrm{Y}$ are represented as matrices, where the $i$-th row of X and Y,
denoted $\mathrm{X}_{i*}$ and $\mathrm{Y}_{i*}$ respectively,
together form a *training example*. The combined data set consists of $N$ such training examples.
Note that the number of columns of X and Y are equal to the number of inputs and outputs of our MLP respectively.
This is a restatement of the observation made in The number of inputs and outputs of our MLP must be equal
to the number of input and output values of the function we are trying to fit. In the following we assume that the MLP only has one
output, thus each row of Y is a single number, denoted $Y_{i*} = Y_i$.
To minimize the error, we need a way to *define* it.
As in mathematical optimization, the error is represented by an objective function, also called *loss* function
or *cost* function. Training a NN therefore amounts to the minimization of this function, which can be written in the
following general way,
!bt
\begin{equation}
\Gamma = \Gamma\bigr(\{\mathrm{W}_l\}, \{\vec{b}_l\}, \mathrm{X}, \mathrm{Y}\bigr)
label{generalCost}
\end{equation}
!et
where $\{\mathrm{W}_l\}$ and $\{\vec{b}_l\}$ are all the weights and biases of the NN respectively.
The value of this function is a measure of how well the NN is able to map $\mathrm{X} \rightarrow \mathrm{Y}$.
By adjusting $\{\mathrm{W}_l\}$ and $\{\vec{b}_l\}$, we try to minimize the value of this function.
The standard cost function used in regression with NNs is the mean-square error,
!bt
\begin{equation}
\Gamma = \frac{1}{2N}\sum_{i=1}^N (Y_i - y_i)^2
label{quadraticCost}
\end{equation}
!et
where $y_i = y_i(W, B, X_{i*})$ is the value predicted by the NN for training example $X_{i*}$.
For a MLP with more than one output, $Y_i$ and $y_i$ are vectors.
The constant $1/2$ is included to cancel out the exponent when this function is differentiated at a later stage
There are a large number of algorithms that can be used to determine the set of weights minimizing the
cost function
Different kinds of gradient descent-based methods are widely used, while higher order methods like
the conjugate gradient algorithm or Newton's method are rarely seen in the literature because they are
too computationally expensive for large NNs and data sets. Therefore, only first-order methods will be discussed here.
The idea behind gradient descent methods is to minimize a function by
iteratively taking steps in the direction of steepest descent towards a minimum in parameter space.
This direction is defined as the negative gradient of the function with respect to
all its parameters. If we define $\vec{\theta} \in \mathbb{R}^d$ as a vector containing all the weights and biases
of a MLP, we get the following iterative scheme,
!bt
\begin{equation}
\vec{\uptheta}_{k+1} = \vec{\uptheta}_{k} - \gamma \nabla_{\vec{\uptheta}_k} \Gamma(\vec{\uptheta})
label{gradientDescent}
\end{equation}
!et
where $\gamma$ is a step size, called the *learning rate* in the context of machine learning. The process is started
by initializing the parameters as random numbers. The value of $\gamma$ is of great importance for the algorithm to converge,
and is allowed to change at every iteration. Note that convergence to
a minimum is not guaranteed without certain assumptions on the function $\Gamma$ and the particular choices of $\gamma$.
Also, the obtained minimum is generally local, not global.
Different ways to update $\gamma$ is discussed below.
===== Gradient descent variants =====
There are three versions of gradient descent, which differ in
the number of training examples we present to the NN before updating the parameters.
According to neural network terminology, the process of adjusting the parameters based on all the training examples,
either in batches or all at once, is called an *epoch*.
In *batch* gradient descent,
we compute the gradient of the cost function for the *whole* data set before updating, also called
*offline learning*.
This approach can be very slow and is intractable for datasets that do not fit in memory. Furthermore, there is a risk
of performing redundant computations if many similar examples are present in the data set.
In contrast, *stochastic* gradient descent (SDG) performs a parameter update for *each* training example.
SDG avoids redundant gradient calculations and is therefore faster than batch gradient descent. However, the accuracy
of each update is lower compared to offline learning, which can lead to quite large error oscillations. The concept of having
more than one parameter update per epoch is called *online learning*.
Finally, we have *mini-batch* gradient descent, which is a mix of the two other approaches. The reference set
is divided into $n$ equally sized mini-batches and a parameter update is performed for each mini-batch. This is usually the
algorithm of choice, as it has the optimal trade-off between speed and accuracy. Normal mini-batch sizes range
between 50 and 256, but should in some degree be tailored to each problem.
We realize that SDG can be seen as
a variant of mini-batch gradient descent with a mini-batch size of 1. In the literature, these two methods
are often collectively referred to as SDG.
===== Optimization algorithms =====
In the following we will outline different optimization algorithms that are widely used in neural network research.
They are all variations on the update rule. The main focus of the methods is to find
a proper learning rate. A learning rate that is too small may lead to very slow convergence, while
a learning rate that is too large can cause the loss function to fluctuate around the minimum or even diverge.
=== Momentum ===
Near local minima, the surface area of the cost function (in parameter space) often curve much more steeply in one
direction than in another, forming ravines. SDG will have slow converge in such regions, as it will oscillate across the slopes
while having slow progress along the bottom of the ravine. The Momentum method avoids this problem
by accelerating SDG in the downwards direction, while damping the oscillations. The update rule is
!bt
\begin{equation}
\begin{aligned}
\vec{v}_{k} &= \eta \vec{v}_{k-1} + \gamma \nabla_k \Gamma \\
\vec{\uptheta}_{k+1} &= \vec{\uptheta}_k - \vec{v}_k
\end{aligned}
label{Momentum}
\end{equation}
!et
where $\eta$ is the momentum term, which is usually set to 0.9 or a similar value. The new update vector $\vec{v}_k$
is formed by adding a fraction $\eta$ of the previous update vector $\vec{v}_{k-1}$ to the gradient. In this way, the magnitude
of the update decreases for parameters (dimensions) whose gradients change direction, while it increases for parameters
whose gradients points in the same direction as in the previous step. This helps to push SDG downhill towards the minima.
=== Adagrad ===
In SDG and Momentum, we have one learning rate $\gamma$ that is applied to all the parameters.
Ideally, the learning rate should be adapted to each individual parameter to perform larger or smaller updates
depending on their importance. Adagrad does just that. For brevity, we set
!bt
\begin{equation}
g_{k,i} = \nabla_{\theta_i} \Gamma (\vec{\uptheta})
\end{equation}
!et
to be the gradient of the loss function w.r.t. parameter $\theta_i$ at step $k$. Adagrad adjusts the general learning rate $\gamma$
at each step $k$ for every parameter $\theta_i$ based on the past gradients for that parameter,
!bt
\begin{equation}
\theta_{k+1,i} = \theta_{k,i} - \frac{\gamma}{\sqrt{G_{k,ii} + \epsilon}} \cdot g_{k,i}
\end{equation}
!et
where $\mathrm{G}_k \in \mathbb{R}^{d \times d}$ is a diagonal matrix where each diagonal element $G_{k,ii}$ is the sum of squares
of the gradient with respect to $\theta_i$ up to step $k$. The smoothing constant $\epsilon \sim 10^{-8}$ is present to avoid
division by zero. The vectorized version of this equation yields,
!bt
\begin{equation}
\vec{\uptheta}_{k+1} = \vec{\uptheta}_k - \frac{\gamma}{\sqrt{\mathrm{diag}(\mathrm{G}_k) + \epsilon}} \odot \vec{g}_k
label{Adagrad}
\end{equation}
!et
where $\odot$ stands for element-wise multiplication. In other words, we do not have to manually tune the learning rate,
Adagrad does that for us. A weakness of this method is that the learning rates inevitably shrinks for each step,
resulting in arbitrary small values, at which point the learning stops.
we assume for brevity that all vector operations are element-wise, i.e.\
$\vec{g}\,\vec{h} = \vec{g} \odot \vec{h}$ and $\vec{g}^2 = \vec{g} \odot \vec{g}$.
=== Adadelta ===
Adadelta is an extension of Adagrad that reduces the rate at which the learning rate decreases.
The sum of all past squared gradients is replaced by a *exponentially decaying average* of all the previous squared gradients.
We introduce the vector $\vec{E}[\vec{g}^2]_k$ containing the decaying averages of the gradient
with respect to all the parameters at step $k$.
This vector is defined recursively
as a weighted average of the previous averages $\vec{E}[\vec{g}^2]_{k-1}$ and the current gradient $\vec{g}_k$,
!bt
\begin{equation}
\vec{E}[\vec{g}^2]_k = \eta \vec{E}[\vec{g}^2]_{k-1} + (1 - \eta) \vec{g}^2_k
label{decayingAverageVector}
\end{equation}
!et
where $\eta$ is a decay constant similar to that in the Momentum method, and is usually set to the same value (0.9).
We also define a new parameter update vector $\Delta \vec{\uptheta}_k = -\vec{v}_k$ so that
!bt
\begin{equation}
\vec{\uptheta}_{k+1} = \vec{\uptheta}_k + \Delta \vec{\uptheta}_k
\end{equation}
!et
Replacing the vector $\mathrm{diag}(\mathrm{G}_k)$ in with the decaying average vector
yields the following parameter update vector,
!bt
\begin{equation}
\Delta \vec{\uptheta}_k = -\frac{\gamma}{\sqrt{\vec{E}[\vec{g}^2]_k + \epsilon}} \vec{g}_k
label{preliminiaryAdadelta}
\end{equation}
!et
Additionally, the learning rate $\gamma$ is replaced by a decaying average of previous squared parameter updates
$\vec{E}[\Delta\vec{\uptheta}^2]$ up to step $k-1$, yielding the Adadelta update rule,
!bt
\begin{equation}
\vec{v}_k = \frac{\sqrt{\vec{E}[\Delta\vec{\uptheta}^2]_{k-1} + \epsilon}}{\sqrt{\vec{E}[\vec{g}^2]_k + \epsilon}} \vec{g}_k
\end{equation}
!et
The final replacement is done to obtain correct units for the update vector and to eliminate the learning rate from the equation.
Thus, we do not even need to set a default learning rate with Adadelta.
=== Adam ===
The final algorithm we are going to discuss is Adapte Moment Estimation (Adam). This method
computes adaptive learning rates for each parameter by storing
exponentially decaying averages of both the gradients *and*
the squared gradients
!bt
\begin{equation}
\begin{aligned}
\vec{E}[\vec{g}]_k &= \beta_1 \vec{E}[\vec{g}]_{k-1} + (1 - \beta_1) \vec{g}_k \\
\vec{E}[\vec{g}^2]_k &= \beta_2 \vec{E}[\vec{g}^2]_{k-1} + (1 - \beta_2) \vec{g}^2_k
\end{aligned}
\end{equation}
!et
We set $\vec{a}_k = \vec{E}[\vec{g}]_k$ and $\vec{b}_k = \vec{E}[\vec{g}^2]_k$ so that
!bt
\begin{equation}
\begin{aligned}
\vec{a}_k &= \beta_1 \vec{a}_{k-1} + (1 - \beta_1) \vec{g}_k \\
\vec{b}_k &= \beta_2 \vec{b}_{k-1} + (1 - \beta_2) \vec{g}^2_k
\end{aligned}
\end{equation}
!et
where $\vec{a}_k$ and $\vec{b}_k$ are estimates of the first moment (the mean) and the second moment (the uncentered variance)
of the gradients respectively, hence the name of the method. The elements of these two vectors are initialized as zeros,
which make them biased towards zero, especially during the first time steps. This bias is counteracted by
computing bias-corrected version of the vectors,
!bt
\begin{equation}
\begin{aligned}
\hat{\vec{a}}_k &= \frac{\vec{a}_k}{1 - \beta_1^k} \\
\hat{\vec{b}}_k &= \frac{\vec{b}_k}{1 - \beta_2^k}
\end{aligned}
\end{equation}
!et
The parameter update vector for Adam is obtained by replacing the gradient $\vec{g}_k$ with $\vec{a}_k$ in
yielding the update rule,
!bt
\begin{equation}
\vec{\uptheta}_{k+1} = \vec{\uptheta}_k - \frac{\gamma}{\sqrt{\hat{\vec{b}}_k} + \epsilon} \hat{\vec{a}}_k
label{adamUpdateRule}
\end{equation}
!et
The default values of the hyperparameters are $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$.
=== Which optimizer to use? ===
There is no general answer to the question stated above. The adaptive learning rate methods
like Adagrad, Adadelta and Adam are generally more robust because they do not rely on manual fine-tuning of the learning rate.
Also, an adaptive learning rate generally handles *sparse* data better, i.e.\ data where some learning features
are poorly represented.
Adam has been proven to perform favourably over the other optimization methods for certain applications like classification
with MLPs and convolutional NNs. However, the performance of various optimizers should be tested on
the problem at hand to determine which method that works best.
===== Backpropagation =====
In the preceding sections we have discussed different optimizers that implements various update rules for
the parameters of a loss function. In our case, these parameters are the weights and biases of a neural network
and the cost function $\Gamma$ is the mean-square-error
!bt
\begin{equation}
\Gamma = \frac{1}{2} (Y_i - y_i)^2
\end{equation}
!et
here written as a function of only *one* training example $i$.
What we have not discussed yet, is
how we calculate the gradient of the cost function, i.e. how we obtain the partial derivatives $g_{ij}$ and $h_i$
with respect to all the weights and biases respectively,
!bt
\begin{align}
g_{ij} &= \frac{\partial \Gamma}{\partial w_{ij}} label{weightDerivative} \\
h_i &= \frac{\partial \Gamma}{\partial b_i} label{biasDerivative}
\end{align}
!et
where we have dropped layer indicies for clarity.
A common method to obtain these derivatives is *backpropagation*.
In backpropagation, a training example is propagated forward through the NN to produce an output.
This output is compared to the desired output (target values), and the error is then propagated *backwards*
through the layers to obtain the amount of which each parameter should be adjusted,
hence the name.
The method is essentially an implementation of the chain rule, and will allow us to calculate
the partial derivatives of the cost with respect to all the parameters, thereby obtaining the gradient of the network.
=== Forward propagation ===
!bt
\begin{equation}
A_i = \{j:w_{ij}\}
\end{equation}
!et
as the set $\{j\}$ of nodes anterior to node $i$ and connected to node $i$ with weights $w_{ij}$, in addition to
!bt
\begin{equation}
P_j = \{i:w_{ij}\}
\end{equation}
!et
as the set $\{i\}$ of nodes posterior to node $j$ and connected to node $j$ with weights $w_{ij}$.
=== Backward propagation ===
The second step is to backpropagate the error of the NN output. The weight derivative
can be expanded into two factors by use of the chain rule,
!bt
\begin{equation}
g_{ij} = \frac{\partial \Gamma}{\partial u_i}\frac{\partial u_i}{\partial w_{ij}}
label{weightDerivativeExpanded}
\end{equation}
!et
Now we move in the opposite direction of the feed-forward stage: First we differentiate the cost w.r.t
the input of neuron $i$, then we differentiate the input w.r.t. weight $w_{ij}$ connecting neurons $j$ (in the preceding layer)
and $i$.
The first term on the r.h.s. is defined as the the error $\delta_i$ of node $i$,
!bt
\begin{equation}
\delta_i = \frac{\partial\Gamma}{\partial u_i}
label{neuronError}
\end{equation}
!et
This definition can be justified by the following argument.
To change the value of a cost function, we need to change the outputs of the neurons in the network.
Changing the input $u_j$ to neuron $j$ by a small amount $\Delta u_j$ results in the output
!bt
\begin{equation}
y_j = f_j(u_j + \Delta u_j)
\end{equation}
!et
This change will propagate through subsequent layers in the network, finally causing the overall cost to change
by an amount $\frac{\partial \Gamma}{\partial u_j}\Delta u_j$. If $\partial \Gamma / \partial u_j$ is close to zero,
we are not able improve the cost much by perturbing the weighted input $u_j$; the neuron is already quite near the optimal value.
This is a heuristic argument for $\partial \Gamma / \partial u_j$ to be a measure of the error of the neuron.
!bt
\begin{equation}
\frac{\partial u_i}{\partial w_{ij}} = \frac{\partial}{\partial w_{ij}} \sum_{m\in A_i} w_{im}y_m = y_j
label{derivativeSecondTerm}
\end{equation}
!et
Combining the the two terms
!bt
\begin{equation}
g_{ij} = \delta_i y_j
label{weightGradient}
\end{equation}
!et
To compute this quantity, we thus need to know the outputs and the errors of all nodes in the network.
The outputs are generated during the feed-forward stage
!bt
\begin{equation}
y_i = f_i(u_i) = f_i\left(\sum_{j\in A_i} w_{ij}y_j + b_i\right)
label{forwardProp}
\end{equation}
!et
and need to be stored for all nodes.
The errors are obtained by backpropagating the error of the output neuron $o$,
!bt
\begin{equation}
\delta_o = \frac{\partial \Gamma}{\partial u_o} = \frac{\partial \Gamma}{\partial y_o}\frac{\partial y_o}{\partial u_o}
= (Y - y_o) \frac{\partial y_o}{\partial u_o}
\end{equation}
!et
We remember that the output neuron has the identity activation function $y_o = f_o(u_o) = u_o$,
which reduces the error to
!bt
\begin{equation}
\delta_o = Y - y_o
\end{equation}
!et
This error is then propagated backwards through the network, layer by layer.
The error of each neuron in layer $l$ thus depends on the errors of all neurons in layer $l+1$.
Consequentially, the error of an arbitrary hidden neuron can be written as a recursive equation
!bt
\begin{equation}
\delta_j = \frac{\partial \Gamma}{\partial u_j} =
\sum_{i\in P_j} \frac{\partial \Gamma}{\partial u_i}\frac{\partial u_i}{\partial y_j}\frac{\partial y_j}{\partial u_j}
label{errorTerms}
\end{equation}
!et
The second term is the derivative of the net input of all posterior
nodes $\{i\}$ w.r.t. the output of node $j$,
!bt
\begin{equation}
\frac{\partial u_i}{\partial y_j} = \frac{\partial}{y_j}\left(\sum_{m\in A_i} w_{im}y_m + b_i\right) = w_{ij}
\end{equation}
!et
while the third is the derivative of node $j$'s activation function w.r.t. its net input:
!bt
\begin{equation}
\frac{\partial y_j}{\partial u_j} = \frac{\partial f_j(u_j)}{\partial u_j} \equiv y^\prime_j
\end{equation}
!et
Putting all the pieces together we obtain
!bt
\begin{equation}
\delta_j = y^\prime_j\sum_{i\in P_j}w_{ij} \delta_i
label{backprop}
\end{equation}
!et
This expression is illustrated in
Each hidden neuron $j$ receives a weighted sum
of the errors of all nodes in the posterior layer. Then we differentiate in the backwards direction (compare with
the output $y_j$ of node $j$ is differentiated w.r.t. its input $u_j$.
The errors are propagated backwards through the whole NN until we reach the weights connecting the input layer
and the first hidden layer.
By propagating the error
of only one output neuron, we thus obtain the errors of all the neurons at once. This is the main strength of the backpropagation
algorithm, and the reason for its popularity in neural network research.
!bt
\begin{equation}
h_j = \frac{\partial \Gamma}{\partial b_j} = \sum_{i\in P_j} \frac{\partial \Gamma}{\partial u_i}
\frac{\partial u_i}{\partial y_j} \frac{\partial y_j}{\partial u_j} \frac{\partial u_j}{\partial b_j}
\end{equation}
!et
The only new term here is the last one:
!bt
\begin{equation}
\frac{\partial u_j}{\partial b_j} = \frac{\partial}{\partial b_j}\left(\sum_{m\in A_j} w_{jm}y_m + b_j\right) = 1
\end{equation}
!et
Consequentially, the bias gradient is simply the error of each neuron:
!bt
\begin{equation}
h_j = \delta_j
\end{equation}
!et
=== Matrix notation ===
This is easily done also for the backpropagation case. We have the
set of vectors $\vec{b}_l$ and $\vec{y}_l$ for $l = 1,\dots ,L+1$,
where $L$ is the number of hidden layers. We now extend this set with the vectors $\vec{\updelta}_l$
and $\vec{u}_l$, i.e.\ the errors and inputs (or *preactivations*) of layer $l$
respectively. These are all column vectors of size $N_l \times 1$.
!bt
\begin{equation}
\sum_j w_{ij} y_j \: \Rightarrow \: \mathrm{W} \vec{y}
\end{equation}
!et
we have
!bt
\begin{equation}
\sum_i w_{ij} \delta_i \: \Rightarrow \: \mathrm{W}^T \vec{\delta}
\end{equation}
!et
i.e.\ the weight matrices employed in backpropagation are the transpose of the matrices used in forward activation.
Thus, for nodes $\{j\}$ in layer $l$ and nodes $\{i\}$ in layer $l+1$, the vectorized equation for the error of each hidden neuron is
!bt
\begin{equation}
\delta_j = y^\prime_j\sum_{i\in P_j}w_{ij} \delta_i \: \Rightarrow \:
\vec{\updelta}_l = \vec{y}^\prime_l \odot (W_{l+1} \vec{\updelta}_{l+1})
\end{equation}
!et
where $\odot$ signifies element-wise multiplication. The expression for the weight gradients is converted
into an outer product of the errors of layer $l+1$ and the outputs of layer $l$,
!bt
\begin{equation}
g_{ij} = \delta_i y_j \Rightarrow \mathrm{G}_l = \vec{\delta}_{l+1} \vec{y}^T_l
\end{equation}
!et
where $\mathrm{G}_l$ is a matrix containing all gradients for the weights connecting layer $l$ and $l+1$.
The corresponding matrix for the biases is a $N_l \times 1$ column vector $\vec{H}_l$,
!bt
\begin{equation}
\vec{H}_l = \vec{\updelta}_l
\end{equation}
!et
The set $\{\mathrm{G}_l, \vec{H}_l\}$ for $l=1,\dots,L+1$ thus make up the total gradient of the cost function
=== Training algorithm ===
We are now ready to express the complete training algorithm for a MLP with backpropagation using
matrix-vector notation. We introduce $\mathrm{G}_{l,i}$ and $\vec{H}_{l,i}$ as the weight gradient matrix and
the bias gradient vector for training example $i$.
* Input a set of $n$ training examples.
* Initialize the input layer:
!bt
\begin{equation}
\vec{y}_0 = \vec{x}
\end{equation}
!et
* Propagate the activity forward. For $l = 1,\dots,L+1$:
!bt
\begin{align}
\vec{u}_l &= \mathrm{W}_l\vec{y}_{l-1} + \vec{b}_l \\ \vec{y}_l &= f_l(\vec{u}_l) label{forwardPropMatrix}
\end{align}
!et
Store all vectors $\vec{u}_l$ and $\vec{y}_l$.
* Calculate and store the error in the output layer:
!bt
\begin{equation}
\vec{\updelta}_{L+1} = \vec{Y} - \vec{y}_{L+1}
\end{equation}
!et
* Backpropagate the error. For $l = L, L-1, \dots ,1$:
!bt
\begin{equation}
\vec{\updelta}_l = \vec{y}^\prime_l \odot (\mathrm{W}^T_{l+1}\vec{\updelta}_{l+1}) \cdot label{backPropMatrix}
\end{equation}
!et
Store all errors $\vec{\updelta}_l$.
* Compute and store the weight and bias gradients. \\ For $l = L+1, L, \dots ,1$:
!bt
\begin{equation}
\mathrm{G}_{l,i} = \vec{\updelta}_l \vec{y}_{l-1}^T, \quad \vec{H}_{l,i} = \vec{\updelta}_l label{weightUpdate}
\end{equation}
!et
Calculate the total gradients of all training examples $i$. \\ For $l = L+1, L, \dots ,1$:
!bt
\begin{equation}
\mathrm{G}_l = \sum_i^n \mathrm{G}_{l,i} \quad \mathrm{and} \quad \vec{H}_l = \sum_i^n \vec{H}_{l,i}
\end{equation}
!et
Update weights and biases with an update rule of choice.
The value of $n$ depends on the learning paradigm that is used, described in
doc/src/NeuralNet/dill
+1044
View File
File diff suppressed because it is too large Load Diff