From e49b6c90d6474da2ad6ae8accafd442ae1f58d11 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 31 Oct 2017 10:06:22 +0100 Subject: [PATCH] update on neural net --- doc/src/NeuralNet/dill | 1044 ---------------------------------------- 1 file changed, 1044 deletions(-) delete mode 100644 doc/src/NeuralNet/dill diff --git a/doc/src/NeuralNet/dill b/doc/src/NeuralNet/dill deleted file mode 100644 index 83de48058..000000000 --- a/doc/src/NeuralNet/dill +++ /dev/null @@ -1,1044 +0,0 @@ -# #ifdef LATEX2DOCONCE -This is the result of the doconce latex2doconce program. -The translation from LaTeX is just a helper. The text must -be carefully examined! (Be prepared that some text might also -be lost in the translation - in seldom cases.) - - -## search for CHECK to see if auto editing was correct - -# #endif -Machine learning is the science of giving computers the ability to learn without being explicitly programmed. -The idea is that there exist generic algorithms which can be used to find patterns in a broad class of data sets without -having to write code specifically for each problem. The algorithm will build its own logic based on the data. - -Machine learning is a subfield of computer science, and is closely related to computational statistics. -It evolved from the study of pattern recognition in artificial intelligence (AI) research, and has made contributions to -AI tasks like computer vision cite{Krizhevsky12}, natural language processing cite{Collobert11} -and speech recognition cite{Hinton12}. It has also, especially in later years, -found applications in a wide variety of other areas, including bioinformatics, economy, physics, finance and marketing -cite{LeCun15}. - -The approaches to machine learning are many, but are often split into two main categories. -In *supervised learning* we know the answer to a problem, -and let the computer deduce the logic behind it. On the other hand, *unsupervised learning* -is a method for finding patterns and relationship in data sets without any prior knowledge of the system. -Some authours also operate with a third category, namely *reinforcement learning*. This is a paradigm -of learning inspired by behavioural psychology, where learning is achieved by trial-and-error, -solely from rewards and punishment. - -Another way to categorize machine learning tasks is to consider the desired output of a system. -Some of the most common tasks are: - - * Classification: Outputs are divided into two or more classes. The goal is to produce a model that assigns inputs into one of these classes. An example is to identify digits based on pictures of hand-written ones. Classification is typically supervised learning. - - * Regression: Finding a functional relationship between an input data set and a reference data set. The goal is to construct a function that maps input data to continuous output values. - - * Clustering: Data are divided into groups with certain common traits, without knowing the different groups beforehand. It is thus a form of unsupervised learning. - - -\autoref{fig:machineLearningDiagram} gives an (incomplete) overview of machine learning methods with some selected applications. -We will not go into further detail on all these approaches to machine learning, as this work is only concerned with -one specific area: artificial neural networks. The rest of this chapter is devoted to the ideas behind this machine learning -approach. -FIGURE: [Figures/Theory/neuron.pdf, width=400 frac=\linewidth]{Figures/Theory/MachineLearningDiagram.png} - \caption{Overview of various approaches to machine learning and their applications. - Source: "isaziconsulting.co.za": "http://www.isaziconsulting.co.za/machinelearning.html".} - label{fig:machineLearningDiagram} -\end{figure} - -======= Artificial neurons} label{sec:ANN ======= -The field of artificial neural networks has a long history of development, and is closely connected with -the advancement of computer science and computers in general. A model of artificial neurons -was first developed by McCulloch and Pitts in 1943 cite{McCulloch43} to study signal processing in the brain and -has later been refined by others. The general idea is to mimic neural networks in the human brain, which -is composed of billions of neurons that communicate with each other by sending electrical signals. -Each neuron accumulates its incoming signals, -which must exceed an activation threshold to yield an output. If the threshold is not overcome, the neuron -remains inactive, i.e.\ has zero output. - -This behaviour has inspired a simple mathematical model for an artificial neuron cite{Rojas96}, - -!bt -\begin{equation} - y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u) - label{artificialNeuron} -\end{equation} -!et -Here, the output $y$ of the neuron is the value of its activation function, which have as input -a weighted sum of signals $x_i, \dots ,x_n$ received by $n$ other neurons. -The model is illustrated in \autoref{fig:neuronModel}. We will now take a closer look at if and how -this mathematical model is biologically plausible. -# original latex figure with width=0.8] Mathematical model of an artificial neuron. The neuron receives input signals $x_i,\dots,x_n$ from $n$ other neurons. Each signal $x_i$ is associated with a weight $w_i$, and the neuron accumulates all input signals as a weighted sum $u$. This sum is then used as input to its activation function $f$, which serves as the neuron's output signal.} label{fig:neuronModel - -===== Biological model ===== - -Artificial neurons are designed to mimic certain aspects of their biological counterparts cite{Kriesel07}. -A schematic model of a biological nervous cell is depicted in \autoref{fig:neuronBiological}. -The *dendrites* in a neuron acts as the input vector, and allow the cell to receive signals from a large number -of neighbouring neurons. As in the mathematical model, each dendrite is associated with the multiplication of a -''weight value'', which is achieved by an increase or decrease of chemical neurotransmitters that amplifies or -weakens the input signals. The cell can also transmit signal inhibitors (oppositely charged ions) to -accomplish ''negative'' weight values. - -The positive and negative ions arriving from the dendrites are mixed together in the solution inside the *soma*, -which corresponds to the summation in (ref{artificialNeuron}). Finally, the *axon* serves -as the activation function by sampling the electrical potential of this solution. If the potential -reaches a certain strength, the axon transmits a signal pulse down its own length. The axon is connected -to another neuron's dendrites (or other parts of the cell), enabling the neurons to communicate. -FIGURE: [Figures/Theory/neuron_anatomy.jpg, width=400 frac=0.9] Biological model of a nervous cell. The various parts of the cell are briefly explained in the text. Source: "askabiologist.asu.edu": "https://askabiologist.asu.edu/neuron-anatomy".} label{fig:neuronBiological - -This discussion of nervous cells helps, to a certain degree, to validate the mathematical form (ref{artificialNeuron}) -of artificial neurons. However, we note that the above description is a highly simplified picture of the great complexity -of biological neurons, which is not yet fully understood. Our mathematical neurons are thus only caricatures of nature. -Next, we will look at how networks of these mathematical objects can be formed to emulate the neural networks -in the brain. - -======= Neural network types ======= - -An artificial neural network (NN), is a computational model that consists of layers of connected neurons, or *nodes*. -It is supposed to mimic a biological nervous system by letting each neuron interact with other neurons -by sending signals in the form of mathematical functions between layers. -Each node is modelled according to (ref{artificialNeuron}). -A wide variety of different NNs have -been developed, but most of them consist of an input layer, an output layer and eventual layers in-between, called -*hidden layers*. All layers can contain an arbitrary number of nodes, and each connection between two nodes -is associated with a weight variable. - -The main factor that separates the different types are how the neurons are *connected*. -This section contains a short presentation of some of the most common types of ANNs, before we move on to -a more detailed description of the NN architecture used in this thesis. - -===== Feed-forward neural networks ===== -The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network, -the information moves in only one direction: forward through the layers. An example FFNN is shown -in \autoref{fig:networkGeneral}, consisting of an input layer, two hidden layers and an output layer. -Nodes are represented by circles, while the arrows display the connections between the nodes, including the -direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here. -We observe that each node in a layer is connected to *all* nodes in the subsequent layer, -making this a so-called *fully-connected* FFNN. -This is the type of NN that is used in the present work, -and will be further investigated in \autoref{sec:MLP}. -FIGURE: [Figures/Theory/networkGeneral.pdf, width=400 frac=0.8] Example of a fully-connected feed-forward neural network with an input layer, two hidden layers and an output layer. Each node in a layer is connected to *all* nodes in the subsequent layer, and information only flows forward through the layers, hence the name.} label{fig:networkGeneral - -A different variant of FFNNs are *convolutional neural networks* (CNNs) cite{LeCun99}, which have a connectivity pattern -inspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli from -small sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strong -spatially local correlation present in natural images. The response of each neuron can be approximated mathematically -as a convolution operation. - -CNNs emulate the behaviour of neurons in the visual cortex by enforcing a *local* connectivity pattern -between nodes of adjacent layers: Each node -in a convolutional layer is connected only to a subset of the nodes in the previous layer, -in contrast to the fully-connected FFNN in \autoref{fig:networkGeneral}. -Often, CNNs -consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end, -which gathers all the local data and produces the outputs. They have wide applications in image and video recognition -cite{LeCun15}. - -===== Recurrent neural networks ===== - -So far we have only mentioned NNs where information flows in one direction: forward. *Recurrent neural networks* on -the other hand, have connections between nodes that form directed *cycles*. This creates a form of -internal memory which are able to capture information on what has been calculated before; the output is dependent -on the previous computations. Recurrent NNs make use of sequential information by performing the same task for -every element in a sequence, where each element depends on previous elements. An example of such information is -sentences, making recurrent NNs especially well-suited for handwriting and speech recognition. - -===== Other types of networks ===== - -There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation -in multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers: -an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' here -means that each node in the output layer has a linear activation function). The layers are normally fully-connected and -there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as -a separate type of NN due the unusual activation functions (common activation functions are presented in -\autoref{sec:activationFunctions}). - -Other types of NNs could also be mentioned, but are outside the scope of this work. We will now move on to a detailed description -of how a fully-connected FFNN works, and how it can be used to interpolate data sets. - -======= Multilayer perceptron} label{sec:MLP ======= -In this thesis we use fully-connected feed-forward neural networks with three -or more layers (an input layer, one or more hidden layers and an output layer), mainly -consisting of neurons that have non-linear activation functions. -Such networks are often called *multilayer perceptrons* (MLPs)% -[^footnote1][^footnote1]: The terms ''feed-forward neural network'' and ''multilayer perceptron'' are used interchangeably in the -literature, although the MLP is just one type of FFNN, namely a fully-connected one with mainly non-linear neurons.. -The *perceptron* was first introduced by Rosenblatt in 1958 cite{Rosenblatt58}, and was a FFNN made up of an input layer -and an output layer only. It served as a *binary classifier*, i.e.\ a function that maps a real-valued vector $\vec{x}$ -to a single binary value $f(\vec{x})$, employing a linear activation function. The addition of hidden layers and -non-linear activation functions led to the MLP, which could produce an arbitrary number of continuous output values. - -===== Why multilayer perceptrons?} label{sec:whyMLP ===== -We have chosen to use MLPs to interpolate data sets for the construction of interatomic potentials. -Other NN types could also have been used, but our choice is well justified. According to the -*Universal approximation theorem* cite{Hornik89}, a feed-forward neural network with just a single hidden layer containing -a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, -assuming the activation function for the hidden layer is a ''non-constant, bounded and monotonically-increasing continuous function''. -Note that the requirements on the activation function only applies to the hidden layer, the output nodes are always -assumed to be linear, so as to not restrict the range of output values. - -The only requirement on the NN information flow is that it must be feed-forward, which is not satisfied by recurrent NNs. -Anyhow, computing energies and forces in MD simulations at a given time step do not require any knowledge of -the earlier states of the system, i.e.\ no memory or recurrency is involved. -Further, it is not obvious how local connectivity should be helpful in a MD context, thus -we might as well use the simplest multilayer FFNN available, the MLP. - -We note that this theorem is only applicable to a NN with *one* hidden layer. -Therefore, we can easily construct a NN -that employs activation functions which do not satisfy the above requirements, as long as we have at least one layer -with activation functions that *do*. Furthermore, although the universal approximation theorem -lays the theoretical foundation for regression with neural networks, it does not say anything about how things work in practice: -A NN can still be able to approximate a given function reasonably well without having the flexibility to fit *all other* -functions. - -===== Mathematical model} label{sec:MLPmodel ===== -In an MLP, each node (neuron) is modelled according to (ref{artificialNeuron}). To increase the flexibility of the NN, -we also add a *bias* $b$ to this model, resulting in - -!bt -\begin{equation} - y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(u) - label{artificialNeuron2} -\end{equation} -!et -In a FFNN of such neurons, the *inputs* $x_i$ -are the *outputs* of the neurons in the preceding layer. Furthermore, a MLP is fully-connected, -which means that each neuron receives a weighted sum of the outputs of *all* neurons in the previous layer. - -\autoref{fig:networkNotation} displays the same NN as in \autoref{fig:networkGeneral}, now with all nodes labeled by their -output $y$, except for the input nodes, which are labeled with $x$. The input nodes are strictly speaking not -artificial neurons, as they have no activation function. Their only purpose is to forward the NN inputs to the -first hidden layer. A few weights $w$ (blue) and biases $b$ (green) are also included, next to the connections and nodes -which they belong to respectively. -FIGURE: [Figures/Theory/networkWithNotation.pdf, width=400 frac=0.8] Example of a fully-connected feed-forward neural network with an input layer, two hidden layers and an output layer. Nodes are depicted as circles, while the arrows shows the connections between neurons, including their directions. Each connection has a weight $w$ (blue), where the notation explained in the text is applied. Every node is marked with its output $y$ and its associated bias $b$ (green), while all input nodes are labeled with an $x$. Only a few weights and biases are displayed.} label{fig:networkNotation -The following notation is introduced: $y_i^l$ is the output of the $i$-th node in layer $l$, where $l=0$ is the input layer, -$l=1$ is the first hidden layer and so on. The same notation applies to the biases. For the weights, we have that -$w_{ij}^l$ is the weight connecting node $j$ in layer $l-1$ with node $i$ in layer $l$% -[^footnote2][^footnote2]: This order of indicies $i$ and $j$ may seem illogical, but will become useful when converting -to matrix equations below.. -All weights and biases are real-valued numbers. - -The MLP in \autoref{fig:networkNotation} maps the inputs $x_i$ to the output $y_1^3$ via two hidden layers -with an (arbitrary) number of hidden nodes. The hidden neurons have no physical meaning, but have the purpose -of defining the functional form of the NN. In the following we describe how to calculate the output of this MLP. The description -is also valid for a NN with several outputs; the result applies to each output node individually. - -First, for each node $i$ in the first hidden layer, we calculate a weighted sum $u_i^1$ of the input coordinates $x_j$, - -!bt -\begin{equation} - u_i^1 = \sum_{j=1}^2 w_{ij}^1 x_j + b_i^1 -\end{equation} -!et -This value is the argument to the activation function $f_1$ of each neuron $i$, -producing the output $y_i^1$ of all neurons in layer 1, - -!bt -\begin{equation} - y_i^1 = f_1(u_i^1) = f_1\left(\sum_{j=1}^2 w_{ij}^1 x_j + b_i^1\right) - label{outputLayer1} -\end{equation} -!et -where we assume that all nodes in the same layer have identical activation functions, hence the notation $f_l$% -[^footnote3][^footnote3]: Note that for the activation functions, the layer indicies are subscripts rather than superscripts, -as having a single superscript can easily be confused with a square.. -For an arbitrary node $i$ in layer $l$ this generalizes to - -!bt -\begin{equation} - y_i^l = f_l(u_i^l) = f_l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right) - label{generalLayer} -\end{equation} -!et -where $N_l$ is the number of nodes in layer $l$. When the output of all the nodes in the first hidden layer are computed, -the values of the subsequent layer can be calculated and so forth until the output is obtained. -The output of neuron $i$ in layer 2 is thus, - -!bt -\begin{align} - y_i^2 &= f_2\left(\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\right) \\ - &= f_2\left[\sum_{j=1}^3 w_{ij}^2f_1\left(\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\right) + b_i^2\right] - label{outputLayer2} -\end{align} -!et -where we have substituted $y_m^1$ with (ref{outputLayer1}). Finally, the NN output yields, - -!bt -\begin{align} - y_1^3 &= f_3\left(\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\right) \\ - &= f_3\left[\sum_{j=1}^3 w_{1j}^3 f_2\left(\sum_{k=1}^3 w_{jk}^2 f_1\left(\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\right) + b_j^2\right) - + b_1^3\right] -\end{align} -!et -We can generalize this expression to a MLP with $l$ hidden layers. The complete functional form -is, -\begin{flalign} -&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! - \dots \!f_1\!\left(\!\sum_{n=1}^{N_0} \!w_{mn}^1 x_n\! + \!b_m^1\!\right) - \!\dots \!\right) \!+ \!b_k^2\!\right) - \!+ \!b_1^3\!\right] && - label{completeNN} -\end{flalign} -which illustrates a basic property of MLPs: The only independent variables are the input values $x_n$. -This confirms that a MLP, -despite its quite convoluted mathematical form, is nothing more than an analytic function, specifically a -mapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m$. -In our example (\autoref{fig:networkNotation}), $n=2$ and $m=1$. Consequentially, -the number of input and output values of the function we want to fit must be equal to the number of inputs and outputs of our MLP. - -Furthermore, the flexibility and universality of a MLP can be illustrated by realizing that -the expression (ref{completeNN}) is essentially a nested sum of scaled activation functions of the form - -!bt -\begin{equation} - h(x) = c_1 f(c_2 x + c_3) + c_4 -\end{equation} -!et -where the parameters $c_i$ are weights and biases. By adjusting these parameters, the activation functions -can be shifted up and down or left and right, change slope or be rescaled (\autoref{fig:activationsFlex}), -which is the key to the flexibility of a NN. -\begin{figure} - \centering - \includegraphics[width = 0.8\linewidth - -FIGURE: [Figures/Theory/activationFlex.pdf, width=400 frac=1.0] The mathematical expression for a MLP consists of nested terms of the form $h(x) = c_1 f(c_2 x + c_3) + c_4$ (ref{completeNN}), where $f$ is the activation function and $c_i$ are NN parameters. The flexibility of the MLP is shown by adjusting $c_1$, $c_2$, $c_3$ and $c_4$ such that $h(x)$ a) is scaled, b) has a change of slope, c) is shifted left and right, d) is shifted up and down, respectively. Here $f$ is the hyperbolic tangent (ref{tanhActivationFunction}). Reproduced from Behler cite{Behler11general}.} label{fig:activationsFlex - -=== Activation function of output neuron === - -We stated in \autoref{sec:whyMLP} that the output neuron always have a linear activation function for regression to avoid -restricting the output values. This is a requirement for a MLP to operate as a mapping -$\vec{x} \in \mathbb{R}^n \rightarrow \vec{y} \in \mathbb{R}^m$. In this thesis, we will exclusively use a -specific linear activation function for the output neuron $o$, namely the identity function $f(x) = x$, - -!bt -\begin{equation} - f_o = f(u_o) = u_o - label{outputActivation} -\end{equation} -!et - - - -=== Matrix-vector notation === -We can introduce a more convenient notation for the activations in a NN. -If we look at the structure of the NN in \autoref{fig:networkNotation}, we realize that all the weights connecting -two adjacent layers can be represented as a matrix $\mathrm{W}_l$, where the $(i,j)$-th element is -is the weight $w^l_{ij}$ connecting connecting node $j$ -in layer $l-1$ with node $i$ in layer $l$, as above. - -Additionally, we can represent the biases and activations -as layer-wise column vectors $\vec{b}_l$ and $\vec{y}_l$, so that the $i$-th element of each vector -is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively. - -We have that $\mathrm{W}_l$ is a $N_{l-1} \times N_l$ matrix, while $\vec{b}_l$ and $\vec{y}_l$ are $N_l \times 1$ column vectors. -With this notation, the sum in (ref{outputLayer2}) becomes a matrix-vector multiplication, and we can write -the equation for the activations of hidden layer 2 in \autoref{fig:networkNotation} as% -[^footnote4][^footnote4]: For the same reasons as the activation functions above, the layer indicies are changed from superscripts to subscripts - with this matrix-vector notation. - -!bt -\begin{equation} - \vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) = - f_2\left(\left[\begin{array}{ccc} - w^2_{11} &w^2_{12} &w^2_{13} \\ - w^2_{21} &w^2_{22} &w^2_{23} \\ - w^2_{31} &w^2_{32} &w^2_{33} \\ - \end{array} \right] \cdot - \left[\begin{array}{c} - y^1_1 \\ - y^1_2 \\ - y^1_3 \\ - \end{array}\right] + - \left[\begin{array}{c} - b^2_1 \\ - b^2_2 \\ - b^2_3 \\ - \end{array}\right]\right) -\end{equation} -!et -and we see that the activation of node $i$ in layer 2 is - -!bt -\begin{equation} - y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) = - f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right) -\end{equation} -!et -which is in accordance with (ref{outputLayer2}). Note that -This is not just a convenient and compact notation, but also -a useful and intuitive way to think about MLPs: The output is calculated by a series of matrix-vector multiplications -and vector additions that are used as input to the activation functions. For each operation -$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer. Furthermore, it is the easiest way to implement NNs with -TensorFlow, which is discussed in \autoref{sec:TensorFlowCreatingNN}. - - - -======= Activation functions} label{sec:activationFunctions ======= -A property that characterizes a NN, other than its connectivity, is the choice of activation function(s). -As described in \autoref{sec:whyMLP}, the following restrictions are imposed on an activation function for a FFNN -to fulfill the universal approximation theorem cite{Hornik89}: - - * Non-constant - - * Bounded - - * Monotonically-increasing - - * Continous - - -We realize that the second requirement excludes all linear functions. Furthermore, in a MLP with only linear activation functions, each -layer simply performs a linear transformation of its inputs. Consequentially, regardless of the number of layers, -the output of the NN will be nothing but a linear function of the inputs. Thus we need to introduce some kind of -non-linearity to the NN to be able to fit non-linear functions. -# original latex figure with width = 0.7\linewidth - -FIGURE: [Figures/Theory/activationFunctions.pdf, width=400 frac=1.0] Two of the most common activation functions for neural networks, the sigmoid (ref{sigmoidActivationFunction}) and the hyperbolic tangent (ref{tanhActivationFunction}). Both meet the requirements of the universal approximation theorem.} label{fig:SigmoidActivationFunctions - -A group of non-linear functions that meet the above requirements are the *sigmoid* functions. -The two sigmoid functions that are most commonly used as activation functions cite{Rojas96} in -neural networks are the logistic function - -!bt -\begin{equation} - f(x) = \frac{1}{1 + e^{-x}} - label{sigmoidActivationFunction} -\end{equation} -!et -and the hyperbolic tangent - -!bt -\begin{equation} - f(x) = \tanh(x) - label{tanhActivationFunction} -\end{equation} -!et -The logistic functon is often referred to as ''the sigmoid'', which is the naming convention we will use -in the following. Both functions are depicted in \autoref{fig:SigmoidActivationFunctions}. We observe that they -are bounded above by 1, while the lower bound is -1 and 0 for the hyperbolic tangent and sigmoid respectively. This property -keeps the activations from diverging. - -The sigmoid are more biologically plausible because -the output of inactive neurons are zero. Such activation function are called *one-sided*. However, -it has been shown cite{Karlik11} that the hyperbolic tangent -performs better than the sigmoid for training MLPs. This will be further discussed in \autoref{sec:hyperParamsActFunctions}. -Nevertheless, one should assess the problem -at hand when deciding what activation function to use; the performance can vary from problem to problem. -# original latex figure with width = 0.7\linewidth - -FIGURE: [Figures/Theory/reluActivation.pdf, width=400 frac=1.0] The rectifier (ref{reluActivationFunction}) has become the most popular activation function for deep convolutional NNs.} label{fig:reluActivationFunction - -# #if FORMAT in ("latex", "pdflatex") -\noindent -# #endif In later years, the rectifier function (\autoref{fig:reluActivationFunction}) - -!bt -\begin{equation} - f(x) = \max(0,x) - label{reluActivationFunction} -\end{equation} -!et -has become the most popular cite{LeCun15} for *deep neural networks*, i.e.\ NNs with several hidden layers% -[^footnote5][^footnote5]: There is no consensus in the literature on how many hidden layers a NN must possess in order to qualify as ''deep'', -but there seems to be a tendency towards setting the minimum number of layers to three.. -It has been argued to be even more biologically plausible than the sigmoid and also perform better -than the hyperbolic tangent, especially for deep, convolutional NNs cite{Glorot11}. Even though the rectifier is a -piecewise *linear* function, it effectively behaves as a non-linearity. However, it does not have an upper bound, and -therefore does not satisfy the universal approximator requirements. Further, the function values can potentially blow up. -These problems are solved in practice by employing various techniques, illustrating that the universal approximation theorem -should not be used as an absolute guideline for the construction of activation functions. - - -======= Training} label{sec:training ======= -In the beginning of this chapter we defined machine learning as the ''science of giving computers the ability to learn without -being explicitly programmed.'' In the neural network case, this learning is achieved by iteratively feeding the network with data. -With the help of certain learning algorithms, the network will then automatically adjust its parameters, i.e.\ the weights and biases, -to find patterns in this data% -[^footnote6][^footnote6]: in the following, ''weights'' will often be equivalent to all the NN parameters, including the biases.. -The procedure depends on what learning paradigm we work with -(unsupervised or supervised learning), and the type of NN we use. - -In this thesis we employ NNs for regression: We want to interpolate data sets og atomic coordinates and energies. -The goal is to adjust the weights of a MLP so that it accurately represents a function that maps -atomic coordinates to the corresponding potential energies. Interpolation with NNs is essentially an optimization problem: -The weights are typically initialized as random numbers, and then modified iteratively to minimize the error -to a set of expected output values, in our case energies. -This procedure is called *training*. - -Thus, the input values to the function that we want to fit are atomic configurations, -while the output values are the total energies of these configurations. -Together they form what we refer to as a *data set* or a *reference set*. -The output values are also known as the *target values*. -The input data $\mathrm{X}$ and output data $\mathrm{Y}$ are represented as matrices, where the $i$-th row of X and Y, -denoted $\mathrm{X}_{i*}$ and $\mathrm{Y}_{i*}$ respectively, -together form a *training example*. The combined data set consists of $N$ such training examples. - -Note that the number of columns of X and Y are equal to the number of inputs and outputs of our MLP respectively. -This is a restatement of the observation made in \autoref{sec:MLPmodel}: The number of inputs and outputs of our MLP must be equal -to the number of input and output values of the function we are trying to fit. In the following we assume that the MLP only has one -output, thus each row of Y is a single number, denoted $Y_{i*} = Y_i$. - - -=== Cost functions} label{costFunctions === -To minimize the error, we need a way to *define* it. -As in mathematical optimization, the error is represented by an objective function, also called *loss* function -or *cost* function. Training a NN therefore amounts to the minimization of this function, which can be written in the -following general way, - -!bt -\begin{equation} - \Gamma = \Gamma\bigr(\{\mathrm{W}_l\}, \{\vec{b}_l\}, \mathrm{X}, \mathrm{Y}\bigr) - label{generalCost} -\end{equation} -!et -where $\{\mathrm{W}_l\}$ and $\{\vec{b}_l\}$ are all the weights and biases of the NN respectively. -The value of this function is a measure of how well the NN is able to map $\mathrm{X} \rightarrow \mathrm{Y}$. -By adjusting $\{\mathrm{W}_l\}$ and $\{\vec{b}_l\}$, we try to minimize the value of this function. - -The standard cost function used in regression with NNs is the mean-square error, - -!bt -\begin{equation} - \Gamma = \frac{1}{2N}\sum_{i=1}^N (Y_i - y_i)^2 - label{quadraticCost} -\end{equation} -!et -where $y_i = y_i(W, B, X_{i*})$ is the value predicted by the NN for training example $X_{i*}$. -For a MLP with more than one output, $Y_i$ and $y_i$ are vectors. -The constant $1/2$ is included to cancel out the exponent when this function is differentiated at a later stage -(\autoref{sec:backprop}). - -A variety of other cost functions are used for other machine learning tasks. In classification the output is often binary, -hence the mean-square error function is inadequate. We will not go into further detail on these cost functions here, but -\autoref{sec:hyperParamsCostFunction} provides a discussion on an extension of (ref{quadraticCost}) that is relevant for -the construction of NNPs. - - -======= Optimization} label{sec:optimization ======= -There are a large number of algorithms that can be used to determine the set of weights minimizing the -cost function (ref{quadraticCost}). -Different kinds of gradient descent-based methods are widely used, while higher order methods like -the conjugate gradient algorithm or Newton's method are rarely seen in the literature because they are -too computationally expensive for large NNs and data sets. Therefore, only first-order methods will be discussed here. - -The idea behind gradient descent methods is to minimize a function by -iteratively taking steps in the direction of steepest descent towards a minimum in parameter space. -This direction is defined as the negative gradient of the function with respect to -all its parameters. If we define $\vec{\theta} \in \mathbb{R}^d$ as a vector containing all the weights and biases -of a MLP, we get the following iterative scheme, - -!bt -\begin{equation} - \vec{\uptheta}_{k+1} = \vec{\uptheta}_{k} - \gamma \nabla_{\vec{\uptheta}_k} \Gamma(\vec{\uptheta}) - label{gradientDescent} -\end{equation} -!et -where $\gamma$ is a step size, called the *learning rate* in the context of machine learning. The process is started -by initializing the parameters as random numbers. The value of $\gamma$ is of great importance for the algorithm to converge, -and is allowed to change at every iteration. Note that convergence to -a minimum is not guaranteed without certain assumptions on the function $\Gamma$ and the particular choices of $\gamma$. -Also, the obtained minimum is generally local, not global. -Different ways to update $\gamma$ is discussed below. - - -===== Gradient descent variants} label{sec:gradientDescentVariants ===== -There are three versions of gradient descent cite{Ruder16}, which differ in -the number of training examples we present to the NN before updating the parameters. -According to neural network terminology, the process of adjusting the parameters based on all the training examples, -either in batches or all at once, is called an *epoch*. - -In *batch* gradient descent, -we compute the gradient of the cost function for the *whole* data set before updating, also called -*offline learning*. -This approach can be very slow and is intractable for datasets that do not fit in memory. Furthermore, there is a risk -of performing redundant computations if many similar examples are present in the data set. - -In contrast, *stochastic* gradient descent (SDG) performs a parameter update for *each* training example. -SDG avoids redundant gradient calculations and is therefore faster than batch gradient descent. However, the accuracy -of each update is lower compared to offline learning, which can lead to quite large error oscillations. The concept of having -more than one parameter update per epoch is called *online learning*. - -Finally, we have *mini-batch* gradient descent, which is a mix of the two other approaches. The reference set -is divided into $n$ equally sized mini-batches and a parameter update is performed for each mini-batch. This is usually the -algorithm of choice, as it has the optimal trade-off between speed and accuracy. Normal mini-batch sizes range -between 50 and 256, but should in some degree be tailored to each problem. - -We realize that SDG can be seen as -a variant of mini-batch gradient descent with a mini-batch size of 1. In the literature, these two methods -are often collectively referred to as SDG. - - -===== Optimization algorithms} label{sec:optimizationAlgorithms ===== -In the following we will outline different optimization algorithms that are widely used in neural network research. -They are all variations on the update rule (ref{gradientDescent}). The main focus of the methods is to find -a proper learning rate. A learning rate that is too small may lead to very slow convergence, while -a learning rate that is too large can cause the loss function to fluctuate around the minimum or even diverge. - -=== Momentum === - -Near local minima, the surface area of the cost function (in parameter space) often curve much more steeply in one -direction than in another, forming ravines. SDG will have slow converge in such regions, as it will oscillate across the slopes -while having slow progress along the bottom of the ravine. The Momentum method cite{Qian99} avoids this problem -by accelerating SDG in the downwards direction, while damping the oscillations. The update rule is - -!bt -\begin{equation} -\begin{aligned} - \vec{v}_{k} &= \eta \vec{v}_{k-1} + \gamma \nabla_k \Gamma \\ - \vec{\uptheta}_{k+1} &= \vec{\uptheta}_k - \vec{v}_k -\end{aligned} -label{Momentum} -\end{equation} -!et -where $\eta$ is the momentum term, which is usually set to 0.9 or a similar value. The new update vector $\vec{v}_k$ -is formed by adding a fraction $\eta$ of the previous update vector $\vec{v}_{k-1}$ to the gradient. In this way, the magnitude -of the update decreases for parameters (dimensions) whose gradients change direction, while it increases for parameters -whose gradients points in the same direction as in the previous step. This helps to push SDG downhill towards the minima. - -=== Adagrad === - -In SDG and Momentum, we have one learning rate $\gamma$ that is applied to all the parameters. -Ideally, the learning rate should be adapted to each individual parameter to perform larger or smaller updates -depending on their importance. Adagrad cite{Duchi11} does just that. For brevity, we set - -!bt -\begin{equation} - g_{k,i} = \nabla_{\theta_i} \Gamma (\vec{\uptheta}) -\end{equation} -!et -to be the gradient of the loss function w.r.t. parameter $\theta_i$ at step $k$. Adagrad adjusts the general learning rate $\gamma$ -at each step $k$ for every parameter $\theta_i$ based on the past gradients for that parameter, - -!bt -\begin{equation} - \theta_{k+1,i} = \theta_{k,i} - \frac{\gamma}{\sqrt{G_{k,ii} + \epsilon}} \cdot g_{k,i} -\end{equation} -!et -where $\mathrm{G}_k \in \mathbb{R}^{d \times d}$ is a diagonal matrix where each diagonal element $G_{k,ii}$ is the sum of squares -of the gradient with respect to $\theta_i$ up to step $k$. The smoothing constant $\epsilon \sim 10^{-8}$ is present to avoid -division by zero. The vectorized version of this equation yields, - -!bt -\begin{equation} - \vec{\uptheta}_{k+1} = \vec{\uptheta}_k - \frac{\gamma}{\sqrt{\mathrm{diag}(\mathrm{G}_k) + \epsilon}} \odot \vec{g}_k - label{Adagrad} -\end{equation} -!et -where $\odot$ stands for element-wise multiplication. In other words, we do not have to manually tune the learning rate, -Adagrad does that for us. A weakness of this method is that the learning rates inevitably shrinks for each step, -resulting in arbitrary small values, at which point the learning stops. - -For the rest of \autoref{sec:optimizationAlgorithms}, we assume for brevity that all vector operations are element-wise, i.e.\ -$\vec{g}\,\vec{h} = \vec{g} \odot \vec{h}$ and $\vec{g}^2 = \vec{g} \odot \vec{g}$. - -=== Adadelta === - -Adadelta cite{Zeiler12} is an extension of Adagrad that reduces the rate at which the learning rate decreases. -The sum of all past squared gradients is replaced by a *exponentially decaying average* of all the previous squared gradients. -We introduce the vector $\vec{E}[\vec{g}^2]_k$ containing the decaying averages of the gradient -with respect to all the parameters at step $k$. -This vector is defined recursively -as a weighted average of the previous averages $\vec{E}[\vec{g}^2]_{k-1}$ and the current gradient $\vec{g}_k$, - -!bt -\begin{equation} - \vec{E}[\vec{g}^2]_k = \eta \vec{E}[\vec{g}^2]_{k-1} + (1 - \eta) \vec{g}^2_k - label{decayingAverageVector} -\end{equation} -!et -where $\eta$ is a decay constant similar to that in the Momentum method, and is usually set to the same value (0.9). -We also define a new parameter update vector $\Delta \vec{\uptheta}_k = -\vec{v}_k$ so that - -!bt -\begin{equation} - \vec{\uptheta}_{k+1} = \vec{\uptheta}_k + \Delta \vec{\uptheta}_k -\end{equation} -!et -Replacing the vector $\mathrm{diag}(\mathrm{G}_k)$ in (ref{Adagrad}) with the decaying average vector (ref{decayingAverageVector}) -yields the following parameter update vector, - -!bt -\begin{equation} - \Delta \vec{\uptheta}_k = -\frac{\gamma}{\sqrt{\vec{E}[\vec{g}^2]_k + \epsilon}} \vec{g}_k - label{preliminiaryAdadelta} -\end{equation} -!et -Additionally, the learning rate $\gamma$ is replaced by a decaying average of previous squared parameter updates -$\vec{E}[\Delta\vec{\uptheta}^2]$ up to step $k-1$, yielding the Adadelta update rule, - -!bt -\begin{equation} - \vec{v}_k = \frac{\sqrt{\vec{E}[\Delta\vec{\uptheta}^2]_{k-1} + \epsilon}}{\sqrt{\vec{E}[\vec{g}^2]_k + \epsilon}} \vec{g}_k -\end{equation} -!et -The final replacement is done to obtain correct units for the update vector and to eliminate the learning rate from the equation. -Thus, we do not even need to set a default learning rate with Adadelta. - -=== Adam === - -The final algorithm we are going to discuss is Adapte Moment Estimation (Adam) cite{Kingma14}. This method -computes adaptive learning rates for each parameter by storing -exponentially decaying averages of both the gradients *and* -the squared gradients (ref{decayingAverageVector}), - -!bt -\begin{equation} - \begin{aligned} - \vec{E}[\vec{g}]_k &= \beta_1 \vec{E}[\vec{g}]_{k-1} + (1 - \beta_1) \vec{g}_k \\ - \vec{E}[\vec{g}^2]_k &= \beta_2 \vec{E}[\vec{g}^2]_{k-1} + (1 - \beta_2) \vec{g}^2_k - \end{aligned} -\end{equation} -!et -We set $\vec{a}_k = \vec{E}[\vec{g}]_k$ and $\vec{b}_k = \vec{E}[\vec{g}^2]_k$ so that - -!bt -\begin{equation} - \begin{aligned} - \vec{a}_k &= \beta_1 \vec{a}_{k-1} + (1 - \beta_1) \vec{g}_k \\ - \vec{b}_k &= \beta_2 \vec{b}_{k-1} + (1 - \beta_2) \vec{g}^2_k - \end{aligned} -\end{equation} -!et -where $\vec{a}_k$ and $\vec{b}_k$ are estimates of the first moment (the mean) and the second moment (the uncentered variance) -of the gradients respectively, hence the name of the method. The elements of these two vectors are initialized as zeros, -which make them biased towards zero, especially during the first time steps. This bias is counteracted by -computing bias-corrected version of the vectors (derived in cite{Kingma14}), - -!bt -\begin{equation} - \begin{aligned} - \hat{\vec{a}}_k &= \frac{\vec{a}_k}{1 - \beta_1^k} \\ - \hat{\vec{b}}_k &= \frac{\vec{b}_k}{1 - \beta_2^k} - \end{aligned} -\end{equation} -!et -The parameter update vector for Adam is obtained by replacing the gradient $\vec{g}_k$ with $\vec{a}_k$ in (ref{preliminiaryAdadelta}), -yielding the update rule, - -!bt -\begin{equation} - \vec{\uptheta}_{k+1} = \vec{\uptheta}_k - \frac{\gamma}{\sqrt{\hat{\vec{b}}_k} + \epsilon} \hat{\vec{a}}_k - label{adamUpdateRule} -\end{equation} -!et -The default values of the hyperparameters are $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 10^{-8}$. - -=== Which optimizer to use? === -There is no general answer to the question stated above. The adaptive learning rate methods -like Adagrad, Adadelta and Adam are generally more robust because they do not rely on manual fine-tuning of the learning rate. -Also, an adaptive learning rate generally handles *sparse* data better, i.e.\ data where some learning features -are poorly represented. - -Adam has been proven to perform favourably over the other optimization methods for certain applications like classification -with MLPs and convolutional NNs cite{Kingma14}. However, the performance of various optimizers should be tested on -the problem at hand to determine which method that works best. - - - - -===== Backpropagation} label{sec:backprop ===== -In the preceding sections we have discussed different optimizers that implements various update rules for -the parameters of a loss function. In our case, these parameters are the weights and biases of a neural network -and the cost function $\Gamma$ is the mean-square-error (ref{quadraticCost}) - -!bt -\begin{equation} - \Gamma = \frac{1}{2} (Y_i - y_i)^2 -\end{equation} -!et -here written as a function of only *one* training example $i$. -What we have not discussed yet, is -how we calculate the gradient of the cost function, i.e.\ how we obtain the partial derivatives $g_{ij}$ and $h_i$ -with respect to all the weights and biases respectively, - -!bt -\begin{align} - g_{ij} &= \frac{\partial \Gamma}{\partial w_{ij}} label{weightDerivative} \\ - h_i &= \frac{\partial \Gamma}{\partial b_i} label{biasDerivative} -\end{align} -!et -where we have dropped layer indicies for clarity. -A common method to obtain these derivatives is *backpropagation* cite{Rumelhart86}. -In backpropagation, a training example is propagated forward through the NN to produce an output. -This output is compared to the desired output (target values), and the error is then propagated *backwards* -through the layers to obtain the amount of which each parameter should be adjusted, -hence the name. -The method is essentially an implementation of the chain rule, and will allow us to calculate -the partial derivatives of the cost with respect to all the parameters, thereby obtaining the gradient of the network. - -=== 1. Forward propagation === -The first stage of the algorithm is forward propagation, producing an output as described in \autoref{sec:MLPmodel}% -[^footnote7][^footnote7]: We assume that the MLP only has one output. As before, the results are valid for each individual output in -a many-output MLP.. -In the following, we will use the same notation as in \autoref{sec:MLPmodel}. We also define - -!bt -\begin{equation} - A_i = \{j:w_{ij}\} -\end{equation} -!et -as the set $\{j\}$ of nodes anterior to node $i$ and connected to node $i$ with weights $w_{ij}$, in addition to - -!bt -\begin{equation} - P_j = \{i:w_{ij}\} -\end{equation} -!et -as the set $\{i\}$ of nodes posterior to node $j$ and connected to node $j$ with weights $w_{ij}$. - -=== 2. Backward propagation === -The second step is to backpropagate the error of the NN output. The weight derivative (ref{weightDerivative}) -can be expanded into two factors by use of the chain rule, - -!bt -\begin{equation} - g_{ij} = \frac{\partial \Gamma}{\partial u_i}\frac{\partial u_i}{\partial w_{ij}} - label{weightDerivativeExpanded} -\end{equation} -!et -Now we move in the opposite direction of the feed-forward stage: First we differentiate the cost w.r.t -the input of neuron $i$, then we differentiate the input w.r.t. weight $w_{ij}$ connecting neurons $j$ (in the preceding layer) -and $i$. -The first term on the r.h.s. is defined as the the error $\delta_i$ of node $i$, - -!bt -\begin{equation} - \delta_i = \frac{\partial\Gamma}{\partial u_i} - label{neuronError} -\end{equation} -!et -This definition can be justified by the following argument. -To change the value of a cost function (ref{generalCost}), we need to change the outputs of the neurons in the network. -Changing the input $u_j$ to neuron $j$ by a small amount $\Delta u_j$ results in the output - -!bt -\begin{equation} - y_j = f_j(u_j + \Delta u_j) -\end{equation} -!et -This change will propagate through subsequent layers in the network, finally causing the overall cost to change -by an amount $\frac{\partial \Gamma}{\partial u_j}\Delta u_j$. If $\partial \Gamma / \partial u_j$ is close to zero, -we are not able improve the cost much by perturbing the weighted input $u_j$; the neuron is already quite near the optimal value. -This is a heuristic argument for $\partial \Gamma / \partial u_j$ to be a measure of the error of the neuron. - -# #if FORMAT in ("latex", "pdflatex") -\noindent -# #endif The second term on the r.h.s. of (ref{weightDerivativeExpanded}) is - -!bt -\begin{equation} - \frac{\partial u_i}{\partial w_{ij}} = \frac{\partial}{\partial w_{ij}} \sum_{m\in A_i} w_{im}y_m = y_j - label{derivativeSecondTerm} -\end{equation} -!et -Combining the the two terms (ref{neuronError}) and (ref{derivativeSecondTerm}) yields - -!bt -\begin{equation} - g_{ij} = \delta_i y_j - label{weightGradient} -\end{equation} -!et -To compute this quantity, we thus need to know the outputs and the errors of all nodes in the network. -The outputs are generated during the feed-forward stage (ref{generalLayer}), - -!bt -\begin{equation} - y_i = f_i(u_i) = f_i\left(\sum_{j\in A_i} w_{ij}y_j + b_i\right) - label{forwardProp} -\end{equation} -!et -and need to be stored for all nodes. -The errors are obtained by backpropagating the error of the output neuron $o$, - -!bt -\begin{equation} - \delta_o = \frac{\partial \Gamma}{\partial u_o} = \frac{\partial \Gamma}{\partial y_o}\frac{\partial y_o}{\partial u_o} - = (Y - y_o) \frac{\partial y_o}{\partial u_o} -\end{equation} -!et -We remember from \autoref{sec:MLPmodel} that the output neuron has the identity activation function $y_o = f_o(u_o) = u_o$, -which reduces the error to - -!bt -\begin{equation} - \delta_o = Y - y_o -\end{equation} -!et -This error is then propagated backwards through the network, layer by layer. -The error of each neuron in layer $l$ thus depends on the errors of all neurons in layer $l+1$. -Consequentially, the error of an arbitrary hidden neuron can be written as a recursive equation (\autoref{fig:backprop}), - -!bt -\begin{equation} - \delta_j = \frac{\partial \Gamma}{\partial u_j} = - \sum_{i\in P_j} \frac{\partial \Gamma}{\partial u_i}\frac{\partial u_i}{\partial y_j}\frac{\partial y_j}{\partial u_j} - label{errorTerms} -\end{equation} -!et -# original latex figure with width = 0.9\linewidth - -FIGURE: [Figures/Theory/backprop.pdf, width=400 frac=1.0] Illustration of the backpropagation algorithm. The direction of information flow is opposite of \autoref{fig:neuronModel}. A hidden neuron $j$ receives a weighted sum of errors of all nodes in the posterior layer. Then, we differentiate in the backwards direction: the activation function (output) $y_j$ of node $j$ is differentiated w.r.t. to its net input $u_j$.} label{fig:backprop -We observe that also the inputs $u_i$ to all nodes need to be stored during forward propagation for -this scheme to work. -The first term on the r.h.s. of (ref{errorTerms}) is the error $\delta_i$ of node $i$, which is assumed to be known. -As long as there are no cycles in the network, there is an ordering -of nodes from the output back to the input that respects this condition. It is -therefore valid only for feed-forward NNs. - -The second term is the derivative of the net input of all posterior -nodes $\{i\}$ w.r.t. the output of node $j$, - -!bt -\begin{equation} - \frac{\partial u_i}{\partial y_j} = \frac{\partial}{y_j}\left(\sum_{m\in A_i} w_{im}y_m + b_i\right) = w_{ij} -\end{equation} -!et -while the third is the derivative of node $j$'s activation function w.r.t. its net input: - -!bt -\begin{equation} - \frac{\partial y_j}{\partial u_j} = \frac{\partial f_j(u_j)}{\partial u_j} \equiv y^\prime_j -\end{equation} -!et -Putting all the pieces together we obtain - -!bt -\begin{equation} - \delta_j = y^\prime_j\sum_{i\in P_j}w_{ij} \delta_i - label{backprop} -\end{equation} -!et -This expression is illustrated in \autoref{fig:backprop}. -Each hidden neuron $j$ receives a weighted sum -of the errors of all nodes in the posterior layer. Then we differentiate in the backwards direction (compare with -\autoref{fig:neuronModel}): -the output $y_j$ of node $j$ is differentiated w.r.t. its input $u_j$. - -The errors are propagated backwards through the whole NN until we reach the weights connecting the input layer -and the first hidden layer. -By propagating the error -of only one output neuron, we thus obtain the errors of all the neurons at once. This is the main strength of the backpropagation -algorithm, and the reason for its popularity in neural network research. - -# #if FORMAT in ("latex", "pdflatex") -\noindent -# #endif For the biases, we have (ref{biasDerivative}), - -!bt -\begin{equation} - h_j = \frac{\partial \Gamma}{\partial b_j} = \sum_{i\in P_j} \frac{\partial \Gamma}{\partial u_i} - \frac{\partial u_i}{\partial y_j} \frac{\partial y_j}{\partial u_j} \frac{\partial u_j}{\partial b_j} -\end{equation} -!et -The only new term here is the last one: - -!bt -\begin{equation} - \frac{\partial u_j}{\partial b_j} = \frac{\partial}{\partial b_j}\left(\sum_{m\in A_j} w_{jm}y_m + b_j\right) = 1 -\end{equation} -!et -Consequentially, the bias gradient is simply the error of each neuron: - -!bt -\begin{equation} - h_j = \delta_j -\end{equation} -!et - -=== Matrix notation === - -In \autoref{sec:MLPmodel} we rewrote the forward propagation framework as matrix-vector equations. -This is easily done also for the backpropagation case. From \autoref{sec:MLPmodel} we have the -set of vectors $\vec{b}_l$ and $\vec{y}_l$ for $l = 1,\dots ,L+1$, -where $L$ is the number of hidden layers. We now extend this set with the vectors $\vec{\updelta}_l$ -and $\vec{u}_l$, i.e.\ the errors and inputs (or *preactivations*) of layer $l$ -respectively. These are all column vectors of size $N_l \times 1$. - -# #if FORMAT in ("latex", "pdflatex") -\noindent -# #endif Comparing the sums in (ref{forwardProp}) and (ref{backprop}), we realize that since - -!bt -\begin{equation} - \sum_j w_{ij} y_j \: \Rightarrow \: \mathrm{W} \vec{y} -\end{equation} -!et -we have - -!bt -\begin{equation} - \sum_i w_{ij} \delta_i \: \Rightarrow \: \mathrm{W}^T \vec{\delta} -\end{equation} -!et -i.e.\ the weight matrices employed in backpropagation are the transpose of the matrices used in forward activation. -Thus, for nodes $\{j\}$ in layer $l$ and nodes $\{i\}$ in layer $l+1$, the vectorized equation for the error of each hidden neuron is - -!bt -\begin{equation} - \delta_j = y^\prime_j\sum_{i\in P_j}w_{ij} \delta_i \: \Rightarrow \: - \vec{\updelta}_l = \vec{y}^\prime_l \odot (W_{l+1} \vec{\updelta}_{l+1}) -\end{equation} -!et -where $\odot$ signifies element-wise multiplication. The expression for the weight gradients (ref{weightGradient}) is converted -into an outer product of the errors of layer $l+1$ and the outputs of layer $l$, - -!bt -\begin{equation} - g_{ij} = \delta_i y_j \Rightarrow \mathrm{G}_l = \vec{\delta}_{l+1} \vec{y}^T_l -\end{equation} -!et -where $\mathrm{G}_l$ is a matrix containing all gradients for the weights connecting layer $l$ and $l+1$. -The corresponding matrix for the biases is a $N_l \times 1$ column vector $\vec{H}_l$, - -!bt -\begin{equation} - \vec{H}_l = \vec{\updelta}_l -\end{equation} -!et -The set $\{\mathrm{G}_l, \vec{H}_l\}$ for $l=1,\dots,L+1$ thus make up the total gradient of the cost function (ref{quadraticCost}). - - -=== Training algorithm === - -We are now ready to express the complete training algorithm for a MLP with backpropagation using -matrix-vector notation. We introduce $\mathrm{G}_{l,i}$ and $\vec{H}_{l,i}$ as the weight gradient matrix and -the bias gradient vector for training example $i$. - - * Input a set of $n$ training examples. - - - - * Initialize the input layer: !bt \begin{equation} \vec{y}_0 = \vec{x} - -\end{equation} -!et - * Propagate the activity forward. For $l = 1,\dots,L+1$: !bt \begin{align} \vec{u}_l &= \mathrm{W}_l\vec{y}_{l-1} + \vec{b}_l \\ \vec{y}_l &= f_l(\vec{u}_l) label{forwardPropMatrix} - -\end{align} -!et - Store all vectors $\vec{u}_l$ and $\vec{y}_l$. - * Calculate and store the error in the output layer: !bt \begin{equation} \vec{\updelta}_{L+1} = \vec{Y} - \vec{y}_{L+1} - -\end{equation} -!et - * Backpropagate the error. For $l = L, L-1, \dots ,1$: !bt \begin{equation} \vec{\updelta}_l = \vec{y}^\prime_l \odot (\mathrm{W}^T_{l+1}\vec{\updelta}_{l+1}) \cdot label{backPropMatrix} - -\end{equation} -!et - Store all errors $\vec{\updelta}_l$. - * Compute and store the weight and bias gradients. \\ For $l = L+1, L, \dots ,1$: !bt \begin{equation} \mathrm{G}_{l,i} = \vec{\updelta}_l \vec{y}_{l-1}^T, \quad \vec{H}_{l,i} = \vec{\updelta}_l label{weightUpdate} - -\end{equation} -!et - - \item Calculate the total gradients of all training examples $i$. \\ For $l = L+1, L, \dots ,1$: !bt \begin{equation} \mathrm{G}_l = \sum_i^n \mathrm{G}_{l,i} \quad \mathrm{and} \quad \vec{H}_l = \sum_i^n \vec{H}_{l,i} - -\end{equation} -!et - \item Update weights and biases with an update rule of choice. - - -The value of $n$ depends on the learning paradigm that is used, described in \autoref{sec:gradientDescentVariants}.