lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a8288fa0df20e81facaf6a36079787ac2cffba9b
FYS-STK4155/doc/pub/week41/html/week41-solarized.html
T
Morten Hjorth-Jensen a8288fa0df added video
2024-10-07 13:19:45 +02:00

2753 lines
119 KiB
HTML
Raw Blame History

<!--
HTML file automatically generated from DocOnce source
(https://github.com/doconce/doconce/)
doconce format html week41.do.txt --pygments_html_style=perldoc --html_style=solarized3 --html_links_in_new_window --html_output=week41-solarized --no_mako
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Neural networks and constructing a neural network code">
<title>Week 41 Neural networks and constructing a neural network code</title>
<link href="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
<style type="text/css">
h1 {color: #b58900;} /* yellow */
/* h1 {color: #cb4b16;} orange */
/* h1 {color: #d33682;} magenta, the original choice of thomasf */
code { padding: 0px; background-color: inherit; }
pre {
border: 0pt solid #93a1a1;
box-shadow: none;
}
.alert-text-small { font-size: 80%; }
.alert-text-large { font-size: 130%; }
.alert-text-normal { font-size: 90%; }
.alert {
padding:8px 35px 8px 14px; margin-bottom:18px;
text-shadow:0 1px 0 rgba(255,255,255,0.5);
border:1px solid #93a1a1;
border-radius: 4px;
-webkit-border-radius: 4px;
-moz-border-radius: 4px;
color: #555;
background-color: #eee8d5;
background-position: 10px 5px;
background-repeat: no-repeat;
background-size: 38px;
padding-left: 55px;
width: 75%;
}
.alert-block {padding-top:14px; padding-bottom:14px}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_question.png); }
div { text-align: justify; text-justify: inter-word; }
.tab {
padding-left: 1.5em;
}
div.toc p,a {
line-height: 1.3;
margin-top: 1.1;
margin-bottom: 1.1;
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 41, October 7-11',
2,
None,
'plan-for-week-41-october-7-11'),
('Material for the lecture on Monday October 7, 2024',
2,
None,
'material-for-the-lecture-on-monday-october-7-2024'),
('Material for the active learning sessions on Tuesday and '
'Wednesday',
2,
None,
'material-for-the-active-learning-sessions-on-tuesday-and-wednesday'),
('Lecture Monday October 7',
2,
None,
'lecture-monday-october-7'),
('Introduction to Neural networks',
2,
None,
'introduction-to-neural-networks'),
('Artificial neurons', 2, None, 'artificial-neurons'),
('Neural network types', 2, None, 'neural-network-types'),
('Feed-forward neural networks',
2,
None,
'feed-forward-neural-networks'),
('Convolutional Neural Network',
2,
None,
'convolutional-neural-network'),
('Recurrent neural networks',
2,
None,
'recurrent-neural-networks'),
('Other types of networks', 2, None, 'other-types-of-networks'),
('Multilayer perceptrons', 2, None, 'multilayer-perceptrons'),
('Why multilayer perceptrons?',
2,
None,
'why-multilayer-perceptrons'),
('Illustration of a single perceptron model and a '
'multi-perceptron model',
2,
None,
'illustration-of-a-single-perceptron-model-and-a-multi-perceptron-model'),
('Examples of XOR, OR and AND gates',
2,
None,
'examples-of-xor-or-and-and-gates'),
('Does Logistic Regression do a better Job?',
2,
None,
'does-logistic-regression-do-a-better-job'),
('Adding Neural Networks', 2, None, 'adding-neural-networks'),
('Mathematics of deep learning',
2,
None,
'mathematics-of-deep-learning'),
('Reminder on books with hands-on material and codes',
2,
None,
'reminder-on-books-with-hands-on-material-and-codes'),
('Reading recommendations', 2, None, 'reading-recommendations'),
('Mathematics of deep learning and neural networks',
2,
None,
'mathematics-of-deep-learning-and-neural-networks'),
('Basics of an NN', 2, None, 'basics-of-an-nn'),
('Overarching view of a neural network',
2,
None,
'overarching-view-of-a-neural-network'),
('The optimization problem', 2, None, 'the-optimization-problem'),
('Parameters of neural networks',
2,
None,
'parameters-of-neural-networks'),
('Other ingredients of a neural network',
2,
None,
'other-ingredients-of-a-neural-network'),
('Other parameters', 2, None, 'other-parameters'),
('Universal approximation theorem',
2,
None,
'universal-approximation-theorem'),
('Some parallels from real analysis',
2,
None,
'some-parallels-from-real-analysis'),
('The approximation theorem in words',
2,
None,
'the-approximation-theorem-in-words'),
('More on the general approximation theorem',
2,
None,
'more-on-the-general-approximation-theorem'),
('Class of functions we can approximate',
2,
None,
'class-of-functions-we-can-approximate'),
('Setting up the equations for a neural network',
2,
None,
'setting-up-the-equations-for-a-neural-network'),
('Layout of a neural network with three hidden layers',
2,
None,
'layout-of-a-neural-network-with-three-hidden-layers'),
('Definitions', 2, None, 'definitions'),
('Inputs to the activation function',
2,
None,
'inputs-to-the-activation-function'),
('Derivatives and the chain rule',
2,
None,
'derivatives-and-the-chain-rule'),
('Derivative of the cost function',
2,
None,
'derivative-of-the-cost-function'),
('Simpler examples first, and automatic differentiation',
2,
None,
'simpler-examples-first-and-automatic-differentiation'),
('Reminder on the chain rule and gradients',
2,
None,
'reminder-on-the-chain-rule-and-gradients'),
('Multivariable functions', 2, None, 'multivariable-functions'),
('Automatic differentiation through examples',
2,
None,
'automatic-differentiation-through-examples'),
('Simple example', 2, None, 'simple-example'),
('Smarter way of evaluating the above function',
2,
None,
'smarter-way-of-evaluating-the-above-function'),
('Reducing the number of operations',
2,
None,
'reducing-the-number-of-operations'),
('Chain rule, forward and reverse modes',
2,
None,
'chain-rule-forward-and-reverse-modes'),
('Forward and reverse modes',
2,
None,
'forward-and-reverse-modes'),
('More complicated function',
2,
None,
'more-complicated-function'),
('Counting the number of floating point operations',
2,
None,
'counting-the-number-of-floating-point-operations'),
('Defining intermediate operations',
2,
None,
'defining-intermediate-operations'),
('New expression for the derivative',
2,
None,
'new-expression-for-the-derivative'),
('Final derivatives', 2, None, 'final-derivatives'),
('In general not this simple',
2,
None,
'in-general-not-this-simple'),
('Automatic differentiation',
2,
None,
'automatic-differentiation'),
('Chain rule', 2, None, 'chain-rule'),
('First network example, simple percepetron with one input',
2,
None,
'first-network-example-simple-percepetron-with-one-input'),
('Layout of a simple neural network with no hidden layer',
2,
None,
'layout-of-a-simple-neural-network-with-no-hidden-layer'),
('Optimizing the parameters',
2,
None,
'optimizing-the-parameters'),
('Adding a hidden layer', 2, None, 'adding-a-hidden-layer'),
('Layout of a simple neural network with one hidden layer',
2,
None,
'layout-of-a-simple-neural-network-with-one-hidden-layer'),
('The derivatives', 2, None, 'the-derivatives'),
('Important observations', 2, None, 'important-observations'),
('The training', 2, None, 'the-training'),
('Code example', 2, None, 'code-example'),
('Exercise 1: Including more data',
2,
None,
'exercise-1-including-more-data'),
('Simple neural network and the back propagation equations',
2,
None,
'simple-neural-network-and-the-back-propagation-equations'),
('Layout of a simple neural network with two input nodes, one '
'hidden layer and one output node',
2,
None,
'layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node'),
('The ouput layer', 2, None, 'the-ouput-layer'),
('Compact expressions', 2, None, 'compact-expressions'),
('Output layer', 2, None, 'output-layer'),
('Explicit derivatives', 2, None, 'explicit-derivatives'),
('Derivatives of the hidden layer',
2,
None,
'derivatives-of-the-hidden-layer'),
('Final expression', 2, None, 'final-expression'),
('Completing the list', 2, None, 'completing-the-list'),
('Final expressions for the biases of the hidden layer',
2,
None,
'final-expressions-for-the-biases-of-the-hidden-layer'),
('Gradient expressions', 2, None, 'gradient-expressions'),
('Exercise 2: Extended program',
2,
None,
'exercise-2-extended-program'),
('Getting serious, the back propagation equations for a neural '
'network',
2,
None,
'getting-serious-the-back-propagation-equations-for-a-neural-network'),
('Analyzing the last results',
2,
None,
'analyzing-the-last-results'),
('More considerations', 2, None, 'more-considerations'),
('Derivatives in terms of $z_j^L$',
2,
None,
'derivatives-in-terms-of-z-j-l'),
('Bringing it together', 2, None, 'bringing-it-together'),
('Final back propagating equation',
2,
None,
'final-back-propagating-equation'),
('Using the chain rule and summing over all $k$ entries',
2,
None,
'using-the-chain-rule-and-summing-over-all-k-entries'),
('Setting up the back propagation algorithm',
2,
None,
'setting-up-the-back-propagation-algorithm'),
('Setting up the back propagation algorithm, part 2',
2,
None,
'setting-up-the-back-propagation-algorithm-part-2'),
('Setting up the Back propagation algorithm, part 3',
2,
None,
'setting-up-the-back-propagation-algorithm-part-3'),
('Updating the gradients', 2, None, 'updating-the-gradients'),
('Activation functions', 3, None, 'activation-functions'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'activation-functions-logistic-and-hyperbolic-ones'),
('Relevance', 3, None, 'relevance'),
('Fine-tuning neural network hyperparameters',
2,
None,
'fine-tuning-neural-network-hyperparameters'),
('Hidden layers', 2, None, 'hidden-layers'),
('Vanishing gradients', 2, None, 'vanishing-gradients'),
('Exploding gradients', 2, None, 'exploding-gradients'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'is-the-logistic-activation-function-sigmoid-our-choice'),
('Logistic function as the root of problems',
2,
None,
'logistic-function-as-the-root-of-problems'),
('The derivative of the Logistic funtion',
2,
None,
'the-derivative-of-the-logistic-funtion'),
('Insights from the paper by Glorot and Bengio',
2,
None,
'insights-from-the-paper-by-glorot-and-bengio'),
('The RELU function family', 2, None, 'the-relu-function-family'),
('ELU function', 2, None, 'elu-function'),
('Which activation function should we use?',
2,
None,
'which-activation-function-should-we-use'),
('More on activation functions, output layers',
2,
None,
'more-on-activation-functions-output-layers'),
('Batch Normalization', 2, None, 'batch-normalization'),
('Dropout', 2, None, 'dropout'),
('Gradient Clipping', 2, None, 'gradient-clipping'),
('A top-down perspective on Neural networks',
2,
None,
'a-top-down-perspective-on-neural-networks'),
('More top-down perspectives',
2,
None,
'more-top-down-perspectives'),
('Limitations of supervised learning with deep networks',
2,
None,
'limitations-of-supervised-learning-with-deep-networks'),
('Limitations of NNs', 2, None, 'limitations-of-nns'),
('Homogeneous data', 2, None, 'homogeneous-data'),
('More limitations', 2, None, 'more-limitations')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
<center>
<h1>Week 41 Neural networks and constructing a neural network code</h1>
</center> <!-- document title -->
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b>
</center>
<!-- institution -->
<center>
<b>Department of Physics, University of Oslo</b>
</center>
<br>
<center>
<h4>Week 41</h4>
</center> <!-- date -->
<br>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="plan-for-week-41-october-7-11">Plan for week 41, October 7-11 </h2>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="material-for-the-lecture-on-monday-october-7-2024">Material for the lecture on Monday October 7, 2024 </h2>
<ol>
<li> Neural Networks, setting up the basic steps, from the simple perceptron model to the multi-layer perceptron model.</li>
<li> Building our own Feed-forward Neural Network</li>
<ul>
<li> Video of lecture notes at <a href="https://youtu.be/pMRUbf9E-gM" target="_blank"><tt>https://youtu.be/pMRUbf9E-gM</tt></a></li>
<li> Whiteboard notes at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOctober7.pdf" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2024/NotesOctober7.pdf</tt></a></li>
</ul>
</ol>
<div class="alert alert-block alert-block alert-text-normal">
<b>Readings and Videos:</b>
<p>
<ol>
<li> These lecture notes</li>
<li> Rashcka et al chapter 11</li>
<li> For neural networks we recommend Goodfellow et al chapter 6.
<ol type="a"></li>
<li> Neural Networks demystified at <a href="https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs" target="_blank"><tt>https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PLiaHhY2iBX9hdHaRr6b7XevZtgZRa1PoU&ab_channel=WelchLabs</tt></a></li>
</ol>
<li> Building Neural Networks from scratch at <a href="https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex" target="_blank"><tt>https://www.youtube.com/watch?v=Wo5dMEP_BbI&list=PLQVvvaa0QuDcjD5BAw2DxE6OF2tius3V3&ab_channel=sentdex</tt></a></li>
<li> Video on Neural Networks at <a href="https://www.youtube.com/watch?v=CqOfi41LfDw" target="_blank"><tt>https://www.youtube.com/watch?v=CqOfi41LfDw</tt></a></li>
<li> Video on the back propagation algorithm at <a href="https://www.youtube.com/watch?v=Ilg3gGewQ5U" target="_blank"><tt>https://www.youtube.com/watch?v=Ilg3gGewQ5U</tt></a></li>
</ol>
<p>We also recommend Michael Nielsen's intuitive approach to the neural networks and the universal approximation theorem, see the slides at <a href="http://neuralnetworksanddeeplearning.com/chap4.html" target="_blank"><tt>http://neuralnetworksanddeeplearning.com/chap4.html</tt></a>.</p>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="material-for-the-active-learning-sessions-on-tuesday-and-wednesday">Material for the active learning sessions on Tuesday and Wednesday </h2>
<ul>
<li> Exercise on writing your own stochastic gradient and gradient descent codes. This exercise continues next week with studies of automatic differentiation</li>
<li> One lecture at the beginning of each session on the material from weeks 39 and 40 and how to write your own gradient descent code</li>
<li> Discussion of project 2</li>
<li> Your task before the sessions: revisit the material from weeks 39 and 40 and in particular the material from week 40 on stochastic gradient descent</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lecture-monday-october-7">Lecture Monday October 7 </h2>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="introduction-to-neural-networks">Introduction to Neural networks </h2>
<p>Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="artificial-neurons">Artificial neurons </h2>
<p>The field of artificial neural networks has a long history of
development, and is closely connected with the advancement of computer
science and computers in general. A model of artificial neurons was
first developed by McCulloch and Pitts in 1943 to study signal
processing in the brain and has later been refined by others. The
general idea is to mimic neural networks in the human brain, which is
composed of billions of neurons that communicate with each other by
sending electrical signals. Each neuron accumulates its incoming
signals, which must exceed an activation threshold to yield an
output. If the threshold is not overcome, the neuron remains inactive,
i.e. has zero output.
</p>
<p>This behaviour has inspired a simple mathematical model for an artificial neuron.</p>
$$
\begin{equation}
y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)
\label{artificialNeuron}
\end{equation}
$$
<p>Here, the output \( y \) of the neuron is the value of its activation function, which have as input
a weighted sum of signals \( x_i, \dots ,x_n \) received by \( n \) other neurons.
</p>
<p>Conceptually, it is helpful to divide neural networks into four
categories:
</p>
<ol>
<li> general purpose neural networks for supervised learning,</li>
<li> neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs),</li>
<li> neural networks for sequential data such as Recurrent Neural Networks (RNNs), and</li>
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
<p>In natural science, DNNs and CNNs have already found numerous
applications. In statistical physics, they have been applied to detect
phase transitions in 2D Ising and Potts models, lattice gauge
theories, and different phases of polymers, or solving the
Navier-Stokes equation in weather forecasting. Deep learning has also
found interesting applications in quantum physics. Various quantum
phase transitions can be detected and studied using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive achievements to reveal the
potential of DNNs to facilitate the study of quantum systems.
</p>
<p>In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural.
</p>
<p>The applications are not limited to the natural sciences. There is a
plethora of applications in essentially all disciplines, from the
humanities to life science and medicine.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="neural-network-types">Neural network types </h2>
<p>An artificial neural network (ANN), is a computational model that
consists of layers of connected neurons, or nodes or units. We will
refer to these interchangeably as units or nodes, and sometimes as
neurons.
</p>
<p>It is supposed to mimic a biological nervous system by letting each
neuron interact with other neurons by sending signals in the form of
mathematical functions between layers. A wide variety of different
ANNs have been developed, but most of them consist of an input layer,
an output layer and eventual layers in-between, called <em>hidden
layers</em>. All layers can contain an arbitrary number of nodes, and each
connection between two nodes is associated with a weight variable.
</p>
<p>Neural networks (also called neural nets) are neural-inspired
nonlinear models for supervised learning. As we will see, neural nets
can be viewed as natural, more powerful extensions of supervised
learning methods such as linear and logistic regression and soft-max
methods we discussed earlier.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="feed-forward-neural-networks">Feed-forward neural networks </h2>
<p>The feed-forward neural network (FFNN) was the first and simplest type
of ANNs that were devised. In this network, the information moves in
only one direction: forward through the layers.
</p>
<p>Nodes are represented by circles, while the arrows display the
connections between the nodes, including the direction of information
flow. Additionally, each arrow corresponds to a weight variable
(figure to come). We observe that each node in a layer is connected
to <em>all</em> nodes in the subsequent layer, making this a so-called
<em>fully-connected</em> FFNN.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="convolutional-neural-network">Convolutional Neural Network </h2>
<p>A different variant of FFNNs are <em>convolutional neural networks</em>
(CNNs), which have a connectivity pattern inspired by the animal
visual cortex. Individual neurons in the visual cortex only respond to
stimuli from small sub-regions of the visual field, called a receptive
field. This makes the neurons well-suited to exploit the strong
spatially local correlation present in natural images. The response of
each neuron can be approximated mathematically as a convolution
operation. (figure to come)
</p>
<p>Convolutional neural networks emulate the behaviour of neurons in the
visual cortex by enforcing a <em>local</em> connectivity pattern between
nodes of adjacent layers: Each node in a convolutional layer is
connected only to a subset of the nodes in the previous layer, in
contrast to the fully-connected FFNN. Often, CNNs consist of several
convolutional layers that learn local features of the input, with a
fully-connected layer at the end, which gathers all the local data and
produces the outputs. They have wide applications in image and video
recognition.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="recurrent-neural-networks">Recurrent neural networks </h2>
<p>So far we have only mentioned ANNs where information flows in one
direction: forward. <em>Recurrent neural networks</em> on the other hand,
have connections between nodes that form directed <em>cycles</em>. This
creates a form of internal memory which are able to capture
information on what has been calculated before; the output is
dependent on the previous computations. Recurrent NNs make use of
sequential information by performing the same task for every element
in a sequence, where each element depends on previous elements. An
example of such information is sentences, making recurrent NNs
especially well-suited for handwriting and speech recognition.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="other-types-of-networks">Other types of networks </h2>
<p>There are many other kinds of ANNs that have been developed. One type
that is specifically designed for interpolation in multidimensional
space is the radial basis function (RBF) network. RBFs are typically
made up of three layers: an input layer, a hidden layer with
non-linear radial symmetric activation functions and a linear output
layer (''linear'' here means that each node in the output layer has a
linear activation function). The layers are normally fully-connected
and there are no cycles, thus RBFs can be viewed as a type of
fully-connected FFNN. They are however usually treated as a separate
type of NN due the unusual activation functions.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="multilayer-perceptrons">Multilayer perceptrons </h2>
<p>One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
</p>
<p>Such networks are often called <em>multilayer perceptrons</em> (MLPs).</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-multilayer-perceptrons">Why multilayer perceptrons? </h2>
<p>According to the <em>Universal approximation theorem</em>, a feed-forward
neural network with just a single hidden layer containing a finite
number of neurons can approximate a continuous multidimensional
function to arbitrary accuracy, assuming the activation function for
the hidden layer is a <b>non-constant, bounded and
monotonically-increasing continuous function</b>.
</p>
<p>Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="illustration-of-a-single-perceptron-model-and-a-multi-perceptron-model">Illustration of a single perceptron model and a multi-perceptron model </h2>
<center> <!-- FIGURE -->
<hr class="figure">
<center>
<p class="caption">Figure 1: In a) we show a single perceptron model while in b) we dispay a network with two hidden layers, an input layer and an output layer. </p>
</center>
<p><img src="figures/nns.png" width="600" align="bottom"></p>
</center>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="examples-of-xor-or-and-and-gates">Examples of XOR, OR and AND gates </h2>
<p>Let us first try to fit various gates using standard linear
regression. The gates we are thinking of are the classical XOR, OR and
AND gates, well-known elements in computer science. The tables here
show how we can set up the inputs \( x_1 \) and \( x_2 \) in order to yield a
specific target \( y_i \).
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">Simple code that tests XOR, OR and AND gates with linear regression</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The X.TX matrix:{</span>X.T @ X<span style="color: #CD5555">}&quot;</span>)
Xinv = np.linalg.pinv(X.T @ X)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The invers of X.TX matrix:{</span>Xinv<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># The XOR gate </span>
yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>])
ThetaXOR = Xinv @ X.T @ yXOR
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The values of theta for the XOR gate:{</span>ThetaXOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The linear regression prediction for the XOR gate:{</span>X @ ThetaXOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># The OR gate </span>
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
ThetaOR = Xinv @ X.T @ yOR
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The values of theta for the OR gate:{</span>ThetaOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The linear regression prediction for the OR gate:{</span>X @ ThetaOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># The OR gate </span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
ThetaAND = Xinv @ X.T @ yAND
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The values of theta for the AND gate:{</span>ThetaAND<span style="color: #CD5555">}&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The linear regression prediction for the AND gate:{</span>X @ ThetaAND<span style="color: #CD5555">}&quot;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>What is happening here?</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="does-logistic-regression-do-a-better-job">Does Logistic Regression do a better Job? </h2>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">Simple code that tests XOR and OR gates with linear regression</span>
<span style="color: #CD5555">and logistic regression</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LogisticRegression
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># Design matrix</span>
X = np.array([ [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>], [<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>],[<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>]],dtype=np.float64)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The X.TX matrix:{</span>X.T @ X<span style="color: #CD5555">}&quot;</span>)
Xinv = np.linalg.pinv(X.T @ X)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The invers of X.TX matrix:{</span>Xinv<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># The XOR gate </span>
yXOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">0</span>])
ThetaXOR = Xinv @ X.T @ yXOR
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The values of theta for the XOR gate:{</span>ThetaXOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The linear regression prediction for the XOR gate:{</span>X @ ThetaXOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># The OR gate </span>
yOR = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span> ,<span style="color: #B452CD">1</span>, <span style="color: #B452CD">1</span>])
ThetaOR = Xinv @ X.T @ yOR
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The values of theta for the OR gate:{</span>ThetaOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The linear regression prediction for the OR gate:{</span>X @ ThetaOR<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># The OR gate </span>
yAND = np.array( [ <span style="color: #B452CD">0</span>, <span style="color: #B452CD">0</span> ,<span style="color: #B452CD">0</span>, <span style="color: #B452CD">1</span>])
ThetaAND = Xinv @ X.T @ yAND
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The values of theta for the AND gate:{</span>ThetaAND<span style="color: #CD5555">}&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;The linear regression prediction for the AND gate:{</span>X @ ThetaAND<span style="color: #CD5555">}&quot;</span>)
<span style="color: #228B22"># Now we change to logistic regression</span>
<span style="color: #228B22"># Logistic Regression</span>
logreg = LogisticRegression()
logreg.fit(X, yOR)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Test set accuracy with Logistic Regression for OR gate: {:.2f}&quot;</span>.format(logreg.score(X,yOR)))
logreg.fit(X, yXOR)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Test set accuracy with Logistic Regression for XOR gate: {:.2f}&quot;</span>.format(logreg.score(X,yXOR)))
logreg.fit(X, yAND)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Test set accuracy with Logistic Regression for AND gate: {:.2f}&quot;</span>.format(logreg.score(X,yAND)))
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Not exactly impressive, but somewhat better.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="adding-neural-networks">Adding Neural Networks </h2>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #228B22"># and now neural networks with Scikit-Learn and the XOR</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.neural_network</span> <span style="color: #8B008B; font-weight: bold">import</span> MLPClassifier
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.datasets</span> <span style="color: #8B008B; font-weight: bold">import</span> make_classification
X, yXOR = make_classification(n_samples=<span style="color: #B452CD">100</span>, random_state=<span style="color: #B452CD">1</span>)
FFNN = MLPClassifier(random_state=<span style="color: #B452CD">1</span>, max_iter=<span style="color: #B452CD">300</span>).fit(X, yXOR)
FFNN.predict_proba(X)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">f&quot;Test set accuracy with Feed Forward Neural Network for XOR gate:{</span>FFNN.score(X, yXOR)<span style="color: #CD5555">}&quot;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematics-of-deep-learning">Mathematics of deep learning </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Two recent books online</b>
<p>
<ol>
<li> <a href="https://arxiv.org/abs/2105.04026" target="_blank">The Modern Mathematics of Deep Learning, by Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen</a>, published as <a href="https://doi.org/10.1017/9781009025096.002" target="_blank">Mathematical Aspects of Deep Learning, pp. 1-111. Cambridge University Press, 2022</a></li>
<li> <a href="https://doi.org/10.48550/arXiv.2310.20360" target="_blank">Mathematical Introduction to Deep Learning: Methods, Implementations, and Theory, Arnulf Jentzen, Benno Kuckuck, Philippe von Wurstemberger</a></li>
</ol>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="reminder-on-books-with-hands-on-material-and-codes">Reminder on books with hands-on material and codes </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<ul>
<li> <a href="https://sebastianraschka.com/blog/2022/ml-pytorch-book.html" target="_blank">Sebastian Rashcka et al, Machine learning with Sickit-Learn and PyTorch</a></li>
</ul>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="reading-recommendations">Reading recommendations </h2>
<ol>
<li> Rashkca et al., chapter 11, jupyter-notebook sent separately, from <a href="https://github.com/rasbt/machine-learning-book" target="_blank">GitHub</a></li>
<li> Goodfellow et al, chapter 6 and 7 contain most of the neural network background.</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematics-of-deep-learning-and-neural-networks">Mathematics of deep learning and neural networks </h2>
<p>Neural networks, in its so-called feed-forward form, where each
iterations contains a feed-forward stage and a back-propgagation
stage, consist of series of affine matrix-matrix and matrix-vector
multiplications. The unknown parameters (the so-called biases and
weights which deternine the architecture of a neural network), are
uptaded iteratively using the so-called back-propagation algorithm.
This algorithm corresponds to the so-called reverse mode of
automatic differentation.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="basics-of-an-nn">Basics of an NN </h2>
<p>A neural network consists of a series of hidden layers, in addition to
the input and output layers. Each layer \( l \) has a set of parameters
\( \boldsymbol{\Theta}^{(l)}=(\boldsymbol{W}^{(l)},\boldsymbol{b}^{(l)}) \) which are related to the
parameters in other layers through a series of affine transformations,
for a standard NN these are matrix-matrix and matrix-vector
multiplications. For all layers we will simply use a collective variable \( \boldsymbol{\Theta} \).
</p>
<p>It consist of two basic steps:</p>
<ol>
<li> a feed forward stage which takes a given input and produces a final output which is compared with the target values through our cost/loss function.</li>
<li> a back-propagation state where the unknown parameters \( \boldsymbol{\Theta} \) are updated through the optimization of the their gradients. The expressions for the gradients are obtained via the chain rule, starting from the derivative of the cost/function.</li>
</ol>
<p>These two steps make up one iteration. This iterative process is continued till we reach an eventual stopping criterion.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="overarching-view-of-a-neural-network">Overarching view of a neural network </h2>
<p>The architecture of a neural network defines our model. This model
aims at describing some function \( f(\boldsymbol{x} \) which represents
some final result (outputs or tagrget values) given a specific inpput
\( \boldsymbol{x} \). Note that here \( \boldsymbol{y} \) and \( \boldsymbol{x} \) are not limited to be
vectors.
</p>
<p>The architecture consists of</p>
<ol>
<li> An input and an output layer where the input layer is defined by the inputs \( \boldsymbol{x} \). The output layer produces the model ouput \( \boldsymbol{\tilde{y}} \) which is compared with the target value \( \boldsymbol{y} \)</li>
<li> A given number of hidden layers and neurons/nodes/units for each layer (this may vary)</li>
<li> A given activation function \( \sigma(\boldsymbol{z}) \) with arguments \( \boldsymbol{z} \) to be defined below. The activation functions may differ from layer to layer.</li>
<li> The last layer, normally called <b>output</b> layer has normally an activation function tailored to the specific problem</li>
<li> Finally we define a so-called cost or loss function which is used to gauge the quality of our model.</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-optimization-problem">The optimization problem </h2>
<p>The cost function is a function of the unknown parameters
\( \boldsymbol{\Theta} \) where the latter is a container for all possible
parameters needed to define a neural network
</p>
<p>If we are dealing with a regression task a typical cost/loss function
is the mean squared error
</p>
$$
C(\boldsymbol{\Theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
<p>This function represents one of many possible ways to define
the so-called cost function. Note that here we have assumed a linear dependence in terms of the paramters \( \boldsymbol{\Theta} \). This is in general not the case.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="parameters-of-neural-networks">Parameters of neural networks </h2>
<p>For neural networks the parameters
\( \boldsymbol{\Theta} \) are given by the so-called weights and biases (to be
defined below).
</p>
<p>The weights are given by matrix elements \( w_{ij}^{(l)} \) where the
superscript indicates the layer number. The biases are typically given
by vector elements representing each single node of a given layer,
that is \( b_j^{(l)} \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="other-ingredients-of-a-neural-network">Other ingredients of a neural network </h2>
<p>Having defined the architecture of a neural network, the optimization
of the cost function with respect to the parameters \( \boldsymbol{\Theta} \),
involves the calculations of gradients and their optimization. The
gradients represent the derivatives of a multidimensional object and
are often approximated by various gradient methods, including
</p>
<ol>
<li> various quasi-Newton methods,</li>
<li> plain gradient descent (GD) with a constant learning rate \( \eta \),</li>
<li> GD with momentum and other approximations to the learning rates such as</li>
<ul>
<li> Adapative gradient (ADAgrad)</li>
<li> Root mean-square propagation (RMSprop)</li>
<li> Adaptive gradient with momentum (ADAM) and many other</li>
</ul>
<li> Stochastic gradient descent and various families of learning rate approximations</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="other-parameters">Other parameters </h2>
<p>In addition to the above, there are often additional hyperparamaters
which are included in the setup of a neural network. These will be
discussed below.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="universal-approximation-theorem">Universal approximation theorem </h2>
<p>The universal approximation theorem plays a central role in deep
learning. <a href="https://link.springer.com/article/10.1007/BF02551274" target="_blank">Cybenko (1989)</a> showed
the following:
</p>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>Let \( \sigma \) be any continuous sigmoidal function such that</p>
$$
\sigma(z) = \left\{\begin{array}{cc} 1 & z\rightarrow \infty\\ 0 & z \rightarrow -\infty \end{array}\right.
$$
<p>Given a continuous and deterministic function \( F(\boldsymbol{x}) \) on the unit
cube in \( d \)-dimensions \( F\in [0,1]^d \), \( x\in [0,1]^d \) and a parameter
\( \epsilon >0 \), there is a one-layer (hidden) neural network
\( f(\boldsymbol{x};\boldsymbol{\Theta}) \) with \( \boldsymbol{\Theta}=(\boldsymbol{W},\boldsymbol{b}) \) and \( \boldsymbol{W}\in
\mathbb{R}^{m\times n} \) and \( \boldsymbol{b}\in \mathbb{R}^{n} \), for which
</p>
$$
\vert F(\boldsymbol{x})-f(\boldsymbol{x};\boldsymbol{\Theta})\vert < \epsilon \hspace{0.1cm} \forall \boldsymbol{x}\in[0,1]^d.
$$
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="some-parallels-from-real-analysis">Some parallels from real analysis </h2>
<p>For those of you familiar with for example the <a href="https://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theorem" target="_blank">Stone-Weierstrass
theorem</a>
for polynomial approximations or the convergence criterion for Fourier
series, there are similarities in the derivation of the proof for
neural networks.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-approximation-theorem-in-words">The approximation theorem in words </h2>
<p><b>Any continuous function \( y=F(\boldsymbol{x}) \) supported on the unit cube in
\( d \)-dimensions can be approximated by a one-layer sigmoidal network to
arbitrary accuracy.</b>
</p>
<p><a href="https://www.sciencedirect.com/science/article/abs/pii/089360809190009T" target="_blank">Hornik (1991)</a> extended the theorem by letting any non-constant, bounded activation function to be included using that the expectation value</p>
$$
\mathbb{E}[\vert F(\boldsymbol{x})\vert^2] =\int_{\boldsymbol{x}\in D} \vert F(\boldsymbol{x})\vert^2p(\boldsymbol{x})d\boldsymbol{x} < \infty.
$$
<p>Then we have</p>
$$
\mathbb{E}[\vert F(\boldsymbol{x})-f(\boldsymbol{x};\boldsymbol{\Theta})\vert^2] =\int_{\boldsymbol{x}\in D} \vert F(\boldsymbol{x})-f(\boldsymbol{x};\boldsymbol{\Theta})\vert^2p(\boldsymbol{x})d\boldsymbol{x} < \epsilon.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-the-general-approximation-theorem">More on the general approximation theorem </h2>
<p>None of the proofs give any insight into the relation between the
number of of hidden layers and nodes and the approximation error
\( \epsilon \), nor the magnitudes of \( \boldsymbol{W} \) and \( \boldsymbol{b} \).
</p>
<p>Neural networks (NNs) have what we may call a kind of universality no matter what function we want to compute.</p>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>It does not mean that an NN can be used to exactly compute any function. Rather, we get an approximation that is as good as we want. </p>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="class-of-functions-we-can-approximate">Class of functions we can approximate </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>The class of functions that can be approximated are the continuous ones.
If the function \( F(\boldsymbol{x}) \) is discontinuous, it won't in general be possible to approximate it. However, an NN may still give an approximation even if we fail in some points.
</p>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-equations-for-a-neural-network">Setting up the equations for a neural network </h2>
<p>The questions we want to ask are how do changes in the biases and the
weights in our network change the cost function and how can we use the
final output to modify the weights and biases?
</p>
<p>To derive these equations let us start with a plain regression problem
and define our cost function as
</p>
$$
{\cal C}(\boldsymbol{\Theta}) = \frac{1}{2}\sum_{i=1}^n\left(y_i - \tilde{y}_i\right)^2,
$$
<p>where the $y_i$s are our \( n \) targets (the values we want to
reproduce), while the outputs of the network after having propagated
all inputs \( \boldsymbol{x} \) are given by \( \boldsymbol{\tilde{y}}_i \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="layout-of-a-neural-network-with-three-hidden-layers">Layout of a neural network with three hidden layers </h2>
<br/><br/>
<center>
<p><img src="figures/nn1.png" width="900" align="bottom"></p>
</center>
<br/><br/>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="definitions">Definitions </h2>
<p>With our definition of the targets \( \boldsymbol{y} \), the outputs of the
network \( \boldsymbol{\tilde{y}} \) and the inputs \( \boldsymbol{x} \) we
define now the activation \( z_j^l \) of node/neuron/unit \( j \) of the
\( l \)-th layer as a function of the bias, the weights which add up from
the previous layer \( l-1 \) and the forward passes/outputs
\( \hat{a}^{l-1} \) from the previous layer as
</p>
$$
z_j^l = \sum_{i=1}^{M_{l-1}}w_{ij}^la_i^{l-1}+b_j^l,
$$
<p>where \( b_k^l \) are the biases from layer \( l \). Here \( M_{l-1} \)
represents the total number of nodes/neurons/units of layer \( l-1 \). The
figure in the whiteboard notes illustrates this equation. We can rewrite this in a more
compact form as the matrix-vector products we discussed earlier,
</p>
$$
\hat{z}^l = \left(\hat{W}^l\right)^T\hat{a}^{l-1}+\hat{b}^l.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="inputs-to-the-activation-function">Inputs to the activation function </h2>
<p>With the activation values \( \boldsymbol{z}^l \) we can in turn define the
output of layer \( l \) as \( \boldsymbol{a}^l = f(\boldsymbol{z}^l) \) where \( f \) is our
activation function. In the examples here we will use the sigmoid
function discussed in our logistic regression lectures. We will also use the same activation function \( f \) for all layers
and their nodes. It means we have
</p>
$$
a_j^l = \sigma(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="derivatives-and-the-chain-rule">Derivatives and the chain rule </h2>
<p>From the definition of the activation \( z_j^l \) we have</p>
$$
\frac{\partial z_j^l}{\partial w_{ij}^l} = a_i^{l-1},
$$
<p>and</p>
$$
\frac{\partial z_j^l}{\partial a_i^{l-1}} = w_{ji}^l.
$$
<p>With our definition of the activation function we have that (note that this function depends only on \( z_j^l \))</p>
$$
\frac{\partial a_j^l}{\partial z_j^{l}} = a_j^l(1-a_j^l)=\sigma(z_j^l)(1-\sigma(z_j^l)).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="derivative-of-the-cost-function">Derivative of the cost function </h2>
<p>With these definitions we can now compute the derivative of the cost function in terms of the weights.</p>
<p>Let us specialize to the output layer \( l=L \). Our cost function is</p>
$$
{\cal C}(\boldsymbol{\Theta}^L) = \frac{1}{2}\sum_{i=1}^n\left(y_i - \tilde{y}_i\right)^2=\frac{1}{2}\sum_{i=1}^n\left(a_i^L - y_i\right)^2,
$$
<p>The derivative of this function with respect to the weights is</p>
$$
\frac{\partial{\cal C}(\boldsymbol{\Theta}^L)}{\partial w_{jk}^L} = \left(a_j^L - y_j\right)\frac{\partial a_j^L}{\partial w_{jk}^{L}},
$$
<p>The last partial derivative can easily be computed and reads (by applying the chain rule)</p>
$$
\frac{\partial a_j^L}{\partial w_{jk}^{L}} = \frac{\partial a_j^L}{\partial z_{j}^{L}}\frac{\partial z_j^L}{\partial w_{jk}^{L}}=a_j^L(1-a_j^L)a_k^{L-1}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simpler-examples-first-and-automatic-differentiation">Simpler examples first, and automatic differentiation </h2>
<p>In order to understand the back propagation algorithm and its
derivation (an implementation of the chain rule), let us first digress
with some simple examples. These examples are also meant to motivate
the link with back propagation and <a href="https://en.wikipedia.org/wiki/Automatic_differentiation" target="_blank">automatic differentiation</a>.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="reminder-on-the-chain-rule-and-gradients">Reminder on the chain rule and gradients </h2>
<p>If we have a multivariate function \( f(x,y) \) where \( x=x(t) \) and \( y=y(t) \) are functions of a variable \( t \), we have that the gradient of \( f \) with respect to \( t \) (without the explicit unit vector components)</p>
$$
\frac{df}{dt} = \begin{bmatrix}\frac{\partial f}{\partial x} & \frac{\partial f}{\partial y} \end{bmatrix} \begin{bmatrix}\frac{\partial x}{\partial t} \\ \frac{\partial y}{\partial t} \end{bmatrix}=\frac{\partial f}{\partial x} \frac{\partial x}{\partial t} +\frac{\partial f}{\partial y} \frac{\partial y}{\partial t}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="multivariable-functions">Multivariable functions </h2>
<p>If we have a multivariate function \( f(x,y) \) where \( x=x(t,s) \) and \( y=y(t,s) \) are functions of the variables \( t \) and \( s \), we have that the partial derivatives</p>
$$
\frac{\partial f}{\partial s}=\frac{\partial f}{\partial x}\frac{\partial x}{\partial s}+\frac{\partial f}{\partial y}\frac{\partial y}{\partial s},
$$
<p>and</p>
$$
\frac{\partial f}{\partial t}=\frac{\partial f}{\partial x}\frac{\partial x}{\partial t}+\frac{\partial f}{\partial y}\frac{\partial y}{\partial t}.
$$
<p>the gradient of \( f \) with respect to \( t \) and \( s \) (without the explicit unit vector components)</p>
$$
\frac{df}{d(s,t)} = \begin{bmatrix}\frac{\partial f}{\partial x} & \frac{\partial f}{\partial y} \end{bmatrix} \begin{bmatrix}\frac{\partial x}{\partial s} &\frac{\partial x}{\partial t} \\ \frac{\partial y}{\partial s} & \frac{\partial y}{\partial t} \end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="automatic-differentiation-through-examples">Automatic differentiation through examples </h2>
<p>A great introduction to automatic differentiation is given by Baydin et al., see <a href="https://arxiv.org/abs/1502.05767" target="_blank"><tt>https://arxiv.org/abs/1502.05767</tt></a>.</p>
<p>Automatic differentiation is a represented by a repeated application
of the chain rule on well-known functions and allows for the
calculation of derivatives to numerical precision. It is not the same
as the calculation of symbolic derivatives via for example SymPy, nor
does it use approximative formulae based on Taylor-expansions of a
function around a given value. The latter are error prone due to
truncation errors and values of the step size \( \Delta \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simple-example">Simple example </h2>
<p>Our first example is rather simple,</p>
$$
f(x) =\exp{x^2},
$$
<p>with derivative</p>
$$
f'(x) =2x\exp{x^2}.
$$
<p>We can use SymPy to extract the pertinent lines of Python code through the following simple example</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">__future__</span> <span style="color: #8B008B; font-weight: bold">import</span> division
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sympy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
x = symbols(<span style="color: #CD5555">&#39;x&#39;</span>)
expr = exp(x*x)
simplify(expr)
derivative = diff(expr,x)
<span style="color: #658b00">print</span>(python(expr))
<span style="color: #658b00">print</span>(python(derivative))
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="smarter-way-of-evaluating-the-above-function">Smarter way of evaluating the above function </h2>
<p>If we study this function, we note that we can reduce the number of operations by introducing an intermediate variable</p>
$$
a = x^2,
$$
<p>leading to </p>
$$
f(x) = f(a(x)) = b= \exp{a}.
$$
<p>We now assume that all operations can be counted in terms of equal
floating point operations. This means that in order to calculate
\( f(x) \) we need first to square \( x \) and then compute the exponential. We
have thus two floating point operations only.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="reducing-the-number-of-operations">Reducing the number of operations </h2>
<p>With the introduction of a precalculated quantity \( a \) and thereby \( f(x) \) we have that the derivative can be written as</p>
$$
f'(x) = 2xb,
$$
<p>which reduces the number of operations from four in the orginal
expression to two. This means that if we need to compute \( f(x) \) and
its derivative (a common task in optimizations), we have reduced the
number of operations from six to four in total.
</p>
<p><b>Note</b> that the usage of a symbolic software like SymPy does not
include such simplifications and the calculations of the function and
the derivatives yield in general more floating point operations.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="chain-rule-forward-and-reverse-modes">Chain rule, forward and reverse modes </h2>
<p>In the above example we have introduced the variables \( a \) and \( b \), and our function is</p>
$$
f(x) = f(a(x)) = b= \exp{a},
$$
<p>with \( a=x^2 \). We can decompose the derivative of \( f \) with respect to \( x \) as</p>
$$
\frac{df}{dx}=\frac{df}{db}\frac{db}{da}\frac{da}{dx}.
$$
<p>We note that since \( b=f(x) \) that</p>
$$
\frac{df}{db}=1,
$$
<p>leading to</p>
$$
\frac{df}{dx}=\frac{db}{da}\frac{da}{dx}=2x\exp{x^2},
$$
<p>as before.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="forward-and-reverse-modes">Forward and reverse modes </h2>
<p>We have that </p>
$$
\frac{df}{dx}=\frac{df}{db}\frac{db}{da}\frac{da}{dx},
$$
<p>which we can rewrite either as</p>
$$
\frac{df}{dx}=\left[\frac{df}{db}\frac{db}{da}\right]\frac{da}{dx},
$$
<p>or</p>
$$
\frac{df}{dx}=\frac{df}{db}\left[\frac{db}{da}\frac{da}{dx}\right].
$$
<p>The first expression is called reverse mode (or back propagation)
since we start by evaluating the derivatives at the end point and then
propagate backwards. This is the standard way of evaluating
derivatives (gradients) when optimizing the parameters of a neural
network. In the context of deep learning this is computationally
more efficient since the output of a neural network consists of either
one or some few other output variables.
</p>
<p>The second equation defines the so-called <b>forward mode</b>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-complicated-function">More complicated function </h2>
<p>We increase our ambitions and introduce a slightly more complicated function</p>
$$
f(x) =\sqrt{x^2+exp{x^2}},
$$
<p>with derivative</p>
$$
f'(x) =\frac{x(1+\exp{x^2})}{\sqrt{x^2+exp{x^2}}}.
$$
<p>The corresponding SymPy code reads</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">__future__</span> <span style="color: #8B008B; font-weight: bold">import</span> division
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sympy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
x = symbols(<span style="color: #CD5555">&#39;x&#39;</span>)
expr = sqrt(x*x+exp(x*x))
simplify(expr)
derivative = diff(expr,x)
<span style="color: #658b00">print</span>(python(expr))
<span style="color: #658b00">print</span>(python(derivative))
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="counting-the-number-of-floating-point-operations">Counting the number of floating point operations </h2>
<p>A simple count of operations shows that we need five operations for
the function itself and ten for the derivative. Fifteen operations in total if we wish to proceed with the above codes.
</p>
<p>Can we reduce this to
say half the number of operations?
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="defining-intermediate-operations">Defining intermediate operations </h2>
<p>We can indeed reduce the number of operation to half of those listed in the brute force approach above.
We define the following quantities
</p>
$$
a = x^2,
$$
<p>and</p>
$$
b = \exp{x^2} = \exp{a},
$$
<p>and</p>
$$
c= a+b,
$$
<p>and</p>
$$
d=f(x)=\sqrt{c}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="new-expression-for-the-derivative">New expression for the derivative </h2>
<p>With these definitions we obtain the following partial derivatives </p>
$$
\frac{\partial a}{\partial x} = 2x,
$$
<p>and</p>
$$
\frac{\partial b}{\partial a} = \exp{a},
$$
<p>and</p>
$$
\frac{\partial c}{\partial a} = 1,
$$
<p>and</p>
$$
\frac{\partial c}{\partial b} = 1,
$$
<p>and</p>
$$
\frac{\partial d}{\partial c} = \frac{1}{2\sqrt{c}},
$$
<p>and finally</p>
$$
\frac{\partial f}{\partial d} = 1.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="final-derivatives">Final derivatives </h2>
<p>Our final derivatives are thus</p>
$$
\frac{\partial f}{\partial c} = \frac{\partial f}{\partial d} \frac{\partial d}{\partial c} = \frac{1}{2\sqrt{c}},
$$
$$
\frac{\partial f}{\partial b} = \frac{\partial f}{\partial c} \frac{\partial c}{\partial b} = \frac{1}{2\sqrt{c}},
$$
$$
\frac{\partial f}{\partial a} = \frac{\partial f}{\partial c} \frac{\partial c}{\partial a}+
\frac{\partial f}{\partial b} \frac{\partial b}{\partial a} = \frac{1+\exp{a}}{2\sqrt{c}},
$$
<p>and finally </p>
$$
\frac{\partial f}{\partial x} = \frac{\partial f}{\partial a} \frac{\partial a}{\partial x} = \frac{x(1+\exp{a})}{\sqrt{c}},
$$
<p>which is just</p>
$$
\frac{\partial f}{\partial x} = \frac{x(1+b)}{d},
$$
<p>and requires only three operations if we can reuse all intermediate variables.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="in-general-not-this-simple">In general not this simple </h2>
<p>In general, see the generalization below, unless we can obtain simple
analytical expressions which we can simplify further, the final
implementation of automatic differentiation involves repeated
calculations (and thereby operations) of derivatives of elementary
functions.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="automatic-differentiation">Automatic differentiation </h2>
<p>We can make this example more formal. Automatic differentiation is a
formalization of the previous example (see graph).
</p>
<p>We define \( \boldsymbol{x}\in x_1,\dots, x_l \) input variables to a given function \( f(\boldsymbol{x}) \) and \( x_{l+1},\dots, x_L \) intermediate variables.</p>
<p>In the above example we have only one input variable, \( l=1 \) and four intermediate variables, that is</p>
$$
\begin{bmatrix} x_1=x & x_2 = x^2=a & x_3 =\exp{a}= b & x_4=c=a+b & x_5 = \sqrt{c}=d \end{bmatrix}.
$$
<p>Furthemore, for \( i=l+1, \dots, L \) (here \( i=2,3,4,5 \) and \( f=x_L=d \)), we
define the elementary functions \( g_i(x_{Pa(x_i)}) \) where \( x_{Pa(x_i)} \) are the parent nodes of the variable \( x_i \).
</p>
<p>In our case, we have for example for \( x_3=g_3(x_{Pa(x_i)})=\exp{a} \), that \( g_3=\exp{()} \) and \( x_{Pa(x_3)}=a \).</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="chain-rule">Chain rule </h2>
<p>We can now compute the gradients by back-propagating the derivatives using the chain rule.
We have defined
</p>
$$
\frac{\partial f}{\partial x_L} = 1,
$$
<p>which allows us to find the derivatives of the various variables \( x_i \) as</p>
$$
\frac{\partial f}{\partial x_i} = \sum_{x_j:x_i\in Pa(x_j)}\frac{\partial f}{\partial x_j} \frac{\partial x_j}{\partial x_i}=\sum_{x_j:x_i\in Pa(x_j)}\frac{\partial f}{\partial x_j} \frac{\partial g_j}{\partial x_i}.
$$
<p>Whenever we have a function which can be expressed as a computation
graph and the various functions can be expressed in terms of
elementary functions that are differentiable, then automatic
differentiation works. The functions may not need to be elementary
functions, they could also be computer programs, although not all
programs can be automatically differentiated.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="first-network-example-simple-percepetron-with-one-input">First network example, simple percepetron with one input </h2>
<p>As yet another example we define now a simple perceptron model with
all quantities given by scalars. We consider only one input variable
\( x \) and one target value \( y \). We define an activation function
\( \sigma_1 \) which takes as input
</p>
$$
z_1 = w_1x+b_1,
$$
<p>where \( w_1 \) is the weight and \( b_1 \) is the bias. These are the
parameters we want to optimize. The output is \( a_1=\sigma(z_1) \) (see
graph from whiteboard notes). This output is then fed into the
<b>cost/loss</b> function, which we here for the sake of simplicity just
define as the squared error
</p>
$$
C(x;w_1,b_1)=\frac{1}{2}(a_1-y)^2.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="layout-of-a-simple-neural-network-with-no-hidden-layer">Layout of a simple neural network with no hidden layer </h2>
<br/><br/>
<center>
<p><img src="figures/simplenn1.png" width="900" align="bottom"></p>
</center>
<br/><br/>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="optimizing-the-parameters">Optimizing the parameters </h2>
<p>In setting up the feed forward and back propagation parts of the
algorithm, we need now the derivative of the various variables we want
to train.
</p>
<p>We need</p>
$$
\frac{\partial C}{\partial w_1} \hspace{0.1cm}\mathrm{and}\hspace{0.1cm}\frac{\partial C}{\partial b_1}.
$$
<p>Using the chain rule we find </p>
$$
\frac{\partial C}{\partial w_1}=\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial w_1}=(a_1-y)\sigma_1'x,
$$
<p>and</p>
$$
\frac{\partial C}{\partial b_1}=\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial b_1}=(a_1-y)\sigma_1',
$$
<p>which we later will just define as</p>
$$
\frac{\partial C}{\partial a_1}\frac{\partial a_1}{\partial z_1}=\delta_1.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="adding-a-hidden-layer">Adding a hidden layer </h2>
<p>We change our simple model to (see graph)
a network with just one hidden layer but with scalar variables only.
</p>
<p>Our output variable changes to \( a_2 \) and \( a_1 \) is now the output from the hidden node and \( a_0=x \).
We have then
</p>
$$
z_1 = w_1a_0+b_1 \hspace{0.1cm} \wedge a_1 = \sigma_1(z_1),
$$
$$
z_2 = w_2a_1+b_2 \hspace{0.1cm} \wedge a_2 = \sigma_2(z_2),
$$
<p>and the cost function</p>
$$
C(x;\boldsymbol{\Theta})=\frac{1}{2}(a_2-y)^2,
$$
<p>with \( \boldsymbol{\Theta}=[w_1,w_2,b_1,b_2] \).</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="layout-of-a-simple-neural-network-with-one-hidden-layer">Layout of a simple neural network with one hidden layer </h2>
<br/><br/>
<center>
<p><img src="figures/simplenn2.png" width="900" align="bottom"></p>
</center>
<br/><br/>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-derivatives">The derivatives </h2>
<p>The derivatives are now, using the chain rule again</p>
$$
\frac{\partial C}{\partial w_2}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial w_2}=(a_2-y)\sigma_2'a_1=\delta_2a_1,
$$
$$
\frac{\partial C}{\partial b_2}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial b_2}=(a_2-y)\sigma_2'=\delta_2,
$$
$$
\frac{\partial C}{\partial w_1}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial w_1}=(a_2-y)\sigma_2'a_1\sigma_1'a_0,
$$
$$
\frac{\partial C}{\partial b_1}=\frac{\partial C}{\partial a_2}\frac{\partial a_2}{\partial z_2}\frac{\partial z_2}{\partial a_1}\frac{\partial a_1}{\partial z_1}\frac{\partial z_1}{\partial b_1}=(a_2-y)\sigma_2'\sigma_1'=\delta_1.
$$
<p>Can you generalize this to more than one hidden layer?</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="important-observations">Important observations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>From the above equations we see that the derivatives of the activation
functions play a central role. If they vanish, the training may
stop. This is called the vanishing gradient problem, see discussions below. If they become
large, the parameters \( w_i \) and \( b_i \) may simply go to infinity. This
is referenced as the exploding gradient problem.
</p>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-training">The training </h2>
<p>The training of the parameters is done through various gradient descent approximations with</p>
$$
w_{i}\leftarrow w_{i}- \eta \delta_i a_{i-1},
$$
<p>and</p>
$$
b_i \leftarrow b_i-\eta \delta_i,
$$
<p>with \( \eta \) is the learning rate.</p>
<p>One iteration consists of one feed forward step and one back-propagation step. Each back-propagation step does one update of the parameters \( \boldsymbol{\Theta} \).</p>
<p>For the first hidden layer \( a_{i-1}=a_0=x \) for this simple model.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="code-example">Code example </h2>
<p>The code here implements the above model with one hidden layer and
scalar variables for the same function we studied in the previous
example. The code is however set up so that we can add multiple
inputs \( x \) and target values \( y \). Note also that we have the
possibility of defining a feature matrix \( \boldsymbol{X} \) with more than just
one column for the input values. This will turn useful in our next example. We have also defined matrices and vectors for all of our operations although it is not necessary here.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># We use the Sigmoid function as activation function</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">sigmoid</span>(z):
<span style="color: #8B008B; font-weight: bold">return</span> <span style="color: #B452CD">1.0</span>/(<span style="color: #B452CD">1.0</span>+np.exp(-z))
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">forwardpropagation</span>(x):
<span style="color: #228B22"># weighted sum of inputs to the hidden layer</span>
z_1 = np.matmul(x, w_1) + b_1
<span style="color: #228B22"># activation in the hidden layer</span>
a_1 = sigmoid(z_1)
<span style="color: #228B22"># weighted sum of inputs to the output layer</span>
z_2 = np.matmul(a_1, w_2) + b_2
a_2 = z_2
<span style="color: #8B008B; font-weight: bold">return</span> a_1, a_2
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">backpropagation</span>(x, y):
a_1, a_2 = forwardpropagation(x)
<span style="color: #228B22"># parameter delta for the output layer, note that a_2=z_2 and its derivative wrt z_2 is just 1</span>
delta_2 = a_2 - y
<span style="color: #658b00">print</span>(<span style="color: #B452CD">0.5</span>*((a_2-y)**<span style="color: #B452CD">2</span>))
<span style="color: #228B22"># delta for the hidden layer</span>
delta_1 = np.matmul(delta_2, w_2.T) * a_1 * (<span style="color: #B452CD">1</span> - a_1)
<span style="color: #228B22"># gradients for the output layer</span>
output_weights_gradient = np.matmul(a_1.T, delta_2)
output_bias_gradient = np.sum(delta_2, axis=<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># gradient for the hidden layer</span>
hidden_weights_gradient = np.matmul(x.T, delta_1)
hidden_bias_gradient = np.sum(delta_1, axis=<span style="color: #B452CD">0</span>)
<span style="color: #8B008B; font-weight: bold">return</span> output_weights_gradient, output_bias_gradient, hidden_weights_gradient, hidden_bias_gradient
<span style="color: #228B22"># ensure the same random numbers appear every time</span>
np.random.seed(<span style="color: #B452CD">0</span>)
<span style="color: #228B22"># Input variable</span>
x = np.array([<span style="color: #B452CD">4.0</span>],dtype=np.float64)
<span style="color: #228B22"># Target values</span>
y = <span style="color: #B452CD">2</span>*x+<span style="color: #B452CD">1.0</span>
<span style="color: #228B22"># Defining the neural network, only scalars here</span>
n_inputs = x.shape
n_features = <span style="color: #B452CD">1</span>
n_hidden_neurons = <span style="color: #B452CD">1</span>
n_outputs = <span style="color: #B452CD">1</span>
<span style="color: #228B22"># Initialize the network</span>
<span style="color: #228B22"># weights and bias in the hidden layer</span>
w_1 = np.random.randn(n_features, n_hidden_neurons)
b_1 = np.zeros(n_hidden_neurons) + <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># weights and bias in the output layer</span>
w_2 = np.random.randn(n_hidden_neurons, n_outputs)
b_2 = np.zeros(n_outputs) + <span style="color: #B452CD">0.01</span>
eta = <span style="color: #B452CD">0.1</span>
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(<span style="color: #B452CD">50</span>):
<span style="color: #228B22"># calculate gradients</span>
derivW2, derivB2, derivW1, derivB1 = backpropagation(x, y)
<span style="color: #228B22"># update weights and biases</span>
w_2 -= eta * derivW2
b_2 -= eta * derivB2
w_1 -= eta * derivW1
b_1 -= eta * derivB1
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>We see that after some few iterations (the results do depend on the learning rate however), we get an error which is rather small.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<!-- --- begin exercise --- -->
<h2 id="exercise-1-including-more-data">Exercise 1: Including more data </h2>
<p>Try to increase the amount of input and
target/output data. Try also to perform calculations for more values
of the learning rates. Feel free to add either hyperparameters with an
\( l_1 \) norm or an \( l_2 \) norm and discuss your results.
Discuss your results as functions of the amount of training data and various learning rates.
</p>
<p><b>Challenge:</b> Try to change the activation functions and replace the hard-coded analytical expressions with automatic derivation via either <b>autograd</b> or <b>JAX</b>.</p>
<!-- --- end exercise --- -->
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simple-neural-network-and-the-back-propagation-equations">Simple neural network and the back propagation equations </h2>
<p>Let us now try to increase our level of ambition and attempt at setting
up the equations for a neural network with two input nodes, one hidden
layer with two hidden nodes and one output layer with one output node/neuron only (see graph)..
</p>
<p>We need to define the following parameters and variables with the input layer (layer \( (0) \))
where we label the nodes \( x_0 \) and \( x_1 \)
</p>
$$
x_0 = a_0^{(0)} \wedge x_1 = a_1^{(0)}.
$$
<p>The hidden layer (layer \( (1) \)) has nodes which yield the outputs \( a_0^{(1)} \) and \( a_1^{(1)} \)) with weight \( \boldsymbol{w} \) and bias \( \boldsymbol{b} \) parameters</p>
$$
w_{ij}^{(1)}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)}\right\} \wedge b^{(1)}=\left\{b_0^{(1)},b_1^{(1)}\right\}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="layout-of-a-simple-neural-network-with-two-input-nodes-one-hidden-layer-and-one-output-node">Layout of a simple neural network with two input nodes, one hidden layer and one output node </h2>
<br/><br/>
<center>
<p><img src="figures/simplenn3.png" width="900" align="bottom"></p>
</center>
<br/><br/>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-ouput-layer">The ouput layer </h2>
<p>Finally, we have the ouput layer given by layer label \( (2) \) with output \( a^{(2)} \) and weights and biases to be determined given by the variables</p>
$$
w_{i}^{(2)}=\left\{w_{0}^{(2)},w_{1}^{(2)}\right\} \wedge b^{(2)}.
$$
<p>Our output is \( \tilde{y}=a^{(2)} \) and we define a generic cost function \( C(a^{(2)},y;\boldsymbol{\Theta}) \) where \( y \) is the target value (a scalar here).
The parameters we need to optimize are given by
</p>
$$
\boldsymbol{\Theta}=\left\{w_{00}^{(1)},w_{01}^{(1)},w_{10}^{(1)},w_{11}^{(1)},w_{0}^{(2)},w_{1}^{(2)},b_0^{(1)},b_1^{(1)},b^{(2)}\right\}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="compact-expressions">Compact expressions </h2>
<p>We can define the inputs to the activation functions for the various layers in terms of various matrix-vector multiplications and vector additions.
The inputs to the first hidden layer are
</p>
$$
\begin{bmatrix}z_0^{(1)} \\ z_1^{(1)} \end{bmatrix}=\begin{bmatrix}w_{00}^{(1)} & w_{01}^{(1)}\\ w_{10}^{(1)} &w_{11}^{(1)} \end{bmatrix}\begin{bmatrix}a_0^{(0)} \\ a_1^{(0)} \end{bmatrix}+\begin{bmatrix}b_0^{(1)} \\ b_1^{(1)} \end{bmatrix},
$$
<p>with outputs</p>
$$
\begin{bmatrix}a_0^{(1)} \\ a_1^{(1)} \end{bmatrix}=\begin{bmatrix}\sigma^{(1)}(z_0^{(1)}) \\ \sigma^{(1)}(z_1^{(1)}) \end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="output-layer">Output layer </h2>
<p>For the final output layer we have the inputs to the final activation function </p>
$$
z^{(2)} = w_{0}^{(2)}a_0^{(1)} +w_{1}^{(2)}a_1^{(1)}+b^{(2)},
$$
<p>resulting in the output</p>
$$
a^{(2)}=\sigma^{(2)}(z^{(2)}).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="explicit-derivatives">Explicit derivatives </h2>
<p>In total we have nine parameters which we need to train. Using the
chain rule (or just the back-propagation algorithm) we can find all
derivatives. Since we will use automatic differentiation in reverse
mode, we start with the derivatives of the cost function with respect
to the parameters of the output layer, namely
</p>
$$
\frac{\partial C}{\partial w_{i}^{(2)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial w_{i}^{(2)}}=\delta^{(2)}a_i^{(1)},
$$
<p>with</p>
$$
\delta^{(2)}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
$$
<p>and finally</p>
$$
\frac{\partial C}{\partial b^{(2)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}\frac{\partial z^{(2)}}{\partial b^{(2)}}=\delta^{(2)}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="derivatives-of-the-hidden-layer">Derivatives of the hidden layer </h2>
<p>Using the chain rule we have the following expressions for say one of the weight parameters (it is easy to generalize to the other weight parameters)</p>
$$
\frac{\partial C}{\partial w_{00}^{(1)}}=\frac{\partial C}{\partial a^{(2)}}\frac{\partial a^{(2)}}{\partial z^{(2)}}
\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}}= \delta^{(2)}\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}},
$$
<p>which, noting that</p>
$$
z^{(2)} =w_0^{(2)}a_0^{(1)}+w_1^{(2)}a_1^{(1)}+b^{(2)},
$$
<p>allows us to rewrite </p>
$$
\frac{\partial z^{(2)}}{\partial z_0^{(1)}}\frac{\partial z_0^{(1)}}{\partial w_{00}^{(1)}}=w_0^{(2)}\frac{\partial a_0^{(1)}}{\partial z_0^{(1)}}a_0^{(1)}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="final-expression">Final expression </h2>
<p>Defining</p>
$$
\delta_0^{(1)}=w_0^{(2)}\frac{\partial a_0^{(1)}}{\partial z_0^{(1)}}\delta^{(2)},
$$
<p>we have </p>
$$
\frac{\partial C}{\partial w_{00}^{(1)}}=\delta_0^{(1)}a_0^{(1)}.
$$
<p>Similarly, we obtain</p>
$$
\frac{\partial C}{\partial w_{01}^{(1)}}=\delta_0^{(1)}a_1^{(1)}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="completing-the-list">Completing the list </h2>
<p>Similarly, we find</p>
$$
\frac{\partial C}{\partial w_{10}^{(1)}}=\delta_1^{(1)}a_0^{(1)},
$$
<p>and </p>
$$
\frac{\partial C}{\partial w_{11}^{(1)}}=\delta_1^{(1)}a_1^{(1)},
$$
<p>where we have defined </p>
$$
\delta_1^{(1)}=w_1^{(2)}\frac{\partial a_1^{(1)}}{\partial z_1^{(1)}}\delta^{(2)}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="final-expressions-for-the-biases-of-the-hidden-layer">Final expressions for the biases of the hidden layer </h2>
<p>For the sake of completeness, we list the derivatives of the biases, which are</p>
$$
\frac{\partial C}{\partial b_{0}^{(1)}}=\delta_0^{(1)},
$$
<p>and</p>
$$
\frac{\partial C}{\partial b_{1}^{(1)}}=\delta_1^{(1)}.
$$
<p>As we will see below, these expressions can be generalized in a more compact form. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="gradient-expressions">Gradient expressions </h2>
<p>For this specific model, with just one output node and two hidden
nodes, the gradient descent equations take the following form for output layer
</p>
$$
w_{i}^{(2)}\leftarrow w_{i}^{(2)}- \eta \delta^{(2)} a_{i}^{(1)},
$$
<p>and</p>
$$
b^{(2)} \leftarrow b^{(2)}-\eta \delta^{(2)},
$$
<p>and</p>
$$
w_{ij}^{(1)}\leftarrow w_{ij}^{(1)}- \eta \delta_{i}^{(1)} a_{j}^{(0)},
$$
<p>and</p>
$$
b_{i}^{(1)} \leftarrow b_{i}^{(1)}-\eta \delta_{i}^{(1)},
$$
<p>where \( \eta \) is the learning rate.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<!-- --- begin exercise --- -->
<h2 id="exercise-2-extended-program">Exercise 2: Extended program </h2>
<p>We extend our simple code to a function which depends on two variable \( x_0 \) and \( x_1 \), that is</p>
$$
y=f(x_0,x_1)=x_0^2+3x_0x_1+x_1^2+5.
$$
<p>We feed our network with \( n=100 \) entries \( x_0 \) and \( x_1 \). We have thus two features represented by these variable and an input matrix/design matrix \( \boldsymbol{X}\in \mathbf{R}^{n\times 2} \)</p>
$$
\boldsymbol{X}=\begin{bmatrix} x_{00} & x_{01} \\ x_{00} & x_{01} \\ x_{10} & x_{11} \\ x_{20} & x_{21} \\ \dots & \dots \\ \dots & \dots \\ x_{n-20} & x_{n-21} \\ x_{n-10} & x_{n-11} \end{bmatrix}.
$$
<p>Write a code, based on the previous code examples, which takes as input these data and fit the above function.
You can extend your code to include automatic differentiation.
</p>
<p>With these examples, we are now ready to embark upon the writing of more a general code for neural networks.</p>
<!-- --- end exercise --- -->
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="getting-serious-the-back-propagation-equations-for-a-neural-network">Getting serious, the back propagation equations for a neural network </h2>
<p>Now it is time to move away from one node in each layer only. Our inputs are also represented either by several inputs.</p>
<p>We have thus</p>
$$
\frac{\partial{\cal C}((\boldsymbol{\Theta}^L)}{\partial w_{jk}^L} = \left(a_j^L - y_j\right)a_j^L(1-a_j^L)a_k^{L-1},
$$
<p>Defining</p>
$$
\delta_j^L = a_j^L(1-a_j^L)\left(a_j^L - y_j\right) = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
$$
<p>and using the Hadamard product of two vectors we can write this as</p>
$$
\boldsymbol{\delta}^L = \sigma'(\hat{z}^L)\circ\frac{\partial {\cal C}}{\partial (\boldsymbol{a}^L)}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="analyzing-the-last-results">Analyzing the last results </h2>
<p>This is an important expression. The second term on the right handside
measures how fast the cost function is changing as a function of the $j$th
output activation. If, for example, the cost function doesn't depend
much on a particular output node \( j \), then \( \delta_j^L \) will be small,
which is what we would expect. The first term on the right, measures
how fast the activation function \( f \) is changing at a given activation
value \( z_j^L \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-considerations">More considerations </h2>
<p>Notice that everything in the above equations is easily computed. In
particular, we compute \( z_j^L \) while computing the behaviour of the
network, and it is only a small additional overhead to compute
\( \sigma'(z^L_j) \). The exact form of the derivative with respect to the
output depends on the form of the cost function.
However, provided the cost function is known there should be little
trouble in calculating
</p>
$$
\frac{\partial {\cal C}}{\partial (a_j^L)}
$$
<p>With the definition of \( \delta_j^L \) we have a more compact definition of the derivative of the cost function in terms of the weights, namely</p>
$$
\frac{\partial{\cal C}}{\partial w_{jk}^L} = \delta_j^La_k^{L-1}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="derivatives-in-terms-of-z-j-l">Derivatives in terms of \( z_j^L \) </h2>
<p>It is also easy to see that our previous equation can be written as</p>
$$
\delta_j^L =\frac{\partial {\cal C}}{\partial z_j^L}= \frac{\partial {\cal C}}{\partial a_j^L}\frac{\partial a_j^L}{\partial z_j^L},
$$
<p>which can also be interpreted as the partial derivative of the cost function with respect to the biases \( b_j^L \), namely</p>
$$
\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L}\frac{\partial b_j^L}{\partial z_j^L}=\frac{\partial {\cal C}}{\partial b_j^L},
$$
<p>That is, the error \( \delta_j^L \) is exactly equal to the rate of change of the cost function as a function of the bias. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="bringing-it-together">Bringing it together </h2>
<p>We have now three equations that are essential for the computations of the derivatives of the cost function at the output layer. These equations are needed to start the algorithm and they are</p>
$$
\begin{equation}
\frac{\partial{\cal C}(\hat{W^L})}{\partial w_{jk}^L} = \delta_j^La_k^{L-1},
\label{_auto1}
\end{equation}
$$
<p>and</p>
$$
\begin{equation}
\delta_j^L = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)},
\label{_auto2}
\end{equation}
$$
<p>and</p>
$$
\begin{equation}
\delta_j^L = \frac{\partial {\cal C}}{\partial b_j^L},
\label{_auto3}
\end{equation}
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="final-back-propagating-equation">Final back propagating equation </h2>
<p>We have that (replacing \( L \) with a general layer \( l \))</p>
$$
\delta_j^l =\frac{\partial {\cal C}}{\partial z_j^l}.
$$
<p>We want to express this in terms of the equations for layer \( l+1 \).</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="using-the-chain-rule-and-summing-over-all-k-entries">Using the chain rule and summing over all \( k \) entries </h2>
<p>We obtain</p>
$$
\delta_j^l =\sum_k \frac{\partial {\cal C}}{\partial z_k^{l+1}}\frac{\partial z_k^{l+1}}{\partial z_j^{l}}=\sum_k \delta_k^{l+1}\frac{\partial z_k^{l+1}}{\partial z_j^{l}},
$$
<p>and recalling that</p>
$$
z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},
$$
<p>with \( M_l \) being the number of nodes in layer \( l \), we obtain</p>
$$
\delta_j^l =\sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l),
$$
<p>This is our final equation.</p>
<p>We are now ready to set up the algorithm for back propagation and learning the weights and biases.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-back-propagation-algorithm">Setting up the back propagation algorithm </h2>
<p>The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.</p>
<p><b>First</b>, we set up the input data \( \hat{x} \) and the activations
\( \hat{z}_1 \) of the input layer and compute the activation function and
the pertinent outputs \( \hat{a}^1 \).
</p>
<p><b>Secondly</b>, we perform then the feed forward till we reach the output
layer and compute all \( \hat{z}_l \) of the input layer and compute the
activation function and the pertinent outputs \( \hat{a}^l \) for
\( l=1,2,3,\dots,L \).
</p>
<p><b>Notation</b>: The first hidden layer has \( l=1 \) as label and the final output layer has \( l=L \).</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-back-propagation-algorithm-part-2">Setting up the back propagation algorithm, part 2 </h2>
<p>Thereafter we compute the ouput error \( \hat{\delta}^L \) by computing all</p>
$$
\delta_j^L = \sigma'(z_j^L)\frac{\partial {\cal C}}{\partial (a_j^L)}.
$$
<p>Then we compute the back propagate error for each \( l=L-1,L-2,\dots,1 \) as</p>
$$
\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}\sigma'(z_j^l).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-back-propagation-algorithm-part-3">Setting up the Back propagation algorithm, part 3 </h2>
<p>Finally, we update the weights and the biases using gradient descent
for each \( l=L-1,L-2,\dots,1 \) and update the weights and biases
according to the rules
</p>
$$
w_{jk}^l\leftarrow = w_{jk}^l- \eta \delta_j^la_k^{l-1},
$$
$$
b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l,
$$
<p>with \( \eta \) being the learning rate.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="updating-the-gradients">Updating the gradients </h2>
<p>With the back propagate error for each \( l=L-1,L-2,\dots,1 \) as</p>
$$
\delta_j^l = \sum_k \delta_k^{l+1}w_{kj}^{l+1}sigma'(z_j^l),
$$
<p>we update the weights and the biases using gradient descent for each \( l=L-1,L-2,\dots,1 \) and update the weights and biases according to the rules</p>
$$
w_{jk}^l\leftarrow = w_{jk}^l- \eta \delta_j^la_k^{l-1},
$$
$$
b_j^l \leftarrow b_j^l-\eta \frac{\partial {\cal C}}{\partial b_j^l}=b_j^l-\eta \delta_j^l,
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="activation-functions">Activation functions </h3>
<p>A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
</p>
<ul>
<li> Non-constant</li>
<li> Bounded</li>
<li> Monotonically-increasing</li>
<li> Continuous</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="activation-functions-logistic-and-hyperbolic-ones">Activation functions, Logistic and Hyperbolic ones </h3>
<p>The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
</p>
<p>Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic <em>Sigmoid</em>
</p>
$$
f(x) = \frac{1}{1 + e^{-x}},
$$
<p>and the <em>hyperbolic tangent</em> function</p>
$$
f(x) = \tanh(x)
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="relevance">Relevance </h3>
<p>The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called <em>one-sided</em>. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for <em>deep neural networks</em>
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #CD5555">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<span style="color: #CD5555">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #CD5555">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #CD5555">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">math</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">mt</span>
z = numpy.arange(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">.1</span>)
sigma_fn = numpy.vectorize(<span style="color: #8B008B; font-weight: bold">lambda</span> z: <span style="color: #B452CD">1</span>/(<span style="color: #B452CD">1</span>+numpy.exp(-z)))
sigma = sigma_fn(z)
fig = plt.figure()
ax = fig.add_subplot(<span style="color: #B452CD">111</span>)
ax.plot(z, sigma)
ax.set_ylim([-<span style="color: #B452CD">0.1</span>, <span style="color: #B452CD">1.1</span>])
ax.set_xlim([-<span style="color: #B452CD">5</span>,<span style="color: #B452CD">5</span>])
ax.grid(<span style="color: #8B008B; font-weight: bold">True</span>)
ax.set_xlabel(<span style="color: #CD5555">&#39;z&#39;</span>)
ax.set_title(<span style="color: #CD5555">&#39;sigmoid function&#39;</span>)
plt.show()
<span style="color: #CD5555">&quot;&quot;&quot;Step Function&quot;&quot;&quot;</span>
z = numpy.arange(-<span style="color: #B452CD">5</span>, <span style="color: #B452CD">5</span>, <span style="color: #B452CD">.02</span>)
step_fn = numpy.vectorize(<span style="color: #8B008B; font-weight: bold">lambda</span> z: <span style="color: #B452CD">1.0</span> <span style="color: #8B008B; font-weight: bold">if</span> z &gt;= <span style="color: #B452CD">0.0</span> <span style="color: #8B008B; font-weight: bold">else</span> <span style="color: #B452CD">0.0</span>)
step = step_fn(z)
fig = plt.figure()
ax = fig.add_subplot(<span style="color: #B452CD">111</span>)
ax.plot(z, step)
ax.set_ylim([-<span style="color: #B452CD">0.5</span>, <span style="color: #B452CD">1.5</span>])
ax.set_xlim([-<span style="color: #B452CD">5</span>,<span style="color: #B452CD">5</span>])
ax.grid(<span style="color: #8B008B; font-weight: bold">True</span>)
ax.set_xlabel(<span style="color: #CD5555">&#39;z&#39;</span>)
ax.set_title(<span style="color: #CD5555">&#39;step function&#39;</span>)
plt.show()
<span style="color: #CD5555">&quot;&quot;&quot;Sine Function&quot;&quot;&quot;</span>
z = numpy.arange(-<span style="color: #B452CD">2</span>*mt.pi, <span style="color: #B452CD">2</span>*mt.pi, <span style="color: #B452CD">0.1</span>)
t = numpy.sin(z)
fig = plt.figure()
ax = fig.add_subplot(<span style="color: #B452CD">111</span>)
ax.plot(z, t)
ax.set_ylim([-<span style="color: #B452CD">1.0</span>, <span style="color: #B452CD">1.0</span>])
ax.set_xlim([-<span style="color: #B452CD">2</span>*mt.pi,<span style="color: #B452CD">2</span>*mt.pi])
ax.grid(<span style="color: #8B008B; font-weight: bold">True</span>)
ax.set_xlabel(<span style="color: #CD5555">&#39;z&#39;</span>)
ax.set_title(<span style="color: #CD5555">&#39;sine function&#39;</span>)
plt.show()
<span style="color: #CD5555">&quot;&quot;&quot;Plots a graph of the squashing function used by a rectified linear</span>
<span style="color: #CD5555">unit&quot;&quot;&quot;</span>
z = numpy.arange(-<span style="color: #B452CD">2</span>, <span style="color: #B452CD">2</span>, <span style="color: #B452CD">.1</span>)
zero = numpy.zeros(<span style="color: #658b00">len</span>(z))
y = numpy.max([zero, z], axis=<span style="color: #B452CD">0</span>)
fig = plt.figure()
ax = fig.add_subplot(<span style="color: #B452CD">111</span>)
ax.plot(z, y)
ax.set_ylim([-<span style="color: #B452CD">2.0</span>, <span style="color: #B452CD">2.0</span>])
ax.set_xlim([-<span style="color: #B452CD">2.0</span>, <span style="color: #B452CD">2.0</span>])
ax.grid(<span style="color: #8B008B; font-weight: bold">True</span>)
ax.set_xlabel(<span style="color: #CD5555">&#39;z&#39;</span>)
ax.set_title(<span style="color: #CD5555">&#39;Rectified linear unit&#39;</span>)
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="fine-tuning-neural-network-hyperparameters">Fine-tuning neural network hyperparameters </h2>
<p>The flexibility of neural networks is also one of their main
drawbacks: there are many hyperparameters to tweak. Not only can you
use any imaginable network topology (how neurons/nodes are
interconnected), but even in a simple FFNN you can change the number
of layers, the number of neurons per layer, the type of activation
function to use in each layer, the weight initialization logic, the
stochastic gradient optmized and much more. How do you know what
combination of hyperparameters is the best for your task?
</p>
<ul>
<li> You can use grid search with cross-validation to find the right hyperparameters.</li>
</ul>
<p>However,since there are many hyperparameters to tune, and since
training a neural network on a large dataset takes a lot of time, you
will only be able to explore a tiny part of the hyperparameter space.
</p>
<ul>
<li> You can use randomized search.</li>
<li> Or use tools like <a href="http://oscar.calldesk.ai/" target="_blank">Oscar</a>, which implements more complex algorithms to help you find a good set of hyperparameters quickly.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="hidden-layers">Hidden layers </h2>
<p>For many problems you can start with just one or two hidden layers and
it will work just fine. For the MNIST data set you ca easily get a
high accuracy using just one hidden layer with a few hundred neurons.
You can reach for this data set above 98% accuracy using two hidden
layers with the same total amount of neurons, in roughly the same
amount of training time.
</p>
<p>For more complex problems, you can gradually ramp up the number of
hidden layers, until you start overfitting the training set. Very
complex tasks, such as large image classification or speech
recognition, typically require networks with dozens of layers and they
need a huge amount of training data. However, you will rarely have to
train such networks from scratch: it is much more common to reuse
parts of a pretrained state-of-the-art network that performs a similar
task.
</p>
<!-- !split -->
<h2 id="vanishing-gradients">Vanishing gradients </h2>
<p>The Back propagation algorithm we derived above works by going from
the output layer to the input layer, propagating the error gradient on
the way. Once the algorithm has computed the gradient of the cost
function with regards to each parameter in the network, it uses these
gradients to update each parameter with a Gradient Descent (GD) step.
</p>
<p>Unfortunately for us, the gradients often get smaller and smaller as
the algorithm progresses down to the first hidden layers. As a result,
the GD update leaves the lower layer connection weights virtually
unchanged, and training never converges to a good solution. This is
known in the literature as <b>the vanishing gradients problem</b>.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="exploding-gradients">Exploding gradients </h2>
<p>In other cases, the opposite can happen, namely the the gradients can
grow bigger and bigger. The result is that many of the layers get
large updates of the weights the algorithm diverges. This is the
<b>exploding gradients problem</b>, which is mostly encountered in
recurrent neural networks. More generally, deep neural networks suffer
from unstable gradients, different layers may learn at widely
different speeds
</p>
<!-- !split -->
<h2 id="is-the-logistic-activation-function-sigmoid-our-choice">Is the Logistic activation function (Sigmoid) our choice? </h2>
<p>Although this unfortunate behavior has been empirically observed for
quite a while (it was one of the reasons why deep neural networks were
mostly abandoned for a long time), it is only around 2010 that
significant progress was made in understanding it.
</p>
<p>A paper titled <a href="http://proceedings.mlr.press/v9/glorot10a.html" target="_blank">Understanding the Difficulty of Training Deep
Feedforward Neural Networks by Xavier Glorot and Yoshua Bengio</a> found that
the problems with the popular logistic
sigmoid activation function and the weight initialization technique
that was most popular at the time, namely random initialization using
a normal distribution with a mean of 0 and a standard deviation of
1.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="logistic-function-as-the-root-of-problems">Logistic function as the root of problems </h2>
<p>They showed that with this activation function and this
initialization scheme, the variance of the outputs of each layer is
much greater than the variance of its inputs. Going forward in the
network, the variance keeps increasing after each layer until the
activation function saturates at the top layers. This is actually made
worse by the fact that the logistic function has a mean of 0.5, not 0
(the hyperbolic tangent function has a mean of 0 and behaves slightly
better than the logistic function in deep networks).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-derivative-of-the-logistic-funtion">The derivative of the Logistic funtion </h2>
<p>Looking at the logistic activation function, when inputs become large
(negative or positive), the function saturates at 0 or 1, with a
derivative extremely close to 0. Thus when backpropagation kicks in,
it has virtually no gradient to propagate back through the network,
and what little gradient exists keeps getting diluted as
backpropagation progresses down through the top layers, so there is
really nothing left for the lower layers.
</p>
<p>In their paper, Glorot and Bengio propose a way to significantly
alleviate this problem. We need the signal to flow properly in both
directions: in the forward direction when making predictions, and in
the reverse direction when backpropagating gradients. We don&#8217;t want
the signal to die out, nor do we want it to explode and saturate. For
the signal to flow properly, the authors argue that we need the
variance of the outputs of each layer to be equal to the variance of
its inputs, and we also need the gradients to have equal variance
before and after flowing through a layer in the reverse direction.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="insights-from-the-paper-by-glorot-and-bengio">Insights from the paper by Glorot and Bengio </h2>
<p>One of the insights in the 2010 paper by Glorot and Bengio was that
the vanishing/exploding gradients problems were in part due to a poor
choice of activation function. Until then most people had assumed that
if Nature had chosen to use roughly sigmoid activation functions in
biological neurons, they must be an excellent choice. But it turns out
that other activation functions behave much better in deep neural
networks, in particular the ReLU activation function, mostly because
it does not saturate for positive values (and also because it is quite
fast to compute).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-relu-function-family">The RELU function family </h2>
<p>The ReLU activation function suffers from a problem known as the dying
ReLUs: during training, some neurons effectively die, meaning they
stop outputting anything other than 0.
</p>
<p>In some cases, you may find that half of your network&#8217;s neurons are
dead, especially if you used a large learning rate. During training,
if a neuron&#8217;s weights get updated such that the weighted sum of the
neuron&#8217;s inputs is negative, it will start outputting 0. When this
happen, the neuron is unlikely to come back to life since the gradient
of the ReLU function is 0 when its input is negative.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="elu-function">ELU function </h2>
<p>To solve this problem, nowadays practitioners use a variant of the
ReLU function, such as the leaky ReLU discussed above or the so-called
exponential linear unit (ELU) function
</p>
$$
ELU(z) = \left\{\begin{array}{cc} \alpha\left( \exp{(z)}-1\right) & z < 0,\\ z & z \ge 0.\end{array}\right.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="which-activation-function-should-we-use">Which activation function should we use? </h2>
<p>In general it seems that the ELU activation function is better than
the leaky ReLU function (and its variants), which is better than
ReLU. ReLU performs better than \( \tanh \) which in turn performs better
than the logistic function.
</p>
<p>If runtime performance is an issue, then you may opt for the leaky
ReLU function over the ELU function If you don&#8217;t want to tweak yet
another hyperparameter, you may just use the default \( \alpha \) of
\( 0.01 \) for the leaky ReLU, and \( 1 \) for ELU. If you have spare time and
computing power, you can use cross-validation or bootstrap to evaluate
other activation functions.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-activation-functions-output-layers">More on activation functions, output layers </h2>
<p>In most cases you can use the ReLU activation function in the hidden
layers (or one of its variants).
</p>
<p>It is a bit faster to compute than other activation functions, and the
gradient descent optimization does in general not get stuck.
</p>
<b>For the output layer:</b>
<ul>
<li> For classification the softmax activation function is generally a good choice for classification tasks (when the classes are mutually exclusive).</li>
<li> For regression tasks, you can simply use no activation function at all.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="batch-normalization">Batch Normalization </h2>
<p>Batch Normalization aims to address the vanishing/exploding gradients
problems, and more generally the problem that the distribution of each
layer&#8217;s inputs changes during training, as the parameters of the
previous layers change.
</p>
<p>The technique consists of adding an operation in the model just before
the activation function of each layer, simply zero-centering and
normalizing the inputs, then scaling and shifting the result using two
new parameters per layer (one for scaling, the other for shifting). In
other words, this operation lets the model learn the optimal scale and
mean of the inputs for each layer. In order to zero-center and
normalize the inputs, the algorithm needs to estimate the inputs&#8217; mean
and standard deviation. It does so by evaluating the mean and standard
deviation of the inputs over the current mini-batch, from this the
name batch normalization.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="dropout">Dropout </h2>
<p>It is a fairly simple algorithm: at every training step, every neuron
(including the input neurons but excluding the output neurons) has a
probability \( p \) of being temporarily dropped out, meaning it will be
entirely ignored during this training step, but it may be active
during the next step.
</p>
<p>The hyperparameter \( p \) is called the dropout rate, and it is typically
set to 50%. After training, the neurons are not dropped anymore. It
is viewed as one of the most popular regularization techniques.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="gradient-clipping">Gradient Clipping </h2>
<p>A popular technique to lessen the exploding gradients problem is to
simply clip the gradients during backpropagation so that they never
exceed some threshold (this is mostly useful for recurrent neural
networks).
</p>
<p>This technique is called Gradient Clipping.</p>
<p>In general however, Batch
Normalization is preferred.
</p>
<!-- !split -->
<h2 id="a-top-down-perspective-on-neural-networks">A top-down perspective on Neural networks </h2>
<p>The first thing we would like to do is divide the data into two or
three parts. A training set, a validation or dev (development) set,
and a test set. The test set is the data on which we want to make
predictions. The dev set is a subset of the training data we use to
check how well we are doing out-of-sample, after training the model on
the training dataset. We use the validation error as a proxy for the
test error in order to make tweaks to our model. It is crucial that we
do not use any of the test data to train the algorithm. This is a
cardinal sin in ML. Then:
</p>
<ol>
<li> Estimate optimal error rate</li>
<li> Minimize underfitting (bias) on training data set.</li>
<li> Make sure you are not overfitting.</li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-top-down-perspectives">More top-down perspectives </h2>
<p>If the validation and test sets are drawn from the same distributions,
then a good performance on the validation set should lead to similarly
good performance on the test set.
</p>
<p>However, sometimes
the training data and test data differ in subtle ways because, for
example, they are collected using slightly different methods, or
because it is cheaper to collect data in one way versus another. In
this case, there can be a mismatch between the training and test
data. This can lead to the neural network overfitting these small
differences between the test and training sets, and a poor performance
on the test set despite having a good performance on the validation
set. To rectify this, Andrew Ng suggests making two validation or dev
sets, one constructed from the training data and one constructed from
the test data. The difference between the performance of the algorithm
on these two validation sets quantifies the train-test mismatch. This
can serve as another important diagnostic when using DNNs for
supervised learning.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="limitations-of-supervised-learning-with-deep-networks">Limitations of supervised learning with deep networks </h2>
<p>Like all statistical methods, supervised learning using neural
networks has important limitations. This is especially important when
one seeks to apply these methods, especially to physics problems. Like
all tools, DNNs are not a universal solution. Often, the same or
better performance on a task can be achieved by using a few
hand-engineered features (or even a collection of random
features).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="limitations-of-nns">Limitations of NNs </h2>
<p>Here we list some of the important limitations of supervised neural network based models. </p>
<ul>
<li> <b>Need labeled data</b>. All supervised learning methods, DNNs for supervised learning require labeled data. Often, labeled data is harder to acquire than unlabeled data (e.g. one must pay for human experts to label images).</li>
<li> <b>Supervised neural networks are extremely data intensive.</b> DNNs are data hungry. They perform best when data is plentiful. This is doubly so for supervised methods where the data must also be labeled. The utility of DNNs is extremely limited if data is hard to acquire or the datasets are small (hundreds to a few thousand samples). In this case, the performance of other methods that utilize hand-engineered features can exceed that of DNNs.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="homogeneous-data">Homogeneous data </h2>
<ul>
<li> <b>Homogeneous data.</b> Almost all DNNs deal with homogeneous data of one type. It is very hard to design architectures that mix and match data types (i.e.&nbsp;some continuous variables, some discrete variables, some time series). In applications beyond images, video, and language, this is often what is required. In contrast, ensemble models like random forests or gradient-boosted trees have no difficulty handling mixed data types.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-limitations">More limitations </h2>
<ul>
<li> <b>Many problems are not about prediction.</b> In natural science we are often interested in learning something about the underlying distribution that generates the data. In this case, it is often difficult to cast these ideas in a supervised learning setting. While the problems are related, it is possible to make good predictions with a <em>wrong</em> model. The model might or might not be useful for understanding the underlying science.</li>
</ul>
<p>Some of these remarks are particular to DNNs, others are shared by all supervised learning methods. This motivates the use of unsupervised methods which in part circumvent these problems.</p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink