lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
b59bb722d3d6168ee40141945801ebd3dc77331c
FYS-STK4155/doc/pub/NeuralNet/html/NeuralNet.html
T
mhjensen b59bb722d3 cleaning up
2018-09-28 11:24:22 +02:00

1031 lines
76 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Elements of machine learning">
<title>Data Analysis and Machine Learning: Elements of machine learning</title>
<style type="text/css">
/* bloodish style */
body {
font-family: Helvetica, Verdana, Arial, Sans-serif;
color: #404040;
background: #ffffff;
}
h1 { font-size: 1.8em; color: #8A0808; }
h2 { font-size: 1.6em; color: #8A0808; }
h3 { font-size: 1.4em; color: #8A0808; }
h4 { color: #8A0808; }
a { color: #8A0808; text-decoration:none; }
tt { font-family: "Courier New", Courier; }
/* pre style removed because it will interfer with pygments */
p { text-indent: 0px; }
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
p.caption { width: 80%; font-style: normal; text-align: left; }
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
div { text-align: justify; text-justify: inter-word; }
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Neural networks', 2, None, '___sec0'),
('Artificial neurons', 2, None, '___sec1'),
('Neural network types', 2, None, '___sec2'),
('Feed-forward neural networks', 2, None, '___sec3'),
('Recurrent neural networks', 2, None, '___sec4'),
('Other types of networks', 2, None, '___sec5'),
('Multilayer perceptrons', 2, None, '___sec6'),
('Why multilayer perceptrons?', 2, None, '___sec7'),
('Mathematical model', 2, None, '___sec8'),
('Mathematical model', 2, None, '___sec9'),
('Mathematical model', 2, None, '___sec10'),
('Mathematical model', 2, None, '___sec11'),
('Mathematical model', 2, None, '___sec12'),
('Matrix-vector notation', 3, None, '___sec13'),
('Matrix-vector notation and activation', 3, None, '___sec14'),
('Activation functions', 3, None, '___sec15'),
('Activation functions, Logistic and Hyperbolic ones',
3,
None,
'___sec16'),
('Relevance', 3, None, '___sec17'),
('Setting up a Multi-layer perceptron model',
2,
None,
'___sec18'),
('Two-layer Neural Network', 2, None, '___sec19')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
<center><h1>Data Analysis and Machine Learning: Elements of machine learning</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 28, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- add own code for DNN -->
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Neural networks </h2>
<p>
Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
biological system, wherein neurons interact by sending signals in the
form of mathematical functions between layers. All layers can contain
an arbitrary number of neurons, and each connection is represented by
a weight variable.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Artificial neurons </h2>
<p>
The field of artificial neural networks has a long history of
development, and is closely connected with the advancement of computer
science and computers in general. A model of artificial neurons was
first developed by McCulloch and Pitts in 1943 to study signal
processing in the brain and has later been refined by others. The
general idea is to mimic neural networks in the human brain, which is
composed of billions of neurons that communicate with each other by
sending electrical signals. Each neuron accumulates its incoming
signals, which must exceed an activation threshold to yield an
output. If the threshold is not overcome, the neuron remains inactive,
i.e. has zero output.
<p>
This behaviour has inspired a simple mathematical model for an artificial neuron.
$$
\begin{equation}
y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)
\label{artificialNeuron}
\end{equation}
$$
Here, the output \( y \) of the neuron is the value of its activation function, which have as input
a weighted sum of signals \( x_i, \dots ,x_n \) received by \( n \) other neurons.
<p>
Conceptually, it is helpful to divide neural networks into four
categories:
<ol>
<li> general purpose neural networks for supervised learning,</li>
<li> neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs),</li>
<li> neural networks for sequential data such as Recurrent Neural Networks (RNNs), and</li>
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
In natural science, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers, or solving the Navier-Stokes equation in weather forecasting.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
achievements to reveal the potential of DNNs to facilitate the study
of quantum systems.
<p>
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails.
<p>
The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that
consists of layers of connected neurons, or <em>nodes</em>. It is supposed
to mimic a biological nervous system by letting each neuron interact
with other neurons by sending signals in the form of mathematical
functions between layers. A wide variety of different NNs have been
developed, but most of them consist of an input layer, an output layer
and eventual layers in-between, called <em>hidden layers</em>. All layers can
contain an arbitrary number of nodes, and each connection between two
nodes is associated with a weight variable.
<p>
Neural networks (also called neural nets) are neural-inspired
nonlinear models for supervised learning. As we will see, neural nets
can be viewed as natural, more powerful extensions of supervised
learning methods such as linear and logistic regression and soft-max
methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
the information moves in only one direction: forward through the layers.
<p>
Nodes are represented by circles, while the arrows display the connections between the nodes, including the
direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here.
We observe that each node in a layer is connected to <em>all</em> nodes in the subsequent layer,
making this a so-called <em>fully-connected</em> FFNN.
<p>
A different variant of FFNNs are <em>convolutional neural networks</em> (CNNs), which have a connectivity pattern
inspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli from
small sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strong
spatially local correlation present in natural images. The response of each neuron can be approximated mathematically
as a convolution operation.
<p>
CNNs emulate the behaviour of neurons in the visual cortex by enforcing a <em>local</em> connectivity pattern
between nodes of adjacent layers: Each node
in a convolutional layer is connected only to a subset of the nodes in the previous layer,
in contrast to the fully-connected FFNN.
Often, CNNs
consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end,
which gathers all the local data and produces the outputs. They have wide applications in image and video recognition
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Recurrent neural networks </h2>
<p>
So far we have only mentioned NNs where information flows in one direction: forward. <em>Recurrent neural networks</em> on
the other hand, have connections between nodes that form directed <em>cycles</em>. This creates a form of
internal memory which are able to capture information on what has been calculated before; the output is dependent
on the previous computations. Recurrent NNs make use of sequential information by performing the same task for
every element in a sequence, where each element depends on previous elements. An example of such information is
sentences, making recurrent NNs especially well-suited for handwriting and speech recognition.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Other types of networks </h2>
<p>
There are many other kinds of NNs that have been developed. One type that is specifically designed for interpolation
in multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers:
an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' here
means that each node in the output layer has a linear activation function). The layers are normally fully-connected and
there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated as
a separate type of NN due the unusual activation functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Multilayer perceptrons </h2>
<p>
One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
activation functions.
<p>
Such networks are often called <em>multilayer perceptrons</em> (MLPs)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Why multilayer perceptrons? </h2>
<p>
According to the <em>Universal approximation theorem</em>, a feed-forward neural network with just a single hidden layer containing
a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy,
assuming the activation function for the hidden layer is a <b>non-constant, bounded and monotonically-increasing continuous function</b>.
<p>
Note that the requirements on the activation function only applies to
the hidden layer, the output nodes are always assumed to be linear, so
as to not restrict the range of output values.
<p>
We note that this theorem is only applicable to an NN with <em>one</em> hidden
layer. Therefore, we can easily construct an NN that employs
activation functions which do not satisfy the above requirements, as
long as we have at least one layer with activation functions that
<em>do</em>. Furthermore, although the universal approximation theorem lays
the theoretical foundation for regression with neural networks, it
does not say anything about how things work in practice: A neural
network can still be able to approximate a given function reasonably
well without having the flexibility to fit <em>all other</em> functions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Mathematical model </h2>
$$
\begin{equation}
y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(u)
\label{artificialNeuron2}
\end{equation}
$$
<p>
In an FFNN of such neurons, the <em>inputs</em> \( x_i \) are the <em>outputs</em> of
the neurons in the preceding layer. Furthermore, an MLP is
fully-connected, which means that each neuron receives a weighted sum
of the outputs of <em>all</em> neurons in the previous layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Mathematical model </h2>
<p>
First, for each node \( i \) in the first hidden layer, we calculate a weighted sum \( u_i^1 \) of the input coordinates \( x_j \),
$$
\begin{equation}
u_i^1 = \sum_{j=1}^2 w_{ij}^1 x_j + b_i^1
\label{_auto1}
\end{equation}
$$
This value is the argument to the activation function \( f_1 \) of each neuron \( i \),
producing the output \( y_i^1 \) of all neurons in layer 1,
$$
\begin{equation}
y_i^1 = f_1(u_i^1) = f_1\left(\sum_{j=1}^2 w_{ij}^1 x_j + b_i^1\right)
\label{outputLayer1}
\end{equation}
$$
<p>
where we assume that all nodes in the same layer have identical
activation functions, hence the notation \( f_l \)
$$
\begin{equation}
y_i^l = f_l(u_i^l) = f_l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)
\label{generalLayer}
\end{equation}
$$
<p>
where \( N_l \) is the number of nodes in layer \( l \). When the output of
all the nodes in the first hidden layer are computed, the values of
the subsequent layer can be calculated and so forth until the output
is obtained.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Mathematical model </h2>
<p>
The output of neuron \( i \) in layer 2 is thus,
$$
\begin{align}
y_i^2 &= f_2\left(\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\right)
\label{_auto2}\\
&= f_2\left[\sum_{j=1}^3 w_{ij}^2f_1\left(\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]
\label{outputLayer2}
\end{align}
$$
where we have substituted \( y_m^1 \) with. Finally, the NN output yields,
$$
\begin{align}
y_1^3 &= f_3\left(\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\right)
\label{_auto3}\\
&= f_3\left[\sum_{j=1}^3 w_{1j}^3 f_2\left(\sum_{k=1}^3 w_{jk}^2 f_1\left(\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\right) + b_j^2\right)
+ b_1^3\right]
\label{_auto4}
\end{align}
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Mathematical model </h2>
<p>
We can generalize this expression to an MLP with \( l \) hidden
layers. The complete functional form is,
$$
\begin{align}
&y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\!
\dots \!f_1\!\left(\!\sum_{n=1}^{N_0} \!w_{mn}^1 x_n\! + \!b_m^1\!\right)
\!\dots \!\right) \!+ \!b_k^2\!\right)
\!+ \!b_1^3\!\right] &&
\label{completeNN}
\end{align}
$$
<p>
which illustrates a basic property of MLPs: The only independent
variables are the input values \( x_n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Mathematical model </h2>
<p>
This confirms that an MLP, despite its quite convoluted mathematical
form, is nothing more than an analytic function, specifically a
mapping of real-valued vectors \( \vec{x} \in \mathbb{R}^n \rightarrow
\vec{y} \in \mathbb{R}^m \). In our example, \( n=2 \) and
\( m=1 \). Consequentially, the number of input and output values of the
function we want to fit must be equal to the number of inputs and
outputs of our MLP.
<p>
Furthermore, the flexibility and universality of a MLP can be
illustrated by realizing that the expression is essentially a nested
sum of scaled activation functions of the form
$$
\begin{equation}
h(x) = c_1 f(c_2 x + c_3) + c_4
\label{_auto5}
\end{equation}
$$
<p>
where the parameters \( c_i \) are weights and biases. By adjusting these
parameters, the activation functions can be shifted up and down or
left and right, change slope or be rescaled which is the key to the
flexibility of a neural network.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec13">Matrix-vector notation </h3>
<p>
We can introduce a more convenient notation for the activations in a NN.
<p>
Additionally, we can represent the biases and activations
as layer-wise column vectors \( \vec{b}_l \) and \( \vec{y}_l \), so that the \( i \)-th element of each vector
is the bias \( b_i^l \) and activation \( y_i^l \) of node \( i \) in layer \( l \) respectively.
<p>
We have that \( \mathrm{W}_l \) is a \( N_{l-1} \times N_l \) matrix, while \( \vec{b}_l \) and \( \vec{y}_l \) are \( N_l \times 1 \) column vectors.
With this notation, the sum in becomes a matrix-vector multiplication, and we can write
the equation for the activations of hidden layer 2 in
$$
\begin{equation}
\vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) =
f_2\left(\left[\begin{array}{ccc}
w^2_{11} &w^2_{12} &w^2_{13} \\
w^2_{21} &w^2_{22} &w^2_{23} \\
w^2_{31} &w^2_{32} &w^2_{33} \\
\end{array} \right] \cdot
\left[\begin{array}{c}
y^1_1 \\
y^1_2 \\
y^1_3 \\
\end{array}\right] +
\left[\begin{array}{c}
b^2_1 \\
b^2_2 \\
b^2_3 \\
\end{array}\right]\right).
\label{_auto6}
\end{equation}
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec14">Matrix-vector notation and activation </h3>
<p>
The activation of node \( i \) in layer 2 is
$$
\begin{equation}
y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =
f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).
\label{_auto7}
\end{equation}
$$
<p>
This is not just a convenient and compact notation, but also a useful
and intuitive way to think about MLPs: The output is calculated by a
series of matrix-vector multiplications and vector additions that are
used as input to the activation functions. For each operation
\( \mathrm{W}_l \vec{y}_{l-1} \) we move forward one layer.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec15">Activation functions </h3>
<p>
A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
for a FFNN to fulfill the universal approximation theorem
<ul>
<li> Non-constant</li>
<li> Bounded</li>
<li> Monotonically-increasing</li>
<li> Continuous</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec16">Activation functions, Logistic and Hyperbolic ones </h3>
<p>
The second requirement excludes all linear functions. Furthermore, in
a MLP with only linear activation functions, each layer simply
performs a linear transformation of its inputs.
<p>
Regardless of the number of layers, the output of the NN will be
nothing but a linear function of the inputs. Thus we need to introduce
some kind of non-linearity to the NN to be able to fit non-linear
functions Typical examples are the logistic <em>Sigmoid</em>
$$
\begin{equation}
f(x) = \frac{1}{1 + e^{-x}},
\label{sigmoidActivationFunction}
\end{equation}
$$
and the <em>hyperbolic tangent</em> function
$$
\begin{equation}
f(x) = \tanh(x)
\label{tanhActivationFunction}
\end{equation}
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h3 id="___sec17">Relevance </h3>
<p>
The <em>sigmoid</em> function are more biologically plausible because the
output of inactive neurons are zero. Such activation function are
called <em>one-sided</em>. However, it has been shown that the hyperbolic
tangent performs better than the sigmoid for training MLPs. has
become the most popular for <em>deep neural networks</em>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<span style="color: #BA2121; font-style: italic">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #BA2121; font-style: italic">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #BA2121; font-style: italic">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">mt</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.1</span>)
sigma_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>numpy<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z)))
sigma <span style="color: #666666">=</span> sigma_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, sigma)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;sigmoid function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Step Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.02</span>)
step_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1.0</span> <span style="color: #008000; font-weight: bold">if</span> z <span style="color: #666666">&gt;=</span> <span style="color: #666666">0.0</span> <span style="color: #008000; font-weight: bold">else</span> <span style="color: #666666">0.0</span>)
step <span style="color: #666666">=</span> step_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, step)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.5</span>, <span style="color: #666666">1.5</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;step function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Sine Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">0.1</span>)
t <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>sin(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, t)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-1.0</span>, <span style="color: #666666">1.0</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi,<span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;sine function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Plots a graph of the squashing function used by a rectified linear</span>
<span style="color: #BA2121; font-style: italic">unit&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-2</span>, <span style="color: #666666">2</span>, <span style="color: #666666">.1</span>)
zero <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>zeros(<span style="color: #008000">len</span>(z))
y <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>max([zero, z], axis<span style="color: #666666">=0</span>)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, y)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-2.0</span>, <span style="color: #666666">2.0</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-2.0</span>, <span style="color: #666666">2.0</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;Rectified linear unit&#39;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec18">Setting up a Multi-layer perceptron model </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">scipy</span> <span style="color: #008000; font-weight: bold">import</span> optimize
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Neural_Network</span>(<span style="color: #008000">object</span>):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, Lambda<span style="color: #666666">=0</span>):
<span style="color: #408080; font-style: italic">#Define Hyperparameters</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize <span style="color: #666666">=</span> <span style="color: #666666">2</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize <span style="color: #666666">=</span> <span style="color: #666666">3</span>
<span style="color: #408080; font-style: italic">#Weights (parameters)</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>W1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize,<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize)
<span style="color: #008000">self</span><span style="color: #666666">.</span>W2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize,<span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize)
<span style="color: #408080; font-style: italic">#Regularization Parameter:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda <span style="color: #666666">=</span> Lambda
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">forward</span>(<span style="color: #008000">self</span>, X):
<span style="color: #408080; font-style: italic">#Propogate inputs though network</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(X, <span style="color: #008000">self</span><span style="color: #666666">.</span>W1)
<span style="color: #008000">self</span><span style="color: #666666">.</span>a2 <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoid(<span style="color: #008000">self</span><span style="color: #666666">.</span>z2)
<span style="color: #008000">self</span><span style="color: #666666">.</span>z3 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(<span style="color: #008000">self</span><span style="color: #666666">.</span>a2, <span style="color: #008000">self</span><span style="color: #666666">.</span>W2)
yHat <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoid(<span style="color: #008000">self</span><span style="color: #666666">.</span>z3)
<span style="color: #008000; font-weight: bold">return</span> yHat
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(<span style="color: #008000">self</span>, z):
<span style="color: #408080; font-style: italic">#Apply sigmoid activation function to scalar, vector, or matrix</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoidPrime</span>(<span style="color: #008000">self</span>,z):
<span style="color: #408080; font-style: italic">#Gradient of sigmoid</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z)<span style="color: #666666">/</span>((<span style="color: #666666">1+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z))<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">costFunction</span>(<span style="color: #008000">self</span>, X, y):
<span style="color: #408080; font-style: italic">#Compute cost for given X,y, use weights already stored in class.</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>yHat <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>forward(X)
J <span style="color: #666666">=</span> <span style="color: #666666">0.5*</span><span style="color: #008000">sum</span>((y<span style="color: #666666">-</span><span style="color: #008000">self</span><span style="color: #666666">.</span>yHat)<span style="color: #666666">**2</span>)<span style="color: #666666">/</span>X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda<span style="color: #666666">/2</span>)<span style="color: #666666">*</span>(np<span style="color: #666666">.</span>sum(<span style="color: #008000">self</span><span style="color: #666666">.</span>W1<span style="color: #666666">**2</span>)<span style="color: #666666">+</span>np<span style="color: #666666">.</span>sum(<span style="color: #008000">self</span><span style="color: #666666">.</span>W2<span style="color: #666666">**2</span>))
<span style="color: #008000; font-weight: bold">return</span> J
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">costFunctionPrime</span>(<span style="color: #008000">self</span>, X, y):
<span style="color: #408080; font-style: italic">#Compute derivative with respect to W and W2 for a given X and y:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>yHat <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>forward(X)
delta3 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>multiply(<span style="color: #666666">-</span>(y<span style="color: #666666">-</span><span style="color: #008000">self</span><span style="color: #666666">.</span>yHat), <span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoidPrime(<span style="color: #008000">self</span><span style="color: #666666">.</span>z3))
<span style="color: #408080; font-style: italic">#Add gradient of regularization term:</span>
dJdW2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(<span style="color: #008000">self</span><span style="color: #666666">.</span>a2<span style="color: #666666">.</span>T, delta3)<span style="color: #666666">/</span>X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>W2
delta2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(delta3, <span style="color: #008000">self</span><span style="color: #666666">.</span>W2<span style="color: #666666">.</span>T)<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoidPrime(<span style="color: #008000">self</span><span style="color: #666666">.</span>z2)
<span style="color: #408080; font-style: italic">#Add gradient of regularization term:</span>
dJdW1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(X<span style="color: #666666">.</span>T, delta2)<span style="color: #666666">/</span>X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>W1
<span style="color: #008000; font-weight: bold">return</span> dJdW1, dJdW2
<span style="color: #408080; font-style: italic">#Helper functions for interacting with other methods/classes</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">getParams</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic">#Get W1 and W2 Rolled into vector:</span>
params <span style="color: #666666">=</span> np<span style="color: #666666">.</span>concatenate((<span style="color: #008000">self</span><span style="color: #666666">.</span>W1<span style="color: #666666">.</span>ravel(), <span style="color: #008000">self</span><span style="color: #666666">.</span>W2<span style="color: #666666">.</span>ravel()))
<span style="color: #008000; font-weight: bold">return</span> params
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">setParams</span>(<span style="color: #008000">self</span>, params):
<span style="color: #408080; font-style: italic">#Set W1 and W2 using single parameter vector:</span>
W1_start <span style="color: #666666">=</span> <span style="color: #666666">0</span>
W1_end <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize
<span style="color: #008000">self</span><span style="color: #666666">.</span>W1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>reshape(params[W1_start:W1_end], \
(<span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize, <span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize))
W2_end <span style="color: #666666">=</span> W1_end <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize
<span style="color: #008000">self</span><span style="color: #666666">.</span>W2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>reshape(params[W1_end:W2_end], \
(<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize, <span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">computeGradients</span>(<span style="color: #008000">self</span>, X, y):
dJdW1, dJdW2 <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>costFunctionPrime(X, y)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>concatenate((dJdW1<span style="color: #666666">.</span>ravel(), dJdW2<span style="color: #666666">.</span>ravel()))
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">trainer</span>(<span style="color: #008000">object</span>):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, N):
<span style="color: #408080; font-style: italic">#Make Local reference to network:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>N <span style="color: #666666">=</span> N
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">callbackF</span>(<span style="color: #008000">self</span>, params):
<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>setParams(params)
<span style="color: #008000">self</span><span style="color: #666666">.</span>J<span style="color: #666666">.</span>append(<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>costFunction(<span style="color: #008000">self</span><span style="color: #666666">.</span>X, <span style="color: #008000">self</span><span style="color: #666666">.</span>y))
<span style="color: #008000">self</span><span style="color: #666666">.</span>testJ<span style="color: #666666">.</span>append(<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>costFunction(<span style="color: #008000">self</span><span style="color: #666666">.</span>testX, <span style="color: #008000">self</span><span style="color: #666666">.</span>testY))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">costFunctionWrapper</span>(<span style="color: #008000">self</span>, params, X, y):
<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>setParams(params)
cost <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>costFunction(X, y)
grad <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>computeGradients(X,y)
<span style="color: #008000; font-weight: bold">return</span> cost, grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">train</span>(<span style="color: #008000">self</span>, trainX, trainY, testX, testY):
<span style="color: #408080; font-style: italic">#Make an internal variable for the callback function:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>X <span style="color: #666666">=</span> trainX
<span style="color: #008000">self</span><span style="color: #666666">.</span>y <span style="color: #666666">=</span> trainY
<span style="color: #008000">self</span><span style="color: #666666">.</span>testX <span style="color: #666666">=</span> testX
<span style="color: #008000">self</span><span style="color: #666666">.</span>testY <span style="color: #666666">=</span> testY
<span style="color: #408080; font-style: italic">#Make empty list to store training costs:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>J <span style="color: #666666">=</span> []
<span style="color: #008000">self</span><span style="color: #666666">.</span>testJ <span style="color: #666666">=</span> []
params0 <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>getParams()
options <span style="color: #666666">=</span> {<span style="color: #BA2121">&#39;maxiter&#39;</span>: <span style="color: #666666">200</span>, <span style="color: #BA2121">&#39;disp&#39;</span> : <span style="color: #008000">True</span>}
_res <span style="color: #666666">=</span> optimize<span style="color: #666666">.</span>minimize(<span style="color: #008000">self</span><span style="color: #666666">.</span>costFunctionWrapper, params0, jac<span style="color: #666666">=</span><span style="color: #008000">True</span>, method<span style="color: #666666">=</span><span style="color: #BA2121">&#39;BFGS&#39;</span>, \
args<span style="color: #666666">=</span>(trainX, trainY), options<span style="color: #666666">=</span>options, callback<span style="color: #666666">=</span><span style="color: #008000">self</span><span style="color: #666666">.</span>callbackF)
<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>setParams(_res<span style="color: #666666">.</span>x)
<span style="color: #008000">self</span><span style="color: #666666">.</span>optimizationResults <span style="color: #666666">=</span> _res
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec19">Two-layer Neural Network </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic">#sigmoid</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">nonlin</span>(x, deriv<span style="color: #666666">=</span><span style="color: #008000">False</span>):
<span style="color: #008000; font-weight: bold">if</span> (deriv<span style="color: #666666">==</span><span style="color: #008000">True</span>):
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">*</span>(<span style="color: #666666">1-</span>x)
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x))
<span style="color: #408080; font-style: italic">#input data</span>
x<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>,<span style="color: #666666">0</span>,<span style="color: #666666">1</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1</span>],[<span style="color: #666666">1</span>,<span style="color: #666666">0</span>,<span style="color: #666666">1</span>],[<span style="color: #666666">1</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1</span>]])
<span style="color: #408080; font-style: italic">#output data</span>
y<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1</span>,<span style="color: #666666">0</span>])<span style="color: #666666">.</span>T
<span style="color: #408080; font-style: italic">#seed random numbers to make calculation</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">1</span>)
<span style="color: #408080; font-style: italic">#initialize weights with mean=0</span>
syn0<span style="color: #666666">=2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>random((<span style="color: #666666">3</span>,<span style="color: #666666">4</span>))<span style="color: #666666">-1</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10000</span>):
<span style="color: #408080; font-style: italic">#forward propogation</span>
l0<span style="color: #666666">=</span>x
l1<span style="color: #666666">=</span>nonlin(np<span style="color: #666666">.</span>dot(l0,syn0))
l1_error<span style="color: #666666">=</span>y<span style="color: #666666">-</span>l1
<span style="color: #408080; font-style: italic">#multiply error by slope of sigmoid at values of l1</span>
l1_delta<span style="color: #666666">=</span>l1_error<span style="color: #666666">*</span>nonlin(l1,<span style="color: #008000">True</span>)
<span style="color: #408080; font-style: italic">#update weights</span>
syn0<span style="color: #666666">+=</span>np<span style="color: #666666">.</span>dot(l0<span style="color: #666666">.</span>T, l1_delta)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Output after training: &quot;</span>,l1 )
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">random</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Network</span>(<span style="color: #008000">object</span>):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_init_</span>(<span style="color: #008000">self</span>, sizes):
<span style="color: #008000">self</span><span style="color: #666666">.</span>num_layers<span style="color: #666666">=</span><span style="color: #008000">len</span>(sizes)
<span style="color: #008000">self</span><span style="color: #666666">.</span>sizes<span style="color: #666666">=</span>sizes
<span style="color: #008000">self</span><span style="color: #666666">.</span>biases<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(y,<span style="color: #666666">1</span>) <span style="color: #008000; font-weight: bold">for</span> y <span style="color: #AA22FF; font-weight: bold">in</span> sizes[<span style="color: #666666">1</span>:]]
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(y,x) <span style="color: #008000; font-weight: bold">for</span> x,y <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(sizes[:<span style="color: #666666">-1</span>], sizes[<span style="color: #666666">1</span>:])]
<span style="color: #408080; font-style: italic">#sizes is the number of neurons in each layer</span>
<span style="color: #408080; font-style: italic">#for example, say n_1st_layer=3, n_2nd_layer=3, n_3rd_layer=1, then net=Network([3,3,1])</span>
<span style="color: #408080; font-style: italic">#The biases and weights are initialized randomly, using Gaussian distributions of mean=0, stdev=1</span>
<span style="color: #408080; font-style: italic">#z is a vector (or a np.array)</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">feedforward</span>(<span style="color: #008000">self</span>,a):
<span style="color: #408080; font-style: italic">#returns output w/ &#39;a&#39; as an input</span>
<span style="color: #008000; font-weight: bold">for</span> b, w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>biases, <span style="color: #008000">self</span><span style="color: #666666">.</span>weights):
a<span style="color: #666666">=</span>sigmoid(np<span style="color: #666666">.</span>dot(w,b)<span style="color: #666666">+</span>b)
<span style="color: #008000; font-weight: bold">return</span> a
<span style="color: #408080; font-style: italic">#Apply a Stochastic Gradient Descent (SGD) method:</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SGD</span>(<span style="color: #008000">self</span>, training_data, epochs, mini_batch_size, eta, test_data<span style="color: #666666">=</span><span style="color: #008000">None</span>):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Trains network using batches incorporating SGD. The network will be evaluated against the</span>
<span style="color: #BA2121; font-style: italic"> test data after each epoch, with partial progress being printed out (this is useful for tracking,</span>
<span style="color: #BA2121; font-style: italic"> but slows the process.)&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">if</span> test_data: n_test<span style="color: #666666">=</span><span style="color: #008000">len</span>(test_data)
n<span style="color: #666666">=</span><span style="color: #008000">len</span>(training_data)
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">xrange</span>(epochs):
random<span style="color: #666666">.</span>shuffle(training_data)
mini_batches<span style="color: #666666">=</span>[training_data[k:k<span style="color: #666666">+</span>mini_batch_size] <span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">xrange</span>(o,n,mini_batch_size)]
<span style="color: #008000; font-weight: bold">for</span> mini_batch <span style="color: #AA22FF; font-weight: bold">in</span> mini_batches:
<span style="color: #008000">self</span><span style="color: #666666">.</span>update_mini_batch(mini_batch, eta)
<span style="color: #008000; font-weight: bold">if</span> test_data:
<span style="color: #008000; font-weight: bold">print</span> (<span style="color: #BA2121">&quot;Epoch {0}: {1}/{2}&quot;</span><span style="color: #666666">.</span>format(j, <span style="color: #008000">self</span><span style="color: #666666">.</span>evaluate(test_data), n_test))
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">print</span> (<span style="color: #BA2121">&quot;Epoch {0} complete&quot;</span><span style="color: #666666">.</span>format(j))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_mini_batch</span>(<span style="color: #008000">self</span>, mini_batch, eta):
<span style="color: #408080; font-style: italic">#updates w and b using backpropagation to a single mini batch. eta is the learning rate.&quot;</span>
nabla_b<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>zeros(b<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> b <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>biases]
nabla_w<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>zeros(w<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights]
<span style="color: #008000; font-weight: bold">for</span> x,y <span style="color: #AA22FF; font-weight: bold">in</span> mini_batch:
delta_nabla_b, delta_nabla_w<span style="color: #666666">=</span><span style="color: #008000">self</span><span style="color: #666666">.</span>backprop(x,y)
nabla_b<span style="color: #666666">=</span>[nb<span style="color: #666666">+</span>dnb <span style="color: #008000; font-weight: bold">for</span> nb, dnb <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(nabla_b, delta_nabla_b)]
nabla_w<span style="color: #666666">=</span>[nw<span style="color: #666666">+</span>dnw <span style="color: #008000; font-weight: bold">for</span> nw, dnw <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(nabla_w, delta_nabla_w)]
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights<span style="color: #666666">=</span>[w<span style="color: #666666">-</span>(eta<span style="color: #666666">/</span><span style="color: #008000">len</span>(mini_batch))<span style="color: #666666">*</span>nw <span style="color: #008000; font-weight: bold">for</span> w, nw <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>weights, nabla_w)]
<span style="color: #008000">self</span><span style="color: #666666">.</span>biases<span style="color: #666666">=</span>[b<span style="color: #666666">-</span>(eta<span style="color: #666666">/</span><span style="color: #008000">len</span>(mini_batch))<span style="color: #666666">*</span>nb <span style="color: #008000; font-weight: bold">for</span> b, nb <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>biases, nabla_b)]
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">backprop</span>(<span style="color: #008000">self</span>, x, y):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return a tuple ``(nabla_b, nabla_w)`` representing the</span>
<span style="color: #BA2121; font-style: italic"> gradient for the cost function C_x. ``nabla_b`` and</span>
<span style="color: #BA2121; font-style: italic"> ``nabla_w`` are layer-by-layer lists of numpy arrays, similar</span>
<span style="color: #BA2121; font-style: italic"> to ``self.biases`` and ``self.weights``.&quot;&quot;&quot;</span>
nabla_b <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>zeros(b<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> b <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>biases]
nabla_w <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>zeros(w<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights]
<span style="color: #408080; font-style: italic"># feedforward</span>
activation <span style="color: #666666">=</span> x
activations <span style="color: #666666">=</span> [x] <span style="color: #408080; font-style: italic"># list to store all the activations, layer by layer</span>
zs <span style="color: #666666">=</span> [] <span style="color: #408080; font-style: italic"># list to store all the z vectors, layer by layer</span>
<span style="color: #008000; font-weight: bold">for</span> b, w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>biases, <span style="color: #008000">self</span><span style="color: #666666">.</span>weights):
z <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(w, activation)<span style="color: #666666">+</span>b
zs<span style="color: #666666">.</span>append(z)
activation <span style="color: #666666">=</span> sigmoid(z)
activations<span style="color: #666666">.</span>append(activation)
<span style="color: #408080; font-style: italic"># backward pass</span>
delta <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>cost_derivative(activations[<span style="color: #666666">-1</span>], y) <span style="color: #666666">*</span> \
sigmoid_prime(zs[<span style="color: #666666">-1</span>])
nabla_b[<span style="color: #666666">-1</span>] <span style="color: #666666">=</span> delta
nabla_w[<span style="color: #666666">-1</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(delta, activations[<span style="color: #666666">-2</span>]<span style="color: #666666">.</span>transpose())
<span style="color: #408080; font-style: italic"># Note that the variable l in the loop below is used a little</span>
<span style="color: #408080; font-style: italic"># differently to the notation in Chapter 2 of the book. Here,</span>
<span style="color: #408080; font-style: italic"># l = 1 means the last layer of neurons, l = 2 is the</span>
<span style="color: #408080; font-style: italic"># second-last layer, and so on. It&#39;s a renumbering of the</span>
<span style="color: #408080; font-style: italic"># scheme in the book, used here to take advantage of the fact</span>
<span style="color: #408080; font-style: italic"># that Python can use negative indices in lists.</span>
<span style="color: #008000; font-weight: bold">for</span> l <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">xrange</span>(<span style="color: #666666">2</span>, <span style="color: #008000">self</span><span style="color: #666666">.</span>num_layers):
z <span style="color: #666666">=</span> zs[<span style="color: #666666">-</span>l]
sp <span style="color: #666666">=</span> sigmoid_prime(z)
delta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(<span style="color: #008000">self</span><span style="color: #666666">.</span>weights[<span style="color: #666666">-</span>l<span style="color: #666666">+1</span>]<span style="color: #666666">.</span>transpose(), delta) <span style="color: #666666">*</span> sp
nabla_b[<span style="color: #666666">-</span>l] <span style="color: #666666">=</span> delta
nabla_w[<span style="color: #666666">-</span>l] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(delta, activations[<span style="color: #666666">-</span>l<span style="color: #666666">-1</span>]<span style="color: #666666">.</span>transpose())
<span style="color: #008000; font-weight: bold">return</span> (nabla_b, nabla_w)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">evaluate</span>(<span style="color: #008000">self</span>, test_data):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return the number of test inputs for which the neural</span>
<span style="color: #BA2121; font-style: italic"> network outputs the correct result. Note that the neural</span>
<span style="color: #BA2121; font-style: italic"> network&#39;s output is assumed to be the index of whichever</span>
<span style="color: #BA2121; font-style: italic"> neuron in the final layer has the highest activation.&quot;&quot;&quot;</span>
test_results <span style="color: #666666">=</span> [(np<span style="color: #666666">.</span>argmax(<span style="color: #008000">self</span><span style="color: #666666">.</span>feedforward(x)), y)
<span style="color: #008000; font-weight: bold">for</span> (x, y) <span style="color: #AA22FF; font-weight: bold">in</span> test_data]
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">sum</span>(<span style="color: #008000">int</span>(x <span style="color: #666666">==</span> y) <span style="color: #008000; font-weight: bold">for</span> (x, y) <span style="color: #AA22FF; font-weight: bold">in</span> test_results)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">cost_derivative</span>(<span style="color: #008000">self</span>, output_activations, y):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return the vector of partial derivatives \partial C_x /</span>
<span style="color: #BA2121; font-style: italic"> \partial a for the output activations.&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">return</span> (output_activations<span style="color: #666666">-</span>y)
<span style="color: #408080; font-style: italic">#Functions</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(z):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1.0/</span>(<span style="color: #666666">1.0+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid_prime</span>(z):
<span style="color: #008000; font-weight: bold">return</span> sigmoid(z)<span style="color: #666666">*</span>(<span style="color: #666666">1-</span>sigmoid(z))
network<span style="color: #666666">=</span>Network()
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># %load neural-networks-and-deep-learning/src/mnist_loader.py</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic">mnist_loader</span>
<span style="color: #BA2121; font-style: italic">~~~~~~~~~~~~</span>
<span style="color: #BA2121; font-style: italic">A library to load the MNIST image data. For details of the data</span>
<span style="color: #BA2121; font-style: italic">structures that are returned, see the doc strings for ``load_data``</span>
<span style="color: #BA2121; font-style: italic">and ``load_data_wrapper``. In practice, ``load_data_wrapper`` is the</span>
<span style="color: #BA2121; font-style: italic">function usually called by our neural network code.</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic">#### Libraries</span>
<span style="color: #408080; font-style: italic"># Standard library</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pickle</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">gzip</span>
<span style="color: #408080; font-style: italic"># Third-party libraries</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">load_data</span>():
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return the MNIST data as a tuple containing the training data,</span>
<span style="color: #BA2121; font-style: italic"> the validation data, and the test data.</span>
<span style="color: #BA2121; font-style: italic"> The ``training_data`` is returned as a tuple with two entries.</span>
<span style="color: #BA2121; font-style: italic"> The first entry contains the actual training images. This is a</span>
<span style="color: #BA2121; font-style: italic"> numpy ndarray with 50,000 entries. Each entry is, in turn, a</span>
<span style="color: #BA2121; font-style: italic"> numpy ndarray with 784 values, representing the 28 * 28 = 784</span>
<span style="color: #BA2121; font-style: italic"> pixels in a single MNIST image.</span>
<span style="color: #BA2121; font-style: italic"> The second entry in the ``training_data`` tuple is a numpy ndarray</span>
<span style="color: #BA2121; font-style: italic"> containing 50,000 entries. Those entries are just the digit</span>
<span style="color: #BA2121; font-style: italic"> values (0...9) for the corresponding images contained in the first</span>
<span style="color: #BA2121; font-style: italic"> entry of the tuple.</span>
<span style="color: #BA2121; font-style: italic"> The ``validation_data`` and ``test_data`` are similar, except</span>
<span style="color: #BA2121; font-style: italic"> each contains only 10,000 images.</span>
<span style="color: #BA2121; font-style: italic"> This is a nice data format, but for use in neural networks it&#39;s</span>
<span style="color: #BA2121; font-style: italic"> helpful to modify the format of the ``training_data`` a little.</span>
<span style="color: #BA2121; font-style: italic"> That&#39;s done in the wrapper function ``load_data_wrapper()``, see</span>
<span style="color: #BA2121; font-style: italic"> below.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
f <span style="color: #666666">=</span> gzip<span style="color: #666666">.</span>open(<span style="color: #BA2121">&#39;../data/mnist.pkl.gz&#39;</span>, <span style="color: #BA2121">&#39;rb&#39;</span>)
training_data, validation_data, test_data <span style="color: #666666">=</span> cPickle<span style="color: #666666">.</span>load(f)
f<span style="color: #666666">.</span>close()
<span style="color: #008000; font-weight: bold">return</span> (training_data, validation_data, test_data)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">load_data_wrapper</span>():
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return a tuple containing ``(training_data, validation_data,</span>
<span style="color: #BA2121; font-style: italic"> test_data)``. Based on ``load_data``, but the format is more</span>
<span style="color: #BA2121; font-style: italic"> convenient for use in our implementation of neural networks.</span>
<span style="color: #BA2121; font-style: italic"> In particular, ``training_data`` is a list containing 50,000</span>
<span style="color: #BA2121; font-style: italic"> 2-tuples ``(x, y)``. ``x`` is a 784-dimensional numpy.ndarray</span>
<span style="color: #BA2121; font-style: italic"> containing the input image. ``y`` is a 10-dimensional</span>
<span style="color: #BA2121; font-style: italic"> numpy.ndarray representing the unit vector corresponding to the</span>
<span style="color: #BA2121; font-style: italic"> correct digit for ``x``.</span>
<span style="color: #BA2121; font-style: italic"> ``validation_data`` and ``test_data`` are lists containing 10,000</span>
<span style="color: #BA2121; font-style: italic"> 2-tuples ``(x, y)``. In each case, ``x`` is a 784-dimensional</span>
<span style="color: #BA2121; font-style: italic"> numpy.ndarry containing the input image, and ``y`` is the</span>
<span style="color: #BA2121; font-style: italic"> corresponding classification, i.e., the digit values (integers)</span>
<span style="color: #BA2121; font-style: italic"> corresponding to ``x``.</span>
<span style="color: #BA2121; font-style: italic"> Obviously, this means we&#39;re using slightly different formats for</span>
<span style="color: #BA2121; font-style: italic"> the training data and the validation / test data. These formats</span>
<span style="color: #BA2121; font-style: italic"> turn out to be the most convenient for use in our neural network</span>
<span style="color: #BA2121; font-style: italic"> code.&quot;&quot;&quot;</span>
tr_d, va_d, te_d <span style="color: #666666">=</span> load_data()
training_inputs <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>reshape(x, (<span style="color: #666666">784</span>, <span style="color: #666666">1</span>)) <span style="color: #008000; font-weight: bold">for</span> x <span style="color: #AA22FF; font-weight: bold">in</span> tr_d[<span style="color: #666666">0</span>]]
training_results <span style="color: #666666">=</span> [vectorized_result(y) <span style="color: #008000; font-weight: bold">for</span> y <span style="color: #AA22FF; font-weight: bold">in</span> tr_d[<span style="color: #666666">1</span>]]
training_data <span style="color: #666666">=</span> <span style="color: #008000">zip</span>(training_inputs, training_results)
validation_inputs <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>reshape(x, (<span style="color: #666666">784</span>, <span style="color: #666666">1</span>)) <span style="color: #008000; font-weight: bold">for</span> x <span style="color: #AA22FF; font-weight: bold">in</span> va_d[<span style="color: #666666">0</span>]]
validation_data <span style="color: #666666">=</span> <span style="color: #008000">zip</span>(validation_inputs, va_d[<span style="color: #666666">1</span>])
test_inputs <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>reshape(x, (<span style="color: #666666">784</span>, <span style="color: #666666">1</span>)) <span style="color: #008000; font-weight: bold">for</span> x <span style="color: #AA22FF; font-weight: bold">in</span> te_d[<span style="color: #666666">0</span>]]
test_data <span style="color: #666666">=</span> <span style="color: #008000">zip</span>(test_inputs, te_d[<span style="color: #666666">1</span>])
<span style="color: #008000; font-weight: bold">return</span> (training_data, validation_data, test_data)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">vectorized_result</span>(j):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return a 10-dimensional unit vector with a 1.0 in the jth</span>
<span style="color: #BA2121; font-style: italic"> position and zeroes elsewhere. This is used to convert a digit</span>
<span style="color: #BA2121; font-style: italic"> (0...9) into a corresponding desired output from the neural</span>
<span style="color: #BA2121; font-style: italic"> network.&quot;&quot;&quot;</span>
e <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #666666">10</span>, <span style="color: #666666">1</span>))
e[j] <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
<span style="color: #008000; font-weight: bold">return</span> e
net<span style="color: #666666">=</span>network<span style="color: #666666">.</span>Network([<span style="color: #666666">784</span>,<span style="color: #666666">30</span>,<span style="color: #666666">30</span>])
net<span style="color: #666666">.</span>SGD(training_data,<span style="color: #666666">30</span>,<span style="color: #666666">10</span>,<span style="color: #666666">3</span>,test_data<span style="color: #666666">=</span>test_data)
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink