lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updating various chapters with codes

Browse Source
This commit is contained in:
mhjensen
2018-05-30 23:00:43 -04:00
parent 045a91c09b
commit 8bf7984604
95 changed files with 7570 additions and 1710 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/NeuralNet.html
+513 -26
View File
@@ -60,7 +60,8 @@ div { text-align: justify; text-justify: inter-word; }
3,
None,
'___sec17'),
('Relevance', 3, None, '___sec18')]}
('Relevance', 3, None, '___sec18'),
('Two-layer Neural Network', 2, None, '___sec19')]}
end of tocinfo -->
<body>
@@ -102,7 +103,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Nov 26, 2017</h4></center> <!-- date -->
<center><h4>May 30, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -110,16 +111,20 @@ MathJax.Hub.Config({
<h2 id="___sec0">What is Machine Learning? </h2>
<p>
Machine learning is the science of giving computers the ability to learn without being explicitly programmed.
The idea is that there exist generic algorithms which can be used to find patterns in a broad class of data sets without
having to write code specifically for each problem. The algorithm will build its own logic based on the data.
Machine learning is the science of giving computers the ability to
learn without being explicitly programmed. The idea is that there
exist generic algorithms which can be used to find patterns in a broad
class of data sets without having to write code specifically for each
problem. The algorithm will build its own logic based on the data.
<p>
Machine learning is a subfield of computer science, and is closely related to computational statistics.
It evolved from the study of pattern recognition in artificial intelligence (AI) research, and has made contributions to
AI tasks like computer vision, natural language processing
and speech recognition. It has also, especially in later years,
found applications in a wide variety of other areas, including bioinformatics, economy, physics, finance and marketing.
Machine learning is a subfield of computer science, and is closely
related to computational statistics. It evolved from the study of
pattern recognition in artificial intelligence (AI) research, and has
made contributions to AI tasks like computer vision, natural language
processing and speech recognition. It has also, especially in later
years, found applications in a wide variety of other areas, including
bioinformatics, economy, physics, finance and marketing.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -148,14 +153,19 @@ Some of the most common tasks are:
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Artificial neurons </h2>
The field of artificial neural networks has a long history of development, and is closely connected with
the advancement of computer science and computers in general. A model of artificial neurons
was first developed by McCulloch and Pitts in 1943 to study signal processing in the brain and
has later been refined by others. The general idea is to mimic neural networks in the human brain, which
is composed of billions of neurons that communicate with each other by sending electrical signals.
Each neuron accumulates its incoming signals,
which must exceed an activation threshold to yield an output. If the threshold is not overcome, the neuron
remains inactive, i.e. has zero output.
<p>
The field of artificial neural networks has a long history of
development, and is closely connected with the advancement of computer
science and computers in general. A model of artificial neurons was
first developed by McCulloch and Pitts in 1943 to study signal
processing in the brain and has later been refined by others. The
general idea is to mimic neural networks in the human brain, which is
composed of billions of neurons that communicate with each other by
sending electrical signals. Each neuron accumulates its incoming
signals, which must exceed an activation threshold to yield an
output. If the threshold is not overcome, the neuron remains inactive,
i.e. has zero output.
<p>
This behaviour has inspired a simple mathematical model for an artificial neuron.
@@ -170,24 +180,67 @@ $$
Here, the output \( y \) of the neuron is the value of its activation function, which have as input
a weighted sum of signals \( x_i, \dots ,x_n \) received by \( n \) other neurons.
<p>
Conceptually, it is helpful to divide neural networks into four
categories:
<ol>
<li> general purpose neural networks for supervised learning,</li>
<li> neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs),</li>
<li> neural networks for sequential data such as Recurrent Neural Networks (RNNs), and</li>
<li> neural networks for unsupervised learning such as Deep Boltzmann Machines.</li>
</ol>
In physics, DNNs and CNNs have already found numerous applications. In
statistical physics, they have been applied to detect phase
transitions in 2D Ising and Potts models, lattice gauge theories, and
different phases of polymers.
Deep learning has also found interesting applications in quantum
physics. Various quantum phase transitions can be detected and studied
using DNNs and CNNs, including the transverse-field Ising model,
topological phases, and even non-equilibrium many-body
localization. Representing quantum states as DNNs quantum state
tomography are among some of the impressive
achievements to reveal the potential of DNNs to facilitate the study
of quantum systems.
<p>
In quantum information theory, it has been shown that one can perform
gate decompositions with the help of neural. In lattice quantum chromodynamics,
DNNs have been used to learn action parameters in regions of parameter
space where PCA fails. Last but not least,
DNNs also found place in the study of quantum, and in scattering theory to learn
\( s \)-wave scattering length of potentials.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Neural network types </h2>
<p>
An artificial neural network (NN), is a computational model that consists of layers of connected neurons, or <em>nodes</em>.
It is supposed to mimic a biological nervous system by letting each neuron interact with other neurons
by sending signals in the form of mathematical functions between layers.
A wide variety of different NNs have
been developed, but most of them consist of an input layer, an output layer and eventual layers in-between, called
<em>hidden layers</em>. All layers can contain an arbitrary number of nodes, and each connection between two nodes
is associated with a weight variable.
An artificial neural network (NN), is a computational model that
consists of layers of connected neurons, or <em>nodes</em>. It is supposed
to mimic a biological nervous system by letting each neuron interact
with other neurons by sending signals in the form of mathematical
functions between layers. A wide variety of different NNs have been
developed, but most of them consist of an input layer, an output layer
and eventual layers in-between, called <em>hidden layers</em>. All layers can
contain an arbitrary number of nodes, and each connection between two
nodes is associated with a weight variable.
<p>
Neural networks (also called neural nets) are neural-inspired
nonlinear models for supervised learning. As we will see, neural nets
can be viewed as natural, more powerful extensions of supervised
learning methods such as linear and logistic regression and soft-max
methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Feed-forward neural networks </h2>
<p>
The feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network,
the information moves in only one direction: forward through the layers.
@@ -522,11 +575,445 @@ has become the most popular for <em>deep neural networks</em>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;The sigmoid function (or the logistic curve) is a </span>
<span style="color: #BA2121; font-style: italic">function that takes any real number, z, and outputs a number (0,1).</span>
<span style="color: #BA2121; font-style: italic">It is useful in neural networks for assigning weights on a relative scale.</span>
<span style="color: #BA2121; font-style: italic">The value z is the weighted sum of parameters involved in the learning algorithm.&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">mt</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.1</span>)
sigma_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>numpy<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z)))
sigma <span style="color: #666666">=</span> sigma_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, sigma)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.1</span>, <span style="color: #666666">1.1</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;sigmoid function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Step Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-5</span>, <span style="color: #666666">5</span>, <span style="color: #666666">.02</span>)
step_fn <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>vectorize(<span style="color: #008000; font-weight: bold">lambda</span> z: <span style="color: #666666">1.0</span> <span style="color: #008000; font-weight: bold">if</span> z <span style="color: #666666">&gt;=</span> <span style="color: #666666">0.0</span> <span style="color: #008000; font-weight: bold">else</span> <span style="color: #666666">0.0</span>)
step <span style="color: #666666">=</span> step_fn(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, step)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-0.5</span>, <span style="color: #666666">1.5</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-5</span>,<span style="color: #666666">5</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;step function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Sine Function&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi, <span style="color: #666666">0.1</span>)
t <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>sin(z)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, t)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-1.0</span>, <span style="color: #666666">1.0</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-2*</span>mt<span style="color: #666666">.</span>pi,<span style="color: #666666">2*</span>mt<span style="color: #666666">.</span>pi])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;sine function&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Plots a graph of the squashing function used by a rectified linear</span>
<span style="color: #BA2121; font-style: italic">unit&quot;&quot;&quot;</span>
z <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>arange(<span style="color: #666666">-2</span>, <span style="color: #666666">2</span>, <span style="color: #666666">.1</span>)
zero <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>zeros(<span style="color: #008000">len</span>(z))
y <span style="color: #666666">=</span> numpy<span style="color: #666666">.</span>max([zero, z], axis<span style="color: #666666">=0</span>)
fig <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>add_subplot(<span style="color: #666666">111</span>)
ax<span style="color: #666666">.</span>plot(z, y)
ax<span style="color: #666666">.</span>set_ylim([<span style="color: #666666">-2.0</span>, <span style="color: #666666">2.0</span>])
ax<span style="color: #666666">.</span>set_xlim([<span style="color: #666666">-2.0</span>, <span style="color: #666666">2.0</span>])
ax<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&#39;z&#39;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&#39;Rectified linear unit&#39;</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">scipy</span> <span style="color: #008000; font-weight: bold">import</span> optimize
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Neural_Network</span>(<span style="color: #008000">object</span>):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, Lambda<span style="color: #666666">=0</span>):
<span style="color: #408080; font-style: italic">#Define Hyperparameters</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize <span style="color: #666666">=</span> <span style="color: #666666">2</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize <span style="color: #666666">=</span> <span style="color: #666666">3</span>
<span style="color: #408080; font-style: italic">#Weights (parameters)</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>W1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize,<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize)
<span style="color: #008000">self</span><span style="color: #666666">.</span>W2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize,<span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize)
<span style="color: #408080; font-style: italic">#Regularization Parameter:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda <span style="color: #666666">=</span> Lambda
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">forward</span>(<span style="color: #008000">self</span>, X):
<span style="color: #408080; font-style: italic">#Propogate inputs though network</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(X, <span style="color: #008000">self</span><span style="color: #666666">.</span>W1)
<span style="color: #008000">self</span><span style="color: #666666">.</span>a2 <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoid(<span style="color: #008000">self</span><span style="color: #666666">.</span>z2)
<span style="color: #008000">self</span><span style="color: #666666">.</span>z3 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(<span style="color: #008000">self</span><span style="color: #666666">.</span>a2, <span style="color: #008000">self</span><span style="color: #666666">.</span>W2)
yHat <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoid(<span style="color: #008000">self</span><span style="color: #666666">.</span>z3)
<span style="color: #008000; font-weight: bold">return</span> yHat
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(<span style="color: #008000">self</span>, z):
<span style="color: #408080; font-style: italic">#Apply sigmoid activation function to scalar, vector, or matrix</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoidPrime</span>(<span style="color: #008000">self</span>,z):
<span style="color: #408080; font-style: italic">#Gradient of sigmoid</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z)<span style="color: #666666">/</span>((<span style="color: #666666">1+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z))<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">costFunction</span>(<span style="color: #008000">self</span>, X, y):
<span style="color: #408080; font-style: italic">#Compute cost for given X,y, use weights already stored in class.</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>yHat <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>forward(X)
J <span style="color: #666666">=</span> <span style="color: #666666">0.5*</span><span style="color: #008000">sum</span>((y<span style="color: #666666">-</span><span style="color: #008000">self</span><span style="color: #666666">.</span>yHat)<span style="color: #666666">**2</span>)<span style="color: #666666">/</span>X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda<span style="color: #666666">/2</span>)<span style="color: #666666">*</span>(np<span style="color: #666666">.</span>sum(<span style="color: #008000">self</span><span style="color: #666666">.</span>W1<span style="color: #666666">**2</span>)<span style="color: #666666">+</span>np<span style="color: #666666">.</span>sum(<span style="color: #008000">self</span><span style="color: #666666">.</span>W2<span style="color: #666666">**2</span>))
<span style="color: #008000; font-weight: bold">return</span> J
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">costFunctionPrime</span>(<span style="color: #008000">self</span>, X, y):
<span style="color: #408080; font-style: italic">#Compute derivative with respect to W and W2 for a given X and y:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>yHat <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>forward(X)
delta3 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>multiply(<span style="color: #666666">-</span>(y<span style="color: #666666">-</span><span style="color: #008000">self</span><span style="color: #666666">.</span>yHat), <span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoidPrime(<span style="color: #008000">self</span><span style="color: #666666">.</span>z3))
<span style="color: #408080; font-style: italic">#Add gradient of regularization term:</span>
dJdW2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(<span style="color: #008000">self</span><span style="color: #666666">.</span>a2<span style="color: #666666">.</span>T, delta3)<span style="color: #666666">/</span>X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>W2
delta2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(delta3, <span style="color: #008000">self</span><span style="color: #666666">.</span>W2<span style="color: #666666">.</span>T)<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>sigmoidPrime(<span style="color: #008000">self</span><span style="color: #666666">.</span>z2)
<span style="color: #408080; font-style: italic">#Add gradient of regularization term:</span>
dJdW1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(X<span style="color: #666666">.</span>T, delta2)<span style="color: #666666">/</span>X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>Lambda<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>W1
<span style="color: #008000; font-weight: bold">return</span> dJdW1, dJdW2
<span style="color: #408080; font-style: italic">#Helper functions for interacting with other methods/classes</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">getParams</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic">#Get W1 and W2 Rolled into vector:</span>
params <span style="color: #666666">=</span> np<span style="color: #666666">.</span>concatenate((<span style="color: #008000">self</span><span style="color: #666666">.</span>W1<span style="color: #666666">.</span>ravel(), <span style="color: #008000">self</span><span style="color: #666666">.</span>W2<span style="color: #666666">.</span>ravel()))
<span style="color: #008000; font-weight: bold">return</span> params
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">setParams</span>(<span style="color: #008000">self</span>, params):
<span style="color: #408080; font-style: italic">#Set W1 and W2 using single parameter vector:</span>
W1_start <span style="color: #666666">=</span> <span style="color: #666666">0</span>
W1_end <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize
<span style="color: #008000">self</span><span style="color: #666666">.</span>W1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>reshape(params[W1_start:W1_end], \
(<span style="color: #008000">self</span><span style="color: #666666">.</span>inputLayerSize, <span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize))
W2_end <span style="color: #666666">=</span> W1_end <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize<span style="color: #666666">*</span><span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize
<span style="color: #008000">self</span><span style="color: #666666">.</span>W2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>reshape(params[W1_end:W2_end], \
(<span style="color: #008000">self</span><span style="color: #666666">.</span>hiddenLayerSize, <span style="color: #008000">self</span><span style="color: #666666">.</span>outputLayerSize))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">computeGradients</span>(<span style="color: #008000">self</span>, X, y):
dJdW1, dJdW2 <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>costFunctionPrime(X, y)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>concatenate((dJdW1<span style="color: #666666">.</span>ravel(), dJdW2<span style="color: #666666">.</span>ravel()))
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">trainer</span>(<span style="color: #008000">object</span>):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, N):
<span style="color: #408080; font-style: italic">#Make Local reference to network:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>N <span style="color: #666666">=</span> N
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">callbackF</span>(<span style="color: #008000">self</span>, params):
<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>setParams(params)
<span style="color: #008000">self</span><span style="color: #666666">.</span>J<span style="color: #666666">.</span>append(<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>costFunction(<span style="color: #008000">self</span><span style="color: #666666">.</span>X, <span style="color: #008000">self</span><span style="color: #666666">.</span>y))
<span style="color: #008000">self</span><span style="color: #666666">.</span>testJ<span style="color: #666666">.</span>append(<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>costFunction(<span style="color: #008000">self</span><span style="color: #666666">.</span>testX, <span style="color: #008000">self</span><span style="color: #666666">.</span>testY))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">costFunctionWrapper</span>(<span style="color: #008000">self</span>, params, X, y):
<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>setParams(params)
cost <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>costFunction(X, y)
grad <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>computeGradients(X,y)
<span style="color: #008000; font-weight: bold">return</span> cost, grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">train</span>(<span style="color: #008000">self</span>, trainX, trainY, testX, testY):
<span style="color: #408080; font-style: italic">#Make an internal variable for the callback function:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>X <span style="color: #666666">=</span> trainX
<span style="color: #008000">self</span><span style="color: #666666">.</span>y <span style="color: #666666">=</span> trainY
<span style="color: #008000">self</span><span style="color: #666666">.</span>testX <span style="color: #666666">=</span> testX
<span style="color: #008000">self</span><span style="color: #666666">.</span>testY <span style="color: #666666">=</span> testY
<span style="color: #408080; font-style: italic">#Make empty list to store training costs:</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>J <span style="color: #666666">=</span> []
<span style="color: #008000">self</span><span style="color: #666666">.</span>testJ <span style="color: #666666">=</span> []
params0 <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>getParams()
options <span style="color: #666666">=</span> {<span style="color: #BA2121">&#39;maxiter&#39;</span>: <span style="color: #666666">200</span>, <span style="color: #BA2121">&#39;disp&#39;</span> : <span style="color: #008000">True</span>}
_res <span style="color: #666666">=</span> optimize<span style="color: #666666">.</span>minimize(<span style="color: #008000">self</span><span style="color: #666666">.</span>costFunctionWrapper, params0, jac<span style="color: #666666">=</span><span style="color: #008000">True</span>, method<span style="color: #666666">=</span><span style="color: #BA2121">&#39;BFGS&#39;</span>, \
args<span style="color: #666666">=</span>(trainX, trainY), options<span style="color: #666666">=</span>options, callback<span style="color: #666666">=</span><span style="color: #008000">self</span><span style="color: #666666">.</span>callbackF)
<span style="color: #008000">self</span><span style="color: #666666">.</span>N<span style="color: #666666">.</span>setParams(_res<span style="color: #666666">.</span>x)
<span style="color: #008000">self</span><span style="color: #666666">.</span>optimizationResults <span style="color: #666666">=</span> _res
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec19">Two-layer Neural Network </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic">#sigmoid</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">nonlin</span>(x, deriv<span style="color: #666666">=</span><span style="color: #008000">False</span>):
<span style="color: #008000; font-weight: bold">if</span> (deriv<span style="color: #666666">==</span><span style="color: #008000">True</span>):
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">*</span>(<span style="color: #666666">1-</span>x)
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1/</span>(<span style="color: #666666">1+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>x))
<span style="color: #408080; font-style: italic">#input data</span>
x<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array([[<span style="color: #666666">0</span>,<span style="color: #666666">0</span>,<span style="color: #666666">1</span>],[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1</span>],[<span style="color: #666666">1</span>,<span style="color: #666666">0</span>,<span style="color: #666666">1</span>],[<span style="color: #666666">1</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1</span>]])
<span style="color: #408080; font-style: italic">#output data</span>
y<span style="color: #666666">=</span>np<span style="color: #666666">.</span>array([<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1</span>,<span style="color: #666666">0</span>])<span style="color: #666666">.</span>T
<span style="color: #408080; font-style: italic">#seed random numbers to make calculation</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">1</span>)
<span style="color: #408080; font-style: italic">#initialize weights with mean=0</span>
syn0<span style="color: #666666">=2*</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>random((<span style="color: #666666">3</span>,<span style="color: #666666">4</span>))<span style="color: #666666">-1</span>
<span style="color: #008000; font-weight: bold">for</span> <span style="color: #008000">iter</span> <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10000</span>):
<span style="color: #408080; font-style: italic">#forward propogation</span>
l0<span style="color: #666666">=</span>x
l1<span style="color: #666666">=</span>nonlin(np<span style="color: #666666">.</span>dot(l0,syn0))
l1_error<span style="color: #666666">=</span>y<span style="color: #666666">-</span>l1
<span style="color: #408080; font-style: italic">#multiply error by slope of sigmoid at values of l1</span>
l1_delta<span style="color: #666666">=</span>l1_error<span style="color: #666666">*</span>nonlin(l1,<span style="color: #008000">True</span>)
<span style="color: #408080; font-style: italic">#update weights</span>
syn0<span style="color: #666666">+=</span>np<span style="color: #666666">.</span>dot(l0<span style="color: #666666">.</span>T, l1_delta)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Output after training: &quot;</span>,l1 )
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">random</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Network</span>(<span style="color: #008000">object</span>):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_init_</span>(<span style="color: #008000">self</span>, sizes):
<span style="color: #008000">self</span><span style="color: #666666">.</span>num_layers<span style="color: #666666">=</span><span style="color: #008000">len</span>(sizes)
<span style="color: #008000">self</span><span style="color: #666666">.</span>sizes<span style="color: #666666">=</span>sizes
<span style="color: #008000">self</span><span style="color: #666666">.</span>biases<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(y,<span style="color: #666666">1</span>) <span style="color: #008000; font-weight: bold">for</span> y <span style="color: #AA22FF; font-weight: bold">in</span> sizes[<span style="color: #666666">1</span>:]]
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(y,x) <span style="color: #008000; font-weight: bold">for</span> x,y <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(sizes[:<span style="color: #666666">-1</span>], sizes[<span style="color: #666666">1</span>:])]
<span style="color: #408080; font-style: italic">#sizes is the number of neurons in each layer</span>
<span style="color: #408080; font-style: italic">#for example, say n_1st_layer=3, n_2nd_layer=3, n_3rd_layer=1, then net=Network([3,3,1])</span>
<span style="color: #408080; font-style: italic">#The biases and weights are initialized randomly, using Gaussian distributions of mean=0, stdev=1</span>
<span style="color: #408080; font-style: italic">#z is a vector (or a np.array)</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">feedforward</span>(<span style="color: #008000">self</span>,a):
<span style="color: #408080; font-style: italic">#returns output w/ &#39;a&#39; as an input</span>
<span style="color: #008000; font-weight: bold">for</span> b, w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>biases, <span style="color: #008000">self</span><span style="color: #666666">.</span>weights):
a<span style="color: #666666">=</span>sigmoid(np<span style="color: #666666">.</span>dot(w,b)<span style="color: #666666">+</span>b)
<span style="color: #008000; font-weight: bold">return</span> a
<span style="color: #408080; font-style: italic">#Apply a Stochastic Gradient Descent (SGD) method:</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SGD</span>(<span style="color: #008000">self</span>, training_data, epochs, mini_batch_size, eta, test_data<span style="color: #666666">=</span><span style="color: #008000">None</span>):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Trains network using batches incorporating SGD. The network will be evaluated against the</span>
<span style="color: #BA2121; font-style: italic"> test data after each epoch, with partial progress being printed out (this is useful for tracking,</span>
<span style="color: #BA2121; font-style: italic"> but slows the process.)&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">if</span> test_data: n_test<span style="color: #666666">=</span><span style="color: #008000">len</span>(test_data)
n<span style="color: #666666">=</span><span style="color: #008000">len</span>(training_data)
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">xrange</span>(epochs):
random<span style="color: #666666">.</span>shuffle(training_data)
mini_batches<span style="color: #666666">=</span>[training_data[k:k<span style="color: #666666">+</span>mini_batch_size] <span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">xrange</span>(o,n,mini_batch_size)]
<span style="color: #008000; font-weight: bold">for</span> mini_batch <span style="color: #AA22FF; font-weight: bold">in</span> mini_batches:
<span style="color: #008000">self</span><span style="color: #666666">.</span>update_mini_batch(mini_batch, eta)
<span style="color: #008000; font-weight: bold">if</span> test_data:
<span style="color: #008000; font-weight: bold">print</span> (<span style="color: #BA2121">&quot;Epoch {0}: {1}/{2}&quot;</span><span style="color: #666666">.</span>format(j, <span style="color: #008000">self</span><span style="color: #666666">.</span>evaluate(test_data), n_test))
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">print</span> (<span style="color: #BA2121">&quot;Epoch {0} complete&quot;</span><span style="color: #666666">.</span>format(j))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_mini_batch</span>(<span style="color: #008000">self</span>, mini_batch, eta):
<span style="color: #408080; font-style: italic">#updates w and b using backpropagation to a single mini batch. eta is the learning rate.&quot;</span>
nabla_b<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>zeros(b<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> b <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>biases]
nabla_w<span style="color: #666666">=</span>[np<span style="color: #666666">.</span>zeros(w<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights]
<span style="color: #008000; font-weight: bold">for</span> x,y <span style="color: #AA22FF; font-weight: bold">in</span> mini_batch:
delta_nabla_b, delta_nabla_w<span style="color: #666666">=</span><span style="color: #008000">self</span><span style="color: #666666">.</span>backprop(x,y)
nabla_b<span style="color: #666666">=</span>[nb<span style="color: #666666">+</span>dnb <span style="color: #008000; font-weight: bold">for</span> nb, dnb <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(nabla_b, delta_nabla_b)]
nabla_w<span style="color: #666666">=</span>[nw<span style="color: #666666">+</span>dnw <span style="color: #008000; font-weight: bold">for</span> nw, dnw <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(nabla_w, delta_nabla_w)]
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights<span style="color: #666666">=</span>[w<span style="color: #666666">-</span>(eta<span style="color: #666666">/</span><span style="color: #008000">len</span>(mini_batch))<span style="color: #666666">*</span>nw <span style="color: #008000; font-weight: bold">for</span> w, nw <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>weights, nabla_w)]
<span style="color: #008000">self</span><span style="color: #666666">.</span>biases<span style="color: #666666">=</span>[b<span style="color: #666666">-</span>(eta<span style="color: #666666">/</span><span style="color: #008000">len</span>(mini_batch))<span style="color: #666666">*</span>nb <span style="color: #008000; font-weight: bold">for</span> b, nb <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>biases, nabla_b)]
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">backprop</span>(<span style="color: #008000">self</span>, x, y):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return a tuple ``(nabla_b, nabla_w)`` representing the</span>
<span style="color: #BA2121; font-style: italic"> gradient for the cost function C_x. ``nabla_b`` and</span>
<span style="color: #BA2121; font-style: italic"> ``nabla_w`` are layer-by-layer lists of numpy arrays, similar</span>
<span style="color: #BA2121; font-style: italic"> to ``self.biases`` and ``self.weights``.&quot;&quot;&quot;</span>
nabla_b <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>zeros(b<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> b <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>biases]
nabla_w <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>zeros(w<span style="color: #666666">.</span>shape) <span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights]
<span style="color: #408080; font-style: italic"># feedforward</span>
activation <span style="color: #666666">=</span> x
activations <span style="color: #666666">=</span> [x] <span style="color: #408080; font-style: italic"># list to store all the activations, layer by layer</span>
zs <span style="color: #666666">=</span> [] <span style="color: #408080; font-style: italic"># list to store all the z vectors, layer by layer</span>
<span style="color: #008000; font-weight: bold">for</span> b, w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">zip</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>biases, <span style="color: #008000">self</span><span style="color: #666666">.</span>weights):
z <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(w, activation)<span style="color: #666666">+</span>b
zs<span style="color: #666666">.</span>append(z)
activation <span style="color: #666666">=</span> sigmoid(z)
activations<span style="color: #666666">.</span>append(activation)
<span style="color: #408080; font-style: italic"># backward pass</span>
delta <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>cost_derivative(activations[<span style="color: #666666">-1</span>], y) <span style="color: #666666">*</span> \
sigmoid_prime(zs[<span style="color: #666666">-1</span>])
nabla_b[<span style="color: #666666">-1</span>] <span style="color: #666666">=</span> delta
nabla_w[<span style="color: #666666">-1</span>] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(delta, activations[<span style="color: #666666">-2</span>]<span style="color: #666666">.</span>transpose())
<span style="color: #408080; font-style: italic"># Note that the variable l in the loop below is used a little</span>
<span style="color: #408080; font-style: italic"># differently to the notation in Chapter 2 of the book. Here,</span>
<span style="color: #408080; font-style: italic"># l = 1 means the last layer of neurons, l = 2 is the</span>
<span style="color: #408080; font-style: italic"># second-last layer, and so on. It&#39;s a renumbering of the</span>
<span style="color: #408080; font-style: italic"># scheme in the book, used here to take advantage of the fact</span>
<span style="color: #408080; font-style: italic"># that Python can use negative indices in lists.</span>
<span style="color: #008000; font-weight: bold">for</span> l <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">xrange</span>(<span style="color: #666666">2</span>, <span style="color: #008000">self</span><span style="color: #666666">.</span>num_layers):
z <span style="color: #666666">=</span> zs[<span style="color: #666666">-</span>l]
sp <span style="color: #666666">=</span> sigmoid_prime(z)
delta <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(<span style="color: #008000">self</span><span style="color: #666666">.</span>weights[<span style="color: #666666">-</span>l<span style="color: #666666">+1</span>]<span style="color: #666666">.</span>transpose(), delta) <span style="color: #666666">*</span> sp
nabla_b[<span style="color: #666666">-</span>l] <span style="color: #666666">=</span> delta
nabla_w[<span style="color: #666666">-</span>l] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>dot(delta, activations[<span style="color: #666666">-</span>l<span style="color: #666666">-1</span>]<span style="color: #666666">.</span>transpose())
<span style="color: #008000; font-weight: bold">return</span> (nabla_b, nabla_w)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">evaluate</span>(<span style="color: #008000">self</span>, test_data):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return the number of test inputs for which the neural</span>
<span style="color: #BA2121; font-style: italic"> network outputs the correct result. Note that the neural</span>
<span style="color: #BA2121; font-style: italic"> network&#39;s output is assumed to be the index of whichever</span>
<span style="color: #BA2121; font-style: italic"> neuron in the final layer has the highest activation.&quot;&quot;&quot;</span>
test_results <span style="color: #666666">=</span> [(np<span style="color: #666666">.</span>argmax(<span style="color: #008000">self</span><span style="color: #666666">.</span>feedforward(x)), y)
<span style="color: #008000; font-weight: bold">for</span> (x, y) <span style="color: #AA22FF; font-weight: bold">in</span> test_data]
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">sum</span>(<span style="color: #008000">int</span>(x <span style="color: #666666">==</span> y) <span style="color: #008000; font-weight: bold">for</span> (x, y) <span style="color: #AA22FF; font-weight: bold">in</span> test_results)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">cost_derivative</span>(<span style="color: #008000">self</span>, output_activations, y):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return the vector of partial derivatives \partial C_x /</span>
<span style="color: #BA2121; font-style: italic"> \partial a for the output activations.&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">return</span> (output_activations<span style="color: #666666">-</span>y)
<span style="color: #408080; font-style: italic">#Functions</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(z):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1.0/</span>(<span style="color: #666666">1.0+</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>z))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid_prime</span>(z):
<span style="color: #008000; font-weight: bold">return</span> sigmoid(z)<span style="color: #666666">*</span>(<span style="color: #666666">1-</span>sigmoid(z))
network<span style="color: #666666">=</span>Network()
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># %load neural-networks-and-deep-learning/src/mnist_loader.py</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic">mnist_loader</span>
<span style="color: #BA2121; font-style: italic">~~~~~~~~~~~~</span>
<span style="color: #BA2121; font-style: italic">A library to load the MNIST image data. For details of the data</span>
<span style="color: #BA2121; font-style: italic">structures that are returned, see the doc strings for ``load_data``</span>
<span style="color: #BA2121; font-style: italic">and ``load_data_wrapper``. In practice, ``load_data_wrapper`` is the</span>
<span style="color: #BA2121; font-style: italic">function usually called by our neural network code.</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic">#### Libraries</span>
<span style="color: #408080; font-style: italic"># Standard library</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pickle</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">gzip</span>
<span style="color: #408080; font-style: italic"># Third-party libraries</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">load_data</span>():
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return the MNIST data as a tuple containing the training data,</span>
<span style="color: #BA2121; font-style: italic"> the validation data, and the test data.</span>
<span style="color: #BA2121; font-style: italic"> The ``training_data`` is returned as a tuple with two entries.</span>
<span style="color: #BA2121; font-style: italic"> The first entry contains the actual training images. This is a</span>
<span style="color: #BA2121; font-style: italic"> numpy ndarray with 50,000 entries. Each entry is, in turn, a</span>
<span style="color: #BA2121; font-style: italic"> numpy ndarray with 784 values, representing the 28 * 28 = 784</span>
<span style="color: #BA2121; font-style: italic"> pixels in a single MNIST image.</span>
<span style="color: #BA2121; font-style: italic"> The second entry in the ``training_data`` tuple is a numpy ndarray</span>
<span style="color: #BA2121; font-style: italic"> containing 50,000 entries. Those entries are just the digit</span>
<span style="color: #BA2121; font-style: italic"> values (0...9) for the corresponding images contained in the first</span>
<span style="color: #BA2121; font-style: italic"> entry of the tuple.</span>
<span style="color: #BA2121; font-style: italic"> The ``validation_data`` and ``test_data`` are similar, except</span>
<span style="color: #BA2121; font-style: italic"> each contains only 10,000 images.</span>
<span style="color: #BA2121; font-style: italic"> This is a nice data format, but for use in neural networks it&#39;s</span>
<span style="color: #BA2121; font-style: italic"> helpful to modify the format of the ``training_data`` a little.</span>
<span style="color: #BA2121; font-style: italic"> That&#39;s done in the wrapper function ``load_data_wrapper()``, see</span>
<span style="color: #BA2121; font-style: italic"> below.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
f <span style="color: #666666">=</span> gzip<span style="color: #666666">.</span>open(<span style="color: #BA2121">&#39;../data/mnist.pkl.gz&#39;</span>, <span style="color: #BA2121">&#39;rb&#39;</span>)
training_data, validation_data, test_data <span style="color: #666666">=</span> cPickle<span style="color: #666666">.</span>load(f)
f<span style="color: #666666">.</span>close()
<span style="color: #008000; font-weight: bold">return</span> (training_data, validation_data, test_data)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">load_data_wrapper</span>():
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return a tuple containing ``(training_data, validation_data,</span>
<span style="color: #BA2121; font-style: italic"> test_data)``. Based on ``load_data``, but the format is more</span>
<span style="color: #BA2121; font-style: italic"> convenient for use in our implementation of neural networks.</span>
<span style="color: #BA2121; font-style: italic"> In particular, ``training_data`` is a list containing 50,000</span>
<span style="color: #BA2121; font-style: italic"> 2-tuples ``(x, y)``. ``x`` is a 784-dimensional numpy.ndarray</span>
<span style="color: #BA2121; font-style: italic"> containing the input image. ``y`` is a 10-dimensional</span>
<span style="color: #BA2121; font-style: italic"> numpy.ndarray representing the unit vector corresponding to the</span>
<span style="color: #BA2121; font-style: italic"> correct digit for ``x``.</span>
<span style="color: #BA2121; font-style: italic"> ``validation_data`` and ``test_data`` are lists containing 10,000</span>
<span style="color: #BA2121; font-style: italic"> 2-tuples ``(x, y)``. In each case, ``x`` is a 784-dimensional</span>
<span style="color: #BA2121; font-style: italic"> numpy.ndarry containing the input image, and ``y`` is the</span>
<span style="color: #BA2121; font-style: italic"> corresponding classification, i.e., the digit values (integers)</span>
<span style="color: #BA2121; font-style: italic"> corresponding to ``x``.</span>
<span style="color: #BA2121; font-style: italic"> Obviously, this means we&#39;re using slightly different formats for</span>
<span style="color: #BA2121; font-style: italic"> the training data and the validation / test data. These formats</span>
<span style="color: #BA2121; font-style: italic"> turn out to be the most convenient for use in our neural network</span>
<span style="color: #BA2121; font-style: italic"> code.&quot;&quot;&quot;</span>
tr_d, va_d, te_d <span style="color: #666666">=</span> load_data()
training_inputs <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>reshape(x, (<span style="color: #666666">784</span>, <span style="color: #666666">1</span>)) <span style="color: #008000; font-weight: bold">for</span> x <span style="color: #AA22FF; font-weight: bold">in</span> tr_d[<span style="color: #666666">0</span>]]
training_results <span style="color: #666666">=</span> [vectorized_result(y) <span style="color: #008000; font-weight: bold">for</span> y <span style="color: #AA22FF; font-weight: bold">in</span> tr_d[<span style="color: #666666">1</span>]]
training_data <span style="color: #666666">=</span> <span style="color: #008000">zip</span>(training_inputs, training_results)
validation_inputs <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>reshape(x, (<span style="color: #666666">784</span>, <span style="color: #666666">1</span>)) <span style="color: #008000; font-weight: bold">for</span> x <span style="color: #AA22FF; font-weight: bold">in</span> va_d[<span style="color: #666666">0</span>]]
validation_data <span style="color: #666666">=</span> <span style="color: #008000">zip</span>(validation_inputs, va_d[<span style="color: #666666">1</span>])
test_inputs <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>reshape(x, (<span style="color: #666666">784</span>, <span style="color: #666666">1</span>)) <span style="color: #008000; font-weight: bold">for</span> x <span style="color: #AA22FF; font-weight: bold">in</span> te_d[<span style="color: #666666">0</span>]]
test_data <span style="color: #666666">=</span> <span style="color: #008000">zip</span>(test_inputs, te_d[<span style="color: #666666">1</span>])
<span style="color: #008000; font-weight: bold">return</span> (training_data, validation_data, test_data)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">vectorized_result</span>(j):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;Return a 10-dimensional unit vector with a 1.0 in the jth</span>
<span style="color: #BA2121; font-style: italic"> position and zeroes elsewhere. This is used to convert a digit</span>
<span style="color: #BA2121; font-style: italic"> (0...9) into a corresponding desired output from the neural</span>
<span style="color: #BA2121; font-style: italic"> network.&quot;&quot;&quot;</span>
e <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #666666">10</span>, <span style="color: #666666">1</span>))
e[j] <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
<span style="color: #008000; font-weight: bold">return</span> e
net<span style="color: #666666">=</span>network<span style="color: #666666">.</span>Network([<span style="color: #666666">784</span>,<span style="color: #666666">30</span>,<span style="color: #666666">30</span>])
net<span style="color: #666666">.</span>SGD(training_data,<span style="color: #666666">30</span>,<span style="color: #666666">10</span>,<span style="color: #666666">3</span>,test_data<span style="color: #666666">=</span>test_data)
</pre></div>
<p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>