lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f64815fcd255b32e7ed243bc4eccb2bf10ac482e
FYS-STK4155/doc/pub/week44/html/week44.html
T
Morten Hjorth-Jensen f64815fcd2 correcting typos
2024-10-28 05:40:23 +01:00

5210 lines
381 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!--
HTML file automatically generated from DocOnce source
(https://github.com/doconce/doconce/)
doconce format html week44.do.txt --pygments_html_style=default --html_style=bloodish --html_links_in_new_window --html_output=week44 --no_mako
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 44, Convolutional Neural Networks (CNN)">
<title>Week 44, Convolutional Neural Networks (CNN)</title>
<style type="text/css">
/* bloodish style */
body {
font-family: Helvetica, Verdana, Arial, Sans-serif;
color: #404040;
background: #ffffff;
}
h1 { font-size: 1.8em; color: #8A0808; }
h2 { font-size: 1.6em; color: #8A0808; }
h3 { font-size: 1.4em; color: #8A0808; }
h4 { font-size: 1.2em; color: #8A0808; }
a { color: #8A0808; text-decoration:none; }
tt { font-family: "Courier New", Courier; }
p { text-indent: 0px; }
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
p.caption { width: 80%; font-style: normal; text-align: left; }
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa; }div.highlight {
border: 1px solid #cfcfcf;
border-radius: 2px;
line-height: 1.21429em;
}
div.cell {
width: 100%;
padding: 5px 5px 5px 0;
margin: 0;
outline: none;
}
div.input {
page-break-inside: avoid;
box-orient: horizontal;
box-align: stretch;
display: flex;
flex-direction: row;
align-items: stretch;
}
div.inner_cell {
box-orient: vertical;
box-align: stretch;
display: flex;
flex-direction: column;
align-items: stretch;
box-flex: 1;
flex: 1;
}
div.input_area {
border: 1px solid #cfcfcf;
border-radius: 4px;
background: #f7f7f7;
line-height: 1.21429em;
}
div.input_area > div.highlight {
margin: .4em;
border: none;
padding: 0;
background-color: transparent;
}
div.output_wrapper {
position: relative;
box-orient: vertical;
box-align: stretch;
display: flex;
flex-direction: column;
align-items: stretch;
}
.output {
box-orient: vertical;
box-align: stretch;
display: flex;
flex-direction: column;
align-items: stretch;
}
div.output_area {
padding: 0;
page-break-inside: avoid;
box-orient: horizontal;
box-align: stretch;
display: flex;
flex-direction: row;
align-items: stretch;
}
div.output_subarea {
padding: .4em .4em 0 .4em;
box-flex: 1;
flex: 1;
}
div.output_text {
text-align: left;
color: #000;
line-height: 1.21429em;
}
.alert-text-small { font-size: 80%; }
.alert-text-large { font-size: 130%; }
.alert-text-normal { font-size: 90%; }
.alert {
padding:8px 35px 8px 14px; margin-bottom:18px;
text-shadow:0 1px 0 rgba(255,255,255,0.5);
border:1px solid #bababa;
border-radius: 4px;
-webkit-border-radius: 4px;
-moz-border-radius: 4px;
color: #555;
background-color: #f8f8f8;
background-position: 10px 5px;
background-repeat: no-repeat;
background-size: 38px;
padding-left: 55px;
width: 75%;
}
.alert-block {padding-top:14px; padding-bottom:14px}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_question.png); }
div { text-align: justify; text-justify: inter-word; }
.tab {
padding-left: 1.5em;
}
div.toc p,a {
line-height: 1.3;
margin-top: 1.1;
margin-bottom: 1.1;
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 44', 2, None, 'plan-for-week-44'),
('Lab sessions on Tuesday and Wednesday',
2,
None,
'lab-sessions-on-tuesday-and-wednesday'),
('Material for Lecture Monday October 28',
2,
None,
'material-for-lecture-monday-october-28'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
'convolutional-neural-networks-recognizing-images'),
('What is the Difference', 2, None, 'what-is-the-difference'),
('Neural Networks vs CNNs', 2, None, 'neural-networks-vs-cnns'),
('Why CNNS for images, sound files, medical images from CT scans '
'etc?',
2,
None,
'why-cnns-for-images-sound-files-medical-images-from-ct-scans-etc'),
('Regular NNs don’t scale well to full images',
2,
None,
'regular-nns-don-t-scale-well-to-full-images'),
('3D volumes of neurons', 2, None, '3d-volumes-of-neurons'),
('More on Dimensionalities', 2, None, 'more-on-dimensionalities'),
('Further remarks', 2, None, 'further-remarks'),
('Layers used to build CNNs',
2,
None,
'layers-used-to-build-cnns'),
('Transforming images', 2, None, 'transforming-images'),
('CNNs in brief', 2, None, 'cnns-in-brief'),
('A deep CNN model ("From Raschka et '
'al":"https://github.com/rasbt/machine-learning-book")',
2,
None,
'a-deep-cnn-model-from-raschka-et-al-https-github-com-rasbt-machine-learning-book'),
('Key Idea', 2, None, 'key-idea'),
('How to do image compression before the era of deep learning',
2,
None,
'how-to-do-image-compression-before-the-era-of-deep-learning'),
('The SVD example', 2, None, 'the-svd-example'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Mathematics of CNNs', 2, None, 'mathematics-of-cnns'),
('Convolution Examples: Polynomial multiplication',
2,
None,
'convolution-examples-polynomial-multiplication'),
('Efficient Polynomial Multiplication',
2,
None,
'efficient-polynomial-multiplication'),
('Further simplification', 2, None, 'further-simplification'),
('A more efficient way of coding the above Convolution',
2,
None,
'a-more-efficient-way-of-coding-the-above-convolution'),
('Commutative process', 2, None, 'commutative-process'),
('Toeplitz matrices', 2, None, 'toeplitz-matrices'),
('Fourier series and Toeplitz matrices',
2,
None,
'fourier-series-and-toeplitz-matrices'),
('Generalizing the above one-dimensional case',
2,
None,
'generalizing-the-above-one-dimensional-case'),
('Memory considerations', 2, None, 'memory-considerations'),
('Padding', 2, None, 'padding'),
('New vector', 2, None, 'new-vector'),
('Rewriting as dot products',
2,
None,
'rewriting-as-dot-products'),
('Cross correlation', 2, None, 'cross-correlation'),
('Two-dimensional objects', 2, None, 'two-dimensional-objects'),
('CNNs in more detail, simple example',
2,
None,
'cnns-in-more-detail-simple-example'),
('The convolution stage', 2, None, 'the-convolution-stage'),
('Finding the number of parameters',
2,
None,
'finding-the-number-of-parameters'),
('New image (or volume)', 2, None, 'new-image-or-volume'),
('Parameters to train, common settings',
2,
None,
'parameters-to-train-common-settings'),
('Examples of CNN setups', 2, None, 'examples-of-cnn-setups'),
('Summarizing: Performing a general discrete convolution ("From '
'Raschka et '
'al":"https://github.com/rasbt/machine-learning-book")',
2,
None,
'summarizing-performing-a-general-discrete-convolution-from-raschka-et-al-https-github-com-rasbt-machine-learning-book'),
('Pooling', 2, None, 'pooling'),
('Pooling arithmetic', 2, None, 'pooling-arithmetic'),
('Pooling types ("From Raschka et '
'al":"https://github.com/rasbt/machine-learning-book")',
2,
None,
'pooling-types-from-raschka-et-al-https-github-com-rasbt-machine-learning-book'),
('Building convolutional neural networks in Tensorflow and Keras',
2,
None,
'building-convolutional-neural-networks-in-tensorflow-and-keras'),
('Setting it up', 2, None, 'setting-it-up'),
('The MNIST dataset again', 2, None, 'the-mnist-dataset-again'),
('Strong correlations', 2, None, 'strong-correlations'),
('Layers of a CNN', 2, None, 'layers-of-a-cnn'),
('Systematic reduction', 2, None, 'systematic-reduction'),
('Prerequisites: Collect and pre-process data',
2,
None,
'prerequisites-collect-and-pre-process-data'),
('Importing Keras and Tensorflow',
2,
None,
'importing-keras-and-tensorflow'),
('Running with Keras', 2, None, 'running-with-keras'),
('Final part', 2, None, 'final-part'),
('Final visualization', 2, None, 'final-visualization'),
('The CIFAR01 data set', 2, None, 'the-cifar01-data-set'),
('Verifying the data set', 2, None, 'verifying-the-data-set'),
('Set up the model', 2, None, 'set-up-the-model'),
('Add Dense layers on top', 2, None, 'add-dense-layers-on-top'),
('Compile and train the model',
2,
None,
'compile-and-train-the-model'),
('Finally, evaluate the model',
2,
None,
'finally-evaluate-the-model'),
('Building our own CNN code',
2,
None,
'building-our-own-cnn-code'),
('List of contents:', 3, None, 'list-of-contents'),
('Schedulers', 3, None, 'schedulers'),
('Usage of schedulers', 3, None, 'usage-of-schedulers'),
('Cost functions', 3, None, 'cost-functions'),
('Usage of cost functions', 3, None, 'usage-of-cost-functions'),
('Activation functions', 3, None, 'activation-functions'),
('Usage of activation functions',
3,
None,
'usage-of-activation-functions'),
('Convolution', 3, None, 'convolution'),
('Layers', 3, None, 'layers'),
('Convolution2DLayer: convolution in a hidden layer',
3,
None,
'convolution2dlayer-convolution-in-a-hidden-layer'),
('Backpropagation in the convolutional layer',
3,
None,
'backpropagation-in-the-convolutional-layer'),
('Demonstration', 3, None, 'demonstration'),
('Pooling Layer', 3, None, 'pooling-layer'),
('Flattening Layer', 3, None, 'flattening-layer'),
('Fully Connected Layers', 3, None, 'fully-connected-layers'),
('Optimized Convolution2DLayer',
3,
None,
'optimized-convolution2dlayer'),
('The Convolutional Neural Network (CNN)',
3,
None,
'the-convolutional-neural-network-cnn'),
('Usage of CNN code', 3, None, 'usage-of-cnn-code'),
('Additional Remarks', 3, None, 'additional-remarks'),
('Remarks on the speed', 3, None, 'remarks-on-the-speed'),
('Convolution using separable kernels',
3,
None,
'convolution-using-separable-kernels'),
('Convolution in the Fourier domain',
3,
None,
'convolution-in-the-fourier-domain')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "AMS" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- ------------------- main content ---------------------- -->
<center>
<h1>Week 44, Convolutional Neural Networks (CNN)</h1>
</center> <!-- document title -->
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<!-- institution(s) -->
<center>
[1] <b>Department of Physics, University of Oslo</b>
</center>
<center>
[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b>
</center>
<br>
<center>
<h4>October 28-November 1 </h4>
</center> <!-- date -->
<br>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="plan-for-week-44">Plan for week 44 </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Material for the lecture Monday October 28, 2024</b>
<p>
<ol>
<li> Convolutional Neural Networks</li>
<li> Readings and Videos:</li>
<ul>
<li> These lecture notes at <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/pub/week44/ipynb/week44.ipynb" target="_blank"><tt>https://github.com/CompPhysics/MachineLearning/blob/master/doc/pub/week44/ipynb/week44.ipynb</tt></a></li>
<li> For a more in depth discussion on neural networks we recommend Goodfellow et al chapter 9. See also chapter 11 and 12 on practicalities and applications</li>
<li> Reading suggestions for implementation of CNNs see <a href="Rashcka et al.'s chapter 14" target="_blank"><tt>Rashcka et al.'s chapter 14</tt></a>:"https://github.com/rasbt/machine-learning-book/tree/main/ch14".</li>
<li> Video on Deep Learning at <a href="https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi" target="_blank"><tt>https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi</tt></a></li>
<li> Video on Convolutional Neural Networks from MIT at <a href="https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini" target="_blank"><tt>https://www.youtube.com/watch?v=iaSUYvmCekI&ab_channel=AlexanderAmini</tt></a></li>
<li> Video on CNNs from Stanford at <a href="https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering" target="_blank"><tt>https://www.youtube.com/watch?v=bNb2fEVKeEo&list=PLC1qU-LWwrF64f4QKQT-Vg5Wr4qEE1Zxk&index=6&ab_channel=StanfordUniversitySchoolofEngineering</tt></a></li>
</ul>
</ol>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lab-sessions-on-tuesday-and-wednesday">Lab sessions on Tuesday and Wednesday </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<ul>
<li> Main focus is discussion of and work on project 2</li>
<li> If you did not get time to finish the exercises from week 43, you can also keep working on them and hand in this coming Friday
<!-- * <a href="https://youtu.be/EajWMW__k0I" target="_blank">Video of lab session from week 44</a> -->
<!-- * <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/Exercisesweek44.pdf" target="_blank">See also whiteboard notes from lab session week 44</a> --></li>
</ul>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="material-for-lecture-monday-october-28">Material for Lecture Monday October 28 </h2>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="convolutional-neural-networks-recognizing-images">Convolutional Neural Networks (recognizing images) </h2>
<p>Convolutional neural networks (CNNs) were developed during the last
decade of the previous century, with a focus on character recognition
tasks. Nowadays, CNNs are a central element in the spectacular success
of deep learning methods. The success in for example image
classifications have made them a central tool for most machine
learning practitioners.
</p>
<p>CNNs are very similar to ordinary Neural Networks.
They are made up of neurons that have learnable weights and
biases. Each neuron receives some inputs, performs a dot product and
optionally follows it with a non-linearity. The whole network still
expresses a single differentiable score function: from the raw image
pixels on one end to class scores at the other. And they still have a
loss function (for example Softmax) on the last (fully-connected) layer
and all the tips/tricks we developed for learning regular Neural
Networks still apply (back propagation, gradient descent etc etc).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="what-is-the-difference">What is the Difference </h2>
<p><b>CNN architectures make the explicit assumption that
the inputs are images, which allows us to encode certain properties
into the architecture. These then make the forward function more
efficient to implement and vastly reduce the amount of parameters in
the network.</b>
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="neural-networks-vs-cnns">Neural Networks vs CNNs </h2>
<p>Neural networks are defined as <b>affine transformations</b>, that is
a vector is received as input and is multiplied with a matrix of so-called weights (our unknown paramters) to produce an
output (to which a bias vector is usually added before passing the result
through a nonlinear activation function). This is applicable to any type of input, be it an
image, a sound clip or an unordered collection of features: whatever their
dimensionality, their representation can always be flattened into a vector
before the transformation.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-cnns-for-images-sound-files-medical-images-from-ct-scans-etc">Why CNNS for images, sound files, medical images from CT scans etc? </h2>
<p>However, when we consider images, sound clips and many other similar kinds of data, these data have an intrinsic
structure. More formally, they share these important properties:
</p>
<ul>
<li> They are stored as multi-dimensional arrays (think of the pixels of a figure) .</li>
<li> They feature one or more axes for which ordering matters (e.g., width and height axes for an image, time axis for a sound clip).</li>
<li> One axis, called the channel axis, is used to access different views of the data (e.g., the red, green and blue channels of a color image, or the left and right channels of a stereo audio track).</li>
</ul>
<p>These properties are not exploited when an affine transformation is applied; in
fact, all the axes are treated in the same way and the topological information
is not taken into account. Still, taking advantage of the implicit structure of
the data may prove very handy in solving some tasks, like computer vision and
speech recognition, and in these cases it would be best to preserve it. This is
where discrete convolutions come into play.
</p>
<p>A discrete convolution is a linear transformation that preserves this notion of
ordering. It is sparse (only a few input units contribute to a given output
unit) and reuses parameters (the same weights are applied to multiple locations
in the input).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="regular-nns-don-t-scale-well-to-full-images">Regular NNs don&#8217;t scale well to full images </h2>
<p>As an example, consider
an image of size \( 32\times 32\times 3 \) (32 wide, 32 high, 3 color channels), so a
single fully-connected neuron in a first hidden layer of a regular
Neural Network would have \( 32\times 32\times 3 = 3072 \) weights. This amount still
seems manageable, but clearly this fully-connected structure does not
scale to larger images. For example, an image of more respectable
size, say \( 200\times 200\times 3 \), would lead to neurons that have
\( 200\times 200\times 3 = 120,000 \) weights.
</p>
<p>We could have
several such neurons, and the parameters would add up quickly! Clearly,
this full connectivity is wasteful and the huge number of parameters
would quickly lead to possible overfitting.
</p>
<center> <!-- FIGURE -->
<hr class="figure">
<center>
<p class="caption">Figure 1: A regular 3-layer Neural Network. </p>
</center>
<p><img src="figslides/nn.jpeg" width="500" align="bottom"></p>
</center>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="3d-volumes-of-neurons">3D volumes of neurons </h2>
<p>Convolutional Neural Networks take advantage of the fact that the
input consists of images and they constrain the architecture in a more
sensible way.
</p>
<p>In particular, unlike a regular Neural Network, the
layers of a CNN have neurons arranged in 3 dimensions: width,
height, depth. (Note that the word depth here refers to the third
dimension of an activation volume, not to the depth of a full Neural
Network, which can refer to the total number of layers in a network.)
</p>
<p>To understand it better, the above example of an image
with an input volume of
activations has dimensions \( 32\times 32\times 3 \) (width, height,
depth respectively).
</p>
<p>The neurons in a layer will
only be connected to a small region of the layer before it, instead of
all of the neurons in a fully-connected manner. Moreover, the final
output layer could for this specific image have dimensions \( 1\times 1 \times 10 \),
because by the
end of the CNN architecture we will reduce the full image into a
single vector of class scores, arranged along the depth
dimension.
</p>
<center> <!-- FIGURE -->
<hr class="figure">
<center>
<p class="caption">Figure 2: A CNN arranges its neurons in three dimensions (width, height, depth), as visualized in one of the layers. Every layer of a CNN transforms the 3D input volume to a 3D output volume of neuron activations. In this example, the red input layer holds the image, so its width and height would be the dimensions of the image, and the depth would be 3 (Red, Green, Blue channels). </p>
</center>
<p><img src="figslides/cnn.jpeg" width="500" align="bottom"></p>
</center>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-dimensionalities">More on Dimensionalities </h2>
<p>In fields like signal processing (and imaging as well), one designs
so-called filters. These filters are defined by the convolutions and
are often hand-crafted. One may specify filters for smoothing, edge
detection, frequency reshaping, and similar operations. However with
neural networks the idea is to automatically learn the filters and use
many of them in conjunction with non-linear operations (activation
functions).
</p>
<p>As an example consider a neural network operating on sound sequence
data. Assume that we an input vector \( \boldsymbol{x} \) of length \( d=10^6 \). We
construct then a neural network with onle hidden layer only with
\( 10^4 \) nodes. This means that we will have a weight matrix with
\( 10^4\times 10^6=10^{10} \) weights to be determined, together with \( 10^4 \) biases.
</p>
<p>Assume furthermore that we have an output layer which is meant to train whether the sound sequence represents a human voice (true) or something else (false).
It means that we have only one output node. But since this output node connects to \( 10^4 \) nodes in the hidden layer, there are in total \( 10^4 \) weights to be determined for the output layer, plus one bias. In total we have
</p>
$$
\mathrm{NumberParameters}=10^{10}+10^4+10^4+1 \approx 10^{10},
$$
<p>that is ten billion parameters to determine. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="further-remarks">Further remarks </h2>
<p>The main principles that justify convolutions is locality of
information and repetion of patterns within the signal. Sound samples
of the input in adjacent spots are much more likely to affect each
other than those that are very far away. Similarly, sounds are
repeated in multiple times in the signal. While slightly simplistic,
reasoning about such a sound example demonstrates this. The same
principles then apply to images and other similar data.
</p>
<!-- !split -->
<h2 id="layers-used-to-build-cnns">Layers used to build CNNs </h2>
<p>A simple CNN is a sequence of layers, and every layer of a CNN
transforms one volume of activations to another through a
differentiable function. We use three main types of layers to build
CNN architectures: Convolutional Layer, Pooling Layer, and
Fully-Connected Layer (exactly as seen in regular Neural Networks). We
will stack these layers to form a full CNN architecture.
</p>
<p>A simple CNN for image classification could have the architecture:</p>
<ul>
<li> <b>INPUT</b> (\( 32\times 32 \times 3 \)) will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B.</li>
<li> <b>CONV</b> (convolutional )layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as \( [32\times 32\times 12] \) if we decided to use 12 filters.</li>
<li> <b>RELU</b> layer will apply an elementwise activation function, such as the \( max(0,x) \) thresholding at zero. This leaves the size of the volume unchanged (\( [32\times 32\times 12] \)).</li>
<li> <b>POOL</b> (pooling) layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as \( [16\times 16\times 12] \).</li>
<li> <b>FC</b> (i.e. fully-connected) layer will compute the class scores, resulting in volume of size \( [1\times 1\times 10] \), where each of the 10 numbers correspond to a class score, such as among the 10 categories of the MNIST images we considered above . As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="transforming-images">Transforming images </h2>
<p>CNNs transform the original image layer by layer from the original
pixel values to the final class scores.
</p>
<p>Observe that some layers contain
parameters and other don&#8217;t. In particular, the CNN layers perform
transformations that are a function of not only the activations in the
input volume, but also of the parameters (the weights and biases of
the neurons). On the other hand, the RELU/POOL layers will implement a
fixed function. The parameters in the CONV/FC layers will be trained
with gradient descent so that the class scores that the CNN computes
are consistent with the labels in the training set for each image.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="cnns-in-brief">CNNs in brief </h2>
<p>In summary:</p>
<ul>
<li> A CNN architecture is in the simplest case a list of Layers that transform the image volume into an output volume (e.g. holding the class scores)</li>
<li> There are a few distinct types of Layers (e.g. CONV/FC/RELU/POOL are by far the most popular)</li>
<li> Each Layer accepts an input 3D volume and transforms it to an output 3D volume through a differentiable function</li>
<li> Each Layer may or may not have parameters (e.g. CONV/FC do, RELU/POOL don&#8217;t)</li>
<li> Each Layer may or may not have additional hyperparameters (e.g. CONV/FC/POOL do, RELU doesn&#8217;t)</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="a-deep-cnn-model-from-raschka-et-al-https-github-com-rasbt-machine-learning-book">A deep CNN model (<a href="https://github.com/rasbt/machine-learning-book" target="_blank">From Raschka et al</a>) </h2>
<center> <!-- FIGURE -->
<hr class="figure">
<center>
<p class="caption">Figure 3: A deep CNN </p>
</center>
<p><img src="figslides/deepcnn.png" width="500" align="bottom"></p>
</center>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="key-idea">Key Idea </h2>
<p>A dense neural network is representd by an affine operation (like matrix-matrix multiplication) where all parameters are included.</p>
<p>The key idea in CNNs for say imaging is that in images neighbor pixels tend to be related! So we connect
only neighboring neurons in the input instead of connecting all with the first hidden layer.
</p>
<p>We say we perform a filtering (convolution is the mathematical operation). </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="how-to-do-image-compression-before-the-era-of-deep-learning">How to do image compression before the era of deep learning </h2>
<p>The singular-value decomposition (SVD) algorithm has been for decades one of the standard ways of compressing images.
The <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter2.html#the-singular-value-decomposition" target="_blank">lectures on the SVD</a> give many of the essential details concerning the SVD.
</p>
<p>The orthogonal vectors which are obtained from the SVD, can be used to
project down the dimensionality of a given image. In the example here
we gray-scale an image and downsize it.
</p>
<p>This recipe relies on us being able to actually perform the SVD. For
large images, and in particular with many images to reconstruct, using the SVD
may quickly become an overwhelming task. With the advent of efficient deep
learning methods like CNNs and later generative methods, these methods
have become in the last years the premier way of performing image
analysis. In particular for classification problems with labelled images.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-svd-example">The SVD example </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">matplotlib.image</span> <span style="color: #008000; font-weight: bold">import</span> imread
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scipy.linalg</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">ln</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">os</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">PIL</span> <span style="color: #008000; font-weight: bold">import</span> Image
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">math</span> <span style="color: #008000; font-weight: bold">import</span> log10, sqrt
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;figure.figsize&#39;</span>] <span style="color: #666666">=</span> [<span style="color: #666666">16</span>, <span style="color: #666666">8</span>]
<span style="color: #408080; font-style: italic"># Import image</span>
A <span style="color: #666666">=</span> imread(os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>join(<span style="color: #BA2121">&quot;figslides/photo1.jpg&quot;</span>))
X <span style="color: #666666">=</span> A<span style="color: #666666">.</span>dot([<span style="color: #666666">0.299</span>, <span style="color: #666666">0.5870</span>, <span style="color: #666666">0.114</span>]) <span style="color: #408080; font-style: italic"># Convert RGB to grayscale</span>
img <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>imshow(X)
<span style="color: #408080; font-style: italic"># convert to gray</span>
img<span style="color: #666666">.</span>set_cmap(<span style="color: #BA2121">&#39;gray&#39;</span>)
plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;off&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic"># Call image size</span>
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;: </span><span style="color: #BB6688; font-weight: bold">%s</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">%</span><span style="color: #008000">str</span>(X<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># split the matrix into U, S, VT</span>
U, S, VT <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(X,full_matrices<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>)
S <span style="color: #666666">=</span> np<span style="color: #666666">.</span>diag(S)
m <span style="color: #666666">=</span> <span style="color: #666666">800</span> <span style="color: #408080; font-style: italic"># Image&#39;s width</span>
n <span style="color: #666666">=</span> <span style="color: #666666">1200</span> <span style="color: #408080; font-style: italic"># Image&#39;s height</span>
j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #408080; font-style: italic"># Try compression with different k vectors (these represent projections):</span>
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> (<span style="color: #666666">5</span>,<span style="color: #666666">10</span>, <span style="color: #666666">20</span>, <span style="color: #666666">100</span>,<span style="color: #666666">200</span>,<span style="color: #666666">400</span>,<span style="color: #666666">500</span>):
<span style="color: #408080; font-style: italic"># Original size of the image</span>
originalSize <span style="color: #666666">=</span> m <span style="color: #666666">*</span> n
<span style="color: #408080; font-style: italic"># Size after compressed</span>
compressedSize <span style="color: #666666">=</span> k <span style="color: #666666">*</span> (<span style="color: #666666">1</span> <span style="color: #666666">+</span> m <span style="color: #666666">+</span> n)
<span style="color: #408080; font-style: italic"># The projection of the original image</span>
Xapprox <span style="color: #666666">=</span> U[:,:k] <span style="color: #666666">@</span> S[<span style="color: #666666">0</span>:k,:k] <span style="color: #666666">@</span> VT[:k,:]
plt<span style="color: #666666">.</span>figure(j<span style="color: #666666">+1</span>)
j <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
img <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>imshow(Xapprox)
img<span style="color: #666666">.</span>set_cmap(<span style="color: #BA2121">&#39;gray&#39;</span>)
plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;off&#39;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&#39;k = &#39;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(k))
plt<span style="color: #666666">.</span>show()
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Original size of image:&#39;</span>)
<span style="color: #008000">print</span>(originalSize)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Compression rate as Compressed image / Original size:&#39;</span>)
ratio <span style="color: #666666">=</span> compressedSize <span style="color: #666666">*</span> <span style="color: #666666">1.0</span> <span style="color: #666666">/</span> originalSize
<span style="color: #008000">print</span>(ratio)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Compression rate is &#39;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>( <span style="color: #008000">round</span>(ratio <span style="color: #666666">*</span> <span style="color: #666666">100</span> ,<span style="color: #666666">2</span>)) <span style="color: #666666">+</span> <span style="color: #BA2121">&#39;%&#39;</span> )
<span style="color: #408080; font-style: italic"># Estimate MQA</span>
x<span style="color: #666666">=</span> X<span style="color: #666666">.</span>astype(<span style="color: #BA2121">&quot;float&quot;</span>)
y<span style="color: #666666">=</span>Xapprox<span style="color: #666666">.</span>astype(<span style="color: #BA2121">&quot;float&quot;</span>)
err <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum((x <span style="color: #666666">-</span> y) <span style="color: #666666">**</span> <span style="color: #666666">2</span>)
err <span style="color: #666666">/=</span> <span style="color: #008000">float</span>(X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">*</span> Xapprox<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;The mean-square deviation &#39;</span><span style="color: #666666">+</span> <span style="color: #008000">str</span>(<span style="color: #008000">round</span>( err)))
max_pixel <span style="color: #666666">=</span> <span style="color: #666666">255.0</span>
<span style="color: #408080; font-style: italic"># Estimate Signal Noise Ratio</span>
srv <span style="color: #666666">=</span> <span style="color: #666666">20</span> <span style="color: #666666">*</span> (log10(max_pixel <span style="color: #666666">/</span> sqrt(err)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Signa to noise ratio &#39;</span><span style="color: #666666">+</span> <span style="color: #008000">str</span>(<span style="color: #008000">round</span>(srv)) <span style="color: #666666">+</span><span style="color: #BA2121">&#39;dB&#39;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematics-of-cnns">Mathematics of CNNs </h2>
<p>The mathematics of CNNs is based on the mathematical operation of
<b>convolution</b>. In mathematics (in particular in functional analysis),
convolution is represented by mathematical operations (integration,
summation etc) on two functions in order to produce a third function
that expresses how the shape of one gets modified by the other.
Convolution has a plethora of applications in a variety of
disciplines, spanning from statistics to signal processing, computer
vision, solutions of differential equations,linear algebra,
engineering, and yes, machine learning.
</p>
<p>Mathematically, convolution is defined as follows (one-dimensional example):
Let us define a continuous function \( y(t) \) given by
</p>
$$
y(t) = \int x(a) w(t-a) da,
$$
<p>where \( x(a) \) represents a so-called input and \( w(t-a) \) is normally called the weight function or kernel.</p>
<p>The above integral is written in a more compact form as</p>
$$
y(t) = \left(x * w\right)(t).
$$
<p>The discretized version reads</p>
$$
y(t) = \sum_{a=-\infty}^{a=\infty}x(a)w(t-a).
$$
<p>Computing the inverse of the above convolution operations is known as deconvolution and the process is commutative.</p>
<p>How can we use this? And what does it mean? Let us study some familiar examples first.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematics-of-cnns">Mathematics of CNNs </h2>
<p>The mathematics of CNNs is based on the mathematical operation of
<b>convolution</b>. In mathematics (in particular in functional analysis),
convolution is represented by mathematical operations (integration,
summation etc) on two functions in order to produce a third function
that expresses how the shape of one gets modified by the other.
Convolution has a plethora of applications in a variety of
disciplines, spanning from statistics to signal processing, computer
vision, solutions of differential equations,linear algebra,
engineering, and yes, machine learning.
</p>
<p>Mathematically, convolution is defined as follows (one-dimensional example):
Let us define a continuous function \( y(t) \) given by
</p>
$$
y(t) = \int x(a) w(t-a) da,
$$
<p>where \( x(a) \) represents a so-called input and \( w(t-a) \) is normally called the weight function or kernel.</p>
<p>The above integral is written in a more compact form as</p>
$$
y(t) = \left(x * w\right)(t).
$$
<p>The discretized version reads</p>
$$
y(t) = \sum_{a=-\infty}^{a=\infty}x(a)w(t-a).
$$
<p>Computing the inverse of the above convolution operations is known as deconvolution and the process is commutative.</p>
<p>How can we use this? And what does it mean? Let us study some familiar examples first.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="convolution-examples-polynomial-multiplication">Convolution Examples: Polynomial multiplication </h2>
<p>Our first example is that of a multiplication between two polynomials,
which we will rewrite in terms of the mathematics of convolution. In
the final stage, since the problem here is a discrete one, we will
recast the final expression in terms of a matrix-vector
multiplication, where the matrix is a so-called <a href="https://link.springer.com/book/10.1007/978-93-86279-04-0" target="_blank">Toeplitz matrix
</a>.
</p>
<p>Let us look a the following polynomials to second and third order, respectively:</p>
$$
p(t) = \alpha_0+\alpha_1 t+\alpha_2 t^2,
$$
<p>and</p>
$$
s(t) = \beta_0+\beta_1 t+\beta_2 t^2+\beta_3 t^3.
$$
<p>The polynomial multiplication gives us a new polynomial of degree \( 5 \)</p>
$$
z(t) = \delta_0+\delta_1 t+\delta_2 t^2+\delta_3 t^3+\delta_4 t^4+\delta_5 t^5.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="efficient-polynomial-multiplication">Efficient Polynomial Multiplication </h2>
<p>Computing polynomial products can be implemented efficiently if we rewrite the more brute force multiplications using convolution.
We note first that the new coefficients are given as
</p>
$$
\begin{split}
\delta_0=&\alpha_0\beta_0\\
\delta_1=&\alpha_1\beta_0+\alpha_0\beta_1\\
\delta_2=&\alpha_0\beta_2+\alpha_1\beta_1+\alpha_2\beta_0\\
\delta_3=&\alpha_1\beta_2+\alpha_2\beta_1+\alpha_0\beta_3\\
\delta_4=&\alpha_2\beta_2+\alpha_1\beta_3\\
\delta_5=&\alpha_2\beta_3.\\
\end{split}
$$
<p>We note that \( \alpha_i=0 \) except for \( i\in \left\{0,1,2\right\} \) and \( \beta_i=0 \) except for \( i\in\left\{0,1,2,3\right\} \).</p>
<p>We can then rewrite the coefficients \( \delta_j \) using a discrete convolution as</p>
$$
\delta_j = \sum_{i=-\infty}^{i=\infty}\alpha_i\beta_{j-i}=(\alpha * \beta)_j,
$$
<p>or as a double sum with restriction \( l=i+j \)</p>
$$
\delta_l = \sum_{ij}\alpha_i\beta_{j}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="further-simplification">Further simplification </h2>
<p>Although we may have redundant operations with some few zeros for \( \beta_i \), we can rewrite the above sum in a more compact way as </p>
$$
\delta_i = \sum_{k=0}^{k=m-1}\alpha_k\beta_{i-k},
$$
<p>where \( m=3 \) in our case, the maximum length of
the vector \( \alpha \). Note that the vector \( \boldsymbol{\beta} \) has length \( n=4 \). Below we will find an even more efficient representation.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="a-more-efficient-way-of-coding-the-above-convolution">A more efficient way of coding the above Convolution </h2>
<p>Since we only have a finite number of \( \alpha \) and \( \beta \) values
which are non-zero, we can rewrite the above convolution expressions
as a matrix-vector multiplication
</p>
$$
\boldsymbol{\delta}=\begin{bmatrix}\alpha_0 & 0 & 0 & 0 \\
\alpha_1 & \alpha_0 & 0 & 0 \\
\alpha_2 & \alpha_1 & \alpha_0 & 0 \\
0 & \alpha_2 & \alpha_1 & \alpha_0 \\
0 & 0 & \alpha_2 & \alpha_1 \\
0 & 0 & 0 & \alpha_2
\end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \beta_3\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="commutative-process">Commutative process </h2>
<p>The process is commutative and we can easily see that we can rewrite the multiplication in terms of a matrix holding \( \beta \) and a vector holding \( \alpha \).
In this case we have
</p>
$$
\boldsymbol{\delta}=\begin{bmatrix}\beta_0 & 0 & 0 \\
\beta_1 & \beta_0 & 0 \\
\beta_2 & \beta_1 & \beta_0 \\
\beta_3 & \beta_2 & \beta_1 \\
0 & \beta_3 & \beta_2 \\
0 & 0 & \beta_3
\end{bmatrix}\begin{bmatrix} \alpha_0 \\ \alpha_1 \\ \alpha_2\end{bmatrix}.
$$
<p>Note that the use of these matrices is for mathematical purposes only
and not implementation purposes. When implementing the above equation
we do not encode (and allocate memory) the matrices explicitely. We
rather code the convolutions in the minimal memory footprint that they
require.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="toeplitz-matrices">Toeplitz matrices </h2>
<p>The above matrices are examples of so-called <a href="https://link.springer.com/book/10.1007/978-93-86279-04-0" target="_blank">Toeplitz
matrices</a>. A
Toeplitz matrix is a matrix in which each descending diagonal from
left to right is constant. For instance the last matrix, which we
rewrite as
</p>
$$
\boldsymbol{A}=\begin{bmatrix}a_0 & 0 & 0 \\
a_1 & a_0 & 0 \\
a_2 & a_1 & a_0 \\
a_3 & a_2 & a_1 \\
0 & a_3 & a_2 \\
0 & 0 & a_3
\end{bmatrix},
$$
<p>with elements \( a_{ii}=a_{i+1,j+1}=a_{i-j} \) is an example of a Toeplitz
matrix. Such a matrix does not need to be a square matrix. Toeplitz
matrices are also closely connected with Fourier series, because the multiplication operator by a trigonometric
polynomial, compressed to a finite-dimensional space, can be
represented by such a matrix. The example above shows that we can
represent linear convolution as multiplication of a Toeplitz matrix by
a vector.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="fourier-series-and-toeplitz-matrices">Fourier series and Toeplitz matrices </h2>
<p>This is an active and ogoing research area concerning CNNs. The following articles may be of interest</p>
<ol>
<li> <a href="https://www.sciencedirect.com/topics/engineering/convolution-theorem#:~:text=The%20convolution%20theorem%20(together%20with,k%20)%20G%20(%20k%20)%20." target="_blank">Read more about the convolution theorem and Fouriers series</a></li>
<li> <a href="https://www.sciencedirect.com/science/article/pii/S1568494623006257" target="_blank">Fourier Transform Layer</a></li>
</ol>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="generalizing-the-above-one-dimensional-case">Generalizing the above one-dimensional case </h2>
<p>In order to align the above simple case with the more general
convolution cases, we rename \( \boldsymbol{\alpha} \), whose length is \( m=3 \),
with \( \boldsymbol{w} \). We will interpret \( \boldsymbol{w} \) as a weight/filter function
with which we want to perform the convolution with an input variable
\( \boldsymbol{x} \) of length \( n \). We will assume always that the filter
\( \boldsymbol{w} \) has dimensionality \( m \le n \).
</p>
<p>We replace thus \( \boldsymbol{\beta} \) with \( \boldsymbol{x} \) and \( \boldsymbol{\delta} \) with \( \boldsymbol{y} \) and have</p>
$$
y(i)= \left(x*w\right)(i)= \sum_{k=0}^{k=m-1}w(k)x(i-k),
$$
<p>where \( m=3 \) in our case, the maximum length of the vector \( \boldsymbol{w} \).
Here the symbol \( * \) represents the mathematical operation of convolution.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="memory-considerations">Memory considerations </h2>
<p>This expression leaves us however with some terms with negative
indices, for example \( x(-1) \) and \( x(-2) \) which may not be defined. Our
vector \( \boldsymbol{x} \) has components \( x(0) \), \( x(1) \), \( x(2) \) and \( x(3) \).
</p>
<p>The index \( j \) for \( \boldsymbol{x} \) runs from \( j=0 \) to \( j=3 \) since \( \boldsymbol{x} \) is meant to
represent a third-order polynomial.
</p>
<p>Furthermore, the index \( i \) runs from \( i=0 \) to \( i=5 \) since \( \boldsymbol{y} \)
contains the coefficients of a fifth-order polynomial. When \( i=5 \) we
may also have values of \( x(4) \) and \( x(5) \) which are not defined.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="padding">Padding </h2>
<p>The solution to this is what is called <b>padding</b>! We simply define a
new vector \( x \) with two added elements set to zero before \( x(0) \) and
two new elements after \( x(3) \) set to zero. That is, we augment the
length of \( \boldsymbol{x} \) from \( n=4 \) to \( n+2P=8 \), where \( P=2 \) is the padding
constant (a new hyperparameter), see discussions below as well.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="new-vector">New vector </h2>
<p>We have a new vector defined as \( x(0)=0 \), \( x(1)=0 \),
\( x(2)=\beta_0 \), \( x(3)=\beta_1 \), \( x(4)=\beta_2 \), \( x(5)=\beta_3 \),
\( x(6)=0 \), and \( x(7)=0 \).
</p>
<p>We have added four new elements, which
are all zero. The benefit is that we can rewrite the equation for
\( \boldsymbol{y} \), with \( i=0,1,\dots,5 \),
</p>
$$
y(i) = \sum_{k=0}^{k=m-1}w(k)x(i+(m-1)-k).
$$
<p>As an example, we have</p>
$$
y(4)=x(6)w(0)+x(5)w(1)+x(4)w(2)=0\times \alpha_0+\beta_3\alpha_1+\beta_2\alpha_2,
$$
<p>as before except that we have an additional term \( x(6)w(0) \), which is zero.</p>
<p>Similarly, for the fifth-order term we have</p>
$$
y(5)=x(7)w(0)+x(6)w(1)+x(5)w(2)=0\times \alpha_0+0\times\alpha_1+\beta_3\alpha_2.
$$
<p>The zeroth-order term is</p>
$$
y(0)=x(2)w(0)+x(1)w(1)+x(0)w(2)=\beta_0 \alpha_0+0\times\alpha_1+0\times\alpha_2=\alpha_0\beta_0.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="rewriting-as-dot-products">Rewriting as dot products </h2>
<p>If we now flip the filter/weight vector, with the following term as a typical example</p>
$$
y(0)=x(2)w(0)+x(1)w(1)+x(0)w(2)=x(2)\tilde{w}(2)+x(1)\tilde{w}(1)+x(0)\tilde{w}(0),
$$
<p>with \( \tilde{w}(0)=w(2) \), \( \tilde{w}(1)=w(1) \), and \( \tilde{w}(2)=w(0) \), we can then rewrite the above sum as a dot product of
\( x(i:i+(m-1))\tilde{w} \) for element \( y(i) \), where \( x(i:i+(m-1)) \) is simply a patch of \( \boldsymbol{x} \) of size \( m-1 \).
</p>
<p>The padding \( P \) we have introduced for the convolution stage is just
another hyperparameter which is introduced as part of the
architecture. Similarly, below we will also introduce another
hyperparameter called <b>Stride</b> \( S \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="cross-correlation">Cross correlation </h2>
<p>In essentially all applications one uses what is called cross correlation instead of the standard convolution described above.
This means that multiplication is performed in the same direction and instead of the general expression we have discussed above (with infinite sums)
</p>
$$
y(i) = \sum_{k=-\infty}^{k=\infty}w(k)x(i-k),
$$
<p>we have now</p>
$$
y(i) = \sum_{k=-\infty}^{k=\infty}w(k)x(i+k).
$$
<p>Both TensorFlow and PyTorch (as well as our own code example below),
implement the last equation, although it is normally referred to as
convolution. The same padding rules and stride rules discussed below
apply to this expression as well.
</p>
<p>We leave it as an exercise for you to convince yourself that the example we have discussed till now, gives the same final expression using the last expression.</p>
<h2 id="two-dimensional-objects">Two-dimensional objects </h2>
<p>We are now ready to start studying the discrete convolutions relevant for convolutional neural networks.
We often use convolutions over more than one dimension at a time. If
we have a two-dimensional image \( X \) as input, we can have a <b>filter</b>
defined by a two-dimensional <b>kernel/weight/filter</b> \( W \). This leads to an output \( Y \)
</p>
$$
Y(i,j)=(X * W)(i,j) = \sum_m\sum_n X(m,n)W(i-m,j-n).
$$
<p>Convolution is a commutative process, which means we can rewrite this equation as</p>
$$
Y(i,j)=(X * W)(i,j) = \sum_m\sum_n X(i-m,j-n)W(m,n).
$$
<p>Normally the latter is more straightforward to implement in a machine
larning library since there is less variation in the range of values
of \( m \) and \( n \).
</p>
<p>As mentioned above, most deep learning libraries implement
cross-correlation instead of convolution (although it is referred to as
convolution)
</p>
$$
Y(i,j)=(X * W)(i,j) = \sum_m\sum_n X(i+m,j+n)W(m,n).
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="cnns-in-more-detail-simple-example">CNNs in more detail, simple example </h2>
<p>Let assume we have an input matrix \( X \) of dimensionality \( 3\times 3 \)
and a \( 2\times 2 \) filter \( W \) given by the following matrices
</p>
$$
\boldsymbol{X}=\begin{bmatrix}x_{00} & x_{01} & x_{02} \\
x_{10} & x_{11} & x_{12} \\
x_{20} & x_{21} & x_{22} \end{bmatrix},
$$
<p>and </p>
$$
\boldsymbol{W}=\begin{bmatrix}w_{00} & w_{01} \\
w_{10} & w_{11}\end{bmatrix}.
$$
<p>We introduce now the hyperparameter \( S \) <b>stride</b>. Stride represents how the filter \( W \) moves the convolution process on the matrix \( X \).
We strongly recommend the repository on <a href="https://github.com/vdumoulin/conv_arithmetic" target="_blank">Arithmetic of deep learning by Dumoulin and Visin</a>
</p>
<p>Here we set the stride equal to \( S=1 \), which means that, starting with the element \( x_{00} \), the filter will act on \( 2\times 2 \) submatrices each time, starting with the upper corner and moving according to the stride value column by column. </p>
<p>Here we perform the operation</p>
$$
Y_(i,j)=(X * W)(i,j) = \sum_m\sum_n X(i-m,j-n)W(m,n),
$$
<p>and obtain</p>
$$
\boldsymbol{Y}=\begin{bmatrix}x_{00}w_{00}+x_{01}w_{01}+x_{10}w_{10}+x_{11}w_{11} & x_{01}w_{00}+x_{02}w_{01}+x_{11}w_{10}+x_{12}w_{11} \\
x_{10}w_{00}+x_{11}w_{01}+x_{20}w_{10}+x_{21}w_{11} & x_{11}w_{00}+x_{12}w_{01}+x_{21}w_{10}+x_{22}w_{11}\end{bmatrix}.
$$
<p>We can rewrite this operation in terms of a matrix-vector multiplication by defining a new vector where we flatten out the inputs as a vector \( \boldsymbol{X}' \) of length \( 9 \) and
a matrix \( \boldsymbol{W}' \) with dimension \( 4\times 9 \) as
</p>
$$
\boldsymbol{X}'=\begin{bmatrix}x_{00} \\ x_{01} \\ x_{02} \\ x_{10} \\ x_{11} \\ x_{12} \\ x_{20} \\ x_{21} \\ x_{22} \end{bmatrix},
$$
<p>and the new matrix</p>
$$
\boldsymbol{W}'=\begin{bmatrix} w_{00} & w_{01} & 0 & w_{10} & w_{11} & 0 & 0 & 0 & 0 \\
0 & w_{00} & w_{01} & 0 & w_{10} & w_{11} & 0 & 0 & 0 \\
0 & 0 & 0 & w_{00} & w_{01} & 0 & w_{10} & w_{11} & 0 \\
0 & 0 & 0 & 0 & w_{00} & w_{01} & 0 & w_{10} & w_{11}\end{bmatrix}.
$$
<p>We see easily that performing the matrix-vector multiplication \( \boldsymbol{W}'\boldsymbol{X}' \) is the same as the above convolution with stride \( S=1 \), that is</p>
$$
Y=(\boldsymbol{W}*\boldsymbol{X}),
$$
<p>is now given by \( \boldsymbol{W}'\boldsymbol{X}' \) which is a vector of length \( 4 \) instead of the originally resulting \( 2\times 2 \) output matrix.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-convolution-stage">The convolution stage </h2>
<p>The convolution stage, where we apply different filters \( \boldsymbol{W} \) in
order to reduce the dimensionality of an image, adds, in addition to
the weights and biases (to be trained by the back propagation
algorithm) that define the filters, two new hyperparameters, the so-called
<b>padding</b> \( P \) and the stride \( S \).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="finding-the-number-of-parameters">Finding the number of parameters </h2>
<p>In the above example we have an input matrix of dimension \( 3\times
3 \). In general we call the input for an input volume and it is defined
by its width \( H_1 \), height \( H_1 \) and depth \( D_1 \). If we have the
standard three color channels \( D_1=3 \).
</p>
<p>The above example has \( W_1=H_1=3 \) and \( D_1=1 \).</p>
<p>When we introduce the filter we have the following additional hyperparameters</p>
<ol>
<li> \( K \) the number of filters. It is common to perform the convolution of the input several times since by experience shrinking the input too fast does not work well</li>
<li> \( F \) as the filter's spatial extent</li>
<li> \( S \) as the stride parameter</li>
<li> \( P \) as the padding parameter</li>
</ol>
<p>These parameters are defined by the architecture of the network and are not included in the training.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="new-image-or-volume">New image (or volume) </h2>
<p>Acting with the filter on the input volume produces an output volume
which is defined by its width \( W_2 \), its height \( H_2 \) and its depth
\( D_2 \).
</p>
<p>These are defined by the following relations</p>
$$
W_2 = \frac{(W_1-F+2P)}{S}+1,
$$
$$
H_2 = \frac{(H_1-F+2P)}{S}+1,
$$
<p>and \( D_2=K \).</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="parameters-to-train-common-settings">Parameters to train, common settings </h2>
<p>With parameter sharing, the convolution involves thus for each filter \( F\times F\times D_1 \) weights plus one bias parameter.</p>
<p>In total we have</p>
$$
\left(F\times F\times D_1\right) \times K+K_{\mathrm{biases}},
$$
<p>parameters to train by back propagation.</p>
<p>It is common to let \( K \) come in powers of \( 2 \), that is \( 32 \), \( 64 \), \( 128 \) etc.</p>
<div class="alert alert-block alert-block alert-text-normal">
<b>Common settings</b>
<p>
<ol>
<li> \( \begin{array}{c} F=3 &amp; S=1 &amp; P=1 \end{array} \)</li>
<li> \( \begin{array}{c} F=5 &amp; S=1 &amp; P=2 \end{array} \)</li>
<li> \( \begin{array}{c} F=5 &amp; S=2 &amp; P=\mathrm{open} \end{array} \)</li>
<li> \( \begin{array}{c} F=1 &amp; S=1 &amp; P=0 \end{array} \)</li>
</ol>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="examples-of-cnn-setups">Examples of CNN setups </h2>
<p>Let us assume we have an input volume \( V \) given by an image of dimensionality
\( 32\times 32 \times 3 \), that is three color channels and \( 32\times 32 \) pixels.
</p>
<p>We apply a filter of dimension \( 5\times 5 \) ten times with stride \( S=1 \) and padding \( P=0 \).</p>
<p>The output volume is given by \( (32-5)/1+1=28 \), resulting in ten images
of dimensionality \( 28\times 28\times 3 \).
</p>
<p>The total number of parameters to train for each filter is then
\( 5\times 5\times 3+1 \), where the last parameter is the bias. This
gives us \( 76 \) parameters for each filter, leading to a total of \( 760 \)
parameters for the ten filters.
</p>
<p>How many parameters will a filter of dimensionality \( 3\times 3 \)
(adding color channels) result in if we produce \( 32 \) new images? Use \( S=1 \) and \( P=0 \).
</p>
<p>Note that strides constitute a form of <b>subsampling</b>. As an alternative to
being interpreted as a measure of how much the kernel/filter is translated, strides
can also be viewed as how much of the output is retained. For instance, moving
the kernel by hops of two is equivalent to moving the kernel by hops of one but
retaining only odd output elements.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="summarizing-performing-a-general-discrete-convolution-from-raschka-et-al-https-github-com-rasbt-machine-learning-book">Summarizing: Performing a general discrete convolution (<a href="https://github.com/rasbt/machine-learning-book" target="_blank">From Raschka et al</a>) </h2>
<center> <!-- FIGURE -->
<hr class="figure">
<center>
<p class="caption">Figure 4: A deep CNN </p>
</center>
<p><img src="figslides/discreteconv1.png" width="500" align="bottom"></p>
</center>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="pooling">Pooling </h2>
<p>In addition to discrete convolutions themselves, <b>pooling</b> operations
make up another important building block in CNNs. Pooling operations reduce
the size of feature maps by using some function to summarize subregions, such
as taking the average or the maximum value.
</p>
<p>Pooling works by sliding a window across the input and feeding the content of
the window to a <b>pooling function</b>. In some sense, pooling works very much
like a discrete convolution, but replaces the linear combination described by
the kernel with some other function.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="pooling-arithmetic">Pooling arithmetic </h2>
<p>In a neural network, pooling layers provide invariance to small translations of
the input. The most common kind of pooling is <b>max pooling</b>, which
consists in splitting the input in (usually non-overlapping) patches and
outputting the maximum value of each patch. Other kinds of pooling exist, e.g.,
mean or average pooling, which all share the same idea of aggregating the input
locally by applying a non-linearity to the content of some patches.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="pooling-types-from-raschka-et-al-https-github-com-rasbt-machine-learning-book">Pooling types (<a href="https://github.com/rasbt/machine-learning-book" target="_blank">From Raschka et al</a>) </h2>
<center> <!-- FIGURE -->
<hr class="figure">
<center>
<p class="caption">Figure 5: A deep CNN </p>
</center>
<p><img src="figslides/maxpooling.png" width="500" align="bottom"></p>
</center>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="building-convolutional-neural-networks-in-tensorflow-and-keras">Building convolutional neural networks in Tensorflow and Keras </h2>
<p>As discussed above, CNNs are neural networks built from the assumption that the inputs
to the network are 2D images. This is important because the number of features or pixels in images
grows very fast with the image size, and an enormous number of weights and biases are needed in order to build an accurate network.
</p>
<p>As before, we still have our input, a hidden layer and an output. What's novel about convolutional networks
are the <b>convolutional</b> and <b>pooling</b> layers stacked in pairs between the input and the hidden layer.
In addition, the data is no longer represented as a 2D feature matrix, instead each input is a number of 2D
matrices, typically 1 for each color dimension (Red, Green, Blue).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-it-up">Setting it up </h2>
<p>It means that to represent the entire
dataset of images, we require a 4D matrix or <b>tensor</b>. This tensor has the dimensions:
</p>
$$
(n_{inputs},\, n_{pixels, width},\, n_{pixels, height},\, depth) .
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-mnist-dataset-again">The MNIST dataset again </h2>
<p>The MNIST dataset consists of grayscale images with a pixel size of
\( 28\times 28 \), meaning we require \( 28 \times 28 = 724 \) weights to each
neuron in the first hidden layer.
</p>
<p>If we were to analyze images of size \( 128\times 128 \) we would require
\( 128 \times 128 = 16384 \) weights to each neuron. Even worse if we were
dealing with color images, as most images are, we have an image matrix
of size \( 128\times 128 \) for each color dimension (Red, Green, Blue),
meaning 3 times the number of weights \( = 49152 \) are required for every
single neuron in the first hidden layer.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="strong-correlations">Strong correlations </h2>
<p>Images typically have strong local correlations, meaning that a small
part of the image varies little from its neighboring regions. If for
example we have an image of a blue car, we can roughly assume that a
small blue part of the image is surrounded by other blue regions.
</p>
<p>Therefore, instead of connecting every single pixel to a neuron in the
first hidden layer, as we have previously done with deep neural
networks, we can instead connect each neuron to a small part of the
image (in all 3 RGB depth dimensions). The size of each small area is
fixed, and known as a <a href="https://en.wikipedia.org/wiki/Receptive_field" target="_blank">receptive</a>.
</p>
<!-- !split -->
<h2 id="layers-of-a-cnn">Layers of a CNN </h2>
<p>The layers of a convolutional neural network arrange neurons in 3D: width, height and depth.
The input image is typically a square matrix of depth 3.
</p>
<p>A <b>convolution</b> is performed on the image which outputs
a 3D volume of neurons. The weights to the input are arranged in a number of 2D matrices, known as <b>filters</b>.
</p>
<p>Each filter slides along the input image, taking the dot product
between each small part of the image and the filter, in all depth
dimensions. This is then passed through a non-linear function,
typically the <b>Rectified Linear (ReLu)</b> function, which serves as the
activation of the neurons in the first convolutional layer. This is
further passed through a <b>pooling layer</b>, which reduces the size of the
convolutional layer, e.g. by taking the maximum or average across some
small regions, and this serves as input to the next convolutional
layer.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="systematic-reduction">Systematic reduction </h2>
<p>By systematically reducing the size of the input volume, through
convolution and pooling, the network should create representations of
small parts of the input, and then from them assemble representations
of larger areas. The final pooling layer is flattened to serve as
input to a hidden layer, such that each neuron in the final pooling
layer is connected to every single neuron in the hidden layer. This
then serves as input to the output layer, e.g. a softmax output for
classification.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="prerequisites-collect-and-pre-process-data">Prerequisites: Collect and pre-process data </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
<span style="color: #408080; font-style: italic"># ensure the same random numbers appear every time</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
<span style="color: #408080; font-style: italic"># display images in notebook</span>
<span style="color: #666666">%</span>matplotlib inline
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;figure.figsize&#39;</span>] <span style="color: #666666">=</span> (<span style="color: #666666">12</span>,<span style="color: #666666">12</span>)
<span style="color: #408080; font-style: italic"># download MNIST dataset</span>
digits <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_digits()
<span style="color: #408080; font-style: italic"># define inputs and labels</span>
inputs <span style="color: #666666">=</span> digits<span style="color: #666666">.</span>images
labels <span style="color: #666666">=</span> digits<span style="color: #666666">.</span>target
<span style="color: #408080; font-style: italic"># RGB images have a depth of 3</span>
<span style="color: #408080; font-style: italic"># our images are grayscale so they should have a depth of 1</span>
inputs <span style="color: #666666">=</span> inputs[:,:,:,np<span style="color: #666666">.</span>newaxis]
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;inputs = (n_inputs, pixel_width, pixel_height, depth) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(inputs<span style="color: #666666">.</span>shape))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;labels = (n_inputs) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(labels<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># choose some random images to display</span>
n_inputs <span style="color: #666666">=</span> <span style="color: #008000">len</span>(inputs)
indices <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(n_inputs)
random_indices <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>choice(indices, size<span style="color: #666666">=5</span>)
<span style="color: #008000; font-weight: bold">for</span> i, image <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(digits<span style="color: #666666">.</span>images[random_indices]):
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">1</span>, <span style="color: #666666">5</span>, i<span style="color: #666666">+1</span>)
plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;off&#39;</span>)
plt<span style="color: #666666">.</span>imshow(image, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>gray_r, interpolation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;nearest&#39;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Label: </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> digits<span style="color: #666666">.</span>target[random_indices[i]])
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="importing-keras-and-tensorflow">Importing Keras and Tensorflow </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> datasets, layers, models
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Input
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.models</span> <span style="color: #008000; font-weight: bold">import</span> Sequential <span style="color: #408080; font-style: italic">#This allows appending layers to existing models</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.layers</span> <span style="color: #008000; font-weight: bold">import</span> Dense <span style="color: #408080; font-style: italic">#This allows defining the characteristics of a particular layer</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> optimizers <span style="color: #408080; font-style: italic">#This allows using whichever optimiser we want (sgd,adam,RMSprop)</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> regularizers <span style="color: #408080; font-style: italic">#This allows using whichever regularizer we want (l1,l2,l1_l2)</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras.utils</span> <span style="color: #008000; font-weight: bold">import</span> to_categorical <span style="color: #408080; font-style: italic">#This allows using categorical cross entropy as the cost function</span>
<span style="color: #408080; font-style: italic">#from tensorflow.keras import Conv2D</span>
<span style="color: #408080; font-style: italic">#from tensorflow.keras import MaxPooling2D</span>
<span style="color: #408080; font-style: italic">#from tensorflow.keras import Flatten</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #408080; font-style: italic"># representation of labels</span>
labels <span style="color: #666666">=</span> to_categorical(labels)
<span style="color: #408080; font-style: italic"># split into train and test data</span>
<span style="color: #408080; font-style: italic"># one-liner from scikit-learn library</span>
train_size <span style="color: #666666">=</span> <span style="color: #666666">0.8</span>
test_size <span style="color: #666666">=</span> <span style="color: #666666">1</span> <span style="color: #666666">-</span> train_size
X_train, X_test, Y_train, Y_test <span style="color: #666666">=</span> train_test_split(inputs, labels, train_size<span style="color: #666666">=</span>train_size,
test_size<span style="color: #666666">=</span>test_size)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split -->
<h2 id="running-with-keras">Running with Keras </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">create_convolutional_neural_network_keras</span>(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd):
model <span style="color: #666666">=</span> Sequential()
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Conv2D(n_filters, (receptive_field, receptive_field), input_shape<span style="color: #666666">=</span>input_shape, padding<span style="color: #666666">=</span><span style="color: #BA2121">&#39;same&#39;</span>,
activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>, kernel_regularizer<span style="color: #666666">=</span>regularizers<span style="color: #666666">.</span>l2(lmbd)))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>MaxPooling2D(pool_size<span style="color: #666666">=</span>(<span style="color: #666666">2</span>, <span style="color: #666666">2</span>)))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Flatten())
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Dense(n_neurons_connected, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>, kernel_regularizer<span style="color: #666666">=</span>regularizers<span style="color: #666666">.</span>l2(lmbd)))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Dense(n_categories, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;softmax&#39;</span>, kernel_regularizer<span style="color: #666666">=</span>regularizers<span style="color: #666666">.</span>l2(lmbd)))
sgd <span style="color: #666666">=</span> optimizers<span style="color: #666666">.</span>SGD(lr<span style="color: #666666">=</span>eta)
model<span style="color: #666666">.</span>compile(loss<span style="color: #666666">=</span><span style="color: #BA2121">&#39;categorical_crossentropy&#39;</span>, optimizer<span style="color: #666666">=</span>sgd, metrics<span style="color: #666666">=</span>[<span style="color: #BA2121">&#39;accuracy&#39;</span>])
<span style="color: #008000; font-weight: bold">return</span> model
epochs <span style="color: #666666">=</span> <span style="color: #666666">100</span>
batch_size <span style="color: #666666">=</span> <span style="color: #666666">100</span>
input_shape <span style="color: #666666">=</span> X_train<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>:<span style="color: #666666">4</span>]
receptive_field <span style="color: #666666">=</span> <span style="color: #666666">3</span>
n_filters <span style="color: #666666">=</span> <span style="color: #666666">10</span>
n_neurons_connected <span style="color: #666666">=</span> <span style="color: #666666">50</span>
n_categories <span style="color: #666666">=</span> <span style="color: #666666">10</span>
eta_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
lmbd_vals <span style="color: #666666">=</span> np<span style="color: #666666">.</span>logspace(<span style="color: #666666">-5</span>, <span style="color: #666666">1</span>, <span style="color: #666666">7</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="final-part">Final part </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">CNN_keras <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)), dtype<span style="color: #666666">=</span><span style="color: #008000">object</span>)
<span style="color: #008000; font-weight: bold">for</span> i, eta <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(eta_vals):
<span style="color: #008000; font-weight: bold">for</span> j, lmbd <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(lmbd_vals):
CNN <span style="color: #666666">=</span> create_convolutional_neural_network_keras(input_shape, receptive_field,
n_filters, n_neurons_connected, n_categories,
eta, lmbd)
CNN<span style="color: #666666">.</span>fit(X_train, Y_train, epochs<span style="color: #666666">=</span>epochs, batch_size<span style="color: #666666">=</span>batch_size, verbose<span style="color: #666666">=0</span>)
scores <span style="color: #666666">=</span> CNN<span style="color: #666666">.</span>evaluate(X_test, Y_test)
CNN_keras[i][j] <span style="color: #666666">=</span> CNN
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Learning rate = &quot;</span>, eta)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Lambda = &quot;</span>, lmbd)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Test accuracy: </span><span style="color: #BB6688; font-weight: bold">%.3f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> scores[<span style="color: #666666">1</span>])
<span style="color: #008000">print</span>()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="final-visualization">Final visualization </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #408080; font-style: italic"># visual representation of grid search</span>
<span style="color: #408080; font-style: italic"># uses seaborn heatmap, could probably do this in matplotlib</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">seaborn</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sns</span>
sns<span style="color: #666666">.</span>set()
train_accuracy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)))
test_accuracy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(eta_vals), <span style="color: #008000">len</span>(lmbd_vals)))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(eta_vals)):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(lmbd_vals)):
CNN <span style="color: #666666">=</span> CNN_keras[i][j]
train_accuracy[i][j] <span style="color: #666666">=</span> CNN<span style="color: #666666">.</span>evaluate(X_train, Y_train)[<span style="color: #666666">1</span>]
test_accuracy[i][j] <span style="color: #666666">=</span> CNN<span style="color: #666666">.</span>evaluate(X_test, Y_test)[<span style="color: #666666">1</span>]
fig, ax <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(figsize <span style="color: #666666">=</span> (<span style="color: #666666">10</span>, <span style="color: #666666">10</span>))
sns<span style="color: #666666">.</span>heatmap(train_accuracy, annot<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>, ax<span style="color: #666666">=</span>ax, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;viridis&quot;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&quot;Training Accuracy&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;$\eta$&quot;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;$\lambda$&quot;</span>)
plt<span style="color: #666666">.</span>show()
fig, ax <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots(figsize <span style="color: #666666">=</span> (<span style="color: #666666">10</span>, <span style="color: #666666">10</span>))
sns<span style="color: #666666">.</span>heatmap(test_accuracy, annot<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>, ax<span style="color: #666666">=</span>ax, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;viridis&quot;</span>)
ax<span style="color: #666666">.</span>set_title(<span style="color: #BA2121">&quot;Test Accuracy&quot;</span>)
ax<span style="color: #666666">.</span>set_ylabel(<span style="color: #BA2121">&quot;$\eta$&quot;</span>)
ax<span style="color: #666666">.</span>set_xlabel(<span style="color: #BA2121">&quot;$\lambda$&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-cifar01-data-set">The CIFAR01 data set </h2>
<p>The CIFAR10 dataset contains 60,000 color images in 10 classes, with
6,000 images in each class. The dataset is divided into 50,000
training images and 10,000 testing images. The classes are mutually
exclusive and there is no overlap between them.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">tensorflow</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">tf</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">tensorflow.keras</span> <span style="color: #008000; font-weight: bold">import</span> datasets, layers, models
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #408080; font-style: italic"># We import the data set</span>
(train_images, train_labels), (test_images, test_labels) <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>cifar10<span style="color: #666666">.</span>load_data()
<span style="color: #408080; font-style: italic"># Normalize pixel values to be between 0 and 1 by dividing by 255. </span>
train_images, test_images <span style="color: #666666">=</span> train_images <span style="color: #666666">/</span> <span style="color: #666666">255.0</span>, test_images <span style="color: #666666">/</span> <span style="color: #666666">255.0</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="verifying-the-data-set">Verifying the data set </h2>
<p>To verify that the dataset looks correct, let's plot the first 25 images from the training set and display the class name below each image.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">class_names <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;airplane&#39;</span>, <span style="color: #BA2121">&#39;automobile&#39;</span>, <span style="color: #BA2121">&#39;bird&#39;</span>, <span style="color: #BA2121">&#39;cat&#39;</span>, <span style="color: #BA2121">&#39;deer&#39;</span>,
<span style="color: #BA2121">&#39;dog&#39;</span>, <span style="color: #BA2121">&#39;frog&#39;</span>, <span style="color: #BA2121">&#39;horse&#39;</span>, <span style="color: #BA2121">&#39;ship&#39;</span>, <span style="color: #BA2121">&#39;truck&#39;</span>]
plt<span style="color: #666666">.</span>figure(figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">10</span>))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">25</span>):
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">5</span>,<span style="color: #666666">5</span>,i<span style="color: #666666">+1</span>)
plt<span style="color: #666666">.</span>xticks([])
plt<span style="color: #666666">.</span>yticks([])
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">False</span>)
plt<span style="color: #666666">.</span>imshow(train_images[i], cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>binary)
<span style="color: #408080; font-style: italic"># The CIFAR labels happen to be arrays, </span>
<span style="color: #408080; font-style: italic"># which is why you need the extra index</span>
plt<span style="color: #666666">.</span>xlabel(class_names[train_labels[i][<span style="color: #666666">0</span>]])
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="set-up-the-model">Set up the model </h2>
<p>The 6 lines of code below define the convolutional base using a common pattern: a stack of Conv2D and MaxPooling2D layers.</p>
<p>As input, a CNN takes tensors of shape (image_height, image_width, color_channels), ignoring the batch size. If you are new to these dimensions, color_channels refers to (R,G,B). In this example, you will configure our CNN to process inputs of shape (32, 32, 3), which is the format of CIFAR images. You can do this by passing the argument input_shape to our first layer.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">model <span style="color: #666666">=</span> models<span style="color: #666666">.</span>Sequential()
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Conv2D(<span style="color: #666666">32</span>, (<span style="color: #666666">3</span>, <span style="color: #666666">3</span>), activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>, input_shape<span style="color: #666666">=</span>(<span style="color: #666666">32</span>, <span style="color: #666666">32</span>, <span style="color: #666666">3</span>)))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>MaxPooling2D((<span style="color: #666666">2</span>, <span style="color: #666666">2</span>)))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Conv2D(<span style="color: #666666">64</span>, (<span style="color: #666666">3</span>, <span style="color: #666666">3</span>), activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>MaxPooling2D((<span style="color: #666666">2</span>, <span style="color: #666666">2</span>)))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Conv2D(<span style="color: #666666">64</span>, (<span style="color: #666666">3</span>, <span style="color: #666666">3</span>), activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>))
<span style="color: #408080; font-style: italic"># Let&#39;s display the architecture of our model so far.</span>
model<span style="color: #666666">.</span>summary()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>You can see that the output of every Conv2D and MaxPooling2D layer is a 3D tensor of shape (height, width, channels). The width and height dimensions tend to shrink as you go deeper in the network. The number of output channels for each Conv2D layer is controlled by the first argument (e.g., 32 or 64). Typically, as the width and height shrink, you can afford (computationally) to add more output channels in each Conv2D layer.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="add-dense-layers-on-top">Add Dense layers on top </h2>
<p>To complete our model, you will feed the last output tensor from the
convolutional base (of shape (4, 4, 64)) into one or more Dense layers
to perform classification. Dense layers take vectors as input (which
are 1D), while the current output is a 3D tensor. First, you will
flatten (or unroll) the 3D output to 1D, then add one or more Dense
layers on top. CIFAR has 10 output classes, so you use a final Dense
layer with 10 outputs and a softmax activation.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Flatten())
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Dense(<span style="color: #666666">64</span>, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>))
model<span style="color: #666666">.</span>add(layers<span style="color: #666666">.</span>Dense(<span style="color: #666666">10</span>))
Here<span style="color: #BA2121">&#39;s the complete architecture of our model.</span>
model<span style="color: #666666">.</span>summary()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>As you can see, our (4, 4, 64) outputs were flattened into vectors of shape (1024) before going through two Dense layers.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="compile-and-train-the-model">Compile and train the model </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">model<span style="color: #666666">.</span>compile(optimizer<span style="color: #666666">=</span><span style="color: #BA2121">&#39;adam&#39;</span>,
loss<span style="color: #666666">=</span>tf<span style="color: #666666">.</span>keras<span style="color: #666666">.</span>losses<span style="color: #666666">.</span>SparseCategoricalCrossentropy(from_logits<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>),
metrics<span style="color: #666666">=</span>[<span style="color: #BA2121">&#39;accuracy&#39;</span>])
history <span style="color: #666666">=</span> model<span style="color: #666666">.</span>fit(train_images, train_labels, epochs<span style="color: #666666">=10</span>,
validation_data<span style="color: #666666">=</span>(test_images, test_labels))
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="finally-evaluate-the-model">Finally, evaluate the model </h2>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">plt<span style="color: #666666">.</span>plot(history<span style="color: #666666">.</span>history[<span style="color: #BA2121">&#39;accuracy&#39;</span>], label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;accuracy&#39;</span>)
plt<span style="color: #666666">.</span>plot(history<span style="color: #666666">.</span>history[<span style="color: #BA2121">&#39;val_accuracy&#39;</span>], label <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;val_accuracy&#39;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&#39;Epoch&#39;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39;Accuracy&#39;</span>)
plt<span style="color: #666666">.</span>ylim([<span style="color: #666666">0.5</span>, <span style="color: #666666">1</span>])
plt<span style="color: #666666">.</span>legend(loc<span style="color: #666666">=</span><span style="color: #BA2121">&#39;lower right&#39;</span>)
test_loss, test_acc <span style="color: #666666">=</span> model<span style="color: #666666">.</span>evaluate(test_images, test_labels, verbose<span style="color: #666666">=2</span>)
<span style="color: #008000">print</span>(test_acc)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="building-our-own-cnn-code">Building our own CNN code </h2>
<p>Here we present a flexible and readable python code for a CNN
implemented with NumPy. We will present the code, showcase how to use
the codebase and fit a CNN that yields a 99% accuracy on the 28x28
MNIST dataset within reasonable time.
</p>
<b>The codes here were developed by Eric Reber and Gregor Kajda during spring 2023.</b>
<p>The CNN is compatible with all schedulers, cost functions and
activation functions discussed in constructing our neural network
codes.
</p>
<p> The CNN code consists of different types of Layer classes, including
Convolution2DLayer, Pooling2DLayer, FlattenLayer, FullyConnectedLayer
and OutputLayer, which can be added to the CNN object using the
interface of the CNN class. This allows you to easily construct your
own CNN, as well as allowing you to get used to an interface similar
to that of TensorFlow which is used for real world applications.
</p>
<p>Another important feature of this code is that it throws errors if
unreasonable decisions are made (for example using a kernel that is
larger than the image, not using a FlattenLayer, etc), and provides
the user with an informative error message.
</p>
<h3 id="list-of-contents">List of contents: </h3>
<ol>
<li> Schedulers</li>
<li> Activation Functions</li>
<li> Cost Functions</li>
<li> Convolution</li>
<li> Layers</li>
<li> CNN</li>
<li> Some final remarks</li>
</ol>
<h3 id="schedulers">Schedulers </h3>
<p>The code below shows object oriented implementations of the Constant,
Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All
of the classes belong to the shared abstract Scheduler class, and
share the update_change() and reset() methods allowing for any of the
schedulers to be seamlessly used during the training stage, as will
later be shown in the fit() method of the neural
network. Update_change() only has one parameter, the gradient
(\( \delta^{l}_{j}a^{l-1}_k \)), and returns the change which will be
subtracted from the weights. The reset() function takes no parameters,
and resets the desired variables. For Constant and Momentum, reset
does nothing.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Scheduler</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Abstract class for Schedulers</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta):
<span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">=</span> eta
<span style="color: #408080; font-style: italic"># should be overwritten</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">NotImplementedError</span>
<span style="color: #408080; font-style: italic"># overwritten if needed</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">pass</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Constant</span>(Scheduler):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(eta)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">*</span> gradient
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">pass</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Momentum</span>(Scheduler):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta: <span style="color: #008000">float</span>, momentum: <span style="color: #008000">float</span>):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(eta)
<span style="color: #008000">self</span><span style="color: #666666">.</span>momentum <span style="color: #666666">=</span> momentum
<span style="color: #008000">self</span><span style="color: #666666">.</span>change <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
<span style="color: #008000">self</span><span style="color: #666666">.</span>change <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>momentum <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>change <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">*</span> gradient
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>change
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">pass</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Adagrad</span>(Scheduler):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(eta)
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
delta <span style="color: #666666">=</span> <span style="color: #666666">1e-8</span> <span style="color: #408080; font-style: italic"># avoid division ny zero</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((gradient<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], gradient<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]))
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">+=</span> gradient <span style="color: #666666">@</span> gradient<span style="color: #666666">.</span>T
G_t_inverse <span style="color: #666666">=</span> <span style="color: #666666">1</span> <span style="color: #666666">/</span> (
delta <span style="color: #666666">+</span> np<span style="color: #666666">.</span>sqrt(np<span style="color: #666666">.</span>reshape(np<span style="color: #666666">.</span>diagonal(<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t), (<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], <span style="color: #666666">1</span>)))
)
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">*</span> gradient <span style="color: #666666">*</span> G_t_inverse
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">AdagradMomentum</span>(Scheduler):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta, momentum):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(eta)
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>momentum <span style="color: #666666">=</span> momentum
<span style="color: #008000">self</span><span style="color: #666666">.</span>change <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
delta <span style="color: #666666">=</span> <span style="color: #666666">1e-8</span> <span style="color: #408080; font-style: italic"># avoid division ny zero</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((gradient<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], gradient<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]))
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">+=</span> gradient <span style="color: #666666">@</span> gradient<span style="color: #666666">.</span>T
G_t_inverse <span style="color: #666666">=</span> <span style="color: #666666">1</span> <span style="color: #666666">/</span> (
delta <span style="color: #666666">+</span> np<span style="color: #666666">.</span>sqrt(np<span style="color: #666666">.</span>reshape(np<span style="color: #666666">.</span>diagonal(<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t), (<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], <span style="color: #666666">1</span>)))
)
<span style="color: #008000">self</span><span style="color: #666666">.</span>change <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>change <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>momentum <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">*</span> gradient <span style="color: #666666">*</span> G_t_inverse
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>change
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000">self</span><span style="color: #666666">.</span>G_t <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">RMS_prop</span>(Scheduler):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta, rho):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(eta)
<span style="color: #008000">self</span><span style="color: #666666">.</span>rho <span style="color: #666666">=</span> rho
<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">=</span> <span style="color: #666666">0.0</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
delta <span style="color: #666666">=</span> <span style="color: #666666">1e-8</span> <span style="color: #408080; font-style: italic"># avoid division ny zero</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">+</span> (<span style="color: #666666">1</span> <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho) <span style="color: #666666">*</span> gradient <span style="color: #666666">*</span> gradient
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">*</span> gradient <span style="color: #666666">/</span> (np<span style="color: #666666">.</span>sqrt(<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">+</span> delta))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">=</span> <span style="color: #666666">0.0</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Adam</span>(Scheduler):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, eta, rho, rho2):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(eta)
<span style="color: #008000">self</span><span style="color: #666666">.</span>rho <span style="color: #666666">=</span> rho
<span style="color: #008000">self</span><span style="color: #666666">.</span>rho2 <span style="color: #666666">=</span> rho2
<span style="color: #008000">self</span><span style="color: #666666">.</span>moment <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>n_epochs <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">update_change</span>(<span style="color: #008000">self</span>, gradient):
delta <span style="color: #666666">=</span> <span style="color: #666666">1e-8</span> <span style="color: #408080; font-style: italic"># avoid division ny zero</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>moment <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>moment <span style="color: #666666">+</span> (<span style="color: #666666">1</span> <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho) <span style="color: #666666">*</span> gradient
<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho2 <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">+</span> (<span style="color: #666666">1</span> <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho2) <span style="color: #666666">*</span> gradient <span style="color: #666666">*</span> gradient
moment_corrected <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>moment <span style="color: #666666">/</span> (<span style="color: #666666">1</span> <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho<span style="color: #666666">**</span><span style="color: #008000">self</span><span style="color: #666666">.</span>n_epochs)
second_corrected <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">/</span> (<span style="color: #666666">1</span> <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>rho2<span style="color: #666666">**</span><span style="color: #008000">self</span><span style="color: #666666">.</span>n_epochs)
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>eta <span style="color: #666666">*</span> moment_corrected <span style="color: #666666">/</span> (np<span style="color: #666666">.</span>sqrt(second_corrected <span style="color: #666666">+</span> delta))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">reset</span>(<span style="color: #008000">self</span>):
<span style="color: #008000">self</span><span style="color: #666666">.</span>n_epochs <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>moment <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>second <span style="color: #666666">=</span> <span style="color: #666666">0</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="usage-of-schedulers">Usage of schedulers </h3>
<p>To initalize a scheduler, simply create the object and pass in the necessary parameters such as the learning rate and the momentum as shown below. As the Scheduler class is an abstract class it should not called directly, and will raise an error upon usage.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">momentum_scheduler <span style="color: #666666">=</span> Momentum(eta<span style="color: #666666">=1e-3</span>, momentum<span style="color: #666666">=0.9</span>)
adam_scheduler <span style="color: #666666">=</span> Adam(eta<span style="color: #666666">=1e-3</span>, rho<span style="color: #666666">=0.9</span>, rho2<span style="color: #666666">=0.999</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Here is a small example for how a segment of code using schedulers could look. Switching out the schedulers is simple.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones((<span style="color: #666666">3</span>,<span style="color: #666666">3</span>))
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;Before scheduler:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BB6688; font-weight: bold">{</span>weights<span style="color: #BB6688; font-weight: bold">=}</span><span style="color: #BA2121">&quot;</span>)
epochs <span style="color: #666666">=</span> <span style="color: #666666">10</span>
<span style="color: #008000; font-weight: bold">for</span> e <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(epochs):
gradient <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(<span style="color: #666666">3</span>, <span style="color: #666666">3</span>)
change <span style="color: #666666">=</span> adam_scheduler<span style="color: #666666">.</span>update_change(gradient)
weights <span style="color: #666666">=</span> weights <span style="color: #666666">-</span> change
adam_scheduler<span style="color: #666666">.</span>reset()
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">After scheduler:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BB6688; font-weight: bold">{</span>weights<span style="color: #BB6688; font-weight: bold">=}</span><span style="color: #BA2121">&quot;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="cost-functions">Cost functions </h3>
<p>In this section we will quickly look at cost functions that can be
used when creating the neural network. Every cost function takes the
target vector as its parameter, and returns a function valued only at
X such that it may easily be differentiated.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostOLS</span>(target):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Return OLS function valued only at X, so</span>
<span style="color: #BA2121; font-style: italic"> that it may be easily differentiated</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">func</span>(X):
<span style="color: #008000; font-weight: bold">return</span> (<span style="color: #666666">1.0</span> <span style="color: #666666">/</span> target<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]) <span style="color: #666666">*</span> np<span style="color: #666666">.</span>sum((target <span style="color: #666666">-</span> X) <span style="color: #666666">**</span> <span style="color: #666666">2</span>)
<span style="color: #008000; font-weight: bold">return</span> func
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostLogReg</span>(target):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Return Logistic Regression cost function</span>
<span style="color: #BA2121; font-style: italic"> valued only at X, so that it may be easily differentiated</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">func</span>(X):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">-</span>(<span style="color: #666666">1.0</span> <span style="color: #666666">/</span> target<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]) <span style="color: #666666">*</span> np<span style="color: #666666">.</span>sum(
(target <span style="color: #666666">*</span> np<span style="color: #666666">.</span>log(X <span style="color: #666666">+</span> <span style="color: #666666">10e-10</span>)) <span style="color: #666666">+</span> ((<span style="color: #666666">1</span> <span style="color: #666666">-</span> target) <span style="color: #666666">*</span> np<span style="color: #666666">.</span>log(<span style="color: #666666">1</span> <span style="color: #666666">-</span> X <span style="color: #666666">+</span> <span style="color: #666666">10e-10</span>))
)
<span style="color: #008000; font-weight: bold">return</span> func
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">CostCrossEntropy</span>(target):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Return cross entropy cost function valued only at X, so</span>
<span style="color: #BA2121; font-style: italic"> that it may be easily differentiated</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">func</span>(X):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">-</span>(<span style="color: #666666">1.0</span> <span style="color: #666666">/</span> target<span style="color: #666666">.</span>size) <span style="color: #666666">*</span> np<span style="color: #666666">.</span>sum(target <span style="color: #666666">*</span> np<span style="color: #666666">.</span>log(X <span style="color: #666666">+</span> <span style="color: #666666">10e-10</span>))
<span style="color: #008000; font-weight: bold">return</span> func
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="usage-of-cost-functions">Usage of cost functions </h3>
<p>Below we will provide a short example of how these cost function may
be used to obtain results if you wish to test them out on your own
using AutoGrad's automatic differentiation.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
target <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">1</span>, <span style="color: #666666">2</span>, <span style="color: #666666">3</span>]])<span style="color: #666666">.</span>T
a <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">4</span>, <span style="color: #666666">5</span>, <span style="color: #666666">6</span>]])<span style="color: #666666">.</span>T
cost_func <span style="color: #666666">=</span> CostCrossEntropy
cost_func_derivative <span style="color: #666666">=</span> grad(cost_func(target))
valued_at_a <span style="color: #666666">=</span> cost_func_derivative(a)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;Derivative of cost function </span><span style="color: #BB6688; font-weight: bold">{</span>cost_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span><span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121"> valued at a:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BB6688; font-weight: bold">{</span>valued_at_a<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="activation-functions">Activation functions </h3>
<p>Finally, before we look at the layers that make up the neural network,
we will look at the activation functions which can be specified
between the hidden layers and as the output function. Each function
can be valued for any given vector or matrix X, and can be
differentiated via derivate().
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> elementwise_grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">identity</span>(X):
<span style="color: #008000; font-weight: bold">return</span> X
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">sigmoid</span>(X):
<span style="color: #008000; font-weight: bold">try</span>:
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1.0</span> <span style="color: #666666">/</span> (<span style="color: #666666">1</span> <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>X))
<span style="color: #008000; font-weight: bold">except</span> <span style="color: #D2413A; font-weight: bold">FloatingPointError</span>:
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>where(X <span style="color: #666666">&gt;</span> np<span style="color: #666666">.</span>zeros(X<span style="color: #666666">.</span>shape), np<span style="color: #666666">.</span>ones(X<span style="color: #666666">.</span>shape), np<span style="color: #666666">.</span>zeros(X<span style="color: #666666">.</span>shape))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">softmax</span>(X):
X <span style="color: #666666">=</span> X <span style="color: #666666">-</span> np<span style="color: #666666">.</span>max(X, axis<span style="color: #666666">=-1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)
delta <span style="color: #666666">=</span> <span style="color: #666666">10e-10</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>exp(X) <span style="color: #666666">/</span> (np<span style="color: #666666">.</span>sum(np<span style="color: #666666">.</span>exp(X), axis<span style="color: #666666">=-1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>) <span style="color: #666666">+</span> delta)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">RELU</span>(X):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>where(X <span style="color: #666666">&gt;</span> np<span style="color: #666666">.</span>zeros(X<span style="color: #666666">.</span>shape), X, np<span style="color: #666666">.</span>zeros(X<span style="color: #666666">.</span>shape))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">LRELU</span>(X):
delta <span style="color: #666666">=</span> <span style="color: #666666">10e-4</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>where(X <span style="color: #666666">&gt;</span> np<span style="color: #666666">.</span>zeros(X<span style="color: #666666">.</span>shape), X, delta <span style="color: #666666">*</span> X)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">derivate</span>(func):
<span style="color: #008000; font-weight: bold">if</span> func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;RELU&quot;</span>:
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">func</span>(X):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>where(X <span style="color: #666666">&gt;</span> <span style="color: #666666">0</span>, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>)
<span style="color: #008000; font-weight: bold">return</span> func
<span style="color: #008000; font-weight: bold">elif</span> func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;LRELU&quot;</span>:
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">func</span>(X):
delta <span style="color: #666666">=</span> <span style="color: #666666">10e-4</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>where(X <span style="color: #666666">&gt;</span> <span style="color: #666666">0</span>, <span style="color: #666666">1</span>, delta)
<span style="color: #008000; font-weight: bold">return</span> func
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">return</span> elementwise_grad(func)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="usage-of-activation-functions">Usage of activation functions </h3>
<p>Below we present a short demonstration of how to use an activation
function. The derivative of the activation function will be important
when calculating the output delta term during backpropagation. Note
that derivate() can also be used for cost functions for a more
generalized approach.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">z <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">4</span>, <span style="color: #666666">5</span>, <span style="color: #666666">6</span>]])<span style="color: #666666">.</span>T
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;Input to activation function:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BB6688; font-weight: bold">{</span>z<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>)
act_func <span style="color: #666666">=</span> sigmoid
a <span style="color: #666666">=</span> act_func(z)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">Output from </span><span style="color: #BB6688; font-weight: bold">{</span>act_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span><span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121"> activation function:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BB6688; font-weight: bold">{</span>a<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>)
act_func_derivative <span style="color: #666666">=</span> derivate(act_func)
valued_at_z <span style="color: #666666">=</span> act_func_derivative(a)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">Derivative of </span><span style="color: #BB6688; font-weight: bold">{</span>act_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span><span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121"> activation function valued at z:</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BB6688; font-weight: bold">{</span>valued_at_z<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="convolution">Convolution </h3>
<p>In order to construct a convolutional neural network (CNN), it is
crucial to comprehend the fundamental principles of convolution and
how it aids in extracting information from images. Convolution, at its
core, is merely a mathematical operation between two functions that
yields another function. It is represented by an integral between two
functions, which is typically expressed as:
</p>
$$
(f \ast g)(t):=\int_{-\infty}^{\infty} f(\tau) g(t-\tau) d \tau.
$$
<p>Here, \( f \) and \( g \) are the two functions on which we want to perform an
operation. The outcome of the convolution operation is represented by
\( (f \ast g) \), and it is derived by sliding the function \( g \) over \( f \) and
computing the integral of their product at each position. If both
functions are continuous, convolution takes the form shown
above. However, if we discretize both \( f \) and \( g \), the convolution
operation will take the form of a sum between the elements of \( f \) and \( g \):
</p>
$$
(f \ast g)[n]=\sum_{m=0}^{n-1} f(m) g(n-m).
$$
<p>The key idea we utilize to extract the information contained in an
image is to slide an \( m \times n \) matrix \( g \) over an \( m \times n \)
matrix \( f \). In our case, \( f \) represents the image, while \( g \)
represents the kernel, oftentimes called a filter. However, since our
convolution will be a two-dimensional variant, we need to extend our
mathematical formula with an additional summation:
</p>
$$
(f \ast g)(i, j)\sum_{m=0}^{M-1}\sum_{n=0}^{N-1} f(m,n) g(i-m, j-n).
$$
<p>It is imperative to note that the size of the kernel g is
significantly smaller than the size of the input image f, thereby
reducing the amount of computation necessary for feature
extraction. Furthermore, the kernel is usually a trainable parameter
in a convolutional neural network, allowing the network to learn
appropriate kernels for specific tasks.
</p>
<p>To give you an example of how 2D convolution works in practice,
suppose we have an image \( f \) of dimension \( 6 \times 6 \)
</p>
$$
f = \begin{bmatrix}
4 & 1 & 2 & 9 & 8 & 6 \\
9 & 5 & 9 & 5 & 8 & 5 \\
1 & 5 & 9 & 7 & 6 & 4 \\
2 & 9 & 8 & 3 & 7 & 1 \\
8 & 1 & 6 & 4 & 2 & 2 \\
1 & 0 & 5 & 7 & 8 & 2 \\
\end{bmatrix}
$$
<p>and a \( 3 \times 3 \) kernel \( g \) called a low-pass filter. Note that the
kernel is usually rotated by 180 degrees during convolution, however
this has no effect on this kernel.
</p>
$$
g = \frac{1}{9}
\begin{bmatrix}
1 & 1 & 1 \\
1 & 1 & 1 \\
1 & 1 & 1 \\
\end{bmatrix}
$$
<p>In order to filter the image, we have to extract a \( 3 \times 3 \)
element from the upper left corner of \( f \), and perform element-wise
multiplication of the extracted image pixels with the elements of the
kernel \( g \):
</p>
$$
\begin{bmatrix}
4 & 1 & 2 \\
9 & 5 & 9 \\
1 & 5 & 9 \\
\end{bmatrix}
\cdot
\begin{bmatrix}
\frac{1}{9} & \frac{1}{9} & \frac{1}{9} \\
\frac{1}{9} & \frac{1}{9} & \frac{1}{9} \\
\frac{1}{9} & \frac{1}{9} & \frac{1}{9} \\
\end{bmatrix}
=
\begin{bmatrix}
\frac{4}{9} & \frac{1}{9} & \frac{2}{9} \\
\frac{9}{9} & \frac{5}{9} & \frac{9}{9} \\
\frac{1}{9} & \frac{5}{9} & \frac{9}{9} \\
\end {bmatrix}
= \boldsymbol{A}
$$
<p>Then, following the multiplication, we summarize all the elements of the resulting matrix \( \boldsymbol{A} \):</p>
$$
(f \ast g)(0, 0)= \sum_{i=0}^{2} \sum_{j=0}^{2} a_{i,j} = 5,
$$
<p>which corresponds to the first element of the filtered image \( (f \ast g) \).</p>
<p>Here we use a stride of \( S=1 \), a parameter denoted \( S \) which describes how
many indexes we move the kernel \( g \) to the right before repeating the
calculations above for the next \( 3 \times 3 \) element of the image
\( f \). It is usually presumed that \( S=1 \), however, larger values for \( S \)
can be used to reduce the dimentionality of the filtered image such
that the convolution operation is more computationally efficient. In
the context of a convolutional neural network, this will become very
useful.
</p>
<p>The full result of the convolution is:</p>
$$
(f \ast g) =
\begin{bmatrix}
5 & 5.78 & 7 & 6.44 \\
6.33 & 6.67 & 6.89 & 5.11 \\
5.44 & 5.78 & 5.78 & 4 \\
4.44 & 4.78 & 5.56 & 4 \\
\end{bmatrix}
$$
<p>The result is markedly smaller in shape than the original image. This
occurs when using convolution without first padding the image with
additional columns and rows, allowing us to keep the original image
shape after sliding the kernel over the image. How many rows and
columns we wish to pad the image with depends strictly on the shape of
the kernel, as we wish to pad the image with \( r \) additional rows and
\( c \) additional columns.
</p>
$$
r =\lfloor \frac{\mathrm{kernel height}}{2} \rfloor \cdot 2 \\
c =\lfloor \frac{\mathrm{kernel width}}{2} \rfloor \cdot 2
$$
<p>Note the notation \( \lfloor \frac{\mathrm{kernel width}}{2} \rfloor \) means that
we floor the result of the division, meaning we round down to a whole
number in case \( \frac{\mathrm{kernel width}}{2} \) results in a floating point
number.
</p>
<p>Using those simple equations, we find out by how much we have to
extend the dimensions of the original image. Before proceeding,
however, we might ask what we shall fill the additional rows and
columns with? One of the most common approaches to padding is
zero-padding, which as the name suggest, involves filling the rows and
columns with zeros. This is the approach that we will be using for
this demonstration. If we apply this padding to out original \( 6 \times 6 \)
image, the result will be an \( 8 \times 8 \) image as the kernel has a width and
height of 3. Note that the original image is encapsuled by the
zero-padded rows and columns:
</p>
$$
\begin{bmatrix}
0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
0 & 4 & 1 & 2 & 9 & 8 & 6 & 0 \\
0 & 9 & 5 & 9 & 5 & 8 & 5 & 0 \\
0 & 1 & 5 & 9 & 7 & 6 & 4 & 0 \\
0 & 2 & 9 & 8 & 3 & 7 & 1 & 0 \\
0 & 8 & 1 & 6 & 4 & 2 & 2 & 0 \\
0 & 1 & 0 & 5 & 7 & 8 & 2 & 0 \\
0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\
\end{bmatrix}.
$$
<p>Below we have provided code that demonstrates padding and
convolution. As you will see when we run the code, the size of the
image will remain unchanged when using padding.~
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">padding</span>(image, kernel):
<span style="color: #408080; font-style: italic"># calculate r and c</span>
r <span style="color: #666666">=</span> (kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
c <span style="color: #666666">=</span> (kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># padded image dimensions</span>
padded_height <span style="color: #666666">=</span> image<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> r
padded_width <span style="color: #666666">=</span> image<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> c
<span style="color: #408080; font-style: italic"># for more readable code</span>
k_half_height <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
k_half_width <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># zero matrix with padded dimensions</span>
padded_img <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((padded_height, padded_width))
<span style="color: #408080; font-style: italic"># place image into zero matrix</span>
padded_img[k_half_height : padded_height <span style="color: #666666">-</span> k_half_height,
k_half_width : padded_width <span style="color: #666666">-</span> k_half_width] <span style="color: #666666">=</span> image[:, :]
<span style="color: #008000; font-weight: bold">return</span> padded_img
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">convolve</span>(original_image, padded_image, kernel, stride<span style="color: #666666">=1</span>):
<span style="color: #408080; font-style: italic"># rotate kernel by 180 degrees</span>
kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>rot90(np<span style="color: #666666">.</span>rot90(kernel))
<span style="color: #408080; font-style: italic"># note that kernel height // 2 is written as &#39;m&#39;</span>
<span style="color: #408080; font-style: italic"># and kernel width // 2 as &#39;n&#39; in the mathematical notation</span>
m <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
n <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
r <span style="color: #666666">=</span> (kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
c <span style="color: #666666">=</span> (kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># initialize output array</span>
convolved_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(original_image<span style="color: #666666">.</span>shape)
image_height <span style="color: #666666">=</span> original_image<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]
image_width <span style="color: #666666">=</span> original_image<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>]
<span style="color: #408080; font-style: italic"># the convolution</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(m, image_height <span style="color: #666666">+</span> m, stride):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(n, image_width <span style="color: #666666">+</span> n, stride):
convolved_image[i<span style="color: #666666">-</span>m, j<span style="color: #666666">-</span>n] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(
padded_image[i : i <span style="color: #666666">+</span> m, j : j <span style="color: #666666">+</span> n]
<span style="color: #666666">*</span> kernel
)
<span style="color: #008000; font-weight: bold">return</span> convolved_image
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">convolve</span>(image, kernel, stride<span style="color: #666666">=1</span>):
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">2</span>):
kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>rot90(kernel)
k_half_height <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
k_half_width <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
conv_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(image<span style="color: #666666">.</span>shape)
pad_image <span style="color: #666666">=</span> padding(image, kernel)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(k_half_height, conv_image<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> k_half_height, stride):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(k_half_width, conv_image<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> k_half_width, stride):
conv_image[i <span style="color: #666666">-</span> k_half_height, j <span style="color: #666666">-</span> k_half_width] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(
pad_image[
i <span style="color: #666666">-</span> k_half_height : i <span style="color: #666666">+</span> k_half_height <span style="color: #666666">+</span> <span style="color: #666666">1</span>, j <span style="color: #666666">-</span> k_half_width : j <span style="color: #666666">+</span> k_half_width <span style="color: #666666">+</span> <span style="color: #666666">1</span>
]
<span style="color: #666666">*</span> kernel
)
<span style="color: #008000; font-weight: bold">return</span> conv_image
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Fun fact: When filtering images, you will see that convolution
involves rotating the kernel by 180 degrees. However, this is not the
case when applying convolution in a CNN, where the same operation that is not
rotated by 180 degrees is called cross-correlation, which is normally implemented in most libraries.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">original_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">4</span>, <span style="color: #666666">1</span>, <span style="color: #666666">2</span>, <span style="color: #666666">9</span>, <span style="color: #666666">8</span>, <span style="color: #666666">6</span>],
[<span style="color: #666666">9</span>, <span style="color: #666666">5</span>, <span style="color: #666666">9</span>, <span style="color: #666666">5</span>, <span style="color: #666666">8</span>, <span style="color: #666666">5</span>],
[<span style="color: #666666">1</span>, <span style="color: #666666">5</span>, <span style="color: #666666">9</span>, <span style="color: #666666">7</span>, <span style="color: #666666">6</span>, <span style="color: #666666">4</span>],
[<span style="color: #666666">2</span>, <span style="color: #666666">9</span>, <span style="color: #666666">8</span>, <span style="color: #666666">3</span>, <span style="color: #666666">7</span>, <span style="color: #666666">1</span>],
[<span style="color: #666666">8</span>, <span style="color: #666666">1</span>, <span style="color: #666666">6</span>, <span style="color: #666666">4</span>, <span style="color: #666666">2</span>, <span style="color: #666666">2</span>],
[<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">5</span>, <span style="color: #666666">7</span>, <span style="color: #666666">8</span>, <span style="color: #666666">2</span>]])
kernel <span style="color: #666666">=</span> (<span style="color: #666666">1/9</span>)<span style="color: #666666">*</span>np<span style="color: #666666">.</span>ones((<span style="color: #666666">3</span>,<span style="color: #666666">3</span>))
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;</span><span style="color: #BB6688; font-weight: bold">{</span>original_image<span style="color: #666666">.</span>shape<span style="color: #BB6688; font-weight: bold">=}</span><span style="color: #BA2121">&quot;</span>)
<span style="color: #408080; font-style: italic"># note that convolve() performs padding</span>
convolved_image <span style="color: #666666">=</span> convolve(original_image, kernel, stride<span style="color: #666666">=1</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&quot;</span><span style="color: #BB6688; font-weight: bold">{</span>convolved_image<span style="color: #666666">.</span>shape<span style="color: #BB6688; font-weight: bold">=}</span><span style="color: #BA2121">&quot;</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>As you can see, the resulting image is of the same size as the
original image. To round of our demonstration of convolution, we will
present the results of convolution using commonly used kernels. In a
CNN, the values of the kernels are randomly initialized, and then
learned during training. These kernels will extract information
regarding the picture, such as for example the edge detection filter
demonstrated below extracts the edges present in the picture. Of
course, there is no guarantee that the CNN will learn an edge
detection filter, but this should provide some intuiton as to how the
CNN is able to use kernels to make better predictions than a regular
feed forward neural network.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #408080; font-style: italic"># Now an example using a real image and first a gaussian low-pass filter and then a Sobel filter</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">imageio.v3</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">imageio</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">time</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">generate_gauss_mask</span>(sigma, K<span style="color: #666666">=1</span>):
side <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ceil(<span style="color: #666666">1</span> <span style="color: #666666">+</span> <span style="color: #666666">8</span> <span style="color: #666666">*</span> sigma)
y, x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-</span>side <span style="color: #666666">//</span> <span style="color: #666666">2</span> <span style="color: #666666">+</span> <span style="color: #666666">1</span> : (side <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>, <span style="color: #666666">-</span>side <span style="color: #666666">//</span> <span style="color: #666666">2</span> <span style="color: #666666">+</span> <span style="color: #666666">1</span> : (side <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>]
ker_coef <span style="color: #666666">=</span> K <span style="color: #666666">/</span> (<span style="color: #666666">2</span> <span style="color: #666666">*</span> np<span style="color: #666666">.</span>pi <span style="color: #666666">*</span> sigma<span style="color: #666666">**2</span>)
g <span style="color: #666666">=</span> np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>((x<span style="color: #666666">**2</span> <span style="color: #666666">+</span> y<span style="color: #666666">**2</span>) <span style="color: #666666">/</span> (<span style="color: #666666">2.0</span> <span style="color: #666666">*</span> sigma<span style="color: #666666">**2</span>)))
<span style="color: #008000; font-weight: bold">return</span> g, ker_coef
img_path <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;data/IMG-2167.JPG&quot;</span>
image_of_cute_dog <span style="color: #666666">=</span> imageio<span style="color: #666666">.</span>imread(img_path, mode<span style="color: #666666">=</span><span style="color: #BA2121">&#39;L&#39;</span>)
plt<span style="color: #666666">.</span>imshow(image_of_cute_dog, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>, vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, aspect<span style="color: #666666">=</span><span style="color: #BA2121">&quot;auto&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Original image&quot;</span>)
plt<span style="color: #666666">.</span>show()
gauss, kernel <span style="color: #666666">=</span> generate_gauss_mask(sigma<span style="color: #666666">=6</span>)
gauss_kernel <span style="color: #666666">=</span> gauss<span style="color: #666666">*</span>kernel
filtered_image <span style="color: #666666">=</span> convolve(image_of_cute_dog, gauss_kernel)
plt<span style="color: #666666">.</span>imshow(filtered_image, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>, vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, aspect<span style="color: #666666">=</span><span style="color: #BA2121">&quot;auto&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Result of convolution with gauss kernel (blurring filter)&quot;</span>)
plt<span style="color: #666666">.</span>show()
sobel_kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([[<span style="color: #666666">1</span>, <span style="color: #666666">2</span>, <span style="color: #666666">1</span>],
[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, <span style="color: #666666">0</span>],
[<span style="color: #666666">-1</span>, <span style="color: #666666">-2</span>, <span style="color: #666666">-1</span>]])
filtered_image <span style="color: #666666">=</span> convolve(image_of_cute_dog, sobel_kernel)
plt<span style="color: #666666">.</span>imshow(filtered_image, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>, vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, aspect<span style="color: #666666">=</span><span style="color: #BA2121">&quot;auto&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Result of convolution with sobel kernel (edge detection filter)&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="layers">Layers </h3>
<p>The code below initialises global variables for readability and
describes the abstract class Layers. This is not important in order to
understand the CNN, but is benefitial for organizing the code neatly.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">math</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">copy</span> <span style="color: #008000; font-weight: bold">import</span> deepcopy, copy
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">typing</span> <span style="color: #008000; font-weight: bold">import</span> Callable
<span style="color: #408080; font-style: italic"># global variables for index readability</span>
input_index <span style="color: #666666">=</span> <span style="color: #666666">0</span>
node_index <span style="color: #666666">=</span> <span style="color: #666666">1</span>
bias_index <span style="color: #666666">=</span> <span style="color: #666666">1</span>
input_channel_index <span style="color: #666666">=</span> <span style="color: #666666">1</span>
feature_maps_index <span style="color: #666666">=</span> <span style="color: #666666">1</span>
height_index <span style="color: #666666">=</span> <span style="color: #666666">2</span>
width_index <span style="color: #666666">=</span> <span style="color: #666666">3</span>
kernel_feature_maps_index <span style="color: #666666">=</span> <span style="color: #666666">1</span>
kernel_input_channels_index <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Layer</span>:
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, seed):
<span style="color: #008000">self</span><span style="color: #666666">.</span>seed <span style="color: #666666">=</span> seed
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">NotImplementedError</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">NotImplementedError</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights</span>(<span style="color: #008000">self</span>, previous_nodes):
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">NotImplementedError</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="convolution2dlayer-convolution-in-a-hidden-layer">Convolution2DLayer: convolution in a hidden layer </h3>
<p>After establishing the foundational understanding of applying
convolution to spatial data, let us delve into the intricate workings
of a convolutional layer in a Convolutional Neural Network (CNN). The
primary function of convolution, as previously discussed, is to
extract pertinent information from images while simultaneously
decreasing the scale of our data. To initiate the image processing, we
shall begin by partitioning the images into color channels (unless the
image is grayscale), comprising three primary colors: red, green, and
blue. We will subsequently utilize trainable kernels to construct a
higher-dimensional encoding of each channel called feature
maps. Successive layers will receive these feature maps as inputs,
generating further encodings, albeit with reduced dimensions. The term
trainable kernels denotes the initialization of pre-defined
kernel-shaped weights, which we will then train via backpropagation,
similar to how weights are trained in a Feedforward Neural Network.
</p>
<p>To ensure seamless integration between our implementation of the
convolutional layer and popular machine learning frameworks like
Tensorflow (Keras) and PyTorch, we have adopted a design pattern that
mirrors the construction of models using these APIs. This involves
implementing our convolutional layer as a Python class or object,
which allows for a more modular and flexible approach to building
neural networks. By structuring our code in this way, users can easily
incorporate our implementation into their existing machine learning
pipelines without having to make significant changes to their
codebase. Additionally, this design pattern promotes code reusability
and makes it easier to maintain and update our convolutional layer
implementation over time.
</p>
<p>Note that the Convolution2DLayer takes in an activation function as a parameter, as it also performs non-linearity.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Convolution2DLayer</span>(Layer):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
input_channels,
feature_maps,
kernel_height,
kernel_width,
v_stride,
h_stride,
pad,
act_func: Callable,
seed<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>,
reset_weights_independently<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,
):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(seed)
<span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels <span style="color: #666666">=</span> input_channels
<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps <span style="color: #666666">=</span> feature_maps
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">=</span> kernel_height
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width <span style="color: #666666">=</span> kernel_width
<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">=</span> v_stride
<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">=</span> h_stride
<span style="color: #008000">self</span><span style="color: #666666">.</span>pad <span style="color: #666666">=</span> pad
<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func <span style="color: #666666">=</span> act_func
<span style="color: #408080; font-style: italic"># such that the layer can be used on its own</span>
<span style="color: #408080; font-style: italic"># outside of the CNN module</span>
<span style="color: #008000; font-weight: bold">if</span> reset_weights_independently <span style="color: #666666">==</span> <span style="color: #008000; font-weight: bold">True</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>_reset_weights_independently()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch):
<span style="color: #408080; font-style: italic"># note that the shape of X_batch = [inputs, input_maps, img_height, img_width]</span>
<span style="color: #408080; font-style: italic"># pad the input batch</span>
X_batch_padded <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_padding(X_batch)
<span style="color: #408080; font-style: italic"># calculate height_index and width_index after stride</span>
strided_height <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>ceil(X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride))
strided_width <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>ceil(X_batch<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride))
<span style="color: #408080; font-style: italic"># create output array</span>
output <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ndarray(
(
X_batch<span style="color: #666666">.</span>shape[input_index],
<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps,
strided_height,
strided_width,
)
)
<span style="color: #408080; font-style: italic"># save input and output for backpropagation</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward <span style="color: #666666">=</span> X_batch
<span style="color: #008000">self</span><span style="color: #666666">.</span>output_shape <span style="color: #666666">=</span> output<span style="color: #666666">.</span>shape
<span style="color: #408080; font-style: italic"># checking for errors, no need to look here :)</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>_check_for_errors()
<span style="color: #408080; font-style: italic"># convolve input with kernel</span>
<span style="color: #008000; font-weight: bold">for</span> img <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(X_batch<span style="color: #666666">.</span>shape[input_index]):
<span style="color: #008000; font-weight: bold">for</span> chin <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels):
<span style="color: #008000; font-weight: bold">for</span> fmap <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps):
out_h <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> h <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">0</span>, X_batch<span style="color: #666666">.</span>shape[height_index], <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride):
out_w <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">0</span>, X_batch<span style="color: #666666">.</span>shape[width_index], <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride):
output[img, fmap, out_h, out_w] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(
X_batch_padded[
img,
chin,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
<span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel[chin, fmap, :, :]
)
out_w <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
out_h <span style="color: #666666">+=</span> <span style="color: #666666">1</span>
<span style="color: #408080; font-style: italic"># Pay attention to the fact that we&#39;re not rotating the kernel by 180 degrees when filtering the image in</span>
<span style="color: #408080; font-style: italic"># the convolutional layer, as convolution in terms of Machine Learning is a procedure known as cross-correlation</span>
<span style="color: #408080; font-style: italic"># in image processing and signal processing</span>
<span style="color: #408080; font-style: italic"># return a</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func(output <span style="color: #666666">/</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, delta_term_next):
<span style="color: #408080; font-style: italic"># intiate matrices</span>
delta_term <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape))
gradient_kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># pad input for convolution</span>
X_batch_padded <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_padding(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward)
<span style="color: #408080; font-style: italic"># Since an activation function is used at the output of the convolution layer, its derivative</span>
<span style="color: #408080; font-style: italic"># has to be accounted for in the backpropagation -&gt; as if ReLU was a layer on its own.</span>
act_derivative <span style="color: #666666">=</span> derivate(<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func)
delta_term_next <span style="color: #666666">=</span> act_derivative(delta_term_next)
<span style="color: #408080; font-style: italic"># fill in 0&#39;s for values removed by vertical stride in feedforward</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">&gt;</span> <span style="color: #666666">1</span>:
v_ind <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(delta_term_next<span style="color: #666666">.</span>shape[height_index]):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">-</span> <span style="color: #666666">1</span>):
delta_term_next <span style="color: #666666">=</span> np<span style="color: #666666">.</span>insert(
delta_term_next, v_ind, <span style="color: #666666">0</span>, axis<span style="color: #666666">=</span>height_index
)
v_ind <span style="color: #666666">+=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride
<span style="color: #408080; font-style: italic"># fill in 0&#39;s for values removed by horizontal stride in feedforward</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">&gt;</span> <span style="color: #666666">1</span>:
h_ind <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(delta_term_next<span style="color: #666666">.</span>shape[width_index]):
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">-</span> <span style="color: #666666">1</span>):
delta_term_next <span style="color: #666666">=</span> np<span style="color: #666666">.</span>insert(
delta_term_next, h_ind, <span style="color: #666666">0</span>, axis<span style="color: #666666">=</span>width_index
)
h_ind <span style="color: #666666">+=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride
<span style="color: #408080; font-style: italic"># crops out 0-rows and 0-columns</span>
delta_term_next <span style="color: #666666">=</span> delta_term_next[
:,
:,
: <span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[height_index],
: <span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[width_index],
]
<span style="color: #408080; font-style: italic"># the gradient received from the next layer also needs to be padded</span>
delta_term_next <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_padding(delta_term_next)
<span style="color: #408080; font-style: italic"># calculate delta term by convolving next delta term with kernel</span>
<span style="color: #008000; font-weight: bold">for</span> img <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_index]):
<span style="color: #008000; font-weight: bold">for</span> chin <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels):
<span style="color: #008000; font-weight: bold">for</span> fmap <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps):
<span style="color: #008000; font-weight: bold">for</span> h <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[height_index]):
<span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[width_index]):
delta_term[img, chin, h, w] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(
delta_term_next[
img,
fmap,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
<span style="color: #666666">*</span> np<span style="color: #666666">.</span>rot90(np<span style="color: #666666">.</span>rot90(<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel[chin, fmap, :, :]))
)
<span style="color: #408080; font-style: italic"># calculate gradient for kernel for weight update</span>
<span style="color: #408080; font-style: italic"># also via convolution</span>
<span style="color: #008000; font-weight: bold">for</span> chin <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels):
<span style="color: #008000; font-weight: bold">for</span> fmap <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps):
<span style="color: #008000; font-weight: bold">for</span> k_x <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height):
<span style="color: #008000; font-weight: bold">for</span> k_y <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width):
gradient_kernel[chin, fmap, k_x, k_y] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(
X_batch_padded[
img,
chin,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
<span style="color: #666666">*</span> delta_term_next[
img,
fmap,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
)
<span style="color: #408080; font-style: italic"># all kernels are updated with weight gradient of kernel</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel <span style="color: #666666">-=</span> gradient_kernel
<span style="color: #408080; font-style: italic"># return delta term</span>
<span style="color: #008000; font-weight: bold">return</span> delta_term
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_padding</span>(<span style="color: #008000">self</span>, X_batch, batch_type<span style="color: #666666">=</span><span style="color: #BA2121">&quot;image&quot;</span>):
<span style="color: #408080; font-style: italic"># same padding for images</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pad <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;same&quot;</span> <span style="color: #AA22FF; font-weight: bold">and</span> batch_type <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;image&quot;</span>:
padded_height <span style="color: #666666">=</span> X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">+</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
padded_width <span style="color: #666666">=</span> X_batch<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">+</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
half_kernel_height <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">//</span> <span style="color: #666666">2</span>
half_kernel_width <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width <span style="color: #666666">//</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># initialize padded array</span>
X_batch_padded <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ndarray(
(
X_batch<span style="color: #666666">.</span>shape[input_index],
X_batch<span style="color: #666666">.</span>shape[feature_maps_index],
padded_height,
padded_width,
)
)
<span style="color: #408080; font-style: italic"># zero pad all images in X_batch</span>
<span style="color: #008000; font-weight: bold">for</span> img <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(X_batch<span style="color: #666666">.</span>shape[input_index]):
padded_img <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(
(X_batch<span style="color: #666666">.</span>shape[feature_maps_index], padded_height, padded_width)
)
padded_img[
:,
half_kernel_height : padded_height <span style="color: #666666">-</span> half_kernel_height,
half_kernel_width : padded_width <span style="color: #666666">-</span> half_kernel_width,
] <span style="color: #666666">=</span> X_batch[img, :, :, :]
X_batch_padded[img, :, :, :] <span style="color: #666666">=</span> padded_img[:, :, :]
<span style="color: #008000; font-weight: bold">return</span> X_batch_padded
<span style="color: #408080; font-style: italic"># same padding for gradients</span>
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pad <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;same&quot;</span> <span style="color: #AA22FF; font-weight: bold">and</span> batch_type <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;grad&quot;</span>:
padded_height <span style="color: #666666">=</span> X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">+</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
padded_width <span style="color: #666666">=</span> X_batch<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">+</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width <span style="color: #666666">//</span> <span style="color: #666666">2</span>) <span style="color: #666666">*</span> <span style="color: #666666">2</span>
half_kernel_height <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">//</span> <span style="color: #666666">2</span>
half_kernel_width <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width <span style="color: #666666">//</span> <span style="color: #666666">2</span>
<span style="color: #408080; font-style: italic"># initialize padded array</span>
delta_term_padded <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(
(
X_batch<span style="color: #666666">.</span>shape[input_index],
X_batch<span style="color: #666666">.</span>shape[feature_maps_index],
padded_height,
padded_width,
)
)
<span style="color: #408080; font-style: italic"># zero pad delta term</span>
delta_term_padded[
:, :, : X_batch<span style="color: #666666">.</span>shape[height_index], : X_batch<span style="color: #666666">.</span>shape[width_index]
] <span style="color: #666666">=</span> X_batch[:, :, :, :]
<span style="color: #008000; font-weight: bold">return</span> delta_term_padded
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">return</span> X_batch
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights_independently</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># sets seed to remove randomness inbetween runs</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>seed <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>:
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #008000">self</span><span style="color: #666666">.</span>seed)
<span style="color: #408080; font-style: italic"># initializes kernel matrix</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ndarray(
(
<span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels,
<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps,
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
)
)
<span style="color: #408080; font-style: italic"># randomly initializes weights</span>
<span style="color: #008000; font-weight: bold">for</span> chin <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel<span style="color: #666666">.</span>shape[kernel_input_channels_index]):
<span style="color: #008000; font-weight: bold">for</span> fmap <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel<span style="color: #666666">.</span>shape[kernel_feature_maps_index]):
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel[chin, fmap, :, :] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height, <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width
)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights</span>(<span style="color: #008000">self</span>, previous_nodes):
<span style="color: #408080; font-style: italic"># sets weights</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>_reset_weights_independently()
<span style="color: #408080; font-style: italic"># returns shape of output used for subsequent layer&#39;s weight initiation</span>
strided_height <span style="color: #666666">=</span> <span style="color: #008000">int</span>(
np<span style="color: #666666">.</span>ceil(previous_nodes<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride)
)
strided_width <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>ceil(previous_nodes<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride))
next_nodes <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones(
(
previous_nodes<span style="color: #666666">.</span>shape[input_index],
<span style="color: #008000">self</span><span style="color: #666666">.</span>feature_maps,
strided_height,
strided_width,
)
)
<span style="color: #008000; font-weight: bold">return</span> next_nodes <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_check_for_errors</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_channel_index] <span style="color: #666666">!=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels:
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">AssertionError</span>(
<span style="color: #BA2121">f&quot;ERROR: Number of input channels in data (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_channel_index]<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">) is not equal to input channels in Convolution2DLayerOPT (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">)! Please change the number of input channels of the Convolution2DLayer such that they are equal&quot;</span>
)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="backpropagation-in-the-convolutional-layer">Backpropagation in the convolutional layer </h3>
<p>As you may have noticed, we have not yet explained how the
backpropagation algorithm works in a convolutional layer. However,
having covered all other major details about convolutional layers, we
are now prepared to do so. It should come as no surprise that the
calculation of delta terms at each convolutional layer takes the form
of convolution. After the gradient has been propagated backwards
through the flattening layer, where it was reshaped into an
appropriate form, calculating the update value for the kernel is
simply a matter of convolving the output gradient with the input of
the layer for which we are updating the weights. For more detail, this
article serves as an excellent resource, see
<a href="https://pavisj.medium.com/convolutions-and-backpropagations-46026a8f5d2c" target="_blank"><tt>https://pavisj.medium.com/convolutions-and-backpropagations-46026a8f5d2c</tt></a>
</p>
<h3 id="demonstration">Demonstration </h3>
<p>We can use the convolutional layer above to perform a simple convolution on an image of the now familiar cute dog.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">imageio.v3</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">imageio</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">plot_convolution_result</span>(X, layer):
plt<span style="color: #666666">.</span>imshow(X[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, :, :], vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Original image&quot;</span>)
plt<span style="color: #666666">.</span>colorbar()
plt<span style="color: #666666">.</span>show()
conv_result <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_feedforward(X)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Result of convolutional layer&quot;</span>)
plt<span style="color: #666666">.</span>imshow(conv_result[<span style="color: #666666">0</span>, <span style="color: #666666">0</span>, :, :], vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>)
plt<span style="color: #666666">.</span>colorbar()
plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic"># create layer</span>
layer <span style="color: #666666">=</span> Convolution2DLayer(
input_channels<span style="color: #666666">=3</span>,
feature_maps<span style="color: #666666">=1</span>,
kernel_height<span style="color: #666666">=4</span>,
kernel_width<span style="color: #666666">=4</span>,
v_stride<span style="color: #666666">=2</span>,
h_stride<span style="color: #666666">=2</span>,
pad<span style="color: #666666">=</span><span style="color: #BA2121">&quot;same&quot;</span>,
act_func<span style="color: #666666">=</span>identity,
seed<span style="color: #666666">=2023</span>,
)
<span style="color: #408080; font-style: italic"># read in image path, make data correct format</span>
img_path <span style="color: #666666">=</span> img_path <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;data/IMG-2167.JPG&quot;</span>
image_of_cute_dog <span style="color: #666666">=</span> imageio<span style="color: #666666">.</span>imread(img_path)
image_shape <span style="color: #666666">=</span> image_of_cute_dog<span style="color: #666666">.</span>shape
image_of_cute_dog <span style="color: #666666">=</span> image_of_cute_dog<span style="color: #666666">.</span>reshape(<span style="color: #666666">1</span>, image_shape[<span style="color: #666666">0</span>], image_shape[<span style="color: #666666">1</span>], image_shape[<span style="color: #666666">2</span>])
image_of_cute_dog <span style="color: #666666">=</span> image_of_cute_dog<span style="color: #666666">.</span>transpose(<span style="color: #666666">0</span>, <span style="color: #666666">3</span>, <span style="color: #666666">1</span>, <span style="color: #666666">2</span>)
<span style="color: #408080; font-style: italic"># plot the result of the convolution</span>
plot_convolution_result(image_of_cute_dog, layer)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>We cobserve that the result has half the pixels on each axis due to
the fact that we've used a horizontal and vertical stride of 2. The
result of this convolution is not very insightfull, as the kernel has
completely random values for the first feedforward pass. However, as
we perform multiple forward and backward passes, the results of the
convolution should provide identifying features of the image it uses
for classification.
</p>
<p>Note that image data usually comes in many different shapes and sizes,
but for our CNN we require the input data be formatted as \[Number of
inputs, input channels, input height, input width\]. Occasionally, the
data you come accross use will be formatted like this, but on many
occasions reshaping and transposing the dimensions is sadly necessary.
</p>
<h3 id="pooling-layer">Pooling Layer </h3>
<p>The pooling layer is another widely used type of layer in
convolutional neural networks that enables data downsampling to a more
manageable size. Despite recent technological advancements that allow
for convolution without excessive size reduction of the data, the
pooling layer still remains a fundamental component of convolutional
neural networks. It can be used before, after, or in between
convolutional layers, although finding the optimal placement of layers
and network depth requires experimentation to achieve the best
performance for a given problem. The code we provide allows you to
perform two types of pooling known as max pooling and average pooling.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Pooling2DLayer</span>(Layer):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
kernel_height,
kernel_width,
v_stride,
h_stride,
pooling<span style="color: #666666">=</span><span style="color: #BA2121">&quot;max&quot;</span>,
seed<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>,
):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(seed)
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">=</span> kernel_height
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width <span style="color: #666666">=</span> kernel_width
<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">=</span> v_stride
<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">=</span> h_stride
<span style="color: #008000">self</span><span style="color: #666666">.</span>pooling <span style="color: #666666">=</span> pooling
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch):
<span style="color: #408080; font-style: italic"># Saving the input for use in the backwardpass</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward <span style="color: #666666">=</span> X_batch
<span style="color: #408080; font-style: italic"># check if user is silly</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>_check_for_errors()
<span style="color: #408080; font-style: italic"># Computing the size of the feature maps based on kernel size and the stride parameter</span>
strided_height <span style="color: #666666">=</span> (
X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height
) <span style="color: #666666">//</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">+</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">if</span> X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">==</span> X_batch<span style="color: #666666">.</span>shape[width_index]:
strided_width <span style="color: #666666">=</span> strided_height
<span style="color: #008000; font-weight: bold">else</span>:
strided_width <span style="color: #666666">=</span> (
X_batch<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width
) <span style="color: #666666">//</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">+</span> <span style="color: #666666">1</span>
<span style="color: #408080; font-style: italic"># initialize output array</span>
output <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ndarray(
(
X_batch<span style="color: #666666">.</span>shape[input_index],
X_batch<span style="color: #666666">.</span>shape[feature_maps_index],
strided_height,
strided_width,
)
)
<span style="color: #408080; font-style: italic"># select pooling action, either max or average pooling</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pooling <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;max&quot;</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>pooling_action <span style="color: #666666">=</span> np<span style="color: #666666">.</span>max
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pooling <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;average&quot;</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>pooling_action <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean
<span style="color: #408080; font-style: italic"># pool based on kernel size and stride</span>
<span style="color: #008000; font-weight: bold">for</span> img <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(output<span style="color: #666666">.</span>shape[input_index]):
<span style="color: #008000; font-weight: bold">for</span> fmap <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(output<span style="color: #666666">.</span>shape[feature_maps_index]):
<span style="color: #008000; font-weight: bold">for</span> h <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(strided_height):
<span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(strided_width):
output[img, fmap, h, w] <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pooling_action(
X_batch[
img,
fmap,
(h <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride) : (h <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride)
<span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
(w <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride) : (w <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride)
<span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
)
<span style="color: #408080; font-style: italic"># output for feedforward in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> output
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, delta_term_next):
<span style="color: #408080; font-style: italic"># initiate delta term array</span>
delta_term <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape))
<span style="color: #008000; font-weight: bold">for</span> img <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(delta_term_next<span style="color: #666666">.</span>shape[input_index]):
<span style="color: #008000; font-weight: bold">for</span> fmap <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(delta_term_next<span style="color: #666666">.</span>shape[feature_maps_index]):
<span style="color: #008000; font-weight: bold">for</span> h <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">0</span>, delta_term_next<span style="color: #666666">.</span>shape[height_index], <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride):
<span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(
<span style="color: #666666">0</span>, delta_term_next<span style="color: #666666">.</span>shape[width_index], <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride
):
<span style="color: #408080; font-style: italic"># max pooling</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pooling <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;max&quot;</span>:
<span style="color: #408080; font-style: italic"># get window</span>
window <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward[
img,
fmap,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
<span style="color: #408080; font-style: italic"># find max values indices in window</span>
max_h, max_w <span style="color: #666666">=</span> np<span style="color: #666666">.</span>unravel_index(
window<span style="color: #666666">.</span>argmax(), window<span style="color: #666666">.</span>shape
)
<span style="color: #408080; font-style: italic"># set values in new, upsampled delta term</span>
delta_term[
img,
fmap,
(h <span style="color: #666666">+</span> max_h),
(w <span style="color: #666666">+</span> max_w),
] <span style="color: #666666">+=</span> delta_term_next[img, fmap, h, w]
<span style="color: #408080; font-style: italic"># average pooling</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pooling <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;average&quot;</span>:
delta_term[
img,
fmap,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
] <span style="color: #666666">=</span> (
delta_term_next[img, fmap, h, w]
<span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height
<span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width
)
<span style="color: #408080; font-style: italic"># returns input to backpropagation in previous layer</span>
<span style="color: #008000; font-weight: bold">return</span> delta_term
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights</span>(<span style="color: #008000">self</span>, previous_nodes):
<span style="color: #408080; font-style: italic"># calculate strided height, strided width</span>
strided_height <span style="color: #666666">=</span> (
previous_nodes<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height
) <span style="color: #666666">//</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">+</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">if</span> previous_nodes<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">==</span> previous_nodes<span style="color: #666666">.</span>shape[width_index]:
strided_width <span style="color: #666666">=</span> strided_height
<span style="color: #008000; font-weight: bold">else</span>:
strided_width <span style="color: #666666">=</span> (
previous_nodes<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">-</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width
) <span style="color: #666666">//</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">+</span> <span style="color: #666666">1</span>
<span style="color: #408080; font-style: italic"># initiate output array</span>
output <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones(
(
previous_nodes<span style="color: #666666">.</span>shape[input_index],
previous_nodes<span style="color: #666666">.</span>shape[feature_maps_index],
strided_height,
strided_width,
)
)
<span style="color: #408080; font-style: italic"># returns output with shape used for reset weights in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> output
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_check_for_errors</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># check if input is smaller than kernel size -&gt; error</span>
<span style="color: #008000; font-weight: bold">assert</span> (
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">&gt;=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width
), <span style="color: #BA2121">f&quot;ERROR: Pooling kernel width_index (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">) larger than data width_index (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>input<span style="color: #666666">.</span>shape[<span style="color: #666666">2</span>]<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">), please lower the kernel width_index of the Pooling2DLayer&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> (
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">&gt;=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height
), <span style="color: #BA2121">f&quot;ERROR: Pooling kernel height_index (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">) larger than data height_index (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>input<span style="color: #666666">.</span>shape[<span style="color: #666666">3</span>]<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">), please lower the kernel height_index of the Pooling2DLayer&quot;</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="flattening-layer">Flattening Layer </h3>
<p>Before we can begin building our first CNN model, we need to introduce
the flattening layer. As its name suggests, the flattening layer
transforms the data into a one-dimensional vector that can be fed into
the feedforward layers of our network. This layer plays a crucial role
in preparing the data for further processing in the
network. Additionally, the flattening layer is responsible for
reshaping the gradient to the proper shape during
backpropagation. This ensures that the kernels are correctly updated,
allowing for effective learning in the network.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">FlattenLayer</span>(Layer):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(<span style="color: #008000">self</span>, act_func<span style="color: #666666">=</span>LRELU, seed<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(seed)
<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func <span style="color: #666666">=</span> act_func
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch):
<span style="color: #408080; font-style: italic"># save input for backpropagation</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward_shape <span style="color: #666666">=</span> X_batch<span style="color: #666666">.</span>shape
<span style="color: #408080; font-style: italic"># Remember, the data has the following shape: (I, FM, H, W, ) in the convolutional layers</span>
<span style="color: #408080; font-style: italic"># whilst the data has the shape (I, FM * H * W) in the fully connected layers</span>
<span style="color: #408080; font-style: italic"># I = Inputs, FM = Feature Maps, H = Height and W = Width.</span>
X_batch <span style="color: #666666">=</span> X_batch<span style="color: #666666">.</span>reshape(
X_batch<span style="color: #666666">.</span>shape[input_index],
X_batch<span style="color: #666666">.</span>shape[feature_maps_index]
<span style="color: #666666">*</span> X_batch<span style="color: #666666">.</span>shape[height_index]
<span style="color: #666666">*</span> X_batch<span style="color: #666666">.</span>shape[width_index],
)
<span style="color: #408080; font-style: italic"># add bias to a</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix <span style="color: #666666">=</span> X_batch
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones((X_batch<span style="color: #666666">.</span>shape[input_index], <span style="color: #666666">1</span>)) <span style="color: #666666">*</span> <span style="color: #666666">0.01</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">=</span> np<span style="color: #666666">.</span>hstack([bias, X_batch])
<span style="color: #408080; font-style: italic"># return a, the input to feedforward in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, weights_next, delta_term_next):
activation_derivative <span style="color: #666666">=</span> derivate(<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func)
<span style="color: #408080; font-style: italic"># calculate delta term</span>
delta_term <span style="color: #666666">=</span> (
weights_next[bias_index:, :] <span style="color: #666666">@</span> delta_term_next<span style="color: #666666">.</span>T
)<span style="color: #666666">.</span>T <span style="color: #666666">*</span> activation_derivative(<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix)
<span style="color: #408080; font-style: italic"># FlattenLayer does not update weights</span>
<span style="color: #408080; font-style: italic"># reshapes delta layer to convolutional layer data format [Input, Feature_Maps, Height, Width]</span>
<span style="color: #008000; font-weight: bold">return</span> delta_term<span style="color: #666666">.</span>reshape(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward_shape)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights</span>(<span style="color: #008000">self</span>, previous_nodes):
<span style="color: #408080; font-style: italic"># note that the previous nodes to the FlattenLayer are from the convolutional layers</span>
previous_nodes <span style="color: #666666">=</span> previous_nodes<span style="color: #666666">.</span>reshape(
previous_nodes<span style="color: #666666">.</span>shape[input_index],
previous_nodes<span style="color: #666666">.</span>shape[feature_maps_index]
<span style="color: #666666">*</span> previous_nodes<span style="color: #666666">.</span>shape[height_index]
<span style="color: #666666">*</span> previous_nodes<span style="color: #666666">.</span>shape[width_index],
)
<span style="color: #408080; font-style: italic"># return shape used in reset_weights in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> previous_nodes<span style="color: #666666">.</span>shape[node_index]
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">get_prev_a</span>(<span style="color: #008000">self</span>):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="fully-connected-layers">Fully Connected Layers </h3>
<p>Finally, the result from the flatten layer will pass to a series of
fully connected layers, which function as a normal feed forward neural
network. The fully connected layers are split into two classes;
FullyConnectedLayer which acts as a hidden layer, and OutputLayer,
which acts as the single output layer at the end of the CNN. If one
wishes to use this codebase to construct a normal feed forward neural
network, it must start with a FlattenLayer due to techincal details
regarding weight intitialization. However many FullyConnectedLayers
can be added to the CNN, and in each layer the amount of nodes, which
activation function and scheduler to use can be specified. In
practice, the scheduler will be specified in the CNN object
initialization, and inherited if no other scheduler is specified.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">FullyConnectedLayer</span>(Layer):
<span style="color: #408080; font-style: italic"># FullyConnectedLayer per default uses LRELU and Adam scheduler</span>
<span style="color: #408080; font-style: italic"># with an eta of 0.0001, rho of 0.9 and rho2 of 0.999</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
nodes: <span style="color: #008000">int</span>,
act_func: Callable <span style="color: #666666">=</span> LRELU,
scheduler: Scheduler <span style="color: #666666">=</span> Adam(eta<span style="color: #666666">=1e-4</span>, rho<span style="color: #666666">=0.9</span>, rho2<span style="color: #666666">=0.999</span>),
seed: <span style="color: #008000">int</span> <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>,
):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(seed)
<span style="color: #008000">self</span><span style="color: #666666">.</span>nodes <span style="color: #666666">=</span> nodes
<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func <span style="color: #666666">=</span> act_func
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_weight <span style="color: #666666">=</span> copy(scheduler)
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_bias <span style="color: #666666">=</span> copy(scheduler)
<span style="color: #408080; font-style: italic"># initiate matrices for later</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch):
<span style="color: #408080; font-style: italic"># calculate z</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix <span style="color: #666666">=</span> X_batch <span style="color: #666666">@</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights
<span style="color: #408080; font-style: italic"># calculate a, add bias</span>
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones((X_batch<span style="color: #666666">.</span>shape[input_index], <span style="color: #666666">1</span>)) <span style="color: #666666">*</span> <span style="color: #666666">0.01</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func(<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix)
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">=</span> np<span style="color: #666666">.</span>hstack([bias, <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix])
<span style="color: #408080; font-style: italic"># return a, the input for feedforward in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, weights_next, delta_term_next, a_previous, lam):
<span style="color: #408080; font-style: italic"># take the derivative of the activation function</span>
activation_derivative <span style="color: #666666">=</span> derivate(<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func)
<span style="color: #408080; font-style: italic"># calculate the delta term</span>
delta_term <span style="color: #666666">=</span> (
weights_next[bias_index:, :] <span style="color: #666666">@</span> delta_term_next<span style="color: #666666">.</span>T
)<span style="color: #666666">.</span>T <span style="color: #666666">*</span> activation_derivative(<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix)
<span style="color: #408080; font-style: italic"># intitiate matrix to store gradient</span>
<span style="color: #408080; font-style: italic"># note that we exclude the bias term, which we will calculate later</span>
gradient_weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(
(
a_previous<span style="color: #666666">.</span>shape[input_index],
a_previous<span style="color: #666666">.</span>shape[node_index] <span style="color: #666666">-</span> bias_index,
delta_term<span style="color: #666666">.</span>shape[node_index],
)
)
<span style="color: #408080; font-style: italic"># calculate gradient = delta term * previous a</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(delta_term)):
gradient_weights[i, :, :] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>outer(
a_previous[i, bias_index:], delta_term[i, :]
)
<span style="color: #408080; font-style: italic"># sum the gradient, divide by input_index</span>
gradient_weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(gradient_weights, axis<span style="color: #666666">=</span>input_index)
<span style="color: #408080; font-style: italic"># for the bias gradient we do not multiply by previous a</span>
gradient_bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(delta_term, axis<span style="color: #666666">=</span>input_index)<span style="color: #666666">.</span>reshape(
<span style="color: #666666">1</span>, delta_term<span style="color: #666666">.</span>shape[node_index]
)
<span style="color: #408080; font-style: italic"># regularization term</span>
gradient_weights <span style="color: #666666">+=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights[bias_index:, :] <span style="color: #666666">*</span> lam
<span style="color: #408080; font-style: italic"># send gradients into scheduler</span>
<span style="color: #408080; font-style: italic"># returns update matrix which will be used to update the weights and bias</span>
update_matrix <span style="color: #666666">=</span> np<span style="color: #666666">.</span>vstack(
[
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_bias<span style="color: #666666">.</span>update_change(gradient_bias),
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_weight<span style="color: #666666">.</span>update_change(gradient_weights),
]
)
<span style="color: #408080; font-style: italic"># update weights</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights <span style="color: #666666">-=</span> update_matrix
<span style="color: #408080; font-style: italic"># return weights and delta term, input for backpropagation in previous layer</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights, delta_term
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights</span>(<span style="color: #008000">self</span>, previous_nodes):
<span style="color: #408080; font-style: italic"># sets seed to remove randomness inbetween runs</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>seed <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>:
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #008000">self</span><span style="color: #666666">.</span>seed)
<span style="color: #408080; font-style: italic"># add bias, initiate random weights</span>
bias <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(previous_nodes <span style="color: #666666">+</span> bias, <span style="color: #008000">self</span><span style="color: #666666">.</span>nodes)
<span style="color: #408080; font-style: italic"># returns number of nodes, used for reset_weights in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>nodes
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_scheduler</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># resets scheduler per epoch</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_weight<span style="color: #666666">.</span>reset()
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_bias<span style="color: #666666">.</span>reset()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">get_prev_a</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># returns a matrix, used in backpropagation</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">OutputLayer</span>(FullyConnectedLayer):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
nodes: <span style="color: #008000">int</span>,
output_func: Callable <span style="color: #666666">=</span> LRELU,
cost_func: Callable <span style="color: #666666">=</span> CostCrossEntropy,
scheduler: Scheduler <span style="color: #666666">=</span> Adam(eta<span style="color: #666666">=1e-4</span>, rho<span style="color: #666666">=0.9</span>, rho2<span style="color: #666666">=0.999</span>),
seed: <span style="color: #008000">int</span> <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>,
):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(nodes, output_func, copy(scheduler), seed)
<span style="color: #008000">self</span><span style="color: #666666">.</span>cost_func <span style="color: #666666">=</span> cost_func
<span style="color: #408080; font-style: italic"># initiate matrices for later</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #408080; font-style: italic"># decides if the output layer performs binary or multi-class classification</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>_set_pred_format()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch: np<span style="color: #666666">.</span>ndarray):
<span style="color: #408080; font-style: italic"># calculate a, z</span>
<span style="color: #408080; font-style: italic"># note that bias is not added as this would create an extra output class</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix <span style="color: #666666">=</span> X_batch <span style="color: #666666">@</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func(<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix)
<span style="color: #408080; font-style: italic"># returns prediction</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, target, a_previous, lam):
<span style="color: #408080; font-style: italic"># note that in the OutputLayer the activation function is the output function</span>
activation_derivative <span style="color: #666666">=</span> derivate(<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func)
<span style="color: #408080; font-style: italic"># calculate output delta terms</span>
<span style="color: #408080; font-style: italic"># for multi-class or binary classification</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;Multi-class&quot;</span>:
delta_term <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix <span style="color: #666666">-</span> target
<span style="color: #008000; font-weight: bold">else</span>:
cost_func_derivative <span style="color: #666666">=</span> grad(<span style="color: #008000">self</span><span style="color: #666666">.</span>cost_func(target))
delta_term <span style="color: #666666">=</span> activation_derivative(<span style="color: #008000">self</span><span style="color: #666666">.</span>z_matrix) <span style="color: #666666">*</span> cost_func_derivative(
<span style="color: #008000">self</span><span style="color: #666666">.</span>a_matrix
)
<span style="color: #408080; font-style: italic"># intiate matrix that stores gradient</span>
gradient_weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(
(
a_previous<span style="color: #666666">.</span>shape[input_index],
a_previous<span style="color: #666666">.</span>shape[node_index] <span style="color: #666666">-</span> bias_index,
delta_term<span style="color: #666666">.</span>shape[node_index],
)
)
<span style="color: #408080; font-style: italic"># calculate gradient = delta term * previous a</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(delta_term)):
gradient_weights[i, :, :] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>outer(
a_previous[i, bias_index:], delta_term[i, :]
)
<span style="color: #408080; font-style: italic"># sum the gradient, divide by input_index</span>
gradient_weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(gradient_weights, axis<span style="color: #666666">=</span>input_index)
<span style="color: #408080; font-style: italic"># for the bias gradient we do not multiply by previous a</span>
gradient_bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(delta_term, axis<span style="color: #666666">=</span>input_index)<span style="color: #666666">.</span>reshape(
<span style="color: #666666">1</span>, delta_term<span style="color: #666666">.</span>shape[node_index]
)
<span style="color: #408080; font-style: italic"># regularization term</span>
gradient_weights <span style="color: #666666">+=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights[bias_index:, :] <span style="color: #666666">*</span> lam
<span style="color: #408080; font-style: italic"># send gradients into scheduler</span>
<span style="color: #408080; font-style: italic"># returns update matrix which will be used to update the weights and bias</span>
update_matrix <span style="color: #666666">=</span> np<span style="color: #666666">.</span>vstack(
[
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_bias<span style="color: #666666">.</span>update_change(gradient_bias),
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_weight<span style="color: #666666">.</span>update_change(gradient_weights),
]
)
<span style="color: #408080; font-style: italic"># update weights</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights <span style="color: #666666">-=</span> update_matrix
<span style="color: #408080; font-style: italic"># return weights and delta term, input for backpropagation in previous layer</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>weights, delta_term
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_weights</span>(<span style="color: #008000">self</span>, previous_nodes):
<span style="color: #408080; font-style: italic"># sets seed to remove randomness inbetween runs</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>seed <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>:
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #008000">self</span><span style="color: #666666">.</span>seed)
<span style="color: #408080; font-style: italic"># add bias, initiate random weights</span>
bias <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>weights <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>rand(previous_nodes <span style="color: #666666">+</span> bias, <span style="color: #008000">self</span><span style="color: #666666">.</span>nodes)
<span style="color: #408080; font-style: italic"># returns number of nodes, used for reset_weights in next layer</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>nodes
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_reset_scheduler</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># resets scheduler per epoch</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_weight<span style="color: #666666">.</span>reset()
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler_bias<span style="color: #666666">.</span>reset()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_set_pred_format</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># sets prediction format to either regression, binary or multi-class classification</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span> <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #008000; font-weight: bold">None</span> <span style="color: #AA22FF; font-weight: bold">or</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;identity&quot;</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;Regression&quot;</span>
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;sigmoid&quot;</span> <span style="color: #AA22FF; font-weight: bold">or</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func<span style="color: #666666">.</span><span style="color: #19177C">__name__</span> <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;tanh&quot;</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;Binary&quot;</span>
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;Multi-class&quot;</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">get_pred_format</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># returns format of prediction</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="optimized-convolution2dlayer">Optimized Convolution2DLayer </h3>
<p>For our CNN, we have also implemented an optimized version of the
Convolution2DLayer, Convolution2DLayerOPT, which runs much faster. See
VII. Remarks for discussion. This layer will per default be used by
the CNN due to its computational advantages, but is much less
readable. We've documented it such that specially interested students
can understand the principles behind it, but it is not recommended to
read. In short, we reshape and transpose parts of the image such that
the convolutional operation can be swapped out for a simple matrix
multiplication.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">Convolution2DLayerOPT</span>(Convolution2DLayer):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Am optimized version of the convolution layer above which</span>
<span style="color: #BA2121; font-style: italic"> utilizes an approach of extracting windows of size equivalent</span>
<span style="color: #BA2121; font-style: italic"> in size to the filter. The convoution is then performed on those</span>
<span style="color: #BA2121; font-style: italic"> windows instead of a full feature map.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
input_channels,
feature_maps,
kernel_height,
kernel_width,
v_stride,
h_stride,
pad,
act_func: Callable,
seed<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>,
reset_weights_independently<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,
):
<span style="color: #008000">super</span>()<span style="color: #666666">.</span><span style="color: #0000FF">__init__</span>(
input_channels,
feature_maps,
kernel_height,
kernel_width,
v_stride,
h_stride,
pad,
act_func,
seed,
)
<span style="color: #408080; font-style: italic"># true if layer is used outside of CNN</span>
<span style="color: #008000; font-weight: bold">if</span> reset_weights_independently <span style="color: #666666">==</span> <span style="color: #008000; font-weight: bold">True</span>:
<span style="color: #008000">self</span><span style="color: #666666">.</span>_reset_weights_independently()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch):
<span style="color: #408080; font-style: italic"># The optimized _feedforward method is difficult to understand but computationally more efficient</span>
<span style="color: #408080; font-style: italic"># for a more &quot;by the book&quot; approach, please look at the _feedforward method of Convolution2DLayer</span>
<span style="color: #408080; font-style: italic"># save the input for backpropagation</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward <span style="color: #666666">=</span> X_batch
<span style="color: #408080; font-style: italic"># check that there are the correct amount of input channels</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>_check_for_errors()
<span style="color: #408080; font-style: italic"># calculate new shape after stride</span>
strided_height <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>ceil(X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride))
strided_width <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>ceil(X_batch<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride))
<span style="color: #408080; font-style: italic"># get windows of the image for more computationally efficient convolution</span>
<span style="color: #408080; font-style: italic"># the idea is that we want to align the dimensions that we wish to matrix</span>
<span style="color: #408080; font-style: italic"># multiply, then use a simple matrix multiplication instead of convolution.</span>
<span style="color: #408080; font-style: italic"># then, we reshape the size back to its intended shape</span>
windows <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_extract_windows(X_batch)
windows <span style="color: #666666">=</span> windows<span style="color: #666666">.</span>transpose(<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">4</span>)<span style="color: #666666">.</span>reshape(
X_batch<span style="color: #666666">.</span>shape[input_index],
strided_height <span style="color: #666666">*</span> strided_width,
<span style="color: #666666">-1</span>,
)
<span style="color: #408080; font-style: italic"># reshape the kernel for more computationally efficient convolution</span>
kernel <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel
kernel <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>transpose(<span style="color: #666666">0</span>, <span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">1</span>)<span style="color: #666666">.</span>reshape(
kernel<span style="color: #666666">.</span>shape[kernel_input_channels_index]
<span style="color: #666666">*</span> kernel<span style="color: #666666">.</span>shape[height_index]
<span style="color: #666666">*</span> kernel<span style="color: #666666">.</span>shape[width_index],
<span style="color: #666666">-1</span>,
)
<span style="color: #408080; font-style: italic"># use simple matrix calculation to obtain output</span>
output <span style="color: #666666">=</span> (
(windows <span style="color: #666666">@</span> kernel)
<span style="color: #666666">.</span>reshape(
X_batch<span style="color: #666666">.</span>shape[input_index],
strided_height,
strided_width,
<span style="color: #666666">-1</span>,
)
<span style="color: #666666">.</span>transpose(<span style="color: #666666">0</span>, <span style="color: #666666">3</span>, <span style="color: #666666">1</span>, <span style="color: #666666">2</span>)
)
<span style="color: #408080; font-style: italic"># The output is reshaped and rearranged to appropriate shape</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>act_func(
output <span style="color: #666666">/</span> (<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height <span style="color: #666666">*</span> X_batch<span style="color: #666666">.</span>shape[feature_maps_index])
)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, delta_term_next):
<span style="color: #408080; font-style: italic"># The optimized _backpropagate method is difficult to understand but computationally more efficient</span>
<span style="color: #408080; font-style: italic"># for a more &quot;by the book&quot; approach, please look at the _backpropagate method of Convolution2DLayer</span>
act_derivative <span style="color: #666666">=</span> derivate(<span style="color: #008000">self</span><span style="color: #666666">.</span>act_func)
delta_term_next <span style="color: #666666">=</span> act_derivative(delta_term_next)
<span style="color: #408080; font-style: italic"># calculate strided dimensions</span>
strided_height <span style="color: #666666">=</span> <span style="color: #008000">int</span>(
np<span style="color: #666666">.</span>ceil(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride)
)
strided_width <span style="color: #666666">=</span> <span style="color: #008000">int</span>(
np<span style="color: #666666">.</span>ceil(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">/</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride)
)
<span style="color: #408080; font-style: italic"># copy kernel</span>
kernel <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel
<span style="color: #408080; font-style: italic"># get windows, reshape for matrix multiplication</span>
windows <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_extract_windows(<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward, <span style="color: #BA2121">&quot;image&quot;</span>)<span style="color: #666666">.</span>reshape(
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_index]
<span style="color: #666666">*</span> strided_height
<span style="color: #666666">*</span> strided_width,
<span style="color: #666666">-1</span>,
)
<span style="color: #408080; font-style: italic"># initialize output gradient, reshape and transpose into correct shape</span>
<span style="color: #408080; font-style: italic"># for matrix multiplication</span>
output_grad_tr <span style="color: #666666">=</span> delta_term_next<span style="color: #666666">.</span>transpose(<span style="color: #666666">0</span>, <span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">1</span>)<span style="color: #666666">.</span>reshape(
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_index]
<span style="color: #666666">*</span> strided_height
<span style="color: #666666">*</span> strided_width,
<span style="color: #666666">-1</span>,
)
<span style="color: #408080; font-style: italic"># calculate gradient kernel via simple matrix multiplication and reshaping</span>
gradient_kernel <span style="color: #666666">=</span> (
(windows<span style="color: #666666">.</span>T <span style="color: #666666">@</span> output_grad_tr)
<span style="color: #666666">.</span>reshape(
kernel<span style="color: #666666">.</span>shape[kernel_input_channels_index],
kernel<span style="color: #666666">.</span>shape[height_index],
kernel<span style="color: #666666">.</span>shape[width_index],
kernel<span style="color: #666666">.</span>shape[kernel_feature_maps_index],
)
<span style="color: #666666">.</span>transpose(<span style="color: #666666">0</span>, <span style="color: #666666">3</span>, <span style="color: #666666">1</span>, <span style="color: #666666">2</span>)
)
<span style="color: #408080; font-style: italic"># for computing the input gradient</span>
windows_out, upsampled_height, upsampled_width <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_extract_windows(
delta_term_next, <span style="color: #BA2121">&quot;grad&quot;</span>
)
<span style="color: #408080; font-style: italic"># calculate new window dimensions</span>
new_windows_first_dim <span style="color: #666666">=</span> (
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_index]
<span style="color: #666666">*</span> upsampled_height
<span style="color: #666666">*</span> upsampled_width
)
<span style="color: #408080; font-style: italic"># ceil allows for various asymmetric kernels</span>
new_windows_sec_dim <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>ceil(windows_out<span style="color: #666666">.</span>size <span style="color: #666666">/</span> new_windows_first_dim))
<span style="color: #408080; font-style: italic"># reshape for matrix multiplication</span>
windows_out <span style="color: #666666">=</span> windows_out<span style="color: #666666">.</span>transpose(<span style="color: #666666">1</span>, <span style="color: #666666">0</span>, <span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">4</span>)<span style="color: #666666">.</span>reshape(
new_windows_first_dim, new_windows_sec_dim
)
<span style="color: #408080; font-style: italic"># reshape for matrix multiplication</span>
kernel_reshaped <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>reshape(<span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels, <span style="color: #666666">-1</span>)
<span style="color: #408080; font-style: italic"># calculating input gradient for next convolutional layer</span>
input_grad <span style="color: #666666">=</span> (windows_out <span style="color: #666666">@</span> kernel_reshaped<span style="color: #666666">.</span>T)<span style="color: #666666">.</span>reshape(
<span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_index],
upsampled_height,
upsampled_width,
kernel<span style="color: #666666">.</span>shape[kernel_input_channels_index],
)
input_grad <span style="color: #666666">=</span> input_grad<span style="color: #666666">.</span>transpose(<span style="color: #666666">0</span>, <span style="color: #666666">3</span>, <span style="color: #666666">1</span>, <span style="color: #666666">2</span>)
<span style="color: #408080; font-style: italic"># Update the weights in the kernel</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>kernel <span style="color: #666666">-=</span> gradient_kernel
<span style="color: #408080; font-style: italic"># Output the gradient to propagate backwards</span>
<span style="color: #008000; font-weight: bold">return</span> input_grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_extract_windows</span>(<span style="color: #008000">self</span>, X_batch, batch_type<span style="color: #666666">=</span><span style="color: #BA2121">&quot;image&quot;</span>):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Receives as input the X_batch with shape (inputs, feature_maps, image_height, image_width)</span>
<span style="color: #BA2121; font-style: italic"> and extract windows of size kernel_height * kernel_width for every image and every feature_map.</span>
<span style="color: #BA2121; font-style: italic"> It then returns an np.ndarray of shape (image_height * image_width, inputs, feature_maps, kernel_height, kernel_width)</span>
<span style="color: #BA2121; font-style: italic"> which will be used either to filter the images in feedforward or to calculate the gradient.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic"># initialize list of windows</span>
windows <span style="color: #666666">=</span> []
<span style="color: #008000; font-weight: bold">if</span> batch_type <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;image&quot;</span>:
<span style="color: #408080; font-style: italic"># pad the images</span>
X_batch_padded <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_padding(X_batch, batch_type<span style="color: #666666">=</span><span style="color: #BA2121">&quot;image&quot;</span>)
img_height, img_width <span style="color: #666666">=</span> X_batch_padded<span style="color: #666666">.</span>shape[<span style="color: #666666">2</span>:]
<span style="color: #408080; font-style: italic"># For each location in the image...</span>
<span style="color: #008000; font-weight: bold">for</span> h <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(
<span style="color: #666666">0</span>,
X_batch<span style="color: #666666">.</span>shape[height_index],
<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride,
):
<span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(
<span style="color: #666666">0</span>,
X_batch<span style="color: #666666">.</span>shape[width_index],
<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride,
):
<span style="color: #408080; font-style: italic"># ...obtain an image patch of the original size (strided)</span>
<span style="color: #408080; font-style: italic"># get window</span>
window <span style="color: #666666">=</span> X_batch_padded[
:,
:,
h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height,
w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width,
]
<span style="color: #408080; font-style: italic"># append to list of windows</span>
windows<span style="color: #666666">.</span>append(window)
<span style="color: #408080; font-style: italic"># return numpy array instead of list</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>stack(windows)
<span style="color: #408080; font-style: italic"># In order to be able to perform backprogagation by the method of window extraction,</span>
<span style="color: #408080; font-style: italic"># here is a modified approach to extracting the windows which allow for the necessary</span>
<span style="color: #408080; font-style: italic"># upsampling of the gradient in case the on of the stride parameters is larger than one.</span>
<span style="color: #008000; font-weight: bold">if</span> batch_type <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;grad&quot;</span>:
<span style="color: #408080; font-style: italic"># In the case of one of the stride parameters being odd, we have to take some</span>
<span style="color: #408080; font-style: italic"># extra care in calculating the upsampled size of X_batch. We solve this</span>
<span style="color: #408080; font-style: italic"># by simply flooring the result of dividing stride by 2.</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">&lt;</span> <span style="color: #666666">2</span> <span style="color: #AA22FF; font-weight: bold">or</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">%</span> <span style="color: #666666">2</span> <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
v_stride <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">else</span>:
v_stride <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>floor(<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">/</span> <span style="color: #666666">2</span>))
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">&lt;</span> <span style="color: #666666">2</span> <span style="color: #AA22FF; font-weight: bold">or</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">%</span> <span style="color: #666666">2</span> <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
h_stride <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">else</span>:
h_stride <span style="color: #666666">=</span> <span style="color: #008000">int</span>(np<span style="color: #666666">.</span>floor(<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">/</span> <span style="color: #666666">2</span>))
upsampled_height <span style="color: #666666">=</span> (X_batch<span style="color: #666666">.</span>shape[height_index] <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride) <span style="color: #666666">-</span> v_stride
upsampled_width <span style="color: #666666">=</span> (X_batch<span style="color: #666666">.</span>shape[width_index] <span style="color: #666666">*</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride) <span style="color: #666666">-</span> h_stride
<span style="color: #408080; font-style: italic"># When upsampling, we need to insert rows and columns filled with zeros</span>
<span style="color: #408080; font-style: italic"># into each feature map. How many of those we have to insert is purely</span>
<span style="color: #408080; font-style: italic"># dependant on the value of stride parameter in the vertical and horizontal</span>
<span style="color: #408080; font-style: italic"># direction.</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">&gt;</span> <span style="color: #666666">1</span>:
v_ind <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(X_batch<span style="color: #666666">.</span>shape[height_index]):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride <span style="color: #666666">-</span> <span style="color: #666666">1</span>):
X_batch <span style="color: #666666">=</span> np<span style="color: #666666">.</span>insert(X_batch, v_ind, <span style="color: #666666">0</span>, axis<span style="color: #666666">=</span>height_index)
v_ind <span style="color: #666666">+=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">&gt;</span> <span style="color: #666666">1</span>:
h_ind <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(X_batch<span style="color: #666666">.</span>shape[width_index]):
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride <span style="color: #666666">-</span> <span style="color: #666666">1</span>):
X_batch <span style="color: #666666">=</span> np<span style="color: #666666">.</span>insert(X_batch, h_ind, <span style="color: #666666">0</span>, axis<span style="color: #666666">=</span>width_index)
h_ind <span style="color: #666666">+=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride
<span style="color: #408080; font-style: italic"># Since the insertion of zero-filled rows and columns isn&#39;t perfect, we have</span>
<span style="color: #408080; font-style: italic"># to assure that the resulting feature maps will have the expected upsampled height</span>
<span style="color: #408080; font-style: italic"># and width by cutting them og at desired dimensions.</span>
X_batch <span style="color: #666666">=</span> X_batch[:, :, :upsampled_height, :upsampled_width]
X_batch_padded <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_padding(X_batch, batch_type<span style="color: #666666">=</span><span style="color: #BA2121">&quot;grad&quot;</span>)
<span style="color: #408080; font-style: italic"># initialize list of windows</span>
windows <span style="color: #666666">=</span> []
<span style="color: #408080; font-style: italic"># For each location in the image...</span>
<span style="color: #008000; font-weight: bold">for</span> h <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(
<span style="color: #666666">0</span>,
X_batch<span style="color: #666666">.</span>shape[height_index],
<span style="color: #008000">self</span><span style="color: #666666">.</span>v_stride,
):
<span style="color: #008000; font-weight: bold">for</span> w <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(
<span style="color: #666666">0</span>,
X_batch<span style="color: #666666">.</span>shape[width_index],
<span style="color: #008000">self</span><span style="color: #666666">.</span>h_stride,
):
<span style="color: #408080; font-style: italic"># ...obtain an image patch of the original size (strided)</span>
<span style="color: #408080; font-style: italic"># get window</span>
window <span style="color: #666666">=</span> X_batch_padded[
:, :, h : h <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_height, w : w <span style="color: #666666">+</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>kernel_width
]
<span style="color: #408080; font-style: italic"># append window to list</span>
windows<span style="color: #666666">.</span>append(window)
<span style="color: #408080; font-style: italic"># return numpy array, unsampled dimensions</span>
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>stack(windows), upsampled_height, upsampled_width
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_check_for_errors</span>(<span style="color: #008000">self</span>):
<span style="color: #408080; font-style: italic"># compares input channels of data to input channels of Convolution2DLayer</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_channel_index] <span style="color: #666666">!=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels:
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">AssertionError</span>(
<span style="color: #BA2121">f&quot;ERROR: Number of input channels in data (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>X_batch_feedforward<span style="color: #666666">.</span>shape[input_channel_index]<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">) is not equal to input channels in Convolution2DLayerOPT (</span><span style="color: #BB6688; font-weight: bold">{</span><span style="color: #008000">self</span><span style="color: #666666">.</span>input_channels<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">)! Please change the number of input channels of the Convolution2DLayer such that they are equal&quot;</span>
)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="the-convolutional-neural-network-cnn">The Convolutional Neural Network (CNN) </h3>
<p>Finally, we present the code for the CNN. The CNN class organizes all the layers, and allows for training on image data.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">math</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">sys</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">warnings</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad, elementwise_grad
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">random</span> <span style="color: #008000; font-weight: bold">import</span> random, seed
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">copy</span> <span style="color: #008000; font-weight: bold">import</span> deepcopy
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">typing</span> <span style="color: #008000; font-weight: bold">import</span> Tuple, Callable
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.utils</span> <span style="color: #008000; font-weight: bold">import</span> resample
warnings<span style="color: #666666">.</span>simplefilter(<span style="color: #BA2121">&quot;error&quot;</span>)
<span style="color: #008000; font-weight: bold">class</span> <span style="color: #0000FF; font-weight: bold">CNN</span>:
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">__init__</span>(
<span style="color: #008000">self</span>,
cost_func: Callable <span style="color: #666666">=</span> CostCrossEntropy,
scheduler: Scheduler <span style="color: #666666">=</span> Adam(eta<span style="color: #666666">=1e-4</span>, rho<span style="color: #666666">=0.9</span>, rho2<span style="color: #666666">=0.999</span>),
seed: <span style="color: #008000">int</span> <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>,
):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Instantiates CNN object</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I output_func (costFunctions) cost function for feed forward neural network part of CNN,</span>
<span style="color: #BA2121; font-style: italic"> such as &quot;CostLogReg&quot;, &quot;CostOLS&quot; or &quot;CostCrossEntropy&quot;</span>
<span style="color: #BA2121; font-style: italic"> II scheduler (Scheduler) optional parameter, default set to Adam. Can also be set to other</span>
<span style="color: #BA2121; font-style: italic"> schedulers such as AdaGrad, Momentum, RMS_prop and Constant. Note that schedulers have</span>
<span style="color: #BA2121; font-style: italic"> to be instantiated first with proper parameters (for example eta, rho and rho2 for Adam)</span>
<span style="color: #BA2121; font-style: italic"> III seed (int) used for seeding all random operations</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>layers <span style="color: #666666">=</span> <span style="color: #008000">list</span>()
<span style="color: #008000">self</span><span style="color: #666666">.</span>cost_func <span style="color: #666666">=</span> cost_func
<span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler <span style="color: #666666">=</span> scheduler
<span style="color: #008000">self</span><span style="color: #666666">.</span>schedulers_weight <span style="color: #666666">=</span> <span style="color: #008000">list</span>()
<span style="color: #008000">self</span><span style="color: #666666">.</span>schedulers_bias <span style="color: #666666">=</span> <span style="color: #008000">list</span>()
<span style="color: #008000">self</span><span style="color: #666666">.</span>seed <span style="color: #666666">=</span> seed
<span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">add_FullyConnectedLayer</span>(
<span style="color: #008000">self</span>, nodes: <span style="color: #008000">int</span>, act_func<span style="color: #666666">=</span>LRELU, scheduler<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>
) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Add a FullyConnectedLayer to the CNN, i.e. a hidden layer in the feed forward neural</span>
<span style="color: #BA2121; font-style: italic"> network part of the CNN. Often called a Dense layer in literature</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I nodes (int) number of nodes in FullyConnectedLayer</span>
<span style="color: #BA2121; font-style: italic"> II act_func (activationFunctions) activation function of FullyConnectedLayer,</span>
<span style="color: #BA2121; font-style: italic"> such as &quot;sigmoid&quot;, &quot;RELU&quot;, &quot;LRELU&quot;, &quot;softmax&quot; or &quot;identity&quot;</span>
<span style="color: #BA2121; font-style: italic"> III scheduler (Scheduler) optional parameter, default set to Adam. Can also be set to other</span>
<span style="color: #BA2121; font-style: italic"> schedulers such as AdaGrad, Momentum, RMS_prop and Constant</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>layers, <span style="color: #BA2121">&quot;FullyConnectedLayer should follow FlattenLayer in CNN&quot;</span>
<span style="color: #008000; font-weight: bold">if</span> scheduler <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #008000; font-weight: bold">None</span>:
scheduler <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler
layer <span style="color: #666666">=</span> FullyConnectedLayer(nodes, act_func, scheduler, <span style="color: #008000">self</span><span style="color: #666666">.</span>seed)
<span style="color: #008000">self</span><span style="color: #666666">.</span>layers<span style="color: #666666">.</span>append(layer)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">add_OutputLayer</span>(<span style="color: #008000">self</span>, nodes: <span style="color: #008000">int</span>, output_func<span style="color: #666666">=</span>sigmoid, scheduler<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Add an OutputLayer to the CNN, i.e. a the final layer in the feed forward neural</span>
<span style="color: #BA2121; font-style: italic"> network part of the CNN</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I nodes (int) number of nodes in OutputLayer. Set nodes=1 for binary classification and</span>
<span style="color: #BA2121; font-style: italic"> nodes = number of classes for multi-class classification</span>
<span style="color: #BA2121; font-style: italic"> II output_func (activationFunctions) activation function for the output layer, such as</span>
<span style="color: #BA2121; font-style: italic"> &quot;identity&quot; for regression, &quot;sigmoid&quot; for binary classification and &quot;softmax&quot; for multi-class</span>
<span style="color: #BA2121; font-style: italic"> classification</span>
<span style="color: #BA2121; font-style: italic"> III scheduler (Scheduler) optional parameter, default set to Adam. Can also be set to other</span>
<span style="color: #BA2121; font-style: italic"> schedulers such as AdaGrad, Momentum, RMS_prop and Constant</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>layers, <span style="color: #BA2121">&quot;OutputLayer should follow FullyConnectedLayer in CNN&quot;</span>
<span style="color: #008000; font-weight: bold">if</span> scheduler <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #008000; font-weight: bold">None</span>:
scheduler <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>scheduler
output_layer <span style="color: #666666">=</span> OutputLayer(
nodes, output_func, <span style="color: #008000">self</span><span style="color: #666666">.</span>cost_func, scheduler, <span style="color: #008000">self</span><span style="color: #666666">.</span>seed
)
<span style="color: #008000">self</span><span style="color: #666666">.</span>layers<span style="color: #666666">.</span>append(output_layer)
<span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">=</span> output_layer<span style="color: #666666">.</span>get_pred_format()
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">add_FlattenLayer</span>(<span style="color: #008000">self</span>, act_func<span style="color: #666666">=</span>LRELU) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Add a FlattenLayer to the CNN, which flattens the image data such that it is formatted to</span>
<span style="color: #BA2121; font-style: italic"> be used in the feed forward neural network part of the CNN</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>layers<span style="color: #666666">.</span>append(FlattenLayer(act_func<span style="color: #666666">=</span>act_func, seed<span style="color: #666666">=</span><span style="color: #008000">self</span><span style="color: #666666">.</span>seed))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">add_Convolution2DLayer</span>(
<span style="color: #008000">self</span>,
input_channels<span style="color: #666666">=1</span>,
feature_maps<span style="color: #666666">=1</span>,
kernel_height<span style="color: #666666">=3</span>,
kernel_width<span style="color: #666666">=3</span>,
v_stride<span style="color: #666666">=1</span>,
h_stride<span style="color: #666666">=1</span>,
pad<span style="color: #666666">=</span><span style="color: #BA2121">&quot;same&quot;</span>,
act_func<span style="color: #666666">=</span>LRELU,
optimized<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,
) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Add a Convolution2DLayer to the CNN, i.e. a convolutional layer with a 2 dimensional kernel. Should be</span>
<span style="color: #BA2121; font-style: italic"> the first layer added to the CNN</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I input_channels (int) specifies amount of input channels. For monochrome images, use input_channels</span>
<span style="color: #BA2121; font-style: italic"> = 1, and input_channels = 3 for colored images, where each channel represents one of R, G and B</span>
<span style="color: #BA2121; font-style: italic"> II feature_maps (int) amount of feature maps in CNN</span>
<span style="color: #BA2121; font-style: italic"> III kernel_height (int) height of the kernel, also called &#39;convolutional filter&#39; in literature</span>
<span style="color: #BA2121; font-style: italic"> IV kernel_width (int) width of the kernel, also called &#39;convolutional filter&#39; in literature</span>
<span style="color: #BA2121; font-style: italic"> V v_stride (int) value of vertical stride for dimentionality reduction</span>
<span style="color: #BA2121; font-style: italic"> VI h_stride (int) value of horizontal stride for dimentionality reduction</span>
<span style="color: #BA2121; font-style: italic"> VII pad (str) default = &quot;same&quot; ensures output size is the same as input size (given stride=1)</span>
<span style="color: #BA2121; font-style: italic"> VIII act_func (activationFunctions) default = &quot;LRELU&quot;, nonlinear activation function</span>
<span style="color: #BA2121; font-style: italic"> IX optimized (bool) default = True, uses Convolution2DLayerOPT if True which is much faster when</span>
<span style="color: #BA2121; font-style: italic"> compared to Convolution2DLayer, which is a more straightforward, understandable implementation</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">if</span> optimized:
conv_layer <span style="color: #666666">=</span> Convolution2DLayerOPT(
input_channels,
feature_maps,
kernel_height,
kernel_width,
v_stride,
h_stride,
pad,
act_func,
<span style="color: #008000">self</span><span style="color: #666666">.</span>seed,
reset_weights_independently<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>,
)
<span style="color: #008000; font-weight: bold">else</span>:
conv_layer <span style="color: #666666">=</span> Convolution2DLayer(
input_channels,
feature_maps,
kernel_height,
kernel_width,
v_stride,
h_stride,
pad,
act_func,
<span style="color: #008000">self</span><span style="color: #666666">.</span>seed,
reset_weights_independently<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>,
)
<span style="color: #008000">self</span><span style="color: #666666">.</span>layers<span style="color: #666666">.</span>append(conv_layer)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">add_PoolingLayer</span>(
<span style="color: #008000">self</span>, kernel_height<span style="color: #666666">=2</span>, kernel_width<span style="color: #666666">=2</span>, v_stride<span style="color: #666666">=1</span>, h_stride<span style="color: #666666">=1</span>, pooling<span style="color: #666666">=</span><span style="color: #BA2121">&quot;max&quot;</span>
) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Add a Pooling2DLayer to the CNN, i.e. a pooling layer that reduces the dimentionality of</span>
<span style="color: #BA2121; font-style: italic"> the image data. It is not necessary to use a Pooling2DLayer when creating a CNN, but it</span>
<span style="color: #BA2121; font-style: italic"> can be used to speed up the training</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I kernel_height (int) height of the kernel used for pooling</span>
<span style="color: #BA2121; font-style: italic"> II kernel_width (int) width of the kernel used for pooling</span>
<span style="color: #BA2121; font-style: italic"> III v_stride (int) value of vertical stride for dimentionality reduction</span>
<span style="color: #BA2121; font-style: italic"> IV h_stride (int) value of horizontal stride for dimentionality reduction</span>
<span style="color: #BA2121; font-style: italic"> V pooling (str) either &quot;max&quot; or &quot;average&quot;, describes type of pooling performed</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
pooling_layer <span style="color: #666666">=</span> Pooling2DLayer(
kernel_height, kernel_width, v_stride, h_stride, pooling, <span style="color: #008000">self</span><span style="color: #666666">.</span>seed
)
<span style="color: #008000">self</span><span style="color: #666666">.</span>layers<span style="color: #666666">.</span>append(pooling_layer)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">fit</span>(
<span style="color: #008000">self</span>,
X: np<span style="color: #666666">.</span>ndarray,
t: np<span style="color: #666666">.</span>ndarray,
epochs: <span style="color: #008000">int</span> <span style="color: #666666">=</span> <span style="color: #666666">100</span>,
lam: <span style="color: #008000">float</span> <span style="color: #666666">=</span> <span style="color: #666666">0</span>,
batches: <span style="color: #008000">int</span> <span style="color: #666666">=</span> <span style="color: #666666">1</span>,
X_val: np<span style="color: #666666">.</span>ndarray <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>,
t_val: np<span style="color: #666666">.</span>ndarray <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>,
) <span style="color: #666666">-&gt;</span> <span style="color: #008000">dict</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Fits the CNN to input X for a given amount of epochs. Performs feedforward and backpropagation passes,</span>
<span style="color: #BA2121; font-style: italic"> can utilize batches, regulariziation and validation if desired.</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> X (numpy array) with input data in format [images, input channels,</span>
<span style="color: #BA2121; font-style: italic"> image height, image_width]</span>
<span style="color: #BA2121; font-style: italic"> t (numpy array) target labels for input data</span>
<span style="color: #BA2121; font-style: italic"> epochs (int) amount of epochs</span>
<span style="color: #BA2121; font-style: italic"> lam (float) regulariziation term lambda</span>
<span style="color: #BA2121; font-style: italic"> batches (int) amount of batches input data splits into</span>
<span style="color: #BA2121; font-style: italic"> X_val (numpy array) validation data</span>
<span style="color: #BA2121; font-style: italic"> t_val (numpy array) target labels for validation data</span>
<span style="color: #BA2121; font-style: italic"> Returns:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> scores (dict) a dictionary with &quot;train_error&quot;, &quot;train_acc&quot;, &quot;val_error&quot;, val_acc&quot; keys</span>
<span style="color: #BA2121; font-style: italic"> that contain numpy arrays with float values of all accuracies/errors over all epochs.</span>
<span style="color: #BA2121; font-style: italic"> Can be used to create plots. Also used to update the progress bar during training</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic"># setup</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>seed <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>:
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #008000">self</span><span style="color: #666666">.</span>seed)
<span style="color: #408080; font-style: italic"># initialize weights</span>
<span style="color: #008000">self</span><span style="color: #666666">.</span>_initialize_weights(X)
<span style="color: #408080; font-style: italic"># create arrays for score metrics</span>
scores <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_initialize_scores(epochs)
<span style="color: #008000; font-weight: bold">assert</span> batches <span style="color: #666666">&lt;=</span> t<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]
batch_size <span style="color: #666666">=</span> X<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> batches
<span style="color: #008000; font-weight: bold">try</span>:
<span style="color: #008000; font-weight: bold">for</span> epoch <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(epochs):
<span style="color: #008000; font-weight: bold">for</span> batch <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(batches):
<span style="color: #408080; font-style: italic"># minibatch gradient descent</span>
<span style="color: #408080; font-style: italic"># If the for loop has reached the last batch, take all thats left</span>
<span style="color: #008000; font-weight: bold">if</span> batch <span style="color: #666666">==</span> batches <span style="color: #666666">-</span> <span style="color: #666666">1</span>:
X_batch <span style="color: #666666">=</span> X[batch <span style="color: #666666">*</span> batch_size :, :, :, :]
t_batch <span style="color: #666666">=</span> t[batch <span style="color: #666666">*</span> batch_size :, :]
<span style="color: #008000; font-weight: bold">else</span>:
X_batch <span style="color: #666666">=</span> X[
batch <span style="color: #666666">*</span> batch_size : (batch <span style="color: #666666">+</span> <span style="color: #666666">1</span>) <span style="color: #666666">*</span> batch_size, :, :, :
]
t_batch <span style="color: #666666">=</span> t[batch <span style="color: #666666">*</span> batch_size : (batch <span style="color: #666666">+</span> <span style="color: #666666">1</span>) <span style="color: #666666">*</span> batch_size, :]
<span style="color: #008000">self</span><span style="color: #666666">.</span>_feedforward(X_batch)
<span style="color: #008000">self</span><span style="color: #666666">.</span>_backpropagate(t_batch, lam)
<span style="color: #408080; font-style: italic"># reset schedulers for each epoch (some schedulers pass in this call)</span>
<span style="color: #008000; font-weight: bold">for</span> layer <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>layers:
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">isinstance</span>(layer, FullyConnectedLayer):
layer<span style="color: #666666">.</span>_reset_scheduler()
<span style="color: #408080; font-style: italic"># computing performance metrics</span>
scores <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_compute_scores(scores, epoch, X, t, X_val, t_val)
<span style="color: #408080; font-style: italic"># printing progress bar</span>
print_length <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_progress_bar(
epoch,
epochs,
scores,
)
<span style="color: #408080; font-style: italic"># allows for stopping training at any point and seeing the result</span>
<span style="color: #008000; font-weight: bold">except</span> <span style="color: #D2413A; font-weight: bold">KeyboardInterrupt</span>:
<span style="color: #008000; font-weight: bold">pass</span>
<span style="color: #408080; font-style: italic"># visualization of training progression (similiar to tensorflow progression bar)</span>
sys<span style="color: #666666">.</span>stdout<span style="color: #666666">.</span>write(<span style="color: #BA2121">&quot;</span><span style="color: #BB6622; font-weight: bold">\r</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">+</span> <span style="color: #BA2121">&quot; &quot;</span> <span style="color: #666666">*</span> print_length)
sys<span style="color: #666666">.</span>stdout<span style="color: #666666">.</span>flush()
<span style="color: #008000">self</span><span style="color: #666666">.</span>_progress_bar(
epochs,
epochs,
scores,
)
sys<span style="color: #666666">.</span>stdout<span style="color: #666666">.</span>write(<span style="color: #BA2121">&quot;&quot;</span>)
<span style="color: #008000; font-weight: bold">return</span> scores
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_feedforward</span>(<span style="color: #008000">self</span>, X_batch) <span style="color: #666666">-&gt;</span> np<span style="color: #666666">.</span>ndarray:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Performs the feedforward pass for all layers in the CNN. Called from fit()</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
a <span style="color: #666666">=</span> X_batch
<span style="color: #008000; font-weight: bold">for</span> layer <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>layers:
a <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_feedforward(a)
<span style="color: #008000; font-weight: bold">return</span> a
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_backpropagate</span>(<span style="color: #008000">self</span>, t_batch, lam) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Performs backpropagation for all layers in the CNN. Called from fit()</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> <span style="color: #008000">len</span>(<span style="color: #008000">self</span><span style="color: #666666">.</span>layers) <span style="color: #666666">&gt;=</span> <span style="color: #666666">2</span>
reversed_layers <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>layers[::<span style="color: #666666">-1</span>]
<span style="color: #408080; font-style: italic"># for every layer, backwards</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">len</span>(reversed_layers) <span style="color: #666666">-</span> <span style="color: #666666">1</span>):
layer <span style="color: #666666">=</span> reversed_layers[i]
prev_layer <span style="color: #666666">=</span> reversed_layers[i <span style="color: #666666">+</span> <span style="color: #666666">1</span>]
<span style="color: #408080; font-style: italic"># OutputLayer</span>
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">isinstance</span>(layer, OutputLayer):
prev_a <span style="color: #666666">=</span> prev_layer<span style="color: #666666">.</span>get_prev_a()
weights_next, delta_next <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_backpropagate(t_batch, prev_a, lam)
<span style="color: #408080; font-style: italic"># FullyConnectedLayer</span>
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">isinstance</span>(layer, FullyConnectedLayer):
<span style="color: #008000; font-weight: bold">assert</span> (
delta_next <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>
), <span style="color: #BA2121">&quot;No OutputLayer to follow FullyConnectedLayer&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> (
weights_next <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>
), <span style="color: #BA2121">&quot;No OutputLayer to follow FullyConnectedLayer&quot;</span>
prev_a <span style="color: #666666">=</span> prev_layer<span style="color: #666666">.</span>get_prev_a()
weights_next, delta_next <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_backpropagate(
weights_next, delta_next, prev_a, lam
)
<span style="color: #408080; font-style: italic"># FlattenLayer</span>
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">isinstance</span>(layer, FlattenLayer):
<span style="color: #008000; font-weight: bold">assert</span> (
delta_next <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>
), <span style="color: #BA2121">&quot;No FullyConnectedLayer to follow FlattenLayer&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> (
weights_next <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>
), <span style="color: #BA2121">&quot;No FullyConnectedLayer to follow FlattenLayer&quot;</span>
delta_next <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_backpropagate(weights_next, delta_next)
<span style="color: #408080; font-style: italic"># Convolution2DLayer and Convolution2DLayerOPT</span>
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">isinstance</span>(layer, Convolution2DLayer):
<span style="color: #008000; font-weight: bold">assert</span> (
delta_next <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>
), <span style="color: #BA2121">&quot;No FlattenLayer to follow Convolution2DLayer&quot;</span>
delta_next <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_backpropagate(delta_next)
<span style="color: #408080; font-style: italic"># Pooling2DLayer</span>
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">isinstance</span>(layer, Pooling2DLayer):
<span style="color: #008000; font-weight: bold">assert</span> delta_next <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>, <span style="color: #BA2121">&quot;No Layer to follow Pooling2DLayer&quot;</span>
delta_next <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_backpropagate(delta_next)
<span style="color: #408080; font-style: italic"># Catch error</span>
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">raise</span> <span style="color: #D2413A; font-weight: bold">NotImplementedError</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_compute_scores</span>(
<span style="color: #008000">self</span>,
scores: <span style="color: #008000">dict</span>,
epoch: <span style="color: #008000">int</span>,
X: np<span style="color: #666666">.</span>ndarray,
t: np<span style="color: #666666">.</span>ndarray,
X_val: np<span style="color: #666666">.</span>ndarray,
t_val: np<span style="color: #666666">.</span>ndarray,
) <span style="color: #666666">-&gt;</span> <span style="color: #008000">dict</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Computes scores such as training error, training accuracy, validation error</span>
<span style="color: #BA2121; font-style: italic"> and validation accuracy for the CNN depending on if a validation set is used</span>
<span style="color: #BA2121; font-style: italic"> and if the CNN performs classification or regression</span>
<span style="color: #BA2121; font-style: italic"> Returns:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> scores (dict) a dictionary with &quot;train_error&quot;, &quot;train_acc&quot;, &quot;val_error&quot;, val_acc&quot; keys</span>
<span style="color: #BA2121; font-style: italic"> that contain numpy arrays with float values of all accuracies/errors over all epochs.</span>
<span style="color: #BA2121; font-style: italic"> Can be used to create plots. Also used to update the progress bar during training</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
pred_train <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>predict(X)
cost_function_train <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>cost_func(t)
train_error <span style="color: #666666">=</span> cost_function_train(pred_train)
scores[<span style="color: #BA2121">&quot;train_error&quot;</span>][epoch] <span style="color: #666666">=</span> train_error
<span style="color: #008000; font-weight: bold">if</span> X_val <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span> <span style="color: #AA22FF; font-weight: bold">and</span> t_val <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>:
cost_function_val <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>cost_func(t_val)
pred_val <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>predict(X_val)
val_error <span style="color: #666666">=</span> cost_function_val(pred_val)
scores[<span style="color: #BA2121">&quot;val_error&quot;</span>][epoch] <span style="color: #666666">=</span> val_error
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">!=</span> <span style="color: #BA2121">&quot;Regression&quot;</span>:
train_acc <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_accuracy(pred_train, t)
scores[<span style="color: #BA2121">&quot;train_acc&quot;</span>][epoch] <span style="color: #666666">=</span> train_acc
<span style="color: #008000; font-weight: bold">if</span> X_val <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span> <span style="color: #AA22FF; font-weight: bold">and</span> t_val <span style="color: #AA22FF; font-weight: bold">is</span> <span style="color: #AA22FF; font-weight: bold">not</span> <span style="color: #008000; font-weight: bold">None</span>:
val_acc <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_accuracy(pred_val, t_val)
scores[<span style="color: #BA2121">&quot;val_acc&quot;</span>][epoch] <span style="color: #666666">=</span> val_acc
<span style="color: #008000; font-weight: bold">return</span> scores
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_initialize_scores</span>(<span style="color: #008000">self</span>, epochs) <span style="color: #666666">-&gt;</span> <span style="color: #008000">dict</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Initializes scores such as training error, training accuracy, validation error</span>
<span style="color: #BA2121; font-style: italic"> and validation accuracy for the CNN</span>
<span style="color: #BA2121; font-style: italic"> Returns:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> A dictionary with &quot;train_error&quot;, &quot;train_acc&quot;, &quot;val_error&quot;, val_acc&quot; keys that</span>
<span style="color: #BA2121; font-style: italic"> will contain numpy arrays with float values of all accuracies/errors over all epochs</span>
<span style="color: #BA2121; font-style: italic"> when passed through the _compute_scores() function during fit()</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
scores <span style="color: #666666">=</span> <span style="color: #008000">dict</span>()
train_errors <span style="color: #666666">=</span> np<span style="color: #666666">.</span>empty(epochs)
train_errors<span style="color: #666666">.</span>fill(np<span style="color: #666666">.</span>nan)
val_errors <span style="color: #666666">=</span> np<span style="color: #666666">.</span>empty(epochs)
val_errors<span style="color: #666666">.</span>fill(np<span style="color: #666666">.</span>nan)
train_accs <span style="color: #666666">=</span> np<span style="color: #666666">.</span>empty(epochs)
train_accs<span style="color: #666666">.</span>fill(np<span style="color: #666666">.</span>nan)
val_accs <span style="color: #666666">=</span> np<span style="color: #666666">.</span>empty(epochs)
val_accs<span style="color: #666666">.</span>fill(np<span style="color: #666666">.</span>nan)
scores[<span style="color: #BA2121">&quot;train_error&quot;</span>] <span style="color: #666666">=</span> train_errors
scores[<span style="color: #BA2121">&quot;val_error&quot;</span>] <span style="color: #666666">=</span> val_errors
scores[<span style="color: #BA2121">&quot;train_acc&quot;</span>] <span style="color: #666666">=</span> train_accs
scores[<span style="color: #BA2121">&quot;val_acc&quot;</span>] <span style="color: #666666">=</span> val_accs
<span style="color: #008000; font-weight: bold">return</span> scores
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_initialize_weights</span>(<span style="color: #008000">self</span>, X: np<span style="color: #666666">.</span>ndarray) <span style="color: #666666">-&gt;</span> <span style="color: #008000; font-weight: bold">None</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Initializes weights for all layers in CNN</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I X (np.ndarray) input of format [img, feature_maps, height, width]</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
prev_nodes <span style="color: #666666">=</span> X
<span style="color: #008000; font-weight: bold">for</span> layer <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>layers:
prev_nodes <span style="color: #666666">=</span> layer<span style="color: #666666">.</span>_reset_weights(prev_nodes)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">predict</span>(<span style="color: #008000">self</span>, X: np<span style="color: #666666">.</span>ndarray, <span style="color: #666666">*</span>, threshold<span style="color: #666666">=0.5</span>) <span style="color: #666666">-&gt;</span> np<span style="color: #666666">.</span>ndarray:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Predicts output of input X</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I X (np.ndarray) input [img, feature_maps, height, width]</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
prediction <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_feedforward(X)
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;Binary&quot;</span>:
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>where(prediction <span style="color: #666666">&gt;</span> threshold, <span style="color: #666666">1</span>, <span style="color: #666666">0</span>)
<span style="color: #008000; font-weight: bold">elif</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>pred_format <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;Multi-class&quot;</span>:
class_prediction <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(prediction<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(prediction<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>]):
class_prediction[i, np<span style="color: #666666">.</span>argmax(prediction[i, :])] <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">return</span> class_prediction
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">return</span> prediction
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_accuracy</span>(<span style="color: #008000">self</span>, prediction: np<span style="color: #666666">.</span>ndarray, target: np<span style="color: #666666">.</span>ndarray) <span style="color: #666666">-&gt;</span> <span style="color: #008000">float</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Calculates accuracy of given prediction to target</span>
<span style="color: #BA2121; font-style: italic"> Parameters:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> I prediction (np.ndarray): output of predict() fuction</span>
<span style="color: #BA2121; font-style: italic"> (1s and 0s in case of classification, and real numbers in case of regression)</span>
<span style="color: #BA2121; font-style: italic"> II target (np.ndarray): vector of true values (What the network should predict)</span>
<span style="color: #BA2121; font-style: italic"> Returns:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> A floating point number representing the percentage of correctly classified instances.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">assert</span> prediction<span style="color: #666666">.</span>size <span style="color: #666666">==</span> target<span style="color: #666666">.</span>size
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>average((target <span style="color: #666666">==</span> prediction))
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_progress_bar</span>(<span style="color: #008000">self</span>, epoch: <span style="color: #008000">int</span>, epochs: <span style="color: #008000">int</span>, scores: <span style="color: #008000">dict</span>) <span style="color: #666666">-&gt;</span> <span style="color: #008000">int</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Displays progress of training</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
progression <span style="color: #666666">=</span> epoch <span style="color: #666666">/</span> epochs
epoch <span style="color: #666666">-=</span> <span style="color: #666666">1</span>
print_length <span style="color: #666666">=</span> <span style="color: #666666">40</span>
num_equals <span style="color: #666666">=</span> <span style="color: #008000">int</span>(progression <span style="color: #666666">*</span> print_length)
num_not <span style="color: #666666">=</span> print_length <span style="color: #666666">-</span> num_equals
arrow <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;&gt;&quot;</span> <span style="color: #008000; font-weight: bold">if</span> num_equals <span style="color: #666666">&gt;</span> <span style="color: #666666">0</span> <span style="color: #008000; font-weight: bold">else</span> <span style="color: #BA2121">&quot;&quot;</span>
bar <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;[&quot;</span> <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;=&quot;</span> <span style="color: #666666">*</span> (num_equals <span style="color: #666666">-</span> <span style="color: #666666">1</span>) <span style="color: #666666">+</span> arrow <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;-&quot;</span> <span style="color: #666666">*</span> num_not <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;]&quot;</span>
perc_print <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_fmt(progression <span style="color: #666666">*</span> <span style="color: #666666">100</span>, N<span style="color: #666666">=5</span>)
line <span style="color: #666666">=</span> <span style="color: #BA2121">f&quot; </span><span style="color: #BB6688; font-weight: bold">{</span>bar<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">{</span>perc_print<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">% &quot;</span>
<span style="color: #008000; font-weight: bold">for</span> key, score <span style="color: #AA22FF; font-weight: bold">in</span> scores<span style="color: #666666">.</span>items():
<span style="color: #008000; font-weight: bold">if</span> np<span style="color: #666666">.</span>isnan(score[epoch]) <span style="color: #666666">==</span> <span style="color: #008000; font-weight: bold">False</span>:
value <span style="color: #666666">=</span> <span style="color: #008000">self</span><span style="color: #666666">.</span>_fmt(score[epoch], N<span style="color: #666666">=4</span>)
line <span style="color: #666666">+=</span> <span style="color: #BA2121">f&quot;| </span><span style="color: #BB6688; font-weight: bold">{</span>key<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">: </span><span style="color: #BB6688; font-weight: bold">{</span>value<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121"> &quot;</span>
<span style="color: #008000">print</span>(line, end<span style="color: #666666">=</span><span style="color: #BA2121">&quot;</span><span style="color: #BB6622; font-weight: bold">\r</span><span style="color: #BA2121">&quot;</span>)
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">len</span>(line)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">_fmt</span>(<span style="color: #008000">self</span>, value: <span style="color: #008000">int</span>, N<span style="color: #666666">=4</span>) <span style="color: #666666">-&gt;</span> <span style="color: #008000">str</span>:
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Description:</span>
<span style="color: #BA2121; font-style: italic"> ------------</span>
<span style="color: #BA2121; font-style: italic"> Formats decimal numbers for progress bar</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">if</span> value <span style="color: #666666">&gt;</span> <span style="color: #666666">0</span>:
v <span style="color: #666666">=</span> value
<span style="color: #008000; font-weight: bold">elif</span> value <span style="color: #666666">&lt;</span> <span style="color: #666666">0</span>:
v <span style="color: #666666">=</span> <span style="color: #666666">-10</span> <span style="color: #666666">*</span> value
<span style="color: #008000; font-weight: bold">else</span>:
v <span style="color: #666666">=</span> <span style="color: #666666">1</span>
n <span style="color: #666666">=</span> <span style="color: #666666">1</span> <span style="color: #666666">+</span> math<span style="color: #666666">.</span>floor(math<span style="color: #666666">.</span>log10(v))
<span style="color: #008000; font-weight: bold">if</span> n <span style="color: #666666">&gt;=</span> N <span style="color: #666666">-</span> <span style="color: #666666">1</span>:
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #008000">str</span>(<span style="color: #008000">round</span>(value))
<span style="color: #408080; font-style: italic"># or overflow</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #BA2121">f&quot;</span><span style="color: #BB6688; font-weight: bold">{</span>value<span style="color: #BB6688; font-weight: bold">:</span><span style="color: #BA2121">.</span><span style="color: #BB6688; font-weight: bold">{</span>N<span style="color: #666666">-</span>n<span style="color: #666666">-1</span><span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">f</span><span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&quot;</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="usage-of-cnn-code">Usage of CNN code </h3>
<p>Using the CNN codebase is very simple. We begin by initiating a CNN
object, which takes a cost function, a scheduler and a seed as its
arguments. If a scheduler is not provided, it will per default
initiate an Adam scheduler with eta=1e-4, and if a seed is not
provided, the CNN will not be seeded, meaning it will run with a
different random seed every run. Below we demonstrate an initiation of
our CNN.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">adam_scheduler <span style="color: #666666">=</span> Adam(eta<span style="color: #666666">=1e-3</span>, rho<span style="color: #666666">=0.9</span>, rho2<span style="color: #666666">=0.999</span>)
cnn <span style="color: #666666">=</span> CNN(cost_func<span style="color: #666666">=</span>CostCrossEntropy, scheduler<span style="color: #666666">=</span>adam_scheduler, seed<span style="color: #666666">=2023</span>)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Now that we have our CNN object, we can begin to add layers to it!
Many of the add_layer functions have default values, for example
add_Convolution2DLayer() has a default v_stride and h_stride of
1. However, these can of course be set to any value you please. Note
that the input channels of a subsequent convolutional layer must equal
the previous convolutional layer's feature maps.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">cnn<span style="color: #666666">.</span>add_Convolution2DLayer(
input_channels<span style="color: #666666">=1</span>,
feature_maps<span style="color: #666666">=1</span>,
kernel_height<span style="color: #666666">=3</span>,
kernel_width<span style="color: #666666">=3</span>,
act_func<span style="color: #666666">=</span>LRELU,
)
cnn<span style="color: #666666">.</span>add_FlattenLayer()
cnn<span style="color: #666666">.</span>add_FullyConnectedLayer(<span style="color: #666666">30</span>, LRELU)
cnn<span style="color: #666666">.</span>add_FullyConnectedLayer(<span style="color: #666666">20</span>, LRELU)
cnn<span style="color: #666666">.</span>add_OutputLayer(<span style="color: #666666">10</span>, softmax)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Here we have created a CNN with the following architecture:</p>
<ol>
<li> A convolutional layer with 1 input channel, with a kernel height of 2 and a width of 2, which uses LRELU as its non-linearity function. This layer outputs 1 feature map, which feed into the subsequent layer.</li>
<li> A flatten layer</li>
<li> A hidden layer with 30 nodes, with LRELU as its activation function</li>
<li> Another hidden layer but with 20 nodes</li>
<li> The output layer, with softmax as its activation function and 10 nodes. We use 10 nodes because we will be using a dataset with 10 classes.</li>
</ol>
<p>Now, before we can train the model, we need to load in our data. We
will use the MNIST dataset and use 10000 \( 28 \times 28 \) images.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> fetch_openml
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">onehot</span>(target: np<span style="color: #666666">.</span>ndarray):
onehot <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((target<span style="color: #666666">.</span>size, target<span style="color: #666666">.</span>max() <span style="color: #666666">+</span> <span style="color: #666666">1</span>))
onehot[np<span style="color: #666666">.</span>arange(target<span style="color: #666666">.</span>size), target] <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">return</span> onehot
<span style="color: #408080; font-style: italic"># get dataset</span>
dataset <span style="color: #666666">=</span> fetch_openml(<span style="color: #BA2121">&quot;mnist_784&quot;</span>, parser<span style="color: #666666">=</span><span style="color: #BA2121">&quot;auto&quot;</span>)
mnist <span style="color: #666666">=</span> dataset<span style="color: #666666">.</span>data<span style="color: #666666">.</span>to_numpy(dtype<span style="color: #666666">=</span><span style="color: #BA2121">&quot;float&quot;</span>)[:<span style="color: #666666">10000</span>, :]
<span style="color: #408080; font-style: italic"># scale data</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(mnist<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>]):
mnist[:, i] <span style="color: #666666">/=</span> <span style="color: #666666">255</span>
<span style="color: #408080; font-style: italic"># reshape to add single input channel to data shape [inputs, input_channels, height, width]</span>
mnist <span style="color: #666666">=</span> mnist<span style="color: #666666">.</span>reshape(mnist<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], <span style="color: #666666">1</span>, <span style="color: #666666">28</span>, <span style="color: #666666">28</span>)
<span style="color: #408080; font-style: italic"># one hot encode target as we are doing multi-class classification</span>
target <span style="color: #666666">=</span> onehot(np<span style="color: #666666">.</span>array([<span style="color: #008000">int</span>(i) <span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> dataset<span style="color: #666666">.</span>target<span style="color: #666666">.</span>to_numpy()[:<span style="color: #666666">10000</span>]]))
<span style="color: #408080; font-style: italic"># split into training and validation data</span>
x_train, x_val, y_train, y_val <span style="color: #666666">=</span> train_test_split(mnist, target)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Now we may train our model. Note that we can utilize regularization in
the CNN by using the lam (lambda) parameter in fit(), and utilize
different types of gradient descent by specifying the amount of
batches via the batches parameter as shown below.
</p>
<p>The functionfit() returns a score dictionary of the training error and
accuracy (and validation error and accuracy if a validation set is
provided) which can be used to plot the error and accuracy of the
model over epochs.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">scores <span style="color: #666666">=</span> cnn<span style="color: #666666">.</span>fit(
x_train,
y_train,
lam<span style="color: #666666">=1e-5</span>,
batches<span style="color: #666666">=10</span>,
epochs<span style="color: #666666">=100</span>,
X_val<span style="color: #666666">=</span>x_val,
t_val<span style="color: #666666">=</span>y_val,
)
plt<span style="color: #666666">.</span>plot(scores[<span style="color: #BA2121">&quot;train_acc&quot;</span>], label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;Training&quot;</span>)
plt<span style="color: #666666">.</span>plot(scores[<span style="color: #BA2121">&quot;val_acc&quot;</span>], label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;Validation&quot;</span>)
plt<span style="color: #666666">.</span>ylim([<span style="color: #666666">0.8</span>,<span style="color: #666666">1</span>])
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;Epochs&quot;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;Accuracy&quot;</span>)
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Considering we only trained the model for 100 epochs without any tuning of the hyperparameters, this result is pretty good.</p>
<p>The codebase allows for great flexibility in CNN
architectures. Pooling layers can be added before, inbetween or after
convolutional layers, but due to the great optimizations made within
Convolution2DLayerOPT, we recommend using the v_stride and h_stride
parameters in add_Convolution2DLayer() to reduce the dimentionality of
the problem as the pooling layer is slow in comparison. To use the
unoptimized version of Convolution2DLayer, simply pass optimized=False
as an argument in add_Convolution2DLayer().
</p>
<p>If one wishes to perform binary classification using the CNN, simply
use the cost function 'CostLogReg' when initializing the CNN and use 1
node at the OutputLayer.
</p>
<p>Below we have created another, more untraditional architecture using
our code to demonstrate its flexibility and different attributes such
as asymmetric stride that might become useful when constructing your
own CNN.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">adam_scheduler <span style="color: #666666">=</span> Adam(eta<span style="color: #666666">=1e-3</span>, rho<span style="color: #666666">=0.9</span>, rho2<span style="color: #666666">=0.999</span>)
cnn <span style="color: #666666">=</span> CNN(cost_func<span style="color: #666666">=</span>CostCrossEntropy, scheduler<span style="color: #666666">=</span>adam_scheduler, seed<span style="color: #666666">=2023</span>)
cnn<span style="color: #666666">.</span>add_Convolution2DLayer(
input_channels<span style="color: #666666">=1</span>,
feature_maps<span style="color: #666666">=7</span>,
kernel_height<span style="color: #666666">=7</span>,
kernel_width<span style="color: #666666">=1</span>,
act_func<span style="color: #666666">=</span>LRELU,
)
cnn<span style="color: #666666">.</span>add_PoolingLayer(
kernel_height<span style="color: #666666">=2</span>,
kernel_width<span style="color: #666666">=2</span>,
pooling<span style="color: #666666">=</span><span style="color: #BA2121">&quot;average&quot;</span>,
)
cnn<span style="color: #666666">.</span>add_PoolingLayer(
kernel_height<span style="color: #666666">=2</span>,
kernel_width<span style="color: #666666">=2</span>,
pooling<span style="color: #666666">=</span><span style="color: #BA2121">&quot;max&quot;</span>,
)
cnn<span style="color: #666666">.</span>add_Convolution2DLayer(
input_channels<span style="color: #666666">=7</span>,
feature_maps<span style="color: #666666">=1</span>,
kernel_height<span style="color: #666666">=4</span>,
kernel_width<span style="color: #666666">=4</span>,
v_stride<span style="color: #666666">=2</span>,
h_stride<span style="color: #666666">=3</span>,
act_func<span style="color: #666666">=</span>LRELU,
optimized<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>,
)
cnn<span style="color: #666666">.</span>add_Convolution2DLayer(
input_channels<span style="color: #666666">=1</span>,
feature_maps<span style="color: #666666">=1</span>,
kernel_height<span style="color: #666666">=2</span>,
kernel_width<span style="color: #666666">=2</span>,
act_func<span style="color: #666666">=</span>sigmoid,
optimized<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,
)
cnn<span style="color: #666666">.</span>add_PoolingLayer(
kernel_height<span style="color: #666666">=2</span>,
kernel_width<span style="color: #666666">=2</span>,
pooling<span style="color: #666666">=</span><span style="color: #BA2121">&quot;max&quot;</span>
)
cnn<span style="color: #666666">.</span>add_FlattenLayer()
cnn<span style="color: #666666">.</span>add_FullyConnectedLayer(<span style="color: #666666">100</span>, LRELU)
cnn<span style="color: #666666">.</span>add_FullyConnectedLayer(<span style="color: #666666">10</span>, sigmoid)
cnn<span style="color: #666666">.</span>add_FullyConnectedLayer(<span style="color: #666666">101</span>, identity)
cnn<span style="color: #666666">.</span>add_OutputLayer(<span style="color: #666666">10</span>, softmax)
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Here we see the use of asymmetrical 1D kernels such as the \( 7 \times
1 \) kernel in the first convolutional layer, both max and average
pooling, asymmetric stride in the unoptimized convolutional layer,
more pooling, a flatten layer, a hidden layer with 100 nodes using
LRELU, another hidden layer with 10 hidden nodes that uses the sigmoid
activation function, and another hidden layer with 101 nodes which
utilizes no activation function (identity). Finally, we arrive at the
output layer with 10 nodes, which uses softmax as its activation
function.
</p>
<h3 id="additional-remarks">Additional Remarks </h3>
<p>The stride parameter controls the distance between each convolution
and the kernel/filter. If our image is padded, stride is the only
parameter that determines the size of the output from a convolutional
layer. However, if we decide not to perform any padding, the size of
the output feature map depends on both the stride and kernel size. It
is important to note that neither the stride nor the kernel has to be
symmetrical. This means that we can use a rectangular filter if we
choose, and the stride in the vertical direction (axis=0 in Python)
does not need to be the same as the stride in the horizontal direction
(axis=1 in Python). It may even be the case that asymmetric
combinations of stride or kernel dimensions, or both, yield better
results than symmetric values for these parameters.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">convolve</span>(image, kernel, stride<span style="color: #666666">=1</span>):
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">2</span>):
kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>rot90(kernel)
k_half_height <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
k_half_width <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
conv_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(image<span style="color: #666666">.</span>shape)
pad_image <span style="color: #666666">=</span> padding(image, kernel)
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(k_half_height, conv_image<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> k_half_height, stride):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(k_half_width, conv_image<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> k_half_width, stride):
conv_image[i <span style="color: #666666">-</span> k_half_height, j <span style="color: #666666">-</span> k_half_width] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(
pad_image[
i <span style="color: #666666">-</span> k_half_height : i <span style="color: #666666">+</span> k_half_height <span style="color: #666666">+</span> <span style="color: #666666">1</span>, j <span style="color: #666666">-</span> k_half_width : j <span style="color: #666666">+</span> k_half_width <span style="color: #666666">+</span> <span style="color: #666666">1</span>
]
<span style="color: #666666">*</span> kernel
)
<span style="color: #008000; font-weight: bold">return</span> conv_image
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<h3 id="remarks-on-the-speed">Remarks on the speed </h3>
<p>Despite the naive convolution algorithm shown above working finely, it
is extremely slow, requiring approximately 20-30 seconds to process a
single image. The time complexity of 2D convolution, which is O(NMnm),
rapidly becomes a constraint and may, at worst, make computations
infeasible. Consequently, optimizing the naive 2D convolution
algorithm is a necessity, as the execution time of the algorithm
significantly increases as the input data size expands. This can pose
a bottleneck in applications that necessitate real-time processing of
large data volumes, such as image and video processing, deep learning,
and scientific simulations.
</p>
<p>To address this issue, we shall present two widely used optimization
techniques: the separable kernel approach and Fast Fourier Transform
(FFT). Both of these methods can drastically reduce the computational
complexity of convolution and enhance the overall efficiency of
processing substantial data quantities. While we shall refrain from
delving into the intricacies of these algorithms, we strongly
encourage you to examine at least the application of FFT to optimize
computations.
</p>
<h3 id="convolution-using-separable-kernels">Convolution using separable kernels </h3>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">conv2DSep</span>(image, kernel, coef, stride<span style="color: #666666">=1</span>, pad<span style="color: #666666">=</span><span style="color: #BA2121">&quot;zero&quot;</span>):
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">2</span>):
kernel <span style="color: #666666">=</span> np<span style="color: #666666">.</span>rot90(kernel)
<span style="color: #408080; font-style: italic"># The kernel is quadratic, thus we only need one of its dimensions</span>
half_dim <span style="color: #666666">=</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">//</span> <span style="color: #666666">2</span>
ker1 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array(kernel[<span style="color: #666666">0</span>, :])
ker2 <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array(kernel[:, <span style="color: #666666">0</span>])
<span style="color: #008000; font-weight: bold">if</span> pad <span style="color: #666666">==</span> <span style="color: #BA2121">&quot;zero&quot;</span>:
conv_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(image<span style="color: #666666">.</span>shape)
pad_image <span style="color: #666666">=</span> padding(image, kernel)
<span style="color: #008000; font-weight: bold">else</span>:
conv_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros(
(image<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">-</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], image<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">-</span> kernel<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>])
)
pad_image <span style="color: #666666">=</span> image[:, :]
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(half_dim, conv_image<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> half_dim, stride):
<span style="color: #008000; font-weight: bold">for</span> j <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(half_dim, conv_image<span style="color: #666666">.</span>shape[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> half_dim, stride):
conv_image[i <span style="color: #666666">-</span> half_dim, j <span style="color: #666666">-</span> half_dim] <span style="color: #666666">=</span> (
pad_image[
i <span style="color: #666666">-</span> half_dim : i <span style="color: #666666">+</span> half_dim <span style="color: #666666">+</span> <span style="color: #666666">1</span>, j <span style="color: #666666">-</span> half_dim : j <span style="color: #666666">+</span> half_dim <span style="color: #666666">+</span> <span style="color: #666666">1</span>
]
<span style="color: #666666">@</span> ker1
<span style="color: #666666">@</span> ker2<span style="color: #666666">.</span>T
<span style="color: #666666">*</span> coef
)
<span style="color: #008000; font-weight: bold">return</span> conv_image
img_path <span style="color: #666666">=</span> img_path <span style="color: #666666">=</span> <span style="color: #BA2121">&quot;data/IMG-2167.JPG&quot;</span>
image_of_cute_dog <span style="color: #666666">=</span> imageio<span style="color: #666666">.</span>imread(img_path, mode<span style="color: #666666">=</span><span style="color: #BA2121">&quot;L&quot;</span>)
start_time <span style="color: #666666">=</span> time<span style="color: #666666">.</span>time()
filtered_image <span style="color: #666666">=</span> conv2DSep(image_of_cute_dog, kernel<span style="color: #666666">=</span>sobel_kernel, coef<span style="color: #666666">=1</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&#39;Time taken for convolution with seperated kernel on 128x128 image </span><span style="color: #BB6688; font-weight: bold">{</span>time<span style="color: #666666">.</span>time() <span style="color: #666666">-</span> start_time<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&#39;</span>)
plt<span style="color: #666666">.</span>imshow(filtered_image, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>, vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, aspect<span style="color: #666666">=</span><span style="color: #BA2121">&quot;auto&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>By taking advantage of the capabilities of separable kernels, we can
effectively cut the computational expense of filtering an image in
half. Yet, if we seek even more rapid processing, we can turn to the
Fast Fourier Transform (FFT) algorithm provided by the numpy
library. By utilizing FFT to transform the input image and filter into
the frequency domain, we can perform convolution in this domain. This
approach significantly reduces the number of operations needed and
results in a marked speedup relative to other convolution
techniques. In addition, it is worth noting that the FFT is widely
regarded as one of the most critical algorithms developed to date,
with applications ranging from digital signal processing to scientific
computing.
</p>
<h3 id="convolution-in-the-fourier-domain">Convolution in the Fourier domain </h3>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">start_time <span style="color: #666666">=</span> time<span style="color: #666666">.</span>time()
img_fft <span style="color: #666666">=</span> np<span style="color: #666666">.</span>fft<span style="color: #666666">.</span>fft2(image_of_cute_dog)
kernel_fft <span style="color: #666666">=</span> np<span style="color: #666666">.</span>fft<span style="color: #666666">.</span>fft2(sobel_kernel, s<span style="color: #666666">=</span>image_of_cute_dog<span style="color: #666666">.</span>shape)
conv_image <span style="color: #666666">=</span> img_fft <span style="color: #666666">*</span> kernel_fft
filtered_image <span style="color: #666666">=</span> np<span style="color: #666666">.</span>fft<span style="color: #666666">.</span>ifft2(conv_image)
<span style="color: #008000">print</span>(<span style="color: #BA2121">f&#39;Time take for convolution in the fourier domain: </span><span style="color: #BB6688; font-weight: bold">{</span>time<span style="color: #666666">.</span>time() <span style="color: #666666">-</span> start_time<span style="color: #BB6688; font-weight: bold">}</span><span style="color: #BA2121">&#39;</span>)
plt<span style="color: #666666">.</span>imshow(filtered_image<span style="color: #666666">.</span>real, cmap<span style="color: #666666">=</span><span style="color: #BA2121">&quot;gray&quot;</span>, vmin<span style="color: #666666">=0</span>, vmax<span style="color: #666666">=255</span>, aspect<span style="color: #666666">=</span><span style="color: #BA2121">&quot;auto&quot;</span>)
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>It is evident that executing convolution in the Fourier domain yields
the quickest computation time. Nonetheless, one should exercise
caution, particularly when dealing with images of relatively small
dimensions, as one of the other methods may prove to be more
expeditious than FFT-enhanced convolution. The overhead involved in
transferring both the image and filter into the Fourier domain,
followed by their subsequent transformation back into the spatial
domain, results in a minor inconvenience. Therefore, it is imperative
to remain cognizant of this fact when utilizing FFT as the primary
optimization technique.
</p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2024, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink