lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5c6c05efbe058f033f3f35def695f64a7c70b9ce
FYS-STK4155/doc/pub/week41/html/._week41-bs008.html
T
mhjensen 021cd0f677 updating keras
2020-10-09 06:49:00 +02:00

400 lines
24 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks">
<title>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 41', 2, None, '___sec0'),
('Setting up the Back propagation algorithm', 2, None, '___sec1'),
('Setting up a Multi-layer perceptron model for classification',
2,
None,
'___sec2'),
('Defining the cost function', 2, None, '___sec3'),
('Example: binary classification problem', 2, None, '___sec4'),
('The Softmax function', 2, None, '___sec5'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec6'),
('Collect and pre-process data', 2, None, '___sec7'),
('Train and test datasets', 2, None, '___sec8'),
('Define model and architecture', 2, None, '___sec9'),
('Layers', 2, None, '___sec10'),
('Weights and biases', 2, None, '___sec11'),
('Feed-forward pass', 2, None, '___sec12'),
('Matrix multiplications', 2, None, '___sec13'),
('Choose cost function and optimizer', 2, None, '___sec14'),
('Optimizing the cost function', 2, None, '___sec15'),
('Regularization', 2, None, '___sec16'),
('Matrix multiplication', 2, None, '___sec17'),
('Improving performance', 2, None, '___sec18'),
('Full object-oriented implementation', 2, None, '___sec19'),
('Evaluate model performance on test data', 2, None, '___sec20'),
('Adjust hyperparameters', 2, None, '___sec21'),
('Visualization', 2, None, '___sec22'),
('scikit-learn implementation', 2, None, '___sec23'),
('Visualization', 2, None, '___sec24'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec25'),
('Tensorflow', 2, None, '___sec26'),
('Using Keras', 2, None, '___sec27'),
('Collect and pre-process data', 2, None, '___sec28'),
('The Breast Cancer Data, now with Keras', 2, None, '___sec29'),
('Fine-tuning neural network hyperparameters',
2,
None,
'___sec30'),
('Hidden layers', 2, None, '___sec31'),
('Which activation function should I use?', 2, None, '___sec32'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec33'),
('The derivative of the Logistic funtion', 2, None, '___sec34'),
('The RELU function family', 2, None, '___sec35'),
('Which activation function should we use?', 2, None, '___sec36'),
('More on activation functions, output layers',
2,
None,
'___sec37'),
('Batch Normalization', 2, None, '___sec38'),
('Dropout', 2, None, '___sec39'),
('Gradient Clipping', 2, None, '___sec40'),
('A very nice website on Neural Networks', 2, None, '___sec41'),
('A top-down perspective on Neural networks',
2,
None,
'___sec42'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec43'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
'___sec44'),
('Regular NNs don’t scale well to full images',
2,
None,
'___sec45'),
('3D volumes of neurons', 2, None, '___sec46'),
('Layers used to build CNNs', 2, None, '___sec47'),
('Transforming images', 2, None, '___sec48'),
('CNNs in brief', 2, None, '___sec49'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
None,
'___sec50'),
('Setting it up', 2, None, '___sec51'),
('The MNIST dataset again', 2, None, '___sec52'),
('Strong correlations', 2, None, '___sec53'),
('Layers of a CNN', 2, None, '___sec54'),
('Systematic reduction', 2, None, '___sec55'),
('Prerequisites: Collect and pre-process data',
2,
None,
'___sec56'),
('Importing Keras and Tensorflow', 2, None, '___sec57'),
('Running with Keras', 2, None, '___sec58'),
('Final part', 2, None, '___sec59'),
('Final visualization', 2, None, '___sec60'),
('Fun links', 2, None, '___sec61')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week41-bs.html">Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week41-bs001.html#___sec0" style="font-size: 80%;">Plan for week 41</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs002.html#___sec1" style="font-size: 80%;">Setting up the Back propagation algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs003.html#___sec2" style="font-size: 80%;">Setting up a Multi-layer perceptron model for classification</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs004.html#___sec3" style="font-size: 80%;">Defining the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs005.html#___sec4" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs006.html#___sec5" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#___sec6" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs009.html#___sec8" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#___sec9" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#___sec10" style="font-size: 80%;">Layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#___sec11" style="font-size: 80%;">Weights and biases</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs013.html#___sec12" style="font-size: 80%;">Feed-forward pass</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs014.html#___sec13" style="font-size: 80%;">Matrix multiplications</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs015.html#___sec14" style="font-size: 80%;">Choose cost function and optimizer</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs016.html#___sec15" style="font-size: 80%;">Optimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs017.html#___sec16" style="font-size: 80%;">Regularization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs018.html#___sec17" style="font-size: 80%;">Matrix multiplication</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs019.html#___sec18" style="font-size: 80%;">Improving performance</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs020.html#___sec19" style="font-size: 80%;">Full object-oriented implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs021.html#___sec20" style="font-size: 80%;">Evaluate model performance on test data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs022.html#___sec21" style="font-size: 80%;">Adjust hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs023.html#___sec22" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs024.html#___sec23" style="font-size: 80%;">scikit-learn implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs025.html#___sec24" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs026.html#___sec25" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs027.html#___sec26" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs028.html#___sec27" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#___sec28" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#___sec29" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#___sec30" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#___sec31" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#___sec32" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#___sec33" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#___sec34" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#___sec35" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#___sec36" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#___sec37" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#___sec38" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#___sec39" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#___sec40" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#___sec41" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#___sec42" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#___sec43" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs045.html#___sec44" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#___sec45" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#___sec46" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#___sec47" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#___sec48" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#___sec49" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#___sec50" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#___sec51" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#___sec52" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#___sec53" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#___sec54" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#___sec55" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#___sec56" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#___sec57" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#___sec58" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#___sec59" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#___sec60" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#___sec61" style="font-size: 80%;">Fun links</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0008"></a>
<!-- !split -->
<h2 id="___sec7" class="anchor">Collect and pre-process data </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
package. You may also find it for example <a href="http://yann.lecun.com/exdb/mnist/" target="_self">here</a>.
The <em>MNIST</em> (Modified National Institute of Standards and Technology) database is a large database
of handwritten digits that is commonly used for training various image processing systems.
The MNIST dataset consists of 70 000 images of size \( 28\times 28 \) pixels, each labeled from 0 to 9.
The scikit-learn dataset we will use consists of a selection of 1797 images of size \( 8\times 8 \) collected and processed from this database.
<p>
To feed data into a feed-forward neural network we need to represent
the inputs as a design/feature matrix \( X = (n_{inputs}, n_{features}) \). Each
row represents an <em>input</em>, in this case a handwritten digit, and
each column represents a <em>feature</em>, in this case a pixel. The
correct answers, also known as <em>labels</em> or <em>targets</em> are
represented as a 1D array of integers
\( Y = (n_{inputs}) = (5, 3, 1, 8,...) \).
<p>
As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from
measurements of height (in m)
and weight (in kg). If we have measurements of 5 people the design/feature matrix could be for example:
$$ X = \begin{bmatrix}
1.85 &amp; 81\\
1.71 &amp; 65\\
1.95 &amp; 103\\
1.55 &amp; 42\\
1.63 &amp; 56
\end{bmatrix} ,$$
<p>
and the targets would be:
$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$
<p>
Since each input image is a 2D matrix, we need to flatten the image
(i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a
design/feature matrix. This means we lose all spatial information in the
image, such as locality and translational invariance. More complicated
architectures such as Convolutional Neural Networks can take advantage
of such information, and are most commonly applied when analyzing
images.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
<span style="color: #408080; font-style: italic"># ensure the same random numbers appear every time</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
<span style="color: #408080; font-style: italic"># display images in notebook</span>
<span style="color: #666666">%</span>matplotlib inline
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;figure.figsize&#39;</span>] <span style="color: #666666">=</span> (<span style="color: #666666">12</span>,<span style="color: #666666">12</span>)
<span style="color: #408080; font-style: italic"># download MNIST dataset</span>
digits <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_digits()
<span style="color: #408080; font-style: italic"># define inputs and labels</span>
inputs <span style="color: #666666">=</span> digits<span style="color: #666666">.</span>images
labels <span style="color: #666666">=</span> digits<span style="color: #666666">.</span>target
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;inputs = (n_inputs, pixel_width, pixel_height) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(inputs<span style="color: #666666">.</span>shape))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;labels = (n_inputs) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(labels<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># flatten the image</span>
<span style="color: #408080; font-style: italic"># the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64</span>
n_inputs <span style="color: #666666">=</span> <span style="color: #008000">len</span>(inputs)
inputs <span style="color: #666666">=</span> inputs<span style="color: #666666">.</span>reshape(n_inputs, <span style="color: #666666">-1</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;X = (n_inputs, n_features) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(inputs<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># choose some random images to display</span>
indices <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(n_inputs)
random_indices <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>choice(indices, size<span style="color: #666666">=5</span>)
<span style="color: #008000; font-weight: bold">for</span> i, image <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(digits<span style="color: #666666">.</span>images[random_indices]):
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">1</span>, <span style="color: #666666">5</span>, i<span style="color: #666666">+1</span>)
plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;off&#39;</span>)
plt<span style="color: #666666">.</span>imshow(image, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>gray_r, interpolation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;nearest&#39;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Label: </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> digits<span style="color: #666666">.</span>target[random_indices[i]])
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week41-bs007.html">&laquo;</a></li>
<li><a href="._week41-bs000.html">1</a></li>
<li><a href="._week41-bs001.html">2</a></li>
<li><a href="._week41-bs002.html">3</a></li>
<li><a href="._week41-bs003.html">4</a></li>
<li><a href="._week41-bs004.html">5</a></li>
<li><a href="._week41-bs005.html">6</a></li>
<li><a href="._week41-bs006.html">7</a></li>
<li><a href="._week41-bs007.html">8</a></li>
<li class="active"><a href="._week41-bs008.html">9</a></li>
<li><a href="._week41-bs009.html">10</a></li>
<li><a href="._week41-bs010.html">11</a></li>
<li><a href="._week41-bs011.html">12</a></li>
<li><a href="._week41-bs012.html">13</a></li>
<li><a href="._week41-bs013.html">14</a></li>
<li><a href="._week41-bs014.html">15</a></li>
<li><a href="._week41-bs015.html">16</a></li>
<li><a href="._week41-bs016.html">17</a></li>
<li><a href="._week41-bs017.html">18</a></li>
<li><a href="">...</a></li>
<li><a href="._week41-bs062.html">63</a></li>
<li><a href="._week41-bs009.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink