lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5c6c05efbe058f033f3f35def695f64a7c70b9ce
FYS-STK4155/doc/pub/week41/html/._week41-bs045.html
T
mhjensen 021cd0f677 updating keras
2020-10-09 06:49:00 +02:00

346 lines
19 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks">
<title>Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Plan for week 41', 2, None, '___sec0'),
('Setting up the Back propagation algorithm', 2, None, '___sec1'),
('Setting up a Multi-layer perceptron model for classification',
2,
None,
'___sec2'),
('Defining the cost function', 2, None, '___sec3'),
('Example: binary classification problem', 2, None, '___sec4'),
('The Softmax function', 2, None, '___sec5'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec6'),
('Collect and pre-process data', 2, None, '___sec7'),
('Train and test datasets', 2, None, '___sec8'),
('Define model and architecture', 2, None, '___sec9'),
('Layers', 2, None, '___sec10'),
('Weights and biases', 2, None, '___sec11'),
('Feed-forward pass', 2, None, '___sec12'),
('Matrix multiplications', 2, None, '___sec13'),
('Choose cost function and optimizer', 2, None, '___sec14'),
('Optimizing the cost function', 2, None, '___sec15'),
('Regularization', 2, None, '___sec16'),
('Matrix multiplication', 2, None, '___sec17'),
('Improving performance', 2, None, '___sec18'),
('Full object-oriented implementation', 2, None, '___sec19'),
('Evaluate model performance on test data', 2, None, '___sec20'),
('Adjust hyperparameters', 2, None, '___sec21'),
('Visualization', 2, None, '___sec22'),
('scikit-learn implementation', 2, None, '___sec23'),
('Visualization', 2, None, '___sec24'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec25'),
('Tensorflow', 2, None, '___sec26'),
('Using Keras', 2, None, '___sec27'),
('Collect and pre-process data', 2, None, '___sec28'),
('The Breast Cancer Data, now with Keras', 2, None, '___sec29'),
('Fine-tuning neural network hyperparameters',
2,
None,
'___sec30'),
('Hidden layers', 2, None, '___sec31'),
('Which activation function should I use?', 2, None, '___sec32'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec33'),
('The derivative of the Logistic funtion', 2, None, '___sec34'),
('The RELU function family', 2, None, '___sec35'),
('Which activation function should we use?', 2, None, '___sec36'),
('More on activation functions, output layers',
2,
None,
'___sec37'),
('Batch Normalization', 2, None, '___sec38'),
('Dropout', 2, None, '___sec39'),
('Gradient Clipping', 2, None, '___sec40'),
('A very nice website on Neural Networks', 2, None, '___sec41'),
('A top-down perspective on Neural networks',
2,
None,
'___sec42'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec43'),
('Convolutional Neural Networks (recognizing images)',
2,
None,
'___sec44'),
('Regular NNs don’t scale well to full images',
2,
None,
'___sec45'),
('3D volumes of neurons', 2, None, '___sec46'),
('Layers used to build CNNs', 2, None, '___sec47'),
('Transforming images', 2, None, '___sec48'),
('CNNs in brief', 2, None, '___sec49'),
('CNNs in more detail, building convolutional neural networks in '
'Tensorflow and Keras',
2,
None,
'___sec50'),
('Setting it up', 2, None, '___sec51'),
('The MNIST dataset again', 2, None, '___sec52'),
('Strong correlations', 2, None, '___sec53'),
('Layers of a CNN', 2, None, '___sec54'),
('Systematic reduction', 2, None, '___sec55'),
('Prerequisites: Collect and pre-process data',
2,
None,
'___sec56'),
('Importing Keras and Tensorflow', 2, None, '___sec57'),
('Running with Keras', 2, None, '___sec58'),
('Final part', 2, None, '___sec59'),
('Final visualization', 2, None, '___sec60'),
('Fun links', 2, None, '___sec61')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week41-bs.html">Week 41 Tensor flow and Deep Learning, Convolutional Neural Networks</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week41-bs001.html#___sec0" style="font-size: 80%;">Plan for week 41</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs002.html#___sec1" style="font-size: 80%;">Setting up the Back propagation algorithm</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs003.html#___sec2" style="font-size: 80%;">Setting up a Multi-layer perceptron model for classification</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs004.html#___sec3" style="font-size: 80%;">Defining the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs005.html#___sec4" style="font-size: 80%;">Example: binary classification problem</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs006.html#___sec5" style="font-size: 80%;">The Softmax function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs007.html#___sec6" style="font-size: 80%;">Developing a code for doing neural networks with back propagation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs008.html#___sec7" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs009.html#___sec8" style="font-size: 80%;">Train and test datasets</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs010.html#___sec9" style="font-size: 80%;">Define model and architecture</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs011.html#___sec10" style="font-size: 80%;">Layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs012.html#___sec11" style="font-size: 80%;">Weights and biases</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs013.html#___sec12" style="font-size: 80%;">Feed-forward pass</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs014.html#___sec13" style="font-size: 80%;">Matrix multiplications</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs015.html#___sec14" style="font-size: 80%;">Choose cost function and optimizer</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs016.html#___sec15" style="font-size: 80%;">Optimizing the cost function</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs017.html#___sec16" style="font-size: 80%;">Regularization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs018.html#___sec17" style="font-size: 80%;">Matrix multiplication</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs019.html#___sec18" style="font-size: 80%;">Improving performance</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs020.html#___sec19" style="font-size: 80%;">Full object-oriented implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs021.html#___sec20" style="font-size: 80%;">Evaluate model performance on test data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs022.html#___sec21" style="font-size: 80%;">Adjust hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs023.html#___sec22" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs024.html#___sec23" style="font-size: 80%;">scikit-learn implementation</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs025.html#___sec24" style="font-size: 80%;">Visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs026.html#___sec25" style="font-size: 80%;">Building neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs027.html#___sec26" style="font-size: 80%;">Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs028.html#___sec27" style="font-size: 80%;">Using Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs029.html#___sec28" style="font-size: 80%;">Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs030.html#___sec29" style="font-size: 80%;">The Breast Cancer Data, now with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs031.html#___sec30" style="font-size: 80%;">Fine-tuning neural network hyperparameters</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs032.html#___sec31" style="font-size: 80%;">Hidden layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs033.html#___sec32" style="font-size: 80%;">Which activation function should I use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs034.html#___sec33" style="font-size: 80%;">Is the Logistic activation function (Sigmoid) our choice?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs035.html#___sec34" style="font-size: 80%;">The derivative of the Logistic funtion</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs036.html#___sec35" style="font-size: 80%;">The RELU function family</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs037.html#___sec36" style="font-size: 80%;">Which activation function should we use?</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs038.html#___sec37" style="font-size: 80%;">More on activation functions, output layers</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs039.html#___sec38" style="font-size: 80%;">Batch Normalization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs040.html#___sec39" style="font-size: 80%;">Dropout</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs041.html#___sec40" style="font-size: 80%;">Gradient Clipping</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs042.html#___sec41" style="font-size: 80%;">A very nice website on Neural Networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs043.html#___sec42" style="font-size: 80%;">A top-down perspective on Neural networks</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs044.html#___sec43" style="font-size: 80%;">Limitations of supervised learning with deep networks</a></li>
<!-- navigation toc: --> <li><a href="#___sec44" style="font-size: 80%;">Convolutional Neural Networks (recognizing images)</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs046.html#___sec45" style="font-size: 80%;">Regular NNs don’t scale well to full images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs047.html#___sec46" style="font-size: 80%;">3D volumes of neurons</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs048.html#___sec47" style="font-size: 80%;">Layers used to build CNNs</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs049.html#___sec48" style="font-size: 80%;">Transforming images</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs050.html#___sec49" style="font-size: 80%;">CNNs in brief</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs051.html#___sec50" style="font-size: 80%;">CNNs in more detail, building convolutional neural networks in Tensorflow and Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs052.html#___sec51" style="font-size: 80%;">Setting it up</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs053.html#___sec52" style="font-size: 80%;">The MNIST dataset again</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs054.html#___sec53" style="font-size: 80%;">Strong correlations</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs055.html#___sec54" style="font-size: 80%;">Layers of a CNN</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs056.html#___sec55" style="font-size: 80%;">Systematic reduction</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs057.html#___sec56" style="font-size: 80%;">Prerequisites: Collect and pre-process data</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs058.html#___sec57" style="font-size: 80%;">Importing Keras and Tensorflow</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs059.html#___sec58" style="font-size: 80%;">Running with Keras</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs060.html#___sec59" style="font-size: 80%;">Final part</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs061.html#___sec60" style="font-size: 80%;">Final visualization</a></li>
<!-- navigation toc: --> <li><a href="._week41-bs062.html#___sec61" style="font-size: 80%;">Fun links</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0045"></a>
<!-- !split -->
<h2 id="___sec44" class="anchor">Convolutional Neural Networks (recognizing images) </h2>
<p>
Convolutional neural networks (CNNs) were developed during the last
decade of the previous century, with a focus on character recognition
tasks. Nowadays, CNNs are a central element in the spectacular success
of deep learning methods. The success in for example image
classifications have made them a central tool for most machine
learning practitioners.
<p>
CNNs are very similar to ordinary Neural Networks.
They are made up of neurons that have learnable weights and
biases. Each neuron receives some inputs, performs a dot product and
optionally follows it with a non-linearity. The whole network still
expresses a single differentiable score function: from the raw image
pixels on one end to class scores at the other. And they still have a
loss function (for example Softmax) on the last (fully-connected) layer
and all the tips/tricks we developed for learning regular Neural
Networks still apply (back propagation, gradient descent etc etc).
<p>
What is the difference? <b>CNN architectures make the explicit assumption that
the inputs are images, which allows us to encode certain properties
into the architecture. These then make the forward function more
efficient to implement and vastly reduce the amount of parameters in
the network.</b>
<p>
Here we provide only a superficial overview, for the more interested, we recommend highly the course
<a href="https://www.uio.no/studier/emner/matnat/ifi/IN5400/index-eng.html" target="_self">IN5400 &#8211; Machine Learning for Image Analysis</a>
and the slides of <a href="http://cs231n.github.io/convolutional-networks/" target="_self">CS231</a>.
<p>
Another good read is the article here <a href="https://arxiv.org/pdf/1603.07285.pdf" target="_self"><tt>https://arxiv.org/pdf/1603.07285.pdf</tt></a>.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week41-bs044.html">&laquo;</a></li>
<li><a href="._week41-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._week41-bs037.html">38</a></li>
<li><a href="._week41-bs038.html">39</a></li>
<li><a href="._week41-bs039.html">40</a></li>
<li><a href="._week41-bs040.html">41</a></li>
<li><a href="._week41-bs041.html">42</a></li>
<li><a href="._week41-bs042.html">43</a></li>
<li><a href="._week41-bs043.html">44</a></li>
<li><a href="._week41-bs044.html">45</a></li>
<li class="active"><a href="._week41-bs045.html">46</a></li>
<li><a href="._week41-bs046.html">47</a></li>
<li><a href="._week41-bs047.html">48</a></li>
<li><a href="._week41-bs048.html">49</a></li>
<li><a href="._week41-bs049.html">50</a></li>
<li><a href="._week41-bs050.html">51</a></li>
<li><a href="._week41-bs051.html">52</a></li>
<li><a href="._week41-bs052.html">53</a></li>
<li><a href="._week41-bs053.html">54</a></li>
<li><a href="._week41-bs054.html">55</a></li>
<li><a href="">...</a></li>
<li><a href="._week41-bs062.html">63</a></li>
<li><a href="._week41-bs046.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink