lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update on neural network slides

Browse Source
This commit is contained in:
mhjensen
2018-10-16 12:21:57 +02:00
parent 5644101f09
commit ce0c90eccd
74 changed files with 7348 additions and 6706 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/NeuralNet/html/._NeuralNet-bs034.html
+78 -153
View File
@@ -91,53 +91,55 @@ Automatically generated HTML file from DocOnce source
None,
'___sec30'),
('Defining the cost function', 2, None, '___sec31'),
('Example: binary classification problem', 2, None, '___sec32'),
('Developing a code for doing neural networks with back '
'propagation',
2,
None,
'___sec32'),
('Collect and pre-process data', 2, None, '___sec33'),
('Train and test datasets', 2, None, '___sec34'),
('Define model and architecture', 2, None, '___sec35'),
('Layers', 2, None, '___sec36'),
('Weights and biases', 2, None, '___sec37'),
('Feed-forward pass', 2, None, '___sec38'),
('Matrix multiplications', 2, None, '___sec39'),
('Choose cost function and optimizer', 2, None, '___sec40'),
('Optimizing the cost function', 2, None, '___sec41'),
('Regularization', 2, None, '___sec42'),
('Matrix multiplication', 2, None, '___sec43'),
('Improving performance', 2, None, '___sec44'),
('Full object-oriented implementation', 2, None, '___sec45'),
('Evaluate model performance on test data', 2, None, '___sec46'),
('Adjust hyperparameters', 2, None, '___sec47'),
('Visualization', 2, None, '___sec48'),
('scikit-learn implementation', 2, None, '___sec49'),
('Visualization', 2, None, '___sec50'),
'___sec33'),
('Collect and pre-process data', 2, None, '___sec34'),
('Train and test datasets', 2, None, '___sec35'),
('Define model and architecture', 2, None, '___sec36'),
('Layers', 2, None, '___sec37'),
('Weights and biases', 2, None, '___sec38'),
('Feed-forward pass', 2, None, '___sec39'),
('Matrix multiplications', 2, None, '___sec40'),
('Choose cost function and optimizer', 2, None, '___sec41'),
('Optimizing the cost function', 2, None, '___sec42'),
('Regularization', 2, None, '___sec43'),
('Matrix multiplication', 2, None, '___sec44'),
('Improving performance', 2, None, '___sec45'),
('Full object-oriented implementation', 2, None, '___sec46'),
('Evaluate model performance on test data', 2, None, '___sec47'),
('Adjust hyperparameters', 2, None, '___sec48'),
('Visualization', 2, None, '___sec49'),
('scikit-learn implementation', 2, None, '___sec50'),
('Visualization', 2, None, '___sec51'),
('Building neural networks in Tensorflow and Keras',
2,
None,
'___sec51'),
('Tensorflow', 2, None, '___sec52'),
('Collect and pre-process data', 2, None, '___sec53'),
('Using TensorFlow backend', 2, None, '___sec54'),
('Optimizing and using gradient descent', 2, None, '___sec55'),
('Using Keras', 2, None, '___sec56'),
('Which activation function should I use?', 2, None, '___sec57'),
'___sec52'),
('Tensorflow', 2, None, '___sec53'),
('Collect and pre-process data', 2, None, '___sec54'),
('Using TensorFlow backend', 2, None, '___sec55'),
('Optimizing and using gradient descent', 2, None, '___sec56'),
('Using Keras', 2, None, '___sec57'),
('Which activation function should I use?', 2, None, '___sec58'),
('Is the Logistic activation function (Sigmoid) our choice?',
2,
None,
'___sec58'),
('The derivative of the Logistic funtion', 2, None, '___sec59'),
('The RELU function family', 2, None, '___sec60'),
'___sec59'),
('The derivative of the Logistic funtion', 2, None, '___sec60'),
('The RELU function family', 2, None, '___sec61'),
('Which activation function should we use?', 2, None, '___sec62'),
('A top-down perspective on Neural networks',
2,
None,
'___sec61'),
'___sec63'),
('Limitations of supervised learning with deep networks',
2,
None,
'___sec62')]}
'___sec64')]}
end of tocinfo -->
<body>
@@ -207,37 +209,39 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._NeuralNet-bs030.html#___sec29" style="font-size: 80%;"><b>Setting up the Back propagation algorithm</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs031.html#___sec30" style="font-size: 80%;"><b>Setting up a Multi-layer perceptron model for classification</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs032.html#___sec31" style="font-size: 80%;"><b>Defining the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs033.html#___sec32" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec33" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Layers</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Weights and biases</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Improving performance</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Tensorflow</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Using Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>The RELU function family</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs033.html#___sec32" style="font-size: 80%;"><b>Example: binary classification problem</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec33" style="font-size: 80%;"><b>Developing a code for doing neural networks with back propagation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs035.html#___sec34" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs036.html#___sec35" style="font-size: 80%;"><b>Train and test datasets</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs037.html#___sec36" style="font-size: 80%;"><b>Define model and architecture</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs038.html#___sec37" style="font-size: 80%;"><b>Layers</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs039.html#___sec38" style="font-size: 80%;"><b>Weights and biases</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs040.html#___sec39" style="font-size: 80%;"><b>Feed-forward pass</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs041.html#___sec40" style="font-size: 80%;"><b>Matrix multiplications</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs042.html#___sec41" style="font-size: 80%;"><b>Choose cost function and optimizer</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs043.html#___sec42" style="font-size: 80%;"><b>Optimizing the cost function</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs044.html#___sec43" style="font-size: 80%;"><b>Regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs045.html#___sec44" style="font-size: 80%;"><b>Matrix multiplication</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs046.html#___sec45" style="font-size: 80%;"><b>Improving performance</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs047.html#___sec46" style="font-size: 80%;"><b>Full object-oriented implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs048.html#___sec47" style="font-size: 80%;"><b>Evaluate model performance on test data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs049.html#___sec48" style="font-size: 80%;"><b>Adjust hyperparameters</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs050.html#___sec49" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs051.html#___sec50" style="font-size: 80%;"><b>scikit-learn implementation</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs052.html#___sec51" style="font-size: 80%;"><b>Visualization</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs053.html#___sec52" style="font-size: 80%;"><b>Building neural networks in Tensorflow and Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs054.html#___sec53" style="font-size: 80%;"><b>Tensorflow</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs055.html#___sec54" style="font-size: 80%;"><b>Collect and pre-process data</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs056.html#___sec55" style="font-size: 80%;"><b>Using TensorFlow backend</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs057.html#___sec56" style="font-size: 80%;"><b>Optimizing and using gradient descent</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs058.html#___sec57" style="font-size: 80%;"><b>Using Keras</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs059.html#___sec58" style="font-size: 80%;"><b>Which activation function should I use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs060.html#___sec59" style="font-size: 80%;"><b>Is the Logistic activation function (Sigmoid) our choice?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs061.html#___sec60" style="font-size: 80%;"><b>The derivative of the Logistic funtion</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs062.html#___sec61" style="font-size: 80%;"><b>The RELU function family</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs063.html#___sec62" style="font-size: 80%;"><b>Which activation function should we use?</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs064.html#___sec63" style="font-size: 80%;"><b>A top-down perspective on Neural networks</b></a></li>
<!-- navigation toc: --> <li><a href="._NeuralNet-bs065.html#___sec64" style="font-size: 80%;"><b>Limitations of supervised learning with deep networks</b></a></li>
</ul>
</li>
@@ -251,101 +255,22 @@ MathJax.Hub.Config({
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0034"></a>
<!-- !split -->
<!-- !split -->
<h2 id="___sec33" class="anchor">Collect and pre-process data </h2>
<h2 id="___sec33" class="anchor">Developing a code for doing neural networks with back propagation </h2>
<p>
Here we will be using the MNIST dataset, which is readily available through the <b>scikit-learn</b>
package. You may also find it for example <a href="http://yann.lecun.com/exdb/mnist/" target="_self">here</a>.
The <em>MNIST</em> (Modified National Institute of Standards and Technology) database is a large database
of handwritten digits that is commonly used for training various image processing systems.
The MNIST dataset consists of 70 000 images of size 28x28 pixels, each labeled from 0 to 9.
The scikit-learn dataset we will use consists of a selection of 1797 images of size \( 8\times 8 \) collected and processed from this database.
One can identify a set of key steps when using neural networks to solve supervised learning problems:
<p>
To feed data into a feed-forward neural network we need to represent
the inputs as a feature matrix \( X = (n_{inputs}, n_{features}) \). Each
row represents an <em>input</em>, in this case a handwritten digit, and
each column represents a <em>feature</em>, in this case a pixel. The
correct answers, also known as <em>labels</em> or <em>targets</em> are
represented as a 1D array of integers
\( Y = (n_{inputs}) = (5, 3, 1, 8,...) \).
<ol>
<li> Collect and pre-process data</li>
<li> Define model and architecture</li>
<li> Choose cost function and optimizer</li>
<li> Train the model</li>
<li> Evaluate model performance on test data</li>
<li> Adjust hyperparameters (if necessary, network architecture)</li>
</ol>
<p>
As an example, say we want to build a neural network using supervised learning to predict Body-Mass Index (BMI) from
measurements of height (in m)
and weight (in kg). If we have measurements of 5 people the feature matrix could be for example:
$$ X = \begin{bmatrix}
1.85 &amp; 81\\
1.71 &amp; 65\\
1.95 &amp; 103\\
1.55 &amp; 42\\
1.63 &amp; 56
\end{bmatrix} ,$$
<p>
and the targets would be:
$$ Y = (23.7, 22.2, 27.1, 17.5, 21.1) $$
<p>
Since each input image is a 2D matrix, we need to flatten the image
(i.e. "unravel" the 2D matrix into a 1D array) to turn the data into a
feature matrix. This means we lose all spatial information in the
image, such as locality and translational invariance. More complicated
architectures such as Convolutional Neural Networks can take advantage
of such information, and are most commonly applied when analyzing
images.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># import necessary packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> datasets
<span style="color: #408080; font-style: italic"># ensure the same random numbers appear every time</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">0</span>)
<span style="color: #408080; font-style: italic"># display images in notebook</span>
<span style="color: #666666">%</span>matplotlib inline
plt<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;figure.figsize&#39;</span>] <span style="color: #666666">=</span> (<span style="color: #666666">12</span>,<span style="color: #666666">12</span>)
<span style="color: #408080; font-style: italic"># download MNIST dataset</span>
digits <span style="color: #666666">=</span> datasets<span style="color: #666666">.</span>load_digits()
<span style="color: #408080; font-style: italic"># define inputs and labels</span>
inputs <span style="color: #666666">=</span> digits<span style="color: #666666">.</span>images
labels <span style="color: #666666">=</span> digits<span style="color: #666666">.</span>target
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;inputs = (n_inputs, pixel_width, pixel_height) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(inputs<span style="color: #666666">.</span>shape))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;labels = (n_inputs) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(labels<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># flatten the image</span>
<span style="color: #408080; font-style: italic"># the value -1 means dimension is inferred from the remaining dimensions: 8x8 = 64</span>
n_inputs <span style="color: #666666">=</span> <span style="color: #008000">len</span>(inputs)
inputs <span style="color: #666666">=</span> inputs<span style="color: #666666">.</span>reshape(n_inputs, <span style="color: #666666">-1</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;X = (n_inputs, n_features) = &quot;</span> <span style="color: #666666">+</span> <span style="color: #008000">str</span>(inputs<span style="color: #666666">.</span>shape))
<span style="color: #408080; font-style: italic"># choose some random images to display</span>
indices <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(n_inputs)
random_indices <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>choice(indices, size<span style="color: #666666">=5</span>)
<span style="color: #008000; font-weight: bold">for</span> i, image <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">enumerate</span>(digits<span style="color: #666666">.</span>images[random_indices]):
plt<span style="color: #666666">.</span>subplot(<span style="color: #666666">1</span>, <span style="color: #666666">5</span>, i<span style="color: #666666">+1</span>)
plt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&#39;off&#39;</span>)
plt<span style="color: #666666">.</span>imshow(image, cmap<span style="color: #666666">=</span>plt<span style="color: #666666">.</span>cm<span style="color: #666666">.</span>gray_r, interpolation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;nearest&#39;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Label: </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> digits<span style="color: #666666">.</span>target[random_indices[i]])
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
@@ -371,7 +296,7 @@ plt<span style="color: #666666">.</span>show()
<li><a href="._NeuralNet-bs042.html">43</a></li>
<li><a href="._NeuralNet-bs043.html">44</a></li>
<li><a href="">...</a></li>
<li><a href="._NeuralNet-bs063.html">64</a></li>
<li><a href="._NeuralNet-bs065.html">66</a></li>
<li><a href="._NeuralNet-bs035.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->