lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

buidl

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-12-08 06:59:22 +01:00
parent d760cdfa85
commit 7ca589a654
94 changed files with 23333 additions and 2210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter9.html
+53 -38
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>12. Neural networks &#8212; Applied Data Analysis and Machine Learning</title>
<title>13. Neural networks &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -53,8 +53,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="13. Building a Feed Forward Neural Network" href="chapter10.html" />
<link rel="prev" title="11. Basic ideas of the Principal Component Analysis (PCA)" href="chapter8.html" />
<link rel="next" title="14. Building a Feed Forward Neural Network" href="chapter10.html" />
<link rel="prev" title="12. Clustering and Unsupervised Learning" href="clustering.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -199,6 +199,11 @@
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p class="caption" role="heading">
<span class="caption-text">
@@ -208,17 +213,27 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
14. Solving Differential Equations with Deep Learning
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
@@ -294,76 +309,76 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#feed-forward-neural-networks">
12.1. Feed-forward neural networks
13.1. Feed-forward neural networks
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#convolutional-neural-network">
12.2. Convolutional Neural Network
13.2. Convolutional Neural Network
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#recurrent-neural-networks">
12.3. Recurrent neural networks
13.3. Recurrent neural networks
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-types-of-networks">
12.4. Other types of networks
13.4. Other types of networks
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#multilayer-perceptrons">
12.5. Multilayer perceptrons
13.5. Multilayer perceptrons
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#matrix-vector-notation-and-activation">
12.5.1. Matrix-vector notation and activation
13.5.1. Matrix-vector notation and activation
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#activation-functions">
12.5.2. Activation functions
13.5.2. Activation functions
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-multilayer-perceptron-mlp">
12.6. The multilayer perceptron (MLP)
13.6. The multilayer perceptron (MLP)
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#from-one-to-many-layers-the-universal-approximation-theorem">
12.6.1. From one to many layers, the universal approximation theorem
13.6.1. From one to many layers, the universal approximation theorem
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#deriving-the-back-propagation-code-for-a-multilayer-perceptron-model">
12.7. Deriving the back propagation code for a multilayer perceptron model
13.7. Deriving the back propagation code for a multilayer perceptron model
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#derivatives-and-the-chain-rule">
12.7.1. Derivatives and the chain rule
13.7.1. Derivatives and the chain rule
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#bringing-it-together-first-back-propagation-equation">
12.7.2. Bringing it together, first back propagation equation
13.7.2. Bringing it together, first back propagation equation
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#final-back-propagating-equation">
12.7.3. Final back propagating equation
13.7.3. Final back propagating equation
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#setting-up-the-back-propagation-algorithm">
12.7.4. Setting up the Back propagation algorithm
13.7.4. Setting up the Back propagation algorithm
</a>
</li>
</ul>
@@ -380,7 +395,7 @@
<div>
<div class="tex2jax_ignore mathjax_ignore section" id="neural-networks">
<h1><span class="section-number">12. </span>Neural networks<a class="headerlink" href="#neural-networks" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">13. </span>Neural networks<a class="headerlink" href="#neural-networks" title="Permalink to this headline"></a></h1>
<p>Artificial neural networks are computational systems that can learn to
perform tasks by considering examples, generally without being
programmed with any task-specific rules. It is supposed to mimic a
@@ -452,7 +467,7 @@ can be viewed as natural, more powerful extensions of supervised
learning methods such as linear and logistic regression and soft-max
methods we discussed earlier.</p>
<div class="section" id="feed-forward-neural-networks">
<h2><span class="section-number">12.1. </span>Feed-forward neural networks<a class="headerlink" href="#feed-forward-neural-networks" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.1. </span>Feed-forward neural networks<a class="headerlink" href="#feed-forward-neural-networks" title="Permalink to this headline"></a></h2>
<p>The feed-forward neural network (FFNN) was the first and simplest type
of ANNs that were devised. In this network, the information moves in
only one direction: forward through the layers.</p>
@@ -464,7 +479,7 @@ to <em>all</em> nodes in the subsequent layer, making this a so-called
<em>fully-connected</em> FFNN.</p>
</div>
<div class="section" id="convolutional-neural-network">
<h2><span class="section-number">12.2. </span>Convolutional Neural Network<a class="headerlink" href="#convolutional-neural-network" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.2. </span>Convolutional Neural Network<a class="headerlink" href="#convolutional-neural-network" title="Permalink to this headline"></a></h2>
<p>A different variant of FFNNs are <em>convolutional neural networks</em>
(CNNs), which have a connectivity pattern inspired by the animal
visual cortex. Individual neurons in the visual cortex only respond to
@@ -484,7 +499,7 @@ produces the outputs. They have wide applications in image and video
recognition.</p>
</div>
<div class="section" id="recurrent-neural-networks">
<h2><span class="section-number">12.3. </span>Recurrent neural networks<a class="headerlink" href="#recurrent-neural-networks" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.3. </span>Recurrent neural networks<a class="headerlink" href="#recurrent-neural-networks" title="Permalink to this headline"></a></h2>
<p>So far we have only mentioned ANNs where information flows in one
direction: forward. <em>Recurrent neural networks</em> on the other hand,
have connections between nodes that form directed <em>cycles</em>. This
@@ -497,7 +512,7 @@ example of such information is sentences, making recurrent NNs
especially well-suited for handwriting and speech recognition.</p>
</div>
<div class="section" id="other-types-of-networks">
<h2><span class="section-number">12.4. </span>Other types of networks<a class="headerlink" href="#other-types-of-networks" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.4. </span>Other types of networks<a class="headerlink" href="#other-types-of-networks" title="Permalink to this headline"></a></h2>
<p>There are many other kinds of ANNs that have been developed. One type
that is specifically designed for interpolation in multidimensional
space is the radial basis function (RBF) network. RBFs are typically
@@ -510,7 +525,7 @@ fully-connected FFNN. They are however usually treated as a separate
type of NN due the unusual activation functions.</p>
</div>
<div class="section" id="multilayer-perceptrons">
<h2><span class="section-number">12.5. </span>Multilayer perceptrons<a class="headerlink" href="#multilayer-perceptrons" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.5. </span>Multilayer perceptrons<a class="headerlink" href="#multilayer-perceptrons" title="Permalink to this headline"></a></h2>
<p>One uses often so-called fully-connected feed-forward neural networks
with three or more layers (an input layer, one or more hidden layers
and an output layer) consisting of neurons that have non-linear
@@ -680,7 +695,7 @@ the equation for the activations of hidden layer 2 (assuming three nodes for sim
\end{equation}
\end{split}\]</div>
<div class="section" id="matrix-vector-notation-and-activation">
<h3><span class="section-number">12.5.1. </span>Matrix-vector notation and activation<a class="headerlink" href="#matrix-vector-notation-and-activation" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.5.1. </span>Matrix-vector notation and activation<a class="headerlink" href="#matrix-vector-notation-and-activation" title="Permalink to this headline"></a></h3>
<p>The activation of node <span class="math notranslate nohighlight">\(i\)</span> in layer 2 is</p>
<!-- Equation labels as ordinary links -->
<div id="_auto7"></div>
@@ -699,7 +714,7 @@ used as input to the activation functions. For each operation
<span class="math notranslate nohighlight">\(\mathrm{W}_l \hat{y}_{l-1}\)</span> we move forward one layer.</p>
</div>
<div class="section" id="activation-functions">
<h3><span class="section-number">12.5.2. </span>Activation functions<a class="headerlink" href="#activation-functions" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.5.2. </span>Activation functions<a class="headerlink" href="#activation-functions" title="Permalink to this headline"></a></h3>
<p>A property that characterizes a neural network, other than its
connectivity, is the choice of activation function(s). As described
in, the following restrictions are imposed on an activation function
@@ -819,7 +834,7 @@ become the most popular for <em>deep neural networks</em></p>
</div>
</div>
<div class="section" id="the-multilayer-perceptron-mlp">
<h2><span class="section-number">12.6. </span>The multilayer perceptron (MLP)<a class="headerlink" href="#the-multilayer-perceptron-mlp" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.6. </span>The multilayer perceptron (MLP)<a class="headerlink" href="#the-multilayer-perceptron-mlp" title="Permalink to this headline"></a></h2>
<p>The multilayer perceptron is a very popular, and easy to implement approach, to deep learning. It consists of</p>
<ol class="simple">
<li><p>A neural network with one or more layers of nodes between the input and the output nodes.</p></li>
@@ -844,7 +859,7 @@ are learned from data. This results in an important difference between
neural networks and deep learning approaches on one side and methods
like logistic regression or linear regression and their modifications on the other side.</p>
<div class="section" id="from-one-to-many-layers-the-universal-approximation-theorem">
<h3><span class="section-number">12.6.1. </span>From one to many layers, the universal approximation theorem<a class="headerlink" href="#from-one-to-many-layers-the-universal-approximation-theorem" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.6.1. </span>From one to many layers, the universal approximation theorem<a class="headerlink" href="#from-one-to-many-layers-the-universal-approximation-theorem" title="Permalink to this headline"></a></h3>
<p>A neural network with only one layer, what we called the simple
perceptron, is best suited if we have a standard binary model with
clear (linear) boundaries between the outcomes. As such it could
@@ -865,7 +880,7 @@ the potential of being universal approximators.</p>
</div>
</div>
<div class="section" id="deriving-the-back-propagation-code-for-a-multilayer-perceptron-model">
<h2><span class="section-number">12.7. </span>Deriving the back propagation code for a multilayer perceptron model<a class="headerlink" href="#deriving-the-back-propagation-code-for-a-multilayer-perceptron-model" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">13.7. </span>Deriving the back propagation code for a multilayer perceptron model<a class="headerlink" href="#deriving-the-back-propagation-code-for-a-multilayer-perceptron-model" title="Permalink to this headline"></a></h2>
<p>As we have seen now in a feed forward network, we can express the final output of our network in terms of basic matrix-vector multiplications.
The unknowwn quantities are our weights <span class="math notranslate nohighlight">\(w_{ij}\)</span> and we need to find an algorithm for changing them so that our errors are as small as possible.
This leads us to the famous <a class="reference external" href="https://www.nature.com/articles/323533a0">back propagation algorithm</a>.</p>
@@ -912,7 +927,7 @@ and their nodes. It means we have</p>
a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
\]</div>
<div class="section" id="derivatives-and-the-chain-rule">
<h3><span class="section-number">12.7.1. </span>Derivatives and the chain rule<a class="headerlink" href="#derivatives-and-the-chain-rule" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.7.1. </span>Derivatives and the chain rule<a class="headerlink" href="#derivatives-and-the-chain-rule" title="Permalink to this headline"></a></h3>
<p>From the definition of the activation <span class="math notranslate nohighlight">\(z_j^l\)</span> we have</p>
<div class="math notranslate nohighlight">
\[
@@ -946,7 +961,7 @@ a_j^l = f(z_j^l) = \frac{1}{1+\exp{-(z_j^l)}}.
\]</div>
</div>
<div class="section" id="bringing-it-together-first-back-propagation-equation">
<h3><span class="section-number">12.7.2. </span>Bringing it together, first back propagation equation<a class="headerlink" href="#bringing-it-together-first-back-propagation-equation" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.7.2. </span>Bringing it together, first back propagation equation<a class="headerlink" href="#bringing-it-together-first-back-propagation-equation" title="Permalink to this headline"></a></h3>
<p>We have thus</p>
<div class="math notranslate nohighlight">
\[
@@ -1045,7 +1060,7 @@ to do so we need to represent the error in the layer before the final
one <span class="math notranslate nohighlight">\(L-1\)</span> in terms of the errors in the final output layer.</p>
</div>
<div class="section" id="final-back-propagating-equation">
<h3><span class="section-number">12.7.3. </span>Final back propagating equation<a class="headerlink" href="#final-back-propagating-equation" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.7.3. </span>Final back propagating equation<a class="headerlink" href="#final-back-propagating-equation" title="Permalink to this headline"></a></h3>
<p>We have that (replacing <span class="math notranslate nohighlight">\(L\)</span> with a general layer <span class="math notranslate nohighlight">\(l\)</span>)</p>
<div class="math notranslate nohighlight">
\[
@@ -1070,7 +1085,7 @@ z_j^{l+1} = \sum_{i=1}^{M_{l}}w_{ij}^{l+1}a_i^{l}+b_j^{l+1},
<p>We are now ready to set up the algorithm for back propagation and learning the weights and biases.</p>
</div>
<div class="section" id="setting-up-the-back-propagation-algorithm">
<h3><span class="section-number">12.7.4. </span>Setting up the Back propagation algorithm<a class="headerlink" href="#setting-up-the-back-propagation-algorithm" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">13.7.4. </span>Setting up the Back propagation algorithm<a class="headerlink" href="#setting-up-the-back-propagation-algorithm" title="Permalink to this headline"></a></h3>
<p>The four equations provide us with a way of computing the gradient of the cost function. Let us write this out in the form of an algorithm.</p>
<p>First, we set up the input data <span class="math notranslate nohighlight">\(\hat{x}\)</span> and the activations
<span class="math notranslate nohighlight">\(\hat{z}_1\)</span> of the input layer and compute the activation function and
@@ -1134,11 +1149,11 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<div class='prev-next-bottom'>
<div id="prev">
<a class="left-prev" href="chapter8.html" title="previous page">
<a class="left-prev" href="clustering.html" title="previous page">
<i class="prevnext-label fas fa-angle-left"></i>
<div class="prevnext-info">
<p class="prevnext-label">previous</p>
<p class="prevnext-title"><span class="section-number">11. </span>Basic ideas of the Principal Component Analysis (PCA)</p>
<p class="prevnext-title"><span class="section-number">12. </span>Clustering and Unsupervised Learning</p>
</div>
</a>
</div>
@@ -1146,7 +1161,7 @@ Here it is convenient to use stochastic gradient descent (see the examples below
<a class="right-next" href="chapter10.html" title="next page">
<div class="prevnext-info">
<p class="prevnext-label">next</p>
<p class="prevnext-title"><span class="section-number">13. </span>Building a Feed Forward Neural Network</p>
<p class="prevnext-title"><span class="section-number">14. </span>Building a Feed Forward Neural Network</p>
</div>
<i class="prevnext-label fas fa-angle-right"></i>
</a>