lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:05:16 +02:00
parent 30d664a82b
commit ad92592adb
76 changed files with 5646 additions and 2878 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter10.html
+72 -72
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>13. Building a Feed Forward Neural Network &#8212; Applied Data Analysis and Machine Learning</title>
<title>14. Building a Feed Forward Neural Network &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -54,7 +54,7 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="prev" title="12. Neural networks" href="chapter9.html" />
<link rel="prev" title="13. Neural networks" href="chapter9.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -138,17 +138,17 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression, basic Elements
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Resampling Methods
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Ridge and Lasso Regression
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
@@ -158,7 +158,7 @@
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
7. Support Vector Machines, overarching aims
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
@@ -170,12 +170,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
8. Decision trees, overarching aims
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
@@ -187,12 +187,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
10. Basic ideas of the Principal Component Analysis (PCA)
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="Clustering.html">
11. Clustering Analysis
12. Clustering Analysis
</a>
</li>
</ul>
@@ -204,12 +204,12 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
</ul>
@@ -285,151 +285,151 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#defining-the-cost-function">
13.1. Defining the cost function
14.1. Defining the cost function
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#example-binary-classification-problem">
13.1.1. Example: binary classification problem
14.1.1. Example: binary classification problem
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-softmax-function">
13.1.2. The Softmax function
14.1.2. The Softmax function
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#developing-a-code-for-doing-neural-networks-with-back-propagation">
13.2. Developing a code for doing neural networks with back propagation
14.2. Developing a code for doing neural networks with back propagation
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#collect-and-pre-process-data">
13.2.1. Collect and pre-process data
14.2.1. Collect and pre-process data
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#train-and-test-datasets">
13.2.2. Train and test datasets
14.2.2. Train and test datasets
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#define-model-and-architecture">
13.2.3. Define model and architecture
14.2.3. Define model and architecture
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#layers">
13.2.4. Layers
14.2.4. Layers
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#feed-forward-pass">
13.2.5. Feed-forward pass
14.2.5. Feed-forward pass
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#choose-cost-function-and-optimizer">
13.2.6. Choose cost function and optimizer
14.2.6. Choose cost function and optimizer
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#optimizing-the-cost-function">
13.2.7. Optimizing the cost function
14.2.7. Optimizing the cost function
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#regularization">
13.2.8. Regularization
14.2.8. Regularization
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#matrix-multiplication">
13.2.9. Matrix multiplication
14.2.9. Matrix multiplication
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#improving-performance">
13.3. Improving performance
14.3. Improving performance
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#evaluate-model-performance-on-test-data">
13.4. Evaluate model performance on test data
14.4. Evaluate model performance on test data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#adjust-hyperparameters">
13.5. Adjust hyperparameters
14.5. Adjust hyperparameters
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#visualization">
13.6. Visualization
14.6. Visualization
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#scikit-learn-implementation">
13.7. scikit-learn implementation
14.7. scikit-learn implementation
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#id1">
13.8. Visualization
14.8. Visualization
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#building-neural-networks-in-tensorflow-and-keras">
13.9. Building neural networks in Tensorflow and Keras
14.9. Building neural networks in Tensorflow and Keras
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-breast-cancer-data-now-with-keras">
13.10. The Breast Cancer Data, now with Keras
14.10. The Breast Cancer Data, now with Keras
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#fine-tuning-neural-network-hyperparameters">
13.11. Fine-tuning neural network hyperparameters
14.11. Fine-tuning neural network hyperparameters
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#which-activation-function-should-i-use">
13.12. Which activation function should I use?
14.12. Which activation function should I use?
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-relu-function-family">
13.13. The RELU function family
14.13. The RELU function family
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#batch-normalization">
13.14. Batch Normalization
14.14. Batch Normalization
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#dropout">
13.15. Dropout
14.15. Dropout
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#gradient-clipping">
13.16. Gradient Clipping
14.16. Gradient Clipping
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-top-down-perspective-on-neural-networks">
13.17. A top-down perspective on Neural networks
14.17. A top-down perspective on Neural networks
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#limitations-of-supervised-learning-with-deep-networks">
13.18. Limitations of supervised learning with deep networks
14.18. Limitations of supervised learning with deep networks
</a>
</li>
</ul>
@@ -444,7 +444,7 @@
<div>
<div class="section" id="building-a-feed-forward-neural-network">
<h1><span class="section-number">13. </span>Building a Feed Forward Neural Network<a class="headerlink" href="#building-a-feed-forward-neural-network" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">14. </span>Building a Feed Forward Neural Network<a class="headerlink" href="#building-a-feed-forward-neural-network" title="Permalink to this headline"></a></h1>
<p>We are now gong to develop an example based on the MNIST data
base. This is a classification problem and we need to use our
cross-entropy function we discussed in connection with logistic
@@ -472,7 +472,7 @@ P(y = 1 \mid \hat{x}, \hat{\theta}) = 1 - P(y = 0 \mid \hat{x}, \hat{\theta}) ,
<p>where <span class="math notranslate nohighlight">\(y \in \{0, 1\}\)</span> and <span class="math notranslate nohighlight">\(\hat{\theta}\)</span> represents the weights and biases
of our network.</p>
<div class="section" id="defining-the-cost-function">
<h2><span class="section-number">13.1. </span>Defining the cost function<a class="headerlink" href="#defining-the-cost-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.1. </span>Defining the cost function<a class="headerlink" href="#defining-the-cost-function" title="Permalink to this headline"></a></h2>
<p>Our cost function is given as (see the Logistic regression lectures)</p>
<div class="math notranslate nohighlight">
\[
@@ -510,7 +510,7 @@ P(\mathcal{D} \mid \hat{\theta}) = \prod_{i=1}^n \prod_{c=0}^{C-1} [P(y_{ic} = 1
<p>See the logistic regression lectures for a full definition of the cost function.</p>
<p>The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before!</p>
<div class="section" id="example-binary-classification-problem">
<h3><span class="section-number">13.1.1. </span>Example: binary classification problem<a class="headerlink" href="#example-binary-classification-problem" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.1.1. </span>Example: binary classification problem<a class="headerlink" href="#example-binary-classification-problem" title="Permalink to this headline"></a></h3>
<p>As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters <span class="math notranslate nohighlight">\(\beta\)</span> as</p>
<div class="math notranslate nohighlight">
\[
@@ -552,7 +552,7 @@ Our cost function at the final layer <span class="math notranslate nohighlight">
<p>In case we use another activation function than the logistic one, we need to evaluate other derivatives.</p>
</div>
<div class="section" id="the-softmax-function">
<h3><span class="section-number">13.1.2. </span>The Softmax function<a class="headerlink" href="#the-softmax-function" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.1.2. </span>The Softmax function<a class="headerlink" href="#the-softmax-function" title="Permalink to this headline"></a></h3>
<p>In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation <span class="math notranslate nohighlight">\(z_i^l\)</span>, that is we need</p>
<div class="math notranslate nohighlight">
\[
@@ -573,7 +573,7 @@ f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{m=1}^K\exp{(z_m^l)}}.
</div>
</div>
<div class="section" id="developing-a-code-for-doing-neural-networks-with-back-propagation">
<h2><span class="section-number">13.2. </span>Developing a code for doing neural networks with back propagation<a class="headerlink" href="#developing-a-code-for-doing-neural-networks-with-back-propagation" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.2. </span>Developing a code for doing neural networks with back propagation<a class="headerlink" href="#developing-a-code-for-doing-neural-networks-with-back-propagation" title="Permalink to this headline"></a></h2>
<p>One can identify a set of key steps when using neural networks to solve supervised learning problems:</p>
<ol class="simple">
<li><p>Collect and pre-process data</p></li>
@@ -584,7 +584,7 @@ f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{m=1}^K\exp{(z_m^l)}}.
<li><p>Adjust hyperparameters (if necessary, network architecture)</p></li>
</ol>
<div class="section" id="collect-and-pre-process-data">
<h3><span class="section-number">13.2.1. </span>Collect and pre-process data<a class="headerlink" href="#collect-and-pre-process-data" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.1. </span>Collect and pre-process data<a class="headerlink" href="#collect-and-pre-process-data" title="Permalink to this headline"></a></h3>
<p>Here we will be using the MNIST dataset, which is readily available through the <strong>scikit-learn</strong>
package. You may also find it for example <a class="reference external" href="http://yann.lecun.com/exdb/mnist/">here</a>.<br />
The <em>MNIST</em> (Modified National Institute of Standards and Technology) database is a large database
@@ -679,7 +679,7 @@ X = (n_inputs, n_features) = (1797, 64)
</div>
</div>
<div class="section" id="train-and-test-datasets">
<h3><span class="section-number">13.2.2. </span>Train and test datasets<a class="headerlink" href="#train-and-test-datasets" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.2. </span>Train and test datasets<a class="headerlink" href="#train-and-test-datasets" title="Permalink to this headline"></a></h3>
<p>Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.</p>
<p>We will reserve <span class="math notranslate nohighlight">\(80 \%\)</span> of our dataset for training and <span class="math notranslate nohighlight">\(20 \%\)</span> for testing.</p>
<p>It is important that the train and test datasets are drawn randomly from our dataset, to ensure
@@ -728,7 +728,7 @@ Number of test images: 360
</div>
</div>
<div class="section" id="define-model-and-architecture">
<h3><span class="section-number">13.2.3. </span>Define model and architecture<a class="headerlink" href="#define-model-and-architecture" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.3. </span>Define model and architecture<a class="headerlink" href="#define-model-and-architecture" title="Permalink to this headline"></a></h3>
<p>Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation <span class="math notranslate nohighlight">\(y\)</span> of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have</p>
<div class="math notranslate nohighlight">
\[ z = \sum_{i=1}^n w_i a_i ,\]</div>
@@ -758,7 +758,7 @@ We will be using the sigmoid function <span class="math notranslate nohighlight"
<p>which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions.</p>
</div>
<div class="section" id="layers">
<h3><span class="section-number">13.2.4. </span>Layers<a class="headerlink" href="#layers" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.4. </span>Layers<a class="headerlink" href="#layers" title="Permalink to this headline"></a></h3>
<ul class="simple">
<li><p>Input</p></li>
</ul>
@@ -815,7 +815,7 @@ of values. Without it, any input with the value 0 will be mapped to zero (before
</div>
</div>
<div class="section" id="feed-forward-pass">
<h3><span class="section-number">13.2.5. </span>Feed-forward pass<a class="headerlink" href="#feed-forward-pass" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.5. </span>Feed-forward pass<a class="headerlink" href="#feed-forward-pass" title="Permalink to this headline"></a></h3>
<p>Denote <span class="math notranslate nohighlight">\(F\)</span> the number of features, <span class="math notranslate nohighlight">\(H\)</span> the number of hidden neurons and <span class="math notranslate nohighlight">\(C\)</span> the number of categories.<br />
For each input image we calculate a weighted sum of input features (pixel values) to each neuron <span class="math notranslate nohighlight">\(j\)</span> in the hidden layer <span class="math notranslate nohighlight">\(l\)</span>:</p>
<div class="math notranslate nohighlight">
@@ -908,7 +908,7 @@ correct label for image 0: 6
</div>
</div>
<div class="section" id="choose-cost-function-and-optimizer">
<h3><span class="section-number">13.2.6. </span>Choose cost function and optimizer<a class="headerlink" href="#choose-cost-function-and-optimizer" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.6. </span>Choose cost function and optimizer<a class="headerlink" href="#choose-cost-function-and-optimizer" title="Permalink to this headline"></a></h3>
<p>To measure how well our neural network is doing we need to introduce a cost function.<br />
We will call the function that gives the error of a single sample output the <em>loss</em> function, and the function
that gives the total error of our network across all samples the <em>cost</em> function.
@@ -927,7 +927,7 @@ probability of the correct category <span class="math notranslate nohighlight">\
you got the correct label. The probability of category <span class="math notranslate nohighlight">\(c\)</span> is given by the softmax function. The vector <span class="math notranslate nohighlight">\(\hat{\theta}\)</span> represents the parameters of our network, i.e. all the weights and biases.</p>
</div>
<div class="section" id="optimizing-the-cost-function">
<h3><span class="section-number">13.2.7. </span>Optimizing the cost function<a class="headerlink" href="#optimizing-the-cost-function" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.7. </span>Optimizing the cost function<a class="headerlink" href="#optimizing-the-cost-function" title="Permalink to this headline"></a></h3>
<p>The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a <em>local</em> minimum of the cost function.<br />
Each parameter <span class="math notranslate nohighlight">\(\theta\)</span> is iteratively adjusted according to the rule</p>
@@ -953,7 +953,7 @@ We denote each minibatch <span class="math notranslate nohighlight">\(B_k\)</spa
<p>The various optmization methods, with codes and algorithms, are discussed in our lectures on <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html">Gradient descent approaches</a>.</p>
</div>
<div class="section" id="regularization">
<h3><span class="section-number">13.2.8. </span>Regularization<a class="headerlink" href="#regularization" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.8. </span>Regularization<a class="headerlink" href="#regularization" title="Permalink to this headline"></a></h3>
<p>It is common to add an extra term to the cost function, proportional
to the size of the weights. This is equivalent to constraining the
size of the weights, so that they do not grow out of control.
@@ -976,7 +976,7 @@ above. This is a clever use of the chain rule that allows us to
calculate the gradient efficently.</p>
</div>
<div class="section" id="matrix-multiplication">
<h3><span class="section-number">13.2.9. </span>Matrix multiplication<a class="headerlink" href="#matrix-multiplication" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">14.2.9. </span>Matrix multiplication<a class="headerlink" href="#matrix-multiplication" title="Permalink to this headline"></a></h3>
<p>To more efficently train our network these equations are implemented using matrix operations.<br />
The error in the output layer is calculated simply as, with <span class="math notranslate nohighlight">\(\hat{t}\)</span> being our targets,</p>
<div class="math notranslate nohighlight">
@@ -1089,7 +1089,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
</div>
</div>
<div class="section" id="improving-performance">
<h2><span class="section-number">13.3. </span>Improving performance<a class="headerlink" href="#improving-performance" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.3. </span>Improving performance<a class="headerlink" href="#improving-performance" title="Permalink to this headline"></a></h2>
<p>As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.<br />
In order to obtain a network that does something useful, we will have to do a bit more work.</p>
<p>The choice of <em>hyperparameters</em> such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a <em>grid-search</em> is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates <span class="math notranslate nohighlight">\(\eta = 10^{-6}, 10^{-5},...,10^{-1}\)</span> with different regularization parameters <span class="math notranslate nohighlight">\(\lambda = 10^{-6},...,10^{-0}\)</span>.</p>
@@ -1206,7 +1206,7 @@ being realizations of this object with different hyperparameters. An implementat
</div>
</div>
<div class="section" id="evaluate-model-performance-on-test-data">
<h2><span class="section-number">13.4. </span>Evaluate model performance on test data<a class="headerlink" href="#evaluate-model-performance-on-test-data" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.4. </span>Evaluate model performance on test data<a class="headerlink" href="#evaluate-model-performance-on-test-data" title="Permalink to this headline"></a></h2>
<p>To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.<br />
We measure the performance of the network using the <em>accuracy</em> score.<br />
The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of <span class="math notranslate nohighlight">\(1\)</span>.</p>
@@ -1242,7 +1242,7 @@ The accuracy is as you would expect just the number of images correctly labeled
</div>
</div>
<div class="section" id="adjust-hyperparameters">
<h2><span class="section-number">13.5. </span>Adjust hyperparameters<a class="headerlink" href="#adjust-hyperparameters" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.5. </span>Adjust hyperparameters<a class="headerlink" href="#adjust-hyperparameters" title="Permalink to this headline"></a></h2>
<p>We now perform a grid search to find the optimal hyperparameters for the network.<br />
Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around <span class="math notranslate nohighlight">\(98\%\)</span> (<span class="math notranslate nohighlight">\(2\%\)</span> error rate).</p>
<div class="cell docutils container">
@@ -1530,7 +1530,7 @@ Accuracy score on test set: 0.07777777777777778
</div>
</div>
<div class="section" id="visualization">
<h2><span class="section-number">13.6. </span>Visualization<a class="headerlink" href="#visualization" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.6. </span>Visualization<a class="headerlink" href="#visualization" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># visual representation of grid search</span>
@@ -1580,7 +1580,7 @@ Accuracy score on test set: 0.07777777777777778
</div>
</div>
<div class="section" id="scikit-learn-implementation">
<h2><span class="section-number">13.7. </span>scikit-learn implementation<a class="headerlink" href="#scikit-learn-implementation" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.7. </span>scikit-learn implementation<a class="headerlink" href="#scikit-learn-implementation" title="Permalink to this headline"></a></h2>
<p><strong>scikit-learn</strong> focuses more
on traditional machine learning methods, such as regression,
clustering, decision trees, etc. As such, it has only two types of
@@ -1939,7 +1939,7 @@ Accuracy score on test set: 0.1527777777777778
</div>
</div>
<div class="section" id="id1">
<h2><span class="section-number">13.8. </span>Visualization<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.8. </span>Visualization<a class="headerlink" href="#id1" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># optional</span>
@@ -1986,7 +1986,7 @@ Accuracy score on test set: 0.1527777777777778
</div>
</div>
<div class="section" id="building-neural-networks-in-tensorflow-and-keras">
<h2><span class="section-number">13.9. </span>Building neural networks in Tensorflow and Keras<a class="headerlink" href="#building-neural-networks-in-tensorflow-and-keras" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.9. </span>Building neural networks in Tensorflow and Keras<a class="headerlink" href="#building-neural-networks-in-tensorflow-and-keras" title="Permalink to this headline"></a></h2>
<p>Now we want to build on the experience gained from our neural network implementation in NumPy and scikit-learn
and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy
and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer.</p>
@@ -2214,7 +2214,7 @@ If you have Anaconda installed you may run the following command</p>
</div>
</div>
<div class="section" id="the-breast-cancer-data-now-with-keras">
<h2><span class="section-number">13.10. </span>The Breast Cancer Data, now with Keras<a class="headerlink" href="#the-breast-cancer-data-now-with-keras" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.10. </span>The Breast Cancer Data, now with Keras<a class="headerlink" href="#the-breast-cancer-data-now-with-keras" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span>
@@ -2388,7 +2388,7 @@ If you have Anaconda installed you may run the following command</p>
</div>
</div>
<div class="section" id="fine-tuning-neural-network-hyperparameters">
<h2><span class="section-number">13.11. </span>Fine-tuning neural network hyperparameters<a class="headerlink" href="#fine-tuning-neural-network-hyperparameters" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.11. </span>Fine-tuning neural network hyperparameters<a class="headerlink" href="#fine-tuning-neural-network-hyperparameters" title="Permalink to this headline"></a></h2>
<p>The flexibility of neural networks is also one of their main
drawbacks: there are many hyperparameters to tweak. Not only can you
use any imaginable network topology (how neurons/nodes are interconnected),
@@ -2419,7 +2419,7 @@ of training data. However, you will rarely have to train such networks from scra
common to reuse parts of a pretrained state-of-the-art network that performs a similar task.</p>
</div>
<div class="section" id="which-activation-function-should-i-use">
<h2><span class="section-number">13.12. </span>Which activation function should I use?<a class="headerlink" href="#which-activation-function-should-i-use" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.12. </span>Which activation function should I use?<a class="headerlink" href="#which-activation-function-should-i-use" title="Permalink to this headline"></a></h2>
<p>The Back propagation algorithm we derived above works by going from
the output layer to the input layer, propagating the error gradient on
the way. Once the algorithm has computed the gradient of the cost
@@ -2484,7 +2484,7 @@ it does not saturate for positive values (and also because it is quite
fast to compute).</p>
</div>
<div class="section" id="the-relu-function-family">
<h2><span class="section-number">13.13. </span>The RELU function family<a class="headerlink" href="#the-relu-function-family" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.13. </span>The RELU function family<a class="headerlink" href="#the-relu-function-family" title="Permalink to this headline"></a></h2>
<p>The ReLU activation function suffers from a problem known as the dying
ReLUs: during training, some neurons effectively die, meaning they
stop outputting anything other than 0.</p>
@@ -2521,7 +2521,7 @@ bootstrap to evaluate other activation functions.</p>
</ul>
</div>
<div class="section" id="batch-normalization">
<h2><span class="section-number">13.14. </span>Batch Normalization<a class="headerlink" href="#batch-normalization" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.14. </span>Batch Normalization<a class="headerlink" href="#batch-normalization" title="Permalink to this headline"></a></h2>
<p>Batch Normalization
aims to address the vanishing/exploding gradients problems, and more generally the problem that the
distribution of each layer’s inputs changes during training, as the parameters of the previous layers change.</p>
@@ -2534,7 +2534,7 @@ standard deviation. It does so by evaluating the mean and standard deviation of
mini-batch, from this the name batch normalization.</p>
</div>
<div class="section" id="dropout">
<h2><span class="section-number">13.15. </span>Dropout<a class="headerlink" href="#dropout" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.15. </span>Dropout<a class="headerlink" href="#dropout" title="Permalink to this headline"></a></h2>
<p>It is a fairly simple algorithm: at every training step, every neuron (including the input neurons but
excluding the output neurons) has a probability <span class="math notranslate nohighlight">\(p\)</span> of being temporarily dropped out, meaning it will be
entirely ignored during this training step, but it may be active during the next step.</p>
@@ -2543,7 +2543,7 @@ hyperparameter <span class="math notranslate nohighlight">\(p\)</span> is called
It is viewed as one of the most popular regularization techniques.</p>
</div>
<div class="section" id="gradient-clipping">
<h2><span class="section-number">13.16. </span>Gradient Clipping<a class="headerlink" href="#gradient-clipping" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.16. </span>Gradient Clipping<a class="headerlink" href="#gradient-clipping" title="Permalink to this headline"></a></h2>
<p>A popular technique to lessen the exploding gradients problem is to simply clip the gradients during
backpropagation so that they never exceed some threshold (this is mostly useful for recurrent neural
networks).</p>
@@ -2552,7 +2552,7 @@ networks).</p>
Normalization is preferred.</p>
</div>
<div class="section" id="a-top-down-perspective-on-neural-networks">
<h2><span class="section-number">13.17. </span>A top-down perspective on Neural networks<a class="headerlink" href="#a-top-down-perspective-on-neural-networks" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.17. </span>A top-down perspective on Neural networks<a class="headerlink" href="#a-top-down-perspective-on-neural-networks" title="Permalink to this headline"></a></h2>
<p>The first thing we would like to do is divide the data into two or three
parts. A training set, a validation or dev (development) set, and a
test set. The test set is the data on which we want to make
@@ -2586,7 +2586,7 @@ can serve as another important diagnostic when using DNNs for
supervised learning.</p>
</div>
<div class="section" id="limitations-of-supervised-learning-with-deep-networks">
<h2><span class="section-number">13.18. </span>Limitations of supervised learning with deep networks<a class="headerlink" href="#limitations-of-supervised-learning-with-deep-networks" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">14.18. </span>Limitations of supervised learning with deep networks<a class="headerlink" href="#limitations-of-supervised-learning-with-deep-networks" title="Permalink to this headline"></a></h2>
<p>Like all statistical methods, supervised learning using neural
networks has important limitations. This is especially important when
one seeks to apply these methods, especially to physics problems. Like
@@ -2630,7 +2630,7 @@ features).</p>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter9.html" title="previous page"><span class="section-number">12. </span>Neural networks</a>
<a class='left-prev' id="prev-link" href="chapter9.html" title="previous page"><span class="section-number">13. </span>Neural networks</a>
</div>