buidl

2021-12-08 06:59:22 +01:00
parent d760cdfa85
commit 7ca589a654
94 changed files with 23333 additions and 2210 deletions
@@ -5,7 +5,7 @@
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <title>13. Building a Feed Forward Neural Network &#8212; Applied Data Analysis and Machine Learning</title>
+    <title>14. Building a Feed Forward Neural Network &#8212; Applied Data Analysis and Machine Learning</title>
    
  <link href="_static/css/theme.css" rel="stylesheet" />
  <link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -53,8 +53,8 @@
    <script async="async" src="_static/sphinx-thebe.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
-    <link rel="next" title="14. Solving Differential Equations with Deep Learning" href="chapter11.html" />
-    <link rel="prev" title="12. Neural networks" href="chapter9.html" />
+    <link rel="next" title="15. Solving Differential Equations with Deep Learning" href="chapter11.html" />
+    <link rel="prev" title="13. Neural networks" href="chapter9.html" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="docsearch:language" content="en" />
    
@@ -199,6 +199,11 @@
   11. Basic ideas of the Principal Component Analysis (PCA)
  </a>
 </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="clustering.html">
+   12. Clustering and Unsupervised Learning
+  </a>
+ </li>
 </ul>
 <p class="caption" role="heading">
 <span class="caption-text">
@@ -208,17 +213,27 @@
 <ul class="current nav bd-sidenav">
 <li class="toctree-l1">
  <a class="reference internal" href="chapter9.html">
-   12. Neural networks
+   13. Neural networks
  </a>
 </li>
 <li class="toctree-l1 current active">
  <a class="current reference internal" href="#">
-   13. Building a Feed Forward Neural Network
+   14. Building a Feed Forward Neural Network
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="chapter11.html">
-   14. Solving Differential Equations  with Deep Learning
+   15. Solving Differential Equations  with Deep Learning
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter12.html">
+   16. Convolutional Neural Networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter13.html">
+   17. Recurrent neural networks: Overarching view
  </a>
 </li>
 </ul>
@@ -294,151 +309,151 @@
                <ul class="visible nav section-nav flex-column">
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#defining-the-cost-function">
-   13.1. Defining the cost function
+   14.1. Defining the cost function
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#example-binary-classification-problem">
-     13.1.1. Example: binary classification problem
+     14.1.1. Example: binary classification problem
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#the-softmax-function">
-     13.1.2. The Softmax function
+     14.1.2. The Softmax function
    </a>
   </li>
  </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#developing-a-code-for-doing-neural-networks-with-back-propagation">
-   13.2. Developing a code for doing neural networks with back propagation
+   14.2. Developing a code for doing neural networks with back propagation
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#collect-and-pre-process-data">
-     13.2.1. Collect and pre-process data
+     14.2.1. Collect and pre-process data
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#train-and-test-datasets">
-     13.2.2. Train and test datasets
+     14.2.2. Train and test datasets
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#define-model-and-architecture">
-     13.2.3. Define model and architecture
+     14.2.3. Define model and architecture
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#layers">
-     13.2.4. Layers
+     14.2.4. Layers
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#feed-forward-pass">
-     13.2.5. Feed-forward pass
+     14.2.5. Feed-forward pass
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#choose-cost-function-and-optimizer">
-     13.2.6. Choose cost function and optimizer
+     14.2.6. Choose cost function and optimizer
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#optimizing-the-cost-function">
-     13.2.7. Optimizing the cost function
+     14.2.7. Optimizing the cost function
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#regularization">
-     13.2.8. Regularization
+     14.2.8. Regularization
    </a>
   </li>
   <li class="toc-h3 nav-item toc-entry">
    <a class="reference internal nav-link" href="#matrix-multiplication">
-     13.2.9. Matrix  multiplication
+     14.2.9. Matrix  multiplication
    </a>
   </li>
  </ul>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#improving-performance">
-   13.3. Improving performance
+   14.3. Improving performance
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#evaluate-model-performance-on-test-data">
-   13.4. Evaluate model performance on test data
+   14.4. Evaluate model performance on test data
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#adjust-hyperparameters">
-   13.5. Adjust hyperparameters
+   14.5. Adjust hyperparameters
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#visualization">
-   13.6. Visualization
+   14.6. Visualization
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#scikit-learn-implementation">
-   13.7. scikit-learn implementation
+   14.7. scikit-learn implementation
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#id1">
-   13.8. Visualization
+   14.8. Visualization
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#building-neural-networks-in-tensorflow-and-keras">
-   13.9. Building neural networks in Tensorflow and Keras
+   14.9. Building neural networks in Tensorflow and Keras
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#the-breast-cancer-data-now-with-keras">
-   13.10. The Breast Cancer Data, now with Keras
+   14.10. The Breast Cancer Data, now with Keras
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#fine-tuning-neural-network-hyperparameters">
-   13.11. Fine-tuning neural network hyperparameters
+   14.11. Fine-tuning neural network hyperparameters
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#which-activation-function-should-i-use">
-   13.12. Which activation function should I use?
+   14.12. Which activation function should I use?
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#the-relu-function-family">
-   13.13. The RELU function family
+   14.13. The RELU function family
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#batch-normalization">
-   13.14. Batch Normalization
+   14.14. Batch Normalization
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#dropout">
-   13.15. Dropout
+   14.15. Dropout
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#gradient-clipping">
-   13.16. Gradient Clipping
+   14.16. Gradient Clipping
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#a-top-down-perspective-on-neural-networks">
-   13.17. A top-down perspective on Neural networks
+   14.17. A top-down perspective on Neural networks
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
  <a class="reference internal nav-link" href="#limitations-of-supervised-learning-with-deep-networks">
-   13.18. Limitations of supervised learning with deep networks
+   14.18. Limitations of supervised learning with deep networks
  </a>
 </li>
 </ul>
@@ -452,8 +467,9 @@
        
              <div>
                
-  <div class="tex2jax_ignore mathjax_ignore section" id="building-a-feed-forward-neural-network">
-<h1><span class="section-number">13. </span>Building a Feed Forward Neural Network<a class="headerlink" href="#building-a-feed-forward-neural-network" title="Permalink to this headline">¶</a></h1>
+  <!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
+doconce format html chapter10.do.txt --no_mako --><div class="tex2jax_ignore mathjax_ignore section" id="building-a-feed-forward-neural-network">
+<h1><span class="section-number">14. </span>Building a Feed Forward Neural Network<a class="headerlink" href="#building-a-feed-forward-neural-network" title="Permalink to this headline">¶</a></h1>
 <p>We are now gong to develop an example based on the MNIST data
 base. This is a classification problem and we need to use our
 cross-entropy function we discussed in connection with logistic
@@ -481,7 +497,7 @@ P(y = 1 \mid \hat{x}, \hat{\theta}) = 1 - P(y = 0 \mid \hat{x}, \hat{\theta}) ,
 <p>where <span class="math notranslate nohighlight">\(y \in \{0, 1\}\)</span>  and <span class="math notranslate nohighlight">\(\hat{\theta}\)</span> represents the weights and biases
 of our network.</p>
 <div class="section" id="defining-the-cost-function">
-<h2><span class="section-number">13.1. </span>Defining the cost function<a class="headerlink" href="#defining-the-cost-function" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.1. </span>Defining the cost function<a class="headerlink" href="#defining-the-cost-function" title="Permalink to this headline">¶</a></h2>
 <p>Our cost function is given as (see the Logistic regression lectures)</p>
 <div class="math notranslate nohighlight">
 \[
@@ -519,7 +535,7 @@ P(\mathcal{D} \mid \hat{\theta}) = \prod_{i=1}^n \prod_{c=0}^{C-1} [P(y_{ic} = 1
 <p>See the logistic regression lectures for a full definition of the cost function.</p>
 <p>The back propagation equations need now only a small change, namely the definition of a new cost function. We are thus ready to use the same equations as before!</p>
 <div class="section" id="example-binary-classification-problem">
-<h3><span class="section-number">13.1.1. </span>Example: binary classification problem<a class="headerlink" href="#example-binary-classification-problem" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.1.1. </span>Example: binary classification problem<a class="headerlink" href="#example-binary-classification-problem" title="Permalink to this headline">¶</a></h3>
 <p>As an example of the above, relevant for project 2 as well, let us consider a binary class. As discussed in our logistic regression lectures, we defined a cost function in terms of the parameters <span class="math notranslate nohighlight">\(\beta\)</span> as</p>
 <div class="math notranslate nohighlight">
 \[
@@ -561,7 +577,7 @@ Our cost function at the final layer <span class="math notranslate nohighlight">
 <p>In case we use another activation function than the logistic one, we need to evaluate other derivatives.</p>
 </div>
 <div class="section" id="the-softmax-function">
-<h3><span class="section-number">13.1.2. </span>The Softmax function<a class="headerlink" href="#the-softmax-function" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.1.2. </span>The Softmax function<a class="headerlink" href="#the-softmax-function" title="Permalink to this headline">¶</a></h3>
 <p>In case we employ the more general case given by the Softmax equation, we need to evaluate the derivative of the activation function with respect to the activation <span class="math notranslate nohighlight">\(z_i^l\)</span>, that is we need</p>
 <div class="math notranslate nohighlight">
 \[
@@ -582,7 +598,7 @@ f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{m=1}^K\exp{(z_m^l)}}.
 </div>
 </div>
 <div class="section" id="developing-a-code-for-doing-neural-networks-with-back-propagation">
-<h2><span class="section-number">13.2. </span>Developing a code for doing neural networks with back propagation<a class="headerlink" href="#developing-a-code-for-doing-neural-networks-with-back-propagation" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.2. </span>Developing a code for doing neural networks with back propagation<a class="headerlink" href="#developing-a-code-for-doing-neural-networks-with-back-propagation" title="Permalink to this headline">¶</a></h2>
 <p>One can identify a set of key steps when using neural networks to solve supervised learning problems:</p>
 <ol class="simple">
 <li><p>Collect and pre-process data</p></li>
@@ -593,7 +609,7 @@ f(z_i^l) = \frac{\exp{(z_i^l)}}{\sum_{m=1}^K\exp{(z_m^l)}}.
 <li><p>Adjust hyperparameters (if necessary, network architecture)</p></li>
 </ol>
 <div class="section" id="collect-and-pre-process-data">
-<h3><span class="section-number">13.2.1. </span>Collect and pre-process data<a class="headerlink" href="#collect-and-pre-process-data" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.1. </span>Collect and pre-process data<a class="headerlink" href="#collect-and-pre-process-data" title="Permalink to this headline">¶</a></h3>
 <p>Here we will be using the MNIST dataset, which is readily available through the <strong>scikit-learn</strong>
 package. You may also find it for example <a class="reference external" href="http://yann.lecun.com/exdb/mnist/">here</a>.<br />
 The <em>MNIST</em> (Modified National Institute of Standards and Technology) database is a large database
@@ -683,12 +699,12 @@ labels = (n_inputs) = (1797,)
 X = (n_inputs, n_features) = (1797, 64)
 </pre></div>
 </div>
-<img alt="_images/chapter10_33_1.png" src="_images/chapter10_33_1.png" />
+<img alt="_images/chapter10_39_1.png" src="_images/chapter10_39_1.png" />
 </div>
 </div>
 </div>
 <div class="section" id="train-and-test-datasets">
-<h3><span class="section-number">13.2.2. </span>Train and test datasets<a class="headerlink" href="#train-and-test-datasets" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.2. </span>Train and test datasets<a class="headerlink" href="#train-and-test-datasets" title="Permalink to this headline">¶</a></h3>
 <p>Performing analysis before partitioning the dataset is a major error, that can lead to incorrect conclusions.</p>
 <p>We will reserve <span class="math notranslate nohighlight">\(80 \%\)</span> of our dataset for training and <span class="math notranslate nohighlight">\(20 \%\)</span> for testing.</p>
 <p>It is important that the train and test datasets are drawn randomly from our dataset, to ensure
@@ -737,7 +753,7 @@ Number of test images: 360
 </div>
 </div>
 <div class="section" id="define-model-and-architecture">
-<h3><span class="section-number">13.2.3. </span>Define model and architecture<a class="headerlink" href="#define-model-and-architecture" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.3. </span>Define model and architecture<a class="headerlink" href="#define-model-and-architecture" title="Permalink to this headline">¶</a></h3>
 <p>Our simple feed-forward neural network will consist of an <em>input</em> layer, a single <em>hidden</em> layer and an <em>output</em> layer. The activation <span class="math notranslate nohighlight">\(y\)</span> of each neuron is a weighted sum of inputs, passed through an activation function. In case of the simple perceptron model we have</p>
 <div class="math notranslate nohighlight">
 \[ z = \sum_{i=1}^n w_i a_i ,\]</div>
@@ -767,7 +783,7 @@ We will be using the sigmoid function <span class="math notranslate nohighlight"
 <p>which is inspired by probability theory (see logistic regression) and was most commonly used until about 2011. See the discussion below concerning other activation functions.</p>
 </div>
 <div class="section" id="layers">
-<h3><span class="section-number">13.2.4. </span>Layers<a class="headerlink" href="#layers" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.4. </span>Layers<a class="headerlink" href="#layers" title="Permalink to this headline">¶</a></h3>
 <ul class="simple">
 <li><p>Input</p></li>
 </ul>
@@ -824,7 +840,7 @@ of values. Without it, any input with the value 0 will be mapped to zero (before
 </div>
 </div>
 <div class="section" id="feed-forward-pass">
-<h3><span class="section-number">13.2.5. </span>Feed-forward pass<a class="headerlink" href="#feed-forward-pass" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.5. </span>Feed-forward pass<a class="headerlink" href="#feed-forward-pass" title="Permalink to this headline">¶</a></h3>
 <p>Denote <span class="math notranslate nohighlight">\(F\)</span> the number of features, <span class="math notranslate nohighlight">\(H\)</span> the number of hidden neurons and <span class="math notranslate nohighlight">\(C\)</span> the number of categories.<br />
 For each input image we calculate a weighted sum of input features (pixel values) to each neuron <span class="math notranslate nohighlight">\(j\)</span> in the hidden layer <span class="math notranslate nohighlight">\(l\)</span>:</p>
 <div class="math notranslate nohighlight">
@@ -917,7 +933,7 @@ correct label for image 0: 6
 </div>
 </div>
 <div class="section" id="choose-cost-function-and-optimizer">
-<h3><span class="section-number">13.2.6. </span>Choose cost function and optimizer<a class="headerlink" href="#choose-cost-function-and-optimizer" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.6. </span>Choose cost function and optimizer<a class="headerlink" href="#choose-cost-function-and-optimizer" title="Permalink to this headline">¶</a></h3>
 <p>To measure how well our neural network is doing we need to introduce a cost function.<br />
 We will call the function that gives the error of a single sample output the <em>loss</em> function, and the function
 that gives the total error of our network across all samples the <em>cost</em> function.
@@ -936,7 +952,7 @@ probability of the correct category <span class="math notranslate nohighlight">\
 you got the correct label. The probability of category <span class="math notranslate nohighlight">\(c\)</span> is given by the softmax function. The vector <span class="math notranslate nohighlight">\(\hat{\theta}\)</span> represents the parameters of our network, i.e. all the weights and biases.</p>
 </div>
 <div class="section" id="optimizing-the-cost-function">
-<h3><span class="section-number">13.2.7. </span>Optimizing the cost function<a class="headerlink" href="#optimizing-the-cost-function" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.7. </span>Optimizing the cost function<a class="headerlink" href="#optimizing-the-cost-function" title="Permalink to this headline">¶</a></h3>
 <p>The network is trained by finding the weights and biases that minimize the cost function. One of the most widely used classes of methods is <em>gradient descent</em> and its generalizations. The idea behind gradient descent
 is simply to adjust the weights in the direction where the gradient of the cost function is large and negative. This ensures we flow toward a <em>local</em> minimum of the cost function.<br />
 Each parameter <span class="math notranslate nohighlight">\(\theta\)</span> is iteratively adjusted according to the rule</p>
@@ -962,7 +978,7 @@ We denote each minibatch <span class="math notranslate nohighlight">\(B_k\)</spa
 <p>The various optmization  methods, with codes and algorithms,  are discussed in our lectures on <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html">Gradient descent approaches</a>.</p>
 </div>
 <div class="section" id="regularization">
-<h3><span class="section-number">13.2.8. </span>Regularization<a class="headerlink" href="#regularization" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.8. </span>Regularization<a class="headerlink" href="#regularization" title="Permalink to this headline">¶</a></h3>
 <p>It is common to add an extra term to the cost function, proportional
 to the size of the weights.  This is equivalent to constraining the
 size of the weights, so that they do not grow out of control.
@@ -985,7 +1001,7 @@ above. This is a clever use of the chain rule that allows us to
 calculate the gradient efficently.</p>
 </div>
 <div class="section" id="matrix-multiplication">
-<h3><span class="section-number">13.2.9. </span>Matrix  multiplication<a class="headerlink" href="#matrix-multiplication" title="Permalink to this headline">¶</a></h3>
+<h3><span class="section-number">14.2.9. </span>Matrix  multiplication<a class="headerlink" href="#matrix-multiplication" title="Permalink to this headline">¶</a></h3>
 <p>To more efficently train our network these equations are implemented using matrix operations.<br />
 The error in the output layer is calculated simply as, with <span class="math notranslate nohighlight">\(\hat{t}\)</span> being our targets,</p>
 <div class="math notranslate nohighlight">
@@ -1086,7 +1102,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Old accuracy on training data: 0.1440501043841336
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1098,7 +1114,7 @@ the <em>Hadamard product</em>, meaning element-wise multiplication.</p>
 </div>
 </div>
 <div class="section" id="improving-performance">
-<h2><span class="section-number">13.3. </span>Improving performance<a class="headerlink" href="#improving-performance" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.3. </span>Improving performance<a class="headerlink" href="#improving-performance" title="Permalink to this headline">¶</a></h2>
 <p>As we can see the network does not seem to be learning at all. It seems to be just guessing the label for each image.<br />
 In order to obtain a network that does something useful, we will have to do a bit more work.</p>
 <p>The choice of <em>hyperparameters</em> such as learning rate and regularization parameter is hugely influential for the performance of the network. Typically a <em>grid-search</em> is performed, wherein we test different hyperparameters separated by orders of magnitude. For example we could test the learning rates <span class="math notranslate nohighlight">\(\eta = 10^{-6}, 10^{-5},...,10^{-1}\)</span> with different regularization parameters <span class="math notranslate nohighlight">\(\lambda = 10^{-6},...,10^{-0}\)</span>.</p>
@@ -1215,7 +1231,7 @@ being realizations of this object with different hyperparameters. An implementat
 </div>
 </div>
 <div class="section" id="evaluate-model-performance-on-test-data">
-<h2><span class="section-number">13.4. </span>Evaluate model performance on test data<a class="headerlink" href="#evaluate-model-performance-on-test-data" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.4. </span>Evaluate model performance on test data<a class="headerlink" href="#evaluate-model-performance-on-test-data" title="Permalink to this headline">¶</a></h2>
 <p>To measure the performance of our network we evaluate how well it does it data it has never seen before, i.e. the test data.<br />
 We measure the performance of the network using the <em>accuracy</em> score.<br />
 The accuracy is as you would expect just the number of images correctly labeled divided by the total number of images. A perfect classifier will have an accuracy score of <span class="math notranslate nohighlight">\(1\)</span>.</p>
@@ -1251,7 +1267,7 @@ The accuracy is as you would expect just the number of images correctly labeled
 </div>
 </div>
 <div class="section" id="adjust-hyperparameters">
-<h2><span class="section-number">13.5. </span>Adjust hyperparameters<a class="headerlink" href="#adjust-hyperparameters" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.5. </span>Adjust hyperparameters<a class="headerlink" href="#adjust-hyperparameters" title="Permalink to this headline">¶</a></h2>
 <p>We now perform a grid search to find the optimal hyperparameters for the network.<br />
 Note that we are only using 1 layer with 50 neurons, and human performance is estimated to be around <span class="math notranslate nohighlight">\(98\%\)</span> (<span class="math notranslate nohighlight">\(2\%\)</span> error rate).</p>
 <div class="cell docutils container">
@@ -1420,7 +1436,7 @@ Lambda =  10.0
 Accuracy score on test set:  0.21944444444444444
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1429,7 +1445,7 @@ Lambda =  1e-05
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1438,7 +1454,7 @@ Lambda =  0.0001
 Accuracy score on test set:  0.10555555555555556
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1447,7 +1463,7 @@ Lambda =  0.001
 Accuracy score on test set:  0.08888888888888889
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1456,7 +1472,7 @@ Lambda =  0.01
 Accuracy score on test set:  0.11388888888888889
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1465,7 +1481,7 @@ Lambda =  0.1
 Accuracy score on test set:  0.08611111111111111
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1474,7 +1490,7 @@ Lambda =  1.0
 Accuracy score on test set:  0.125
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1483,11 +1499,11 @@ Lambda =  10.0
 Accuracy score on test set:  0.09166666666666666
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1496,11 +1512,11 @@ Lambda =  1e-05
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1509,11 +1525,11 @@ Lambda =  0.0001
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1522,11 +1538,11 @@ Lambda =  0.001
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1535,11 +1551,11 @@ Lambda =  0.01
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1548,7 +1564,7 @@ Lambda =  0.1
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
@@ -1557,11 +1573,11 @@ Lambda =  1.0
 Accuracy score on test set:  0.125
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1570,11 +1586,11 @@ Lambda =  10.0
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1583,11 +1599,11 @@ Lambda =  1e-05
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1596,11 +1612,11 @@ Lambda =  0.0001
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1609,11 +1625,11 @@ Lambda =  0.001
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1622,11 +1638,11 @@ Lambda =  0.01
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1635,11 +1651,11 @@ Lambda =  0.1
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1648,11 +1664,11 @@ Lambda =  1.0
 Accuracy score on test set:  0.07777777777777778
 </pre></div>
 </div>
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:43: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:43: RuntimeWarning: overflow encountered in exp
  exp_term = np.exp(self.z_o)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/1630775253.py:44: RuntimeWarning: invalid value encountered in true_divide
  self.probabilities = exp_term / np.sum(exp_term, axis=1, keepdims=True)
 </pre></div>
 </div>
@@ -1665,7 +1681,7 @@ Accuracy score on test set:  0.07777777777777778
 </div>
 </div>
 <div class="section" id="visualization">
-<h2><span class="section-number">13.6. </span>Visualization<a class="headerlink" href="#visualization" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.6. </span>Visualization<a class="headerlink" href="#visualization" title="Permalink to this headline">¶</a></h2>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># visual representation of grid search</span>
@@ -1705,25 +1721,25 @@ Accuracy score on test set:  0.07777777777777778
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/953065564.py:4: RuntimeWarning: overflow encountered in exp
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/953065564.py:4: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
 </pre></div>
 </div>
-<img alt="_images/chapter10_49_1.png" src="_images/chapter10_49_1.png" />
-<img alt="_images/chapter10_49_2.png" src="_images/chapter10_49_2.png" />
+<img alt="_images/chapter10_59_1.png" src="_images/chapter10_59_1.png" />
+<img alt="_images/chapter10_59_2.png" src="_images/chapter10_59_2.png" />
 </div>
 </div>
 </div>
 <div class="section" id="scikit-learn-implementation">
-<h2><span class="section-number">13.7. </span>scikit-learn implementation<a class="headerlink" href="#scikit-learn-implementation" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.7. </span>scikit-learn implementation<a class="headerlink" href="#scikit-learn-implementation" title="Permalink to this headline">¶</a></h2>
 <p><strong>scikit-learn</strong> focuses more
 on traditional machine learning methods, such as regression,
 clustering, decision trees, etc. As such, it has only two types of
@@ -2058,13 +2074,13 @@ Accuracy score on test set:  0.2
 Learning rate  =  10.0
 Lambda =  0.001
 Accuracy score on test set:  0.10555555555555556
+
+Learning rate  =  10.0
+Lambda =  0.01
+Accuracy score on test set:  0.06388888888888888
 </pre></div>
 </div>
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate  =  10.0
-Lambda =  0.01
-Accuracy score on test set:  0.06388888888888888
-
-Learning rate  =  10.0
 Lambda =  0.1
 Accuracy score on test set:  0.08888888888888889
 </pre></div>
@@ -2082,7 +2098,7 @@ Accuracy score on test set:  0.09166666666666666
 </div>
 </div>
 <div class="section" id="id1">
-<h2><span class="section-number">13.8. </span>Visualization<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.8. </span>Visualization<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># optional</span>
@@ -2123,13 +2139,13 @@ Accuracy score on test set:  0.09166666666666666
 </div>
 </div>
 <div class="cell_output docutils container">
-<img alt="_images/chapter10_53_0.png" src="_images/chapter10_53_0.png" />
-<img alt="_images/chapter10_53_1.png" src="_images/chapter10_53_1.png" />
+<img alt="_images/chapter10_63_0.png" src="_images/chapter10_63_0.png" />
+<img alt="_images/chapter10_63_1.png" src="_images/chapter10_63_1.png" />
 </div>
 </div>
 </div>
 <div class="section" id="building-neural-networks-in-tensorflow-and-keras">
-<h2><span class="section-number">13.9. </span>Building neural networks in Tensorflow and Keras<a class="headerlink" href="#building-neural-networks-in-tensorflow-and-keras" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.9. </span>Building neural networks in Tensorflow and Keras<a class="headerlink" href="#building-neural-networks-in-tensorflow-and-keras" title="Permalink to this headline">¶</a></h2>
 <p>Now we want  to build on the experience gained from our neural network implementation in NumPy and scikit-learn
 and use it to construct a neural network in Tensorflow. Once we have constructed a neural network in NumPy
 and Tensorflow, building one in Keras is really quite trivial, though the performance may suffer.</p>
@@ -2154,23 +2170,8 @@ will give an introduction to the lower level Python Application
 Program Interfaces (APIs), and see how we use them to build our graph.
 Then we will build (effectively) the same graph in Keras, to see just
 how simple solving a machine learning problem can be.</p>
-<p>To install tensorflow on Unix/Linux systems, use pip as</p>
-<div class="cell docutils container">
-<div class="cell_input docutils container">
-<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">pip3</span> <span class="n">install</span> <span class="n">tensorflow</span>
-</pre></div>
-</div>
-</div>
-<div class="cell_output docutils container">
-<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">  File</span><span class="nn"> &quot;/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42331/2357089093.py&quot;</span><span class="gt">, line </span><span class="mi">1</span>
-    <span class="n">pip3</span> <span class="n">install</span> <span class="n">tensorflow</span>
-         <span class="o">^</span>
-<span class="ne">SyntaxError</span>: invalid syntax
-</pre></div>
-</div>
-</div>
-</div>
-<p>and/or if you use <strong>anaconda</strong>, just write (or install from the graphical user interface)
+<p>To install tensorflow on Unix/Linux systems, use pip as <strong>pip3 install tensorflow</strong>
+and/or if you use <strong>anaconda</strong>, just write (or install from the graphical user interface)
 (current release of CPU-only TensorFlow)</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
@@ -2179,6 +2180,14 @@ how simple solving a machine learning problem can be.</p>
 </pre></div>
 </div>
 </div>
+<div class="cell_output docutils container">
+<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">  File</span><span class="nn"> &quot;/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47411/2259440937.py&quot;</span><span class="gt">, line </span><span class="mi">1</span>
+    <span class="n">conda</span> <span class="n">create</span> <span class="o">-</span><span class="n">n</span> <span class="n">tf</span> <span class="n">tensorflow</span>
+          <span class="o">^</span>
+<span class="ne">SyntaxError</span>: invalid syntax
+</pre></div>
+</div>
+</div>
 </div>
 <p>To install the current release of GPU TensorFlow</p>
 <div class="cell docutils container">
@@ -2357,7 +2366,7 @@ If you have Anaconda installed you may run the following command</p>
 </div>
 </div>
 <div class="section" id="the-breast-cancer-data-now-with-keras">
-<h2><span class="section-number">13.10. </span>The Breast Cancer Data, now with Keras<a class="headerlink" href="#the-breast-cancer-data-now-with-keras" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.10. </span>The Breast Cancer Data, now with Keras<a class="headerlink" href="#the-breast-cancer-data-now-with-keras" title="Permalink to this headline">¶</a></h2>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span>
@@ -2531,7 +2540,7 @@ If you have Anaconda installed you may run the following command</p>
 </div>
 </div>
 <div class="section" id="fine-tuning-neural-network-hyperparameters">
-<h2><span class="section-number">13.11. </span>Fine-tuning neural network hyperparameters<a class="headerlink" href="#fine-tuning-neural-network-hyperparameters" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.11. </span>Fine-tuning neural network hyperparameters<a class="headerlink" href="#fine-tuning-neural-network-hyperparameters" title="Permalink to this headline">¶</a></h2>
 <p>The flexibility of neural networks is also one of their main
 drawbacks: there are many hyperparameters to tweak. Not only can you
 use any imaginable network topology (how neurons/nodes are interconnected),
@@ -2562,7 +2571,7 @@ of training data. However, you will rarely have to train such networks from scra
 common to reuse parts of a pretrained state-of-the-art network that performs a similar task.</p>
 </div>
 <div class="section" id="which-activation-function-should-i-use">
-<h2><span class="section-number">13.12. </span>Which activation function should I use?<a class="headerlink" href="#which-activation-function-should-i-use" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.12. </span>Which activation function should I use?<a class="headerlink" href="#which-activation-function-should-i-use" title="Permalink to this headline">¶</a></h2>
 <p>The Back propagation algorithm we derived above works by going from
 the output layer to the input layer, propagating the error gradient on
 the way. Once the algorithm has computed the gradient of the cost
@@ -2627,7 +2636,7 @@ it does not saturate for positive values (and also because it is quite
 fast to compute).</p>
 </div>
 <div class="section" id="the-relu-function-family">
-<h2><span class="section-number">13.13. </span>The RELU function family<a class="headerlink" href="#the-relu-function-family" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.13. </span>The RELU function family<a class="headerlink" href="#the-relu-function-family" title="Permalink to this headline">¶</a></h2>
 <p>The ReLU activation function suffers from a problem known as the dying
 ReLUs: during training, some neurons effectively die, meaning they
 stop outputting anything other than 0.</p>
@@ -2664,7 +2673,7 @@ bootstrap to evaluate other activation functions.</p>
 </ul>
 </div>
 <div class="section" id="batch-normalization">
-<h2><span class="section-number">13.14. </span>Batch Normalization<a class="headerlink" href="#batch-normalization" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.14. </span>Batch Normalization<a class="headerlink" href="#batch-normalization" title="Permalink to this headline">¶</a></h2>
 <p>Batch Normalization
 aims to address the vanishing/exploding gradients problems, and more generally the problem that the
 distribution of each layer’s inputs changes during training, as the parameters of the previous layers change.</p>
@@ -2677,7 +2686,7 @@ standard deviation. It does so by evaluating the mean and standard deviation of
 mini-batch, from this the name batch normalization.</p>
 </div>
 <div class="section" id="dropout">
-<h2><span class="section-number">13.15. </span>Dropout<a class="headerlink" href="#dropout" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.15. </span>Dropout<a class="headerlink" href="#dropout" title="Permalink to this headline">¶</a></h2>
 <p>It is a fairly simple algorithm: at every training step, every neuron (including the input neurons but
 excluding the output neurons) has a probability <span class="math notranslate nohighlight">\(p\)</span> of being temporarily dropped out, meaning it will be
 entirely ignored during this training step, but it may be active during the next step.</p>
@@ -2686,7 +2695,7 @@ hyperparameter <span class="math notranslate nohighlight">\(p\)</span> is called
 It is viewed as one of the most popular regularization techniques.</p>
 </div>
 <div class="section" id="gradient-clipping">
-<h2><span class="section-number">13.16. </span>Gradient Clipping<a class="headerlink" href="#gradient-clipping" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.16. </span>Gradient Clipping<a class="headerlink" href="#gradient-clipping" title="Permalink to this headline">¶</a></h2>
 <p>A popular technique to lessen the exploding gradients problem is to simply clip the gradients during
 backpropagation so that they never exceed some threshold (this is mostly useful for recurrent neural
 networks).</p>
@@ -2695,7 +2704,7 @@ networks).</p>
 Normalization is preferred.</p>
 </div>
 <div class="section" id="a-top-down-perspective-on-neural-networks">
-<h2><span class="section-number">13.17. </span>A top-down perspective on Neural networks<a class="headerlink" href="#a-top-down-perspective-on-neural-networks" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.17. </span>A top-down perspective on Neural networks<a class="headerlink" href="#a-top-down-perspective-on-neural-networks" title="Permalink to this headline">¶</a></h2>
 <p>The first thing we would like to do is divide the data into two or three
 parts. A training set, a validation or dev (development) set, and a
 test set. The test set is the data on which we want to make
@@ -2729,7 +2738,7 @@ can serve as another important diagnostic when using DNNs for
 supervised learning.</p>
 </div>
 <div class="section" id="limitations-of-supervised-learning-with-deep-networks">
-<h2><span class="section-number">13.18. </span>Limitations of supervised learning with deep networks<a class="headerlink" href="#limitations-of-supervised-learning-with-deep-networks" title="Permalink to this headline">¶</a></h2>
+<h2><span class="section-number">14.18. </span>Limitations of supervised learning with deep networks<a class="headerlink" href="#limitations-of-supervised-learning-with-deep-networks" title="Permalink to this headline">¶</a></h2>
 <p>Like all statistical methods, supervised learning using neural
 networks has important limitations. This is especially important when
 one seeks to apply these methods, especially to physics problems. Like
@@ -2782,7 +2791,7 @@ features).</p>
            <i class="prevnext-label fas fa-angle-left"></i>
            <div class="prevnext-info">
                <p class="prevnext-label">previous</p>
-                <p class="prevnext-title"><span class="section-number">12. </span>Neural networks</p>
+                <p class="prevnext-title"><span class="section-number">13. </span>Neural networks</p>
            </div>
        </a>
    </div>
@@ -2790,7 +2799,7 @@ features).</p>
        <a class="right-next" href="chapter11.html" title="next page">
            <div class="prevnext-info">
                <p class="prevnext-label">next</p>
-                <p class="prevnext-title"><span class="section-number">14. </span>Solving Differential Equations  with Deep Learning</p>
+                <p class="prevnext-title"><span class="section-number">15. </span>Solving Differential Equations  with Deep Learning</p>
            </div>
            <i class="prevnext-label fas fa-angle-right"></i>
        </a>