update exercises week 42

2024-10-15 05:47:17 +02:00
parent 0bc2e08294
commit eaa8a92113
8 changed files with 333 additions and 198 deletions
@@ -51,6 +51,8 @@ const thebe_selector_input = "pre"
 const thebe_selector_output = ".output, .cell_output"
 </script>
    <script async="async" src="_static/sphinx-thebe.js"></script>
+    <script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
+    <script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Week 42 Constructing a Neural Network code with examples" href="week42.html" />
@@ -460,8 +462,8 @@ const thebe_selector_output = ".output, .cell_output"
  </a>
 </li>
 <li class="toc-h1 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#exercise-6-training-on-real-data">
-   Exercise 6 - Training on real data
+  <a class="reference internal nav-link" href="#exercise-7-training-on-real-data-optional">
+   Exercise 7 - Training on real data (Optional)
  </a>
 </li>
 </ul>
@@ -525,8 +527,8 @@ const thebe_selector_output = ".output, .cell_output"
  </a>
 </li>
 <li class="toc-h1 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#exercise-6-training-on-real-data">
-   Exercise 6 - Training on real data
+  <a class="reference internal nav-link" href="#exercise-7-training-on-real-data-optional">
+   Exercise 7 - Training on real data (Optional)
  </a>
 </li>
 </ul>
@@ -543,14 +545,15 @@ doconce format html exercisesweek41.do.txt  -->
 <!-- dom:TITLE: Exercises week 41 -->
 <div class="tex2jax_ignore mathjax_ignore section" id="exercises-week-42">
 <h1>Exercises week 42<a class="headerlink" href="#exercises-week-42" title="Permalink to this headline">¶</a></h1>
-<p><strong>October 11-18, 2024</strong></p>
+<p><strong>October 14-18, 2024</strong></p>
 <p>Date: <strong>Deadline is Friday October 18 at midnight</strong></p>
 </div>
 <div class="tex2jax_ignore mathjax_ignore section" id="overarching-aims-of-the-exercises-this-week">
 <h1>Overarching aims of the exercises this week<a class="headerlink" href="#overarching-aims-of-the-exercises-this-week" title="Permalink to this headline">¶</a></h1>
-<p>The aim of the exercises this week is to get started with implementing a neural network. There are a lot of technical and finicky parts of implementing a neutal network, so take your time.</p>
-<p>This week, you will implement only the feed-forward pass and updating the network parameters with simple gradient descent, the gradient will be computed using autograd using code we provide. Next week, you will implement backpropagation. We recommend that you do the exercises this week by editing and running this notebook file, as it includes some checks along the way that you have implemented the pieces of the feed-forward pass correctly, and running small parts of the code at a time will be important for understanding the methods.</p>
-<p>If you have trouble running a notebook, you can run this notebook in google colab instead (<a class="reference external" href="https://colab.research.google.com/drive/1OCQm1tlTWB6hZSf9I7gGUgW9M8SbVeQu#offline=true&amp;sandboxMode=true">https://colab.research.google.com/drive/1OCQm1tlTWB6hZSf9I7gGUgW9M8SbVeQu#offline=true&amp;sandboxMode=true</a>), an updated link will be provided on the course discord (you can also send an email to <a class="reference external" href="mailto:k&#46;h&#46;fredly&#37;&#52;&#48;fys&#46;uio&#46;no">k<span>&#46;</span>h<span>&#46;</span>fredly<span>&#64;</span>fys<span>&#46;</span>uio<span>&#46;</span>no</a> if you encounter any trouble), though we recommend that you set up VSCode and your python environment to run code like this locally.</p>
+<p>This week, you will implement the entire feed-forward pass of a neural network! Next week you will compute the gradient of the network by implementing back-propagation manually, and by using autograd which does back-propagation for you (much easier!). Next week, you will also use the gradient to optimize the network with a gradient method! However, there is an optional exercise this week to get started on training the network and getting good results!</p>
+<p>We recommend that you do the exercises this week by editing and running this notebook file, as it includes some checks along the way that you have implemented the pieces of the feed-forward pass correctly, and running small parts of the code at a time will be important for understanding the methods.</p>
+<p>If you have trouble running a notebook, you can run this notebook in google colab instead (<a class="reference external" href="https://colab.research.google.com/drive/1zKibVQf-iAYaAn2-GlKfgRjHtLnPlBX4#offline=true&amp;sandboxMode=true">https://colab.research.google.com/drive/1zKibVQf-iAYaAn2-GlKfgRjHtLnPlBX4#offline=true&amp;sandboxMode=true</a>), an updated link will be provided on the course discord (you can also send an email to <a class="reference external" href="mailto:k&#46;h&#46;fredly&#37;&#52;&#48;fys&#46;uio&#46;no">k<span>&#46;</span>h<span>&#46;</span>fredly<span>&#64;</span>fys<span>&#46;</span>uio<span>&#46;</span>no</a> if you encounter any trouble), though we recommend that you set up VSCode and your python environment to run code like this locally.</p>
+<p>First, here are some functions you are going to need, don’t change this cell. If you are unable to import autograd, just swap in normal numpy until you want to do the final optional exercise.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>  <span class="c1"># We need to use this numpy wrapper to make automatic differentiation work later</span>
@@ -762,9 +765,9 @@ doconce format html exercisesweek41.do.txt  -->
 <p><strong>a)</strong> Complete the <code class="docutils literal notranslate"><span class="pre">feed_forward</span></code> function which accepts a list of activation functions as an argument, and which evaluates these activation functions at each layer.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
-<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">feed_forward</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">):</span>
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">feed_forward</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">):</span>
    <span class="n">a</span> <span class="o">=</span> <span class="nb">input</span>
-    <span class="k">for</span> <span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">),</span> <span class="n">activation</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">):</span>
+    <span class="k">for</span> <span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">),</span> <span class="n">activation_func</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">):</span>
        <span class="n">z</span> <span class="o">=</span> <span class="o">...</span>
        <span class="n">a</span> <span class="o">=</span> <span class="o">...</span>
    <span class="k">return</span> <span class="n">a</span>
@@ -772,21 +775,22 @@ doconce format html exercisesweek41.do.txt  -->
 </div>
 </div>
 </div>
-<p><strong>b)</strong> Make a list with three activation functions(don’t call them yet! you can make a list with function names as elements, and then call these elements of the list later), two ReLU and one sigmoid. (If you add other functions than the ones defined at the start of the notebook, make sure everything is defined using autograd’s numpy wrapper, like above, since we want to use automatic differentiation on all of these functions later.)</p>
-<p>Then evaluate a network with three layers and these activation functions.</p>
+<p><strong>b)</strong> You are now given a list with three activation functions, two ReLU and one sigmoid. (Don’t call them yet! you can make a list with function names as elements, and then call these elements of the list later. If you add other functions than the ones defined at the start of the notebook, make sure everything is defined using autograd’s numpy wrapper, like above, since we want to use automatic differentiation on all of these functions later.)</p>
+<p>Evaluate a network with three layers and these activation functions.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">network_input_size</span> <span class="o">=</span> <span class="o">...</span>
 <span class="n">layer_output_sizes</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span>
-<span class="n">activations</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span>
+<span class="n">activation_funcs</span> <span class="o">=</span> <span class="p">[</span><span class="n">ReLU</span><span class="p">,</span> <span class="n">ReLU</span><span class="p">,</span> <span class="n">sigmoid</span><span class="p">]</span>
 <span class="n">layers</span> <span class="o">=</span> <span class="o">...</span>

 <span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">)</span>
-<span class="n">feed_forward</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">)</span>
+<span class="n">feed_forward</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">)</span>
 </pre></div>
 </div>
 </div>
 </div>
+<p><strong>c)</strong> How does the output of the network change if you use sigmoid in the hidden layers and ReLU in the output layer?</p>
 </div>
 <div class="tex2jax_ignore mathjax_ignore section" id="exercise-5-processing-multiple-inputs-at-once">
 <h1>Exercise 5 - Processing multiple inputs at once<a class="headerlink" href="#exercise-5-processing-multiple-inputs-at-once" title="Permalink to this headline">¶</a></h1>
@@ -816,9 +820,9 @@ doconce format html exercisesweek41.do.txt  -->
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">inputs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">1000</span><span class="p">,</span> <span class="mi">4</span><span class="p">)</span>


-<span class="k">def</span> <span class="nf">feed_forward_batch</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">):</span>
+<span class="k">def</span> <span class="nf">feed_forward_batch</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">):</span>
    <span class="n">a</span> <span class="o">=</span> <span class="n">inputs</span>
-    <span class="k">for</span> <span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">),</span> <span class="n">activation</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">):</span>
+    <span class="k">for</span> <span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">),</span> <span class="n">activation_func</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">):</span>
        <span class="n">z</span> <span class="o">=</span> <span class="o">...</span>
        <span class="n">a</span> <span class="o">=</span> <span class="o">...</span>
    <span class="k">return</span> <span class="n">a</span>
@@ -831,11 +835,11 @@ doconce format html exercisesweek41.do.txt  -->
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">network_input_size</span> <span class="o">=</span> <span class="o">...</span>
 <span class="n">layer_output_sizes</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span>
-<span class="n">activations</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span>
+<span class="n">activation_funcs</span> <span class="o">=</span> <span class="p">[</span><span class="o">...</span><span class="p">]</span>
 <span class="n">layers</span> <span class="o">=</span> <span class="n">create_layers_batch</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">,</span> <span class="n">layer_output_sizes</span><span class="p">)</span>

 <span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">network_input_size</span><span class="p">)</span>
-<span class="n">feed_forward_batch</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">)</span>
+<span class="n">feed_forward_batch</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">)</span>
 </pre></div>
 </div>
 </div>
@@ -881,7 +885,7 @@ doconce format html exercisesweek41.do.txt  -->
 </div>
 </div>
 </div>
-<p><strong>a)</strong> What should the input size for the network be with this dataset? What should the output shape of the last layer be?</p>
+<p><strong>a)</strong> What should the input size for the network be with this dataset? What should the output size of the last layer be?</p>
 <p><strong>b)</strong> Create a network with two hidden layers, the first with sigmoid activation and the last with softmax, the first layer should have 8 “nodes”, the second has the number of nodes you found in exercise a). Softmax returns a “probability distribution”, in the sense that the numbers in the output are positive and add up to 1 and, their magnitude are in some sense relative to their magnitude before going through the softmax function. Remember to use the batched version of the create_layers and feed forward functions.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
@@ -894,7 +898,7 @@ doconce format html exercisesweek41.do.txt  -->
 <p><strong>c)</strong> Evaluate your model on the entire iris dataset! For later purposes, we will split the data into train and test sets, and compute gradients on smaller batches of the training data. But for now, evaluate the network on the whole thing at once.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
-<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">predictions</span> <span class="o">=</span> <span class="n">feed_forward_batch</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">)</span>
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">predictions</span> <span class="o">=</span> <span class="n">feed_forward_batch</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">)</span>
 </pre></div>
 </div>
 </div>
@@ -908,27 +912,37 @@ doconce format html exercisesweek41.do.txt  -->
 </div>
 </div>
 </div>
-<div class="tex2jax_ignore mathjax_ignore section" id="exercise-6-training-on-real-data">
-<h1>Exercise 6 - Training on real data<a class="headerlink" href="#exercise-6-training-on-real-data" title="Permalink to this headline">¶</a></h1>
+<div class="tex2jax_ignore mathjax_ignore section" id="exercise-7-training-on-real-data-optional">
+<h1>Exercise 7 - Training on real data (Optional)<a class="headerlink" href="#exercise-7-training-on-real-data-optional" title="Permalink to this headline">¶</a></h1>
 <p>To be able to actually do anything useful with your neural network, you need to train it. For this, we need a cost function and a way to take the gradient of the cost function wrt. the network parameters. The following exercises guide you through taking the gradient using autograd, and updating the network parameters using the gradient. Feel free to implement gradient methods like ADAM if you finish everything.</p>
-<p>The cross-entropy loss function can evaluate performance on classification tasks. It sees if your prediction is “most certain” on the correct target.</p>
+<p>Since we are doing a classification task with multiple output classes, we use the cross-entropy loss function, which can evaluate performance on classification tasks. It sees if your prediction is “most certain” on the correct target.</p>
+<div class="cell docutils container">
+<div class="cell_input docutils container">
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">cross_entropy</span><span class="p">(</span><span class="n">predict</span><span class="p">,</span> <span class="n">target</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="o">-</span><span class="n">target</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">predict</span><span class="p">))</span>
+
+
+<span class="k">def</span> <span class="nf">cost</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">,</span> <span class="n">target</span><span class="p">):</span>
+    <span class="n">predict</span> <span class="o">=</span> <span class="n">feed_forward_batch</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">)</span>
+    <span class="k">return</span> <span class="n">cross_entropy</span><span class="p">(</span><span class="n">predict</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+<p>To improve our network on whatever prediction task we have given it, we need to use a sensible cost function, take the gradient of that cost function with respect to our network parameters, the weights and biases, and then update the weights and biases using these gradients. To clarify, we need to find and use these</p>
+<div class="math notranslate nohighlight">
+\[
+\frac{\partial C}{\partial W}, \frac{\partial C}{\partial b}
+\]</div>
+<p>Now we need to compute these gradients. This is pretty hard to do for a neural network, we will use most of next week to do this, but we can also use autograd to just do it for us, which is what we always do in practice. With the code cell below, we create a function which takes all of these gradients for us.</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>


-<span class="k">def</span> <span class="nf">cost</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">,</span> <span class="n">target</span><span class="p">):</span>
-    <span class="n">predict</span> <span class="o">=</span> <span class="n">feed_forward_batch</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">)</span>
-    <span class="k">return</span> <span class="n">cross_entropy</span><span class="p">(</span><span class="n">predict</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
-
-
-<span class="k">def</span> <span class="nf">cross_entropy</span><span class="p">(</span><span class="n">predict</span><span class="p">,</span> <span class="n">target</span><span class="p">):</span>
-    <span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="o">-</span><span class="n">target</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">predict</span><span class="p">))</span>
-
-
 <span class="n">gradient_func</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span>
-    <span class="n">cross_entropy</span><span class="p">,</span> <span class="mi">1</span>
-<span class="p">)</span>  <span class="c1"># Taking the gradient wrt. the second input to the cost function</span>
+    <span class="n">cost</span><span class="p">,</span> <span class="mi">1</span>
+<span class="p">)</span>  <span class="c1"># Taking the gradient wrt. the second input to the cost function, i.e. the layers</span>
 </pre></div>
 </div>
 </div>
@@ -937,7 +951,9 @@ doconce format html exercisesweek41.do.txt  -->
 <p><strong>b)</strong> Use the <code class="docutils literal notranslate"><span class="pre">gradient_func</span></code> function to take the gradient of the cross entropy wrt. the weights and biases of the network. Check the shapes of what’s inside. What does the <code class="docutils literal notranslate"><span class="pre">grad</span></code> func from autograd actually do?</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
-<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">layers_grad</span> <span class="o">=</span> <span class="n">gradient_func</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">,</span> <span class="n">targets</span><span class="p">)</span>  <span class="c1"># Don&#39;t change this</span>
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">layers_grad</span> <span class="o">=</span> <span class="n">gradient_func</span><span class="p">(</span>
+    <span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">,</span> <span class="n">targets</span>
+<span class="p">)</span>  <span class="c1"># Don&#39;t change this</span>
 </pre></div>
 </div>
 </div>
@@ -946,10 +962,10 @@ doconce format html exercisesweek41.do.txt  -->
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">train_network</span><span class="p">(</span>
-    <span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span>
+    <span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">,</span> <span class="n">targets</span><span class="p">,</span> <span class="n">learning_rate</span><span class="o">=</span><span class="mf">0.001</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span>
 <span class="p">):</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">epochs</span><span class="p">):</span>
-        <span class="n">layers_grad</span> <span class="o">=</span> <span class="n">gradient_func</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activations</span><span class="p">,</span> <span class="n">targets</span><span class="p">)</span>
+        <span class="n">layers_grad</span> <span class="o">=</span> <span class="n">gradient_func</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">activation_funcs</span><span class="p">,</span> <span class="n">targets</span><span class="p">)</span>
        <span class="k">for</span> <span class="p">(</span><span class="n">W</span><span class="p">,</span> <span class="n">b</span><span class="p">),</span> <span class="p">(</span><span class="n">W_g</span><span class="p">,</span> <span class="n">b_g</span><span class="p">)</span> <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">layers</span><span class="p">,</span> <span class="n">layers_grad</span><span class="p">):</span>
            <span class="n">W</span> <span class="o">-=</span> <span class="o">...</span>
            <span class="n">b</span> <span class="o">-=</span> <span class="o">...</span>