buidl

2021-12-08 06:59:22 +01:00
parent d760cdfa85
commit 7ca589a654
94 changed files with 23333 additions and 2210 deletions
@@ -199,6 +199,11 @@
   11. Basic ideas of the Principal Component Analysis (PCA)
  </a>
 </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="clustering.html">
+   12. Clustering and Unsupervised Learning
+  </a>
+ </li>
 </ul>
 <p class="caption" role="heading">
 <span class="caption-text">
@@ -208,17 +213,27 @@
 <ul class="nav bd-sidenav">
 <li class="toctree-l1">
  <a class="reference internal" href="chapter9.html">
-   12. Neural networks
+   13. Neural networks
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="chapter10.html">
-   13. Building a Feed Forward Neural Network
+   14. Building a Feed Forward Neural Network
  </a>
 </li>
 <li class="toctree-l1">
  <a class="reference internal" href="chapter11.html">
-   14. Solving Differential Equations  with Deep Learning
+   15. Solving Differential Equations  with Deep Learning
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter12.html">
+   16. Convolutional Neural Networks
+  </a>
+ </li>
+ <li class="toctree-l1">
+  <a class="reference internal" href="chapter13.html">
+   17. Recurrent neural networks: Overarching view
  </a>
 </li>
 </ul>
@@ -330,8 +345,8 @@
  </a>
 </li>
 <li class="toc-h2 nav-item toc-entry">
-  <a class="reference internal nav-link" href="#stochastic-gradient-descent">
-   7.7. Stochastic Gradient Descent
+  <a class="reference internal nav-link" href="#stochastic-gradient-descent-sgd">
+   7.7. Stochastic Gradient Descent (SGD)
  </a>
  <ul class="nav section-nav flex-column">
   <li class="toc-h3 nav-item toc-entry">
@@ -368,6 +383,23 @@
   7.10. Automatic differentiation
  </a>
 </li>
+ <li class="toc-h2 nav-item toc-entry">
+  <a class="reference internal nav-link" href="#using-autograd-with-ols">
+   7.11. Using Autograd with OLS
+  </a>
+  <ul class="nav section-nav flex-column">
+   <li class="toc-h3 nav-item toc-entry">
+    <a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
+     7.11.1. Including Stochastic Gradient Descent with Autograd
+    </a>
+   </li>
+   <li class="toc-h3 nav-item toc-entry">
+    <a class="reference internal nav-link" href="#and-logistic-regression">
+     7.11.2. And Logistic Regression
+    </a>
+   </li>
+  </ul>
+ </li>
 </ul>

            </nav>
@@ -379,7 +411,8 @@
        
              <div>
                
-  <div class="tex2jax_ignore mathjax_ignore section" id="optimization-the-central-part-of-any-machine-learning-algortithm">
+  <!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
+doconce format html chapteroptimization.do.txt  --><div class="tex2jax_ignore mathjax_ignore section" id="optimization-the-central-part-of-any-machine-learning-algortithm">
 <h1><span class="section-number">7. </span>Optimization, the central part of any Machine Learning algortithm<a class="headerlink" href="#optimization-the-central-part-of-any-machine-learning-algortithm" title="Permalink to this headline">¶</a></h1>
 <p>Almost every problem in machine learning and data science starts with
 a dataset <span class="math notranslate nohighlight">\(X\)</span>, a model <span class="math notranslate nohighlight">\(g(\beta)\)</span>, which is a function of the
@@ -750,14 +783,14 @@ which equals</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42573/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
+<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47735/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
  ax = fig.gca(projection=&quot;3d&quot;)
 </pre></div>
 </div>
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fad10f9a280&gt;
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fd098df3280&gt;
 </pre></div>
 </div>
-<img alt="_images/chapteroptimization_56_2.png" src="_images/chapteroptimization_56_2.png" />
+<img alt="_images/chapteroptimization_61_2.png" src="_images/chapteroptimization_61_2.png" />
 </div>
 </div>
 <p>And then as countor plot</p>
@@ -770,7 +803,7 @@ which equals</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<img alt="_images/chapteroptimization_58_0.png" src="_images/chapteroptimization_58_0.png" />
+<img alt="_images/chapteroptimization_63_0.png" src="_images/chapteroptimization_63_0.png" />
 </div>
 </div>
 <p>Find guesses</p>
@@ -812,10 +845,10 @@ which equals</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x7fad20f69be0&gt;]
+<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x7fd0a9063be0&gt;]
 </pre></div>
 </div>
-<img alt="_images/chapteroptimization_64_1.png" src="_images/chapteroptimization_64_1.png" />
+<img alt="_images/chapteroptimization_69_1.png" src="_images/chapteroptimization_69_1.png" />
 </div>
 </div>
 </div>
@@ -1069,16 +1102,14 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.32903042 4.1484256 ]
-[[4.04621521]
- [3.00415763]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.28001319 4.21265216]
+[[3.96987657]
+ [3.02493054]]
+[[3.96987657]
+ [3.02493054]]
 </pre></div>
 </div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.04621521]
- [3.00415763]]
-</pre></div>
-</div>
-<img alt="_images/chapteroptimization_118_2.png" src="_images/chapteroptimization_118_2.png" />
+<img alt="_images/chapteroptimization_123_1.png" src="_images/chapteroptimization_123_1.png" />
 </div>
 </div>
 <p>Alternatively, we can use <strong>Scikit-Learn</strong> as done here</p>
@@ -1104,9 +1135,9 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[3.97230501]
- [3.14741468]]
-[3.94735055] [3.17084902]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.00275135]
+ [2.99724883]]
+[3.97065296] [3.07656896]
 </pre></div>
 </div>
 </div>
@@ -1177,13 +1208,13 @@ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta|
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[3.92343595]
- [3.15258907]]
-[[3.815563  ]
- [3.23522201]]
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.1533795 ]
+ [2.92819235]]
+[[4.06858699]
+ [2.99829953]]
 </pre></div>
 </div>
-<img alt="_images/chapteroptimization_127_1.png" src="_images/chapteroptimization_127_1.png" />
+<img alt="_images/chapteroptimization_132_1.png" src="_images/chapteroptimization_132_1.png" />
 </div>
 </div>
 </div>
@@ -1198,8 +1229,27 @@ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta|
 <li><p>GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.</p></li>
 </ul>
 </div>
-<div class="section" id="stochastic-gradient-descent">
-<h2><span class="section-number">7.7. </span>Stochastic Gradient Descent<a class="headerlink" href="#stochastic-gradient-descent" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="stochastic-gradient-descent-sgd">
+<h2><span class="section-number">7.7. </span>Stochastic Gradient Descent (SGD)<a class="headerlink" href="#stochastic-gradient-descent-sgd" title="Permalink to this headline">¶</a></h2>
+<p>In stochastic gradient descent, the extreme case is the case where we
+have only one batch, that is we include the whole data set.</p>
+<p>This process is called Stochastic Gradient
+Descent (SGD) (or also sometimes on-line gradient descent). This is
+relatively less common to see because in practice due to vectorized
+code optimizations it can be computationally much more efficient to
+evaluate the gradient for 100 examples, than the gradient for one
+example 100 times. Even though SGD technically refers to using a
+single example at a time to evaluate the gradient, you will hear
+people use the term SGD even when referring to mini-batch gradient
+descent (i.e. mentions of MGD for “Minibatch Gradient Descent”, or BGD
+for “Batch gradient descent” are rare to see), where it is usually
+assumed that mini-batches are used. The size of the mini-batch is a
+hyperparameter but it is not very common to cross-validate or bootstrap it. It is
+usually based on memory constraints (if any), or set to some value,
+e.g. 32, 64 or 128. We use powers of 2 in practice because many
+vectorized operation implementations work faster when their inputs are
+sized in powers of 2.</p>
+<p>In our notes with  SGD we mean stochastic gradient descent with mini-batches.</p>
 <p>Stochastic gradient descent (SGD) and variants thereof address some of
 the shortcomings of the Gradient descent method discussed above.</p>
 <p>The underlying idea of SGD comes from the observation that the cost
@@ -1256,7 +1306,7 @@ the number of minibatches, as exemplified in the code below.</p>
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span> 

 <span class="n">n</span> <span class="o">=</span> <span class="mi">100</span> <span class="c1">#100 datapoints </span>
-<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span>   <span class="c1">#size of each minibatch</span>
+<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span>   <span class="c1">#size of each mini-batche</span>
 <span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
 <span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">10</span> <span class="c1">#number of epochs</span>

@@ -1332,38 +1382,39 @@ function.</p>
 </div>
 </div>
 </div>
+<p>We note that we have defined several hyperparameters. These are now the number of epochs, the number of mini-batches and the parameters <span class="math notranslate nohighlight">\(t_0\)</span> and <span class="math notranslate nohighlight">\(t_1\)</span>.</p>
 <div class="section" id="program-for-stochastic-gradient">
 <h3><span class="section-number">7.7.1. </span>Program for stochastic gradient<a class="headerlink" href="#program-for-stochastic-gradient" title="Permalink to this headline">¶</a></h3>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Importing various packages</span>
+<span class="c1"># Importing various packages</span>
 <span class="kn">from</span> <span class="nn">math</span> <span class="kn">import</span> <span class="n">exp</span><span class="p">,</span> <span class="n">sqrt</span>
 <span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
 <span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
 <span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
-<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">SGDRegressor</span>

-<span class="n">m</span> <span class="o">=</span> <span class="mi">100</span>
-<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">m</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
-<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">m</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
+<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>

-<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">m</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
+<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
+<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
 <span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
 <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
 <span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
-<span class="n">sgdreg</span> <span class="o">=</span> <span class="n">SGDRegressor</span><span class="p">(</span><span class="n">max_iter</span> <span class="o">=</span> <span class="mi">50</span><span class="p">,</span> <span class="n">penalty</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">eta0</span><span class="o">=</span><span class="mf">0.1</span><span class="p">)</span>
-<span class="n">sgdreg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">y</span><span class="o">.</span><span class="n">ravel</span><span class="p">())</span>
-<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;sgdreg from scikit&quot;</span><span class="p">)</span>
-<span class="nb">print</span><span class="p">(</span><span class="n">sgdreg</span><span class="o">.</span><span class="n">intercept_</span><span class="p">,</span> <span class="n">sgdreg</span><span class="o">.</span><span class="n">coef_</span><span class="p">)</span>
-
+<span class="c1"># Hessian matrix</span>
+<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
+<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>

 <span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
-<span class="n">eta</span> <span class="o">=</span> <span class="mf">0.1</span>
+<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
 <span class="n">Niterations</span> <span class="o">=</span> <span class="mi">1000</span>


 <span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
-    <span class="n">gradients</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">/</span><span class="n">m</span><span class="o">*</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="p">((</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">-</span><span class="n">y</span><span class="p">)</span>
+    <span class="n">gradients</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="o">*</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="p">((</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">-</span><span class="n">y</span><span class="p">)</span>
    <span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
 <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own gd&quot;</span><span class="p">)</span>
 <span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
@@ -1373,8 +1424,9 @@ function.</p>
 <span class="n">ypredict</span> <span class="o">=</span> <span class="n">Xnew</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
 <span class="n">ypredict2</span> <span class="o">=</span> <span class="n">Xnew</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>

-
 <span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
+<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span>   <span class="c1">#size of each minibatch</span>
+<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
 <span class="n">t0</span><span class="p">,</span> <span class="n">t1</span> <span class="o">=</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">50</span>
 <span class="k">def</span> <span class="nf">learning_schedule</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
    <span class="k">return</span> <span class="n">t0</span><span class="o">/</span><span class="p">(</span><span class="n">t</span><span class="o">+</span><span class="n">t1</span><span class="p">)</span>
@@ -1382,16 +1434,20 @@ function.</p>
 <span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>

 <span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
+<span class="c1"># Can you figure out a better way of setting up the contributions to each batch?</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
-        <span class="n">random_index</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
-        <span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span>
-        <span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span>
-        <span class="n">gradients</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">xi</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="p">((</span><span class="n">xi</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">-</span><span class="n">yi</span><span class="p">)</span>
+        <span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
+        <span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
+        <span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
+        <span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span> <span class="n">xi</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="p">((</span><span class="n">xi</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">-</span><span class="n">yi</span><span class="p">)</span>
        <span class="n">eta</span> <span class="o">=</span> <span class="n">learning_schedule</span><span class="p">(</span><span class="n">epoch</span><span class="o">*</span><span class="n">m</span><span class="o">+</span><span class="n">i</span><span class="p">)</span>
        <span class="n">theta</span> <span class="o">=</span> <span class="n">theta</span> <span class="o">-</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
 <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own sdg&quot;</span><span class="p">)</span>
 <span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>

+
+
+
 <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">xnew</span><span class="p">,</span> <span class="n">ypredict</span><span class="p">,</span> <span class="s2">&quot;r-&quot;</span><span class="p">)</span>
 <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">xnew</span><span class="p">,</span> <span class="n">ypredict2</span><span class="p">,</span> <span class="s2">&quot;b-&quot;</span><span class="p">)</span>
 <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">,</span><span class="s1">&#39;ro&#39;</span><span class="p">)</span>
@@ -1405,23 +1461,24 @@ function.</p>
 </div>
 <div class="cell_output docutils container">
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
-[[4.15629539]
- [2.7293182 ]]
-sgdreg from scikit
-[4.0728785] [2.66410989]
+[[3.99775949]
+ [2.94659383]]
+Eigenvalues of Hessian Matrix:[0.36102113 4.18276924]
 theta from own gd
-[[4.15629539]
- [2.7293182 ]]
+[[3.99775949]
+ [2.94659383]]
+theta from own sdg
+[[3.96489434]
+ [2.98399675]]
 </pre></div>
 </div>
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>theta from own sdg
-[[4.15454301]
- [2.72522848]]
-</pre></div>
-</div>
-<img alt="_images/chapteroptimization_141_2.png" src="_images/chapteroptimization_141_2.png" />
+<img alt="_images/chapteroptimization_148_1.png" src="_images/chapteroptimization_148_1.png" />
 </div>
 </div>
+<p>In the above code, we have use replacement in setting up the
+mini-batches. The discussion
+<a class="reference external" href="https://sebastianraschka.com/faq/docs/sgd-methods.html">here</a> may be
+useful.  More material will be added later.</p>
 </div>
 </div>
 <div class="section" id="momentum-based-gd">
@@ -1538,7 +1595,7 @@ the steep computational price of calculating or approximating
 Hessians.</p>
 <p>Recently, a number of methods have been introduced that accomplish
 this by tracking not only the gradient, but also the second moment of
-the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
+the gradient. These methods include AdaGrad, AdaDelta, Root Mean Squared Propagation (RMS-Prop), and
 ADAM.</p>
 <div class="section" id="rms-prop">
 <h3><span class="section-number">7.8.1. </span>RMS prop<a class="headerlink" href="#rms-prop" title="Permalink to this headline">¶</a></h3>
@@ -1724,7 +1781,7 @@ f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right)
 </div>
 </div>
 <div class="cell_output docutils container">
-<img alt="_images/chapteroptimization_177_0.png" src="_images/chapteroptimization_177_0.png" />
+<img alt="_images/chapteroptimization_188_0.png" src="_images/chapteroptimization_188_0.png" />
 <div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The max absolute difference is: 1.77636e-15
 </pre></div>
 </div>
@@ -1907,28 +1964,42 @@ The analytical gradient of f4 at x = 2.7 is: 13.8759
 </div>
 </div>
 </div>
-<p>1
-8</p>
-<p>&lt;
-&lt;
-&lt;
-!
-!
-C
-O
-D
-E
-_
-B
-L
-O
-C
-K</p>
-<p>p
-y
-c
-o
-d</p>
+<div class="cell docutils container">
+<div class="cell_input docutils container">
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
+<span class="k">def</span> <span class="nf">f6_for</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
+    <span class="n">val</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>
+        <span class="n">val</span> <span class="o">=</span> <span class="n">val</span> <span class="o">+</span> <span class="n">x</span><span class="o">**</span><span class="n">i</span>
+    <span class="k">return</span> <span class="n">val</span>
+
+<span class="k">def</span> <span class="nf">f6_while</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
+    <span class="n">val</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
+    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="mi">10</span><span class="p">:</span>
+        <span class="n">val</span> <span class="o">=</span> <span class="n">val</span> <span class="o">+</span> <span class="n">x</span><span class="o">**</span><span class="n">i</span>
+        <span class="n">i</span> <span class="o">=</span> <span class="n">i</span> <span class="o">+</span> <span class="mi">1</span>
+    <span class="k">return</span> <span class="n">val</span>
+
+<span class="n">f6_for_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f6_for</span><span class="p">)</span>
+<span class="n">f6_while_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f6_while</span><span class="p">)</span>
+
+<span class="n">x</span> <span class="o">=</span> <span class="mf">0.5</span>
+
+<span class="c1"># Print the computed derivaties of f6_for and f6_while</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The computed derivative of f6_for at x = </span><span class="si">%g</span><span class="s2"> is: </span><span class="si">%g</span><span class="s2">&quot;</span><span class="o">%</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">f6_for_grad</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The computed derivative of f6_while at x = </span><span class="si">%g</span><span class="s2"> is: </span><span class="si">%g</span><span class="s2">&quot;</span><span class="o">%</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">f6_while_grad</span><span class="p">(</span><span class="n">x</span><span class="p">)))</span>
+</pre></div>
+</div>
+</div>
+<div class="cell_output docutils container">
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The computed derivative of f6_for at x = 0.5 is: 3.95703
+The computed derivative of f6_while at x = 0.5 is: 3.95703
+</pre></div>
+</div>
+</div>
+</div>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
@@ -1944,7 +2015,7 @@ d</p>
 </div>
 </div>
 <div class="cell_output docutils container">
-<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The analytical derivative of f6 at x = 2.7 is: 37732.5
+<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The analytical derivative of f6 at x = 0.5 is: 3.95703
 </pre></div>
 </div>
 </div>
@@ -1989,49 +2060,31 @@ The analytical derivative of f7 at n = 2 is: 1
 </div>
 </div>
 <p>Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.</p>
-<p>Autograd supports many features. However, there are some functions that are not supported (yet) by Autograd.</p>
-<p>Assigning a value to the variable being differentiated with respect to is an example thereof.</p>
-<div class="cell docutils container">
-<div class="cell_input docutils container">
-<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1">#import autograd.numpy as np</span>
-<span class="c1">#from autograd import grad</span>
-<span class="c1">#def f8(x): # Assume x is an array</span>
-<span class="c1">#    x[2] = 3</span>
-<span class="c1">#    return x*2</span>
-
-<span class="c1">#f8_grad = grad(f8)</span>
-
-<span class="c1">#x = 8.4</span>
-
-<span class="c1">#print(&quot;The derivative of f8 is:&quot;,f8_grad(x))</span>
-</pre></div>
-</div>
-</div>
-</div>
-<p>Here, Autograd tells us that an ‘ArrayBox’ does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.</p>
+<p>Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.</p>
+<p>Assigning a value to the variable being differentiated with respect to</p>
 <div class="cell docutils container">
 <div class="cell_input docutils container">
 <div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
 <span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
-<span class="k">def</span> <span class="nf">f9</span><span class="p">(</span><span class="n">a</span><span class="p">):</span> <span class="c1"># Assume a is an array with 2 elements</span>
-    <span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">])</span>
-    <span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
+<span class="k">def</span> <span class="nf">f8</span><span class="p">(</span><span class="n">x</span><span class="p">):</span> <span class="c1"># Assume x is an array</span>
+    <span class="n">x</span><span class="p">[</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3</span>
+    <span class="k">return</span> <span class="n">x</span><span class="o">*</span><span class="mi">2</span>

-<span class="n">f9_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f9</span><span class="p">)</span>
+<span class="n">f8_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f8</span><span class="p">)</span>

-<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">0.0</span><span class="p">])</span>
+<span class="n">x</span> <span class="o">=</span> <span class="mf">8.4</span>

-<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f9 is:&quot;</span><span class="p">,</span><span class="n">f9_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f8 is:&quot;</span><span class="p">,</span><span class="n">f8_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
 </pre></div>
 </div>
 </div>
 <div class="cell_output docutils container">
 <div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span>---------------------------------------------------------------------------
-AttributeError                            Traceback (most recent call last)
-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42573/546166676.py in &lt;module&gt;
-      9 x = np.array([1.0,0.0])
+TypeError                                 Traceback (most recent call last)
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47735/1122558214.py in &lt;module&gt;
+      9 x = 8.4
     10 
---&gt; 11 print(&quot;The derivative of f9 is:&quot;,f9_grad(x))
+---&gt; 11 print(&quot;The derivative of f8 is:&quot;,f8_grad(x))

 ~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py in nary_f(*args, **kwargs)
     18             else:
@@ -2068,14 +2121,32 @@ AttributeError                            Traceback (most recent call last)
     16             if isinstance(argnum, int):
     17                 x = args[argnum]

-/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_42573/546166676.py in f9(a)
-      3 def f9(a): # Assume a is an array with 2 elements
-      4     b = np.array([1.0,2.0])
----&gt; 5     return a.dot(b)
+/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_47735/1122558214.py in f8(x)
+      2 from autograd import grad
+      3 def f8(x): # Assume x is an array
+----&gt; 4     x[2] = 3
+      5     return x*2
      6 
-      7 f9_grad = grad(f9)

-AttributeError: &#39;ArrayBox&#39; object has no attribute &#39;dot&#39;
+TypeError: &#39;ArrayBox&#39; object does not support item assignment
+</pre></div>
+</div>
+</div>
+</div>
+<p>Here, Autograd tells us that an ‘ArrayBox’ does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.</p>
+<div class="cell docutils container">
+<div class="cell_input docutils container">
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
+<span class="k">def</span> <span class="nf">f9</span><span class="p">(</span><span class="n">a</span><span class="p">):</span> <span class="c1"># Assume a is an array with 2 elements</span>
+    <span class="n">b</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">])</span>
+    <span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
+
+<span class="n">f9_grad</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">f9</span><span class="p">)</span>
+
+<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">1.0</span><span class="p">,</span><span class="mf">0.0</span><span class="p">])</span>
+
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;The derivative of f9 is:&quot;</span><span class="p">,</span><span class="n">f9_grad</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
 </pre></div>
 </div>
 </div>
@@ -2114,7 +2185,192 @@ which also computed the dot product can be used:</p>
 </div>
 </div>
 </div>
-<p>More examples will be added, in particular how to compare autograd with own codes for the gradients.</p>
+</div>
+<div class="section" id="using-autograd-with-ols">
+<h2><span class="section-number">7.11. </span>Using Autograd with OLS<a class="headerlink" href="#using-autograd-with-ols" title="Permalink to this headline">¶</a></h2>
+<p>We conclude the part on optmization by showing how we can make codes
+for linear regression and logistic regression using <strong>autograd</strong>. The
+first example shows results with ordinary leats squares.</p>
+<div class="cell docutils container">
+<div class="cell_input docutils container">
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients for OLS</span>
+<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
+<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
+<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
+
+<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">beta</span><span class="p">):</span>
+    <span class="k">return</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">beta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
+
+<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
+<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+
+<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
+<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
+<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
+<span class="c1"># Hessian matrix</span>
+<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
+<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
+
+<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
+<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">1000</span>
+<span class="c1"># define the gradient</span>
+<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">)</span>
+
+<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
+    <span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
+    <span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own gd&quot;</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
+
+<span class="n">xnew</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">0</span><span class="p">],[</span><span class="mi">2</span><span class="p">]])</span>
+<span class="n">Xnew</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">xnew</span><span class="p">]</span>
+<span class="n">ypredict</span> <span class="o">=</span> <span class="n">Xnew</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
+<span class="n">ypredict2</span> <span class="o">=</span> <span class="n">Xnew</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
+
+<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">xnew</span><span class="p">,</span> <span class="n">ypredict</span><span class="p">,</span> <span class="s2">&quot;r-&quot;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">xnew</span><span class="p">,</span> <span class="n">ypredict2</span><span class="p">,</span> <span class="s2">&quot;b-&quot;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">,</span><span class="s1">&#39;ro&#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span> <span class="mf">15.0</span><span class="p">])</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;$x$&#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;$y$&#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;Random numbers &#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="including-stochastic-gradient-descent-with-autograd">
+<h3><span class="section-number">7.11.1. </span>Including Stochastic Gradient Descent with Autograd<a class="headerlink" href="#including-stochastic-gradient-descent-with-autograd" title="Permalink to this headline">¶</a></h3>
+<p>In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using <strong>autograd</strong>.</p>
+<div class="cell docutils container">
+<div class="cell_input docutils container">
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using SGD</span>
+<span class="c1"># OLS example</span>
+<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
+<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
+<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
+
+<span class="c1"># Note change from previous example</span>
+<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
+
+<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
+<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+
+<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
+<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
+<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
+<span class="c1"># Hessian matrix</span>
+<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
+<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
+
+<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
+<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">1000</span>
+
+<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
+<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
+
+<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
+    <span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
+    <span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own gd&quot;</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
+
+<span class="n">xnew</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mi">0</span><span class="p">],[</span><span class="mi">2</span><span class="p">]])</span>
+<span class="n">Xnew</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">xnew</span><span class="p">]</span>
+<span class="n">ypredict</span> <span class="o">=</span> <span class="n">Xnew</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
+<span class="n">ypredict2</span> <span class="o">=</span> <span class="n">Xnew</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
+
+<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">xnew</span><span class="p">,</span> <span class="n">ypredict</span><span class="p">,</span> <span class="s2">&quot;r-&quot;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">xnew</span><span class="p">,</span> <span class="n">ypredict2</span><span class="p">,</span> <span class="s2">&quot;b-&quot;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span> <span class="p">,</span><span class="s1">&#39;ro&#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span><span class="mf">2.0</span><span class="p">,</span><span class="mi">0</span><span class="p">,</span> <span class="mf">15.0</span><span class="p">])</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;$x$&#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;$y$&#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="sa">r</span><span class="s1">&#39;Random numbers &#39;</span><span class="p">)</span>
+<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
+
+<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
+<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span>   <span class="c1">#size of each minibatch</span>
+<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
+<span class="n">t0</span><span class="p">,</span> <span class="n">t1</span> <span class="o">=</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">50</span>
+<span class="k">def</span> <span class="nf">learning_schedule</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
+    <span class="k">return</span> <span class="n">t0</span><span class="o">/</span><span class="p">(</span><span class="n">t</span><span class="o">+</span><span class="n">t1</span><span class="p">)</span>
+
+<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
+
+<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
+<span class="c1"># Can you figure out a better way of setting up the contributions to each batch?</span>
+    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
+        <span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
+        <span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
+        <span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
+        <span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
+        <span class="n">eta</span> <span class="o">=</span> <span class="n">learning_schedule</span><span class="p">(</span><span class="n">epoch</span><span class="o">*</span><span class="n">m</span><span class="o">+</span><span class="n">i</span><span class="p">)</span>
+        <span class="n">theta</span> <span class="o">=</span> <span class="n">theta</span> <span class="o">-</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own sdg&quot;</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+</div>
+<div class="section" id="and-logistic-regression">
+<h3><span class="section-number">7.11.2. </span>And Logistic Regression<a class="headerlink" href="#and-logistic-regression" title="Permalink to this headline">¶</a></h3>
+<div class="cell docutils container">
+<div class="cell_input docutils container">
+<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
+<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
+
+<span class="k">def</span> <span class="nf">sigmoid</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
+    <span class="k">return</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">x</span> <span class="o">/</span> <span class="mf">2.</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
+
+<span class="k">def</span> <span class="nf">logistic_predictions</span><span class="p">(</span><span class="n">weights</span><span class="p">,</span> <span class="n">inputs</span><span class="p">):</span>
+    <span class="c1"># Outputs probability of a label being true according to logistic model.</span>
+    <span class="k">return</span> <span class="n">sigmoid</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">weights</span><span class="p">))</span>
+
+<span class="k">def</span> <span class="nf">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">):</span>
+    <span class="c1"># Training loss is the negative log-likelihood of the training labels.</span>
+    <span class="n">preds</span> <span class="o">=</span> <span class="n">logistic_predictions</span><span class="p">(</span><span class="n">weights</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
+    <span class="n">label_probabilities</span> <span class="o">=</span> <span class="n">preds</span> <span class="o">*</span> <span class="n">targets</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">preds</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span>
+    <span class="k">return</span> <span class="o">-</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">label_probabilities</span><span class="p">))</span>
+
+<span class="c1"># Build a toy dataset.</span>
+<span class="n">inputs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mf">0.52</span><span class="p">,</span> <span class="mf">1.12</span><span class="p">,</span>  <span class="mf">0.77</span><span class="p">],</span>
+                   <span class="p">[</span><span class="mf">0.88</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.08</span><span class="p">,</span> <span class="mf">0.15</span><span class="p">],</span>
+                   <span class="p">[</span><span class="mf">0.52</span><span class="p">,</span> <span class="mf">0.06</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.30</span><span class="p">],</span>
+                   <span class="p">[</span><span class="mf">0.74</span><span class="p">,</span> <span class="o">-</span><span class="mf">2.49</span><span class="p">,</span> <span class="mf">1.39</span><span class="p">]])</span>
+<span class="n">targets</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">])</span>
+
+<span class="c1"># Define a function that returns gradients of training loss using Autograd.</span>
+<span class="n">training_gradient_fun</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">training_loss</span><span class="p">)</span>
+
+<span class="c1"># Optimize weights using gradient descent.</span>
+<span class="n">weights</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">])</span>
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Initial loss:&quot;</span><span class="p">,</span> <span class="n">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">))</span>
+<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
+    <span class="n">weights</span> <span class="o">-=</span> <span class="n">training_gradient_fun</span><span class="p">(</span><span class="n">weights</span><span class="p">)</span> <span class="o">*</span> <span class="mf">0.01</span>
+
+<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Trained loss:&quot;</span><span class="p">,</span> <span class="n">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">))</span>
+</pre></div>
+</div>
+</div>
+</div>
+</div>
 </div>
 </div>