lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-10-05 07:55:07 +02:00
parent b00518e915
commit 031d352181
31 changed files with 3581 additions and 2153 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapteroptimization.html
+412 -73
View File
@@ -389,22 +389,37 @@ const thebe_selector_output = ".output, .cell_output"
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-autograd-with-ols">
7.11. Using Autograd with OLS
<a class="reference internal nav-link" href="#replace-or-not">
7.11. Replace or not
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-autograd">
7.12. Using Autograd
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#same-code-but-now-with-momentum-gradient-descent">
7.13. Same code but now with momentum gradient descent
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
7.14. Including Stochastic Gradient Descent with Autograd
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
7.11.1. Including Stochastic Gradient Descent with Autograd
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#and-logistic-regression">
7.11.2. And Logistic Regression
<a class="reference internal nav-link" href="#similar-second-order-function-now-problem-but-now-with-adagrad">
7.14.1. Similar (second order function now) problem but now with AdaGrad
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introducing-jax">
7.15. Introducing JAX
</a>
</li>
</ul>
</nav>
@@ -502,22 +517,37 @@ const thebe_selector_output = ".output, .cell_output"
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-autograd-with-ols">
7.11. Using Autograd with OLS
<a class="reference internal nav-link" href="#replace-or-not">
7.11. Replace or not
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-autograd">
7.12. Using Autograd
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#same-code-but-now-with-momentum-gradient-descent">
7.13. Same code but now with momentum gradient descent
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
7.14. Including Stochastic Gradient Descent with Autograd
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#including-stochastic-gradient-descent-with-autograd">
7.11.1. Including Stochastic Gradient Descent with Autograd
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#and-logistic-regression">
7.11.2. And Logistic Regression
<a class="reference internal nav-link" href="#similar-second-order-function-now-problem-but-now-with-adagrad">
7.14.1. Similar (second order function now) problem but now with AdaGrad
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introducing-jax">
7.15. Introducing JAX
</a>
</li>
</ul>
</nav>
@@ -735,7 +765,18 @@ connecting <span class="math notranslate nohighlight">\(x\)</span> and <span cla
<p>The convex subsets of <span class="math notranslate nohighlight">\(\mathbb{R}\)</span> are the intervals of
<span class="math notranslate nohighlight">\(\mathbb{R}\)</span>. Examples of convex sets of <span class="math notranslate nohighlight">\(\mathbb{R}^2\)</span> are the
regular polygons (triangles, rectangles, pentagons, etc…).</p>
<p><strong>Convex function</strong>: Let <span class="math notranslate nohighlight">\(X \subset \mathbb{R}^n\)</span> be a convex set. Assume that the function <span class="math notranslate nohighlight">\(f: X \rightarrow \mathbb{R}\)</span> is continuous, then <span class="math notranslate nohighlight">\(f\)</span> is said to be convex if $<span class="math notranslate nohighlight">\(f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2) \)</span><span class="math notranslate nohighlight">\( for all \)</span>x_1, x_2 \in X<span class="math notranslate nohighlight">\( and for all \)</span>t \in [0,1]<span class="math notranslate nohighlight">\(. If \)</span>\leq<span class="math notranslate nohighlight">\( is replaced with a strict inequaltiy in the definition, we demand \)</span>x_1 \neq x_2<span class="math notranslate nohighlight">\( and \)</span>t\in(0,1)<span class="math notranslate nohighlight">\( then \)</span>f<span class="math notranslate nohighlight">\( is said to be strictly convex. For a single variable function, convexity means that if you draw a straight line connecting \)</span>f(x_1)<span class="math notranslate nohighlight">\( and \)</span>f(x_2)<span class="math notranslate nohighlight">\(, the value of the function on the interval \)</span>[x_1,x_2]$ is always below the line as illustrated below.</p>
<p><strong>Convex function</strong>: Let <span class="math notranslate nohighlight">\(X \subset \mathbb{R}^n\)</span> be a convex
set. Assume that the function <span class="math notranslate nohighlight">\(f: X \rightarrow \mathbb{R}\)</span> is
continuous, then <span class="math notranslate nohighlight">\(f\)</span> is said to be convex if
<span class="math notranslate nohighlight">\(f(tx_1 + (1-t)x_2) \leq tf(x_1) + (1-t)f(x_2)\)</span>
for all
<span class="math notranslate nohighlight">\(x_1, x_2 \in X\)</span> and for all <span class="math notranslate nohighlight">\(t \in [0,1]\)</span>.</p>
<p>If <span class="math notranslate nohighlight">\(\leq\)</span> is replaced with a strict inequality in the
definition, we demand <span class="math notranslate nohighlight">\(x_1 \neq x_2\)</span> and <span class="math notranslate nohighlight">\(t\in(0,1)\)</span> then <span class="math notranslate nohighlight">\(f\)</span> is said
to be strictly convex. For a single variable function, convexity means
that if you draw a straight line connecting <span class="math notranslate nohighlight">\(f(x_1)\)</span> and <span class="math notranslate nohighlight">\(f(x_2)\)</span>, the
value of the function on the interval <span class="math notranslate nohighlight">\([x_1,x_2]\)</span> is always below the
line as discussed below.</p>
<p>In the following we state first and second-order conditions which
ensures convexity of a function <span class="math notranslate nohighlight">\(f\)</span>. We write <span class="math notranslate nohighlight">\(D_f\)</span> to denote the
domain of <span class="math notranslate nohighlight">\(f\)</span>, i.e the subset of <span class="math notranslate nohighlight">\(R^n\)</span> where <span class="math notranslate nohighlight">\(f\)</span> is defined. For more
@@ -746,7 +787,7 @@ all <span class="math notranslate nohighlight">\(x\)</span> in the domain of <sp
is a convex set and $<span class="math notranslate nohighlight">\(f(y) \geq f(x) + \nabla f(x)^T (y-x) \)</span><span class="math notranslate nohighlight">\( holds
for all \)</span>x,y \in D_f<span class="math notranslate nohighlight">\(. This condition means that for a convex function
the first order Taylor expansion (right hand side above) at any point
a global under estimator of the function. To convince yourself you can
is a global under estimator of the function. To convince yourself you can
make a drawing of \)</span>f(x) = x^2+1<span class="math notranslate nohighlight">\( and draw the tangent line to \)</span>f(x)$ and
note that it is always below the graph.</p>
<p><strong>Second order condition.</strong></p>
@@ -899,11 +940,11 @@ which equals</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94582/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_96694/483257001.py:18: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
ax = fig.gca(projection=&quot;3d&quot;)
</pre></div>
</div>
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x128ee8850&gt;
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x11db14850&gt;
</pre></div>
</div>
<img alt="_images/chapteroptimization_61_2.png" src="_images/chapteroptimization_61_2.png" />
@@ -961,7 +1002,7 @@ which equals</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x12946f2e0&gt;]
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[&lt;matplotlib.lines.Line2D at 0x11e09b370&gt;]
</pre></div>
</div>
<img alt="_images/chapteroptimization_69_1.png" src="_images/chapteroptimization_69_1.png" />
@@ -1218,11 +1259,11 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.34158665 3.94915262]
[[3.97117751]
[3.11850274]]
[[3.97117751]
[3.11850274]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[0.2831603 4.55553537]
[[3.91511388]
[3.13030182]]
[[3.91511388]
[3.13030182]]
</pre></div>
</div>
<img alt="_images/chapteroptimization_123_1.png" src="_images/chapteroptimization_123_1.png" />
@@ -1251,9 +1292,9 @@ when <span class="math notranslate nohighlight">\(||\nabla_\beta C(\beta_k) || \
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.40754621]
[2.78752269]]
[4.37713991] [2.77711437]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.1509778 ]
[2.92461411]]
[4.13288373] [2.92817032]
</pre></div>
</div>
</div>
@@ -1324,10 +1365,10 @@ C_{\text{ridge}}(\beta) = \frac{1}{n}||X\beta -\mathbf{y}||^2 + \lambda ||\beta|
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[3.94107596]
[2.96620033]]
[[3.96670977]
[2.94212937]]
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[4.0795449 ]
[2.86893619]]
[[4.04785727]
[2.89298533]]
</pre></div>
</div>
<img alt="_images/chapteroptimization_132_1.png" src="_images/chapteroptimization_132_1.png" />
@@ -1577,15 +1618,15 @@ function.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Own inversion
[[3.95446837]
[3.16961682]]
Eigenvalues of Hessian Matrix:[0.31447174 4.32459186]
[[4.41170104]
[2.6431453 ]]
Eigenvalues of Hessian Matrix:[0.31228042 4.55571665]
theta from own gd
[[3.95446837]
[3.16961682]]
[[4.41170104]
[2.6431453 ]]
theta from own sdg
[[3.91682433]
[3.13655438]]
[[4.39272691]
[2.63430285]]
</pre></div>
</div>
<img alt="_images/chapteroptimization_148_1.png" src="_images/chapteroptimization_148_1.png" />
@@ -2302,8 +2343,15 @@ which also computed the dot product can be used:</p>
</div>
</div>
</div>
<div class="section" id="using-autograd-with-ols">
<h2><span class="section-number">7.11. </span>Using Autograd with OLS<a class="headerlink" href="#using-autograd-with-ols" title="Permalink to this headline"></a></h2>
<div class="section" id="replace-or-not">
<h2><span class="section-number">7.11. </span>Replace or not<a class="headerlink" href="#replace-or-not" title="Permalink to this headline"></a></h2>
<p>In the above code, we have use replacement in setting up the
mini-batches. The discussion
<a class="reference external" href="https://sebastianraschka.com/faq/docs/sgd-methods.html">here</a> may be
useful.</p>
</div>
<div class="section" id="using-autograd">
<h2><span class="section-number">7.12. </span>Using Autograd<a class="headerlink" href="#using-autograd" title="Permalink to this headline"></a></h2>
<p>We conclude the part on optmization by showing how we can make codes
for linear regression and logistic regression using <strong>autograd</strong>. The
first example shows results with ordinary leats squares.</p>
@@ -2362,8 +2410,117 @@ first example shows results with ordinary leats squares.</p>
</div>
</div>
</div>
</div>
<div class="section" id="same-code-but-now-with-momentum-gradient-descent">
<h2><span class="section-number">7.13. </span>Same code but now with momentum gradient descent<a class="headerlink" href="#same-code-but-now-with-momentum-gradient-descent" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients for OLS</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">beta</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">beta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="c1">#+np.random.randn(n,1)</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
<span class="c1"># Hessian matrix</span>
<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">30</span>
<span class="c1"># define the gradient</span>
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">)</span>
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
<span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
<span class="nb">print</span><span class="p">(</span><span class="nb">iter</span><span class="p">,</span><span class="n">gradients</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">gradients</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own gd&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
<span class="c1"># Now improve with momentum gradient descent</span>
<span class="n">change</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">delta_momentum</span> <span class="o">=</span> <span class="mf">0.3</span>
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
<span class="c1"># calculate gradient</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
<span class="c1"># calculate update</span>
<span class="n">new_change</span> <span class="o">=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span><span class="o">+</span><span class="n">delta_momentum</span><span class="o">*</span><span class="n">change</span>
<span class="c1"># take a step</span>
<span class="n">theta</span> <span class="o">-=</span> <span class="n">new_change</span>
<span class="c1"># save the change</span>
<span class="n">change</span> <span class="o">=</span> <span class="n">new_change</span>
<span class="nb">print</span><span class="p">(</span><span class="nb">iter</span><span class="p">,</span><span class="n">gradients</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">gradients</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own gd wth momentum&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>We note indeed a considerable increase in efficiency here, we less iterations needed.
However, if we can invert the Hessian matrix, this is the preferred approach, as shown in the example here.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Newton&#39;s method</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">beta</span><span class="p">):</span>
<span class="k">return</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">beta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
<span class="n">beta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">beta_linreg</span><span class="p">)</span>
<span class="c1"># Hessian matrix</span>
<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
<span class="c1"># Note that here the Hessian does not depend on the parameters beta</span>
<span class="n">invH</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">5</span>
<span class="c1"># define the gradient</span>
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">)</span>
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="n">training_gradient</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
<span class="n">beta</span> <span class="o">-=</span> <span class="n">invH</span> <span class="o">@</span> <span class="n">gradients</span>
<span class="nb">print</span><span class="p">(</span><span class="nb">iter</span><span class="p">,</span><span class="n">gradients</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span><span class="n">gradients</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;beta from own Newton code&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="including-stochastic-gradient-descent-with-autograd">
<h3><span class="section-number">7.11.1. </span>Including Stochastic Gradient Descent with Autograd<a class="headerlink" href="#including-stochastic-gradient-descent-with-autograd" title="Permalink to this headline"></a></h3>
<h2><span class="section-number">7.14. </span>Including Stochastic Gradient Descent with Autograd<a class="headerlink" href="#including-stochastic-gradient-descent-with-autograd" title="Permalink to this headline"></a></h2>
<p>In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using <strong>autograd</strong>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -2444,48 +2601,230 @@ first example shows results with ordinary leats squares.</p>
</div>
</div>
</div>
</div>
<div class="section" id="and-logistic-regression">
<h3><span class="section-number">7.11.2. </span>And Logistic Regression<a class="headerlink" href="#and-logistic-regression" title="Permalink to this headline"></a></h3>
<p>Here we include momentum in the standard gradient descent approach.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using SGD</span>
<span class="c1"># OLS example</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="k">def</span> <span class="nf">sigmoid</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="mf">0.5</span> <span class="o">*</span> <span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">tanh</span><span class="p">(</span><span class="n">x</span> <span class="o">/</span> <span class="mf">2.</span><span class="p">)</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
<span class="c1"># Note change from previous example</span>
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">logistic_predictions</span><span class="p">(</span><span class="n">weights</span><span class="p">,</span> <span class="n">inputs</span><span class="p">):</span>
<span class="c1"># Outputs probability of a label being true according to logistic model.</span>
<span class="k">return</span> <span class="n">sigmoid</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">dot</span><span class="p">(</span><span class="n">inputs</span><span class="p">,</span> <span class="n">weights</span><span class="p">))</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mi">4</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">):</span>
<span class="c1"># Training loss is the negative log-likelihood of the training labels.</span>
<span class="n">preds</span> <span class="o">=</span> <span class="n">logistic_predictions</span><span class="p">(</span><span class="n">weights</span><span class="p">,</span> <span class="n">inputs</span><span class="p">)</span>
<span class="n">label_probabilities</span> <span class="o">=</span> <span class="n">preds</span> <span class="o">*</span> <span class="n">targets</span> <span class="o">+</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">preds</span><span class="p">)</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span> <span class="o">-</span> <span class="n">targets</span><span class="p">)</span>
<span class="k">return</span> <span class="o">-</span><span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="n">label_probabilities</span><span class="p">))</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">]</span>
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
<span class="c1"># Hessian matrix</span>
<span class="n">H</span> <span class="o">=</span> <span class="p">(</span><span class="mf">2.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span> <span class="n">XT_X</span>
<span class="n">EigValues</span><span class="p">,</span> <span class="n">EigVectors</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">eig</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Eigenvalues of Hessian Matrix:</span><span class="si">{</span><span class="n">EigValues</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="c1"># Build a toy dataset.</span>
<span class="n">inputs</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="mf">0.52</span><span class="p">,</span> <span class="mf">1.12</span><span class="p">,</span> <span class="mf">0.77</span><span class="p">],</span>
<span class="p">[</span><span class="mf">0.88</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.08</span><span class="p">,</span> <span class="mf">0.15</span><span class="p">],</span>
<span class="p">[</span><span class="mf">0.52</span><span class="p">,</span> <span class="mf">0.06</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.30</span><span class="p">],</span>
<span class="p">[</span><span class="mf">0.74</span><span class="p">,</span> <span class="o">-</span><span class="mf">2.49</span><span class="p">,</span> <span class="mf">1.39</span><span class="p">]])</span>
<span class="n">targets</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">])</span>
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">eta</span> <span class="o">=</span> <span class="mf">1.0</span><span class="o">/</span><span class="n">np</span><span class="o">.</span><span class="n">max</span><span class="p">(</span><span class="n">EigValues</span><span class="p">)</span>
<span class="n">Niterations</span> <span class="o">=</span> <span class="mi">100</span>
<span class="c1"># Define a function that returns gradients of training loss using Autograd.</span>
<span class="n">training_gradient_fun</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">training_loss</span><span class="p">)</span>
<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># Optimize weights using gradient descent.</span>
<span class="n">weights</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">,</span> <span class="mf">0.0</span><span class="p">])</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Initial loss:&quot;</span><span class="p">,</span> <span class="n">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">100</span><span class="p">):</span>
<span class="n">weights</span> <span class="o">-=</span> <span class="n">training_gradient_fun</span><span class="p">(</span><span class="n">weights</span><span class="p">)</span> <span class="o">*</span> <span class="mf">0.01</span>
<span class="k">for</span> <span class="nb">iter</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">Niterations</span><span class="p">):</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">n</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">y</span><span class="p">,</span> <span class="n">X</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
<span class="n">theta</span> <span class="o">-=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own gd&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Trained loss:&quot;</span><span class="p">,</span> <span class="n">training_loss</span><span class="p">(</span><span class="n">weights</span><span class="p">))</span>
<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1">#size of each minibatch</span>
<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
<span class="n">t0</span><span class="p">,</span> <span class="n">t1</span> <span class="o">=</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">50</span>
<span class="k">def</span> <span class="nf">learning_schedule</span><span class="p">(</span><span class="n">t</span><span class="p">):</span>
<span class="k">return</span> <span class="n">t0</span><span class="o">/</span><span class="p">(</span><span class="n">t</span><span class="o">+</span><span class="n">t1</span><span class="p">)</span>
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">change</span> <span class="o">=</span> <span class="mf">0.0</span>
<span class="n">delta_momentum</span> <span class="o">=</span> <span class="mf">0.3</span>
<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
<span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
<span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
<span class="n">eta</span> <span class="o">=</span> <span class="n">learning_schedule</span><span class="p">(</span><span class="n">epoch</span><span class="o">*</span><span class="n">m</span><span class="o">+</span><span class="n">i</span><span class="p">)</span>
<span class="c1"># calculate update</span>
<span class="n">new_change</span> <span class="o">=</span> <span class="n">eta</span><span class="o">*</span><span class="n">gradients</span><span class="o">+</span><span class="n">delta_momentum</span><span class="o">*</span><span class="n">change</span>
<span class="c1"># take a step</span>
<span class="n">theta</span> <span class="o">-=</span> <span class="n">new_change</span>
<span class="c1"># save the change</span>
<span class="n">change</span> <span class="o">=</span> <span class="n">new_change</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own sdg with momentum&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="similar-second-order-function-now-problem-but-now-with-adagrad">
<h3><span class="section-number">7.14.1. </span>Similar (second order function now) problem but now with AdaGrad<a class="headerlink" href="#similar-second-order-function-now-problem-but-now-with-adagrad" title="Permalink to this headline"></a></h3>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent</span>
<span class="c1"># OLS example</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="c1"># Note change from previous example</span>
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">10000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span><span class="mi">4</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="c1"># +np.random.randn(n,1)</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">,</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="p">]</span>
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># Define parameters for Stochastic Gradient Descent</span>
<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1">#size of each minibatch</span>
<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
<span class="c1"># Guess for unknown parameters theta</span>
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Value for learning rate</span>
<span class="n">eta</span> <span class="o">=</span> <span class="mf">0.01</span>
<span class="c1"># Including AdaGrad parameter to avoid possible division by zero</span>
<span class="n">delta</span> <span class="o">=</span> <span class="mf">1e-8</span>
<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
<span class="c1"># The outer product is calculated from scratch for each epoch</span>
<span class="n">Giter</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
<span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
<span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
<span class="c1"># Calculate the outer product of the gradients</span>
<span class="n">Giter</span> <span class="o">+=</span><span class="n">gradients</span> <span class="o">@</span> <span class="n">gradients</span><span class="o">.</span><span class="n">T</span>
<span class="c1"># Simpler algorithm with only diagonal elements</span>
<span class="n">Ginverse</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">eta</span><span class="o">/</span><span class="p">(</span><span class="n">delta</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">diagonal</span><span class="p">(</span><span class="n">Giter</span><span class="p">)))]</span>
<span class="c1"># compute update</span>
<span class="n">update</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">multiply</span><span class="p">(</span><span class="n">Ginverse</span><span class="p">,</span><span class="n">gradients</span><span class="p">)</span>
<span class="n">theta</span> <span class="o">-=</span> <span class="n">update</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own AdaGrad&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Running this code we note an almost perfect agreement with the results from matrix inversion.</p>
<p>Similarly, here is our implementation of RMSprop.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent</span>
<span class="c1"># OLS example</span>
<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span><span class="p">,</span> <span class="n">seed</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">autograd.numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">autograd</span> <span class="kn">import</span> <span class="n">grad</span>
<span class="c1"># Note change from previous example</span>
<span class="k">def</span> <span class="nf">CostOLS</span><span class="p">(</span><span class="n">y</span><span class="p">,</span><span class="n">X</span><span class="p">,</span><span class="n">theta</span><span class="p">):</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y</span><span class="o">-</span><span class="n">X</span> <span class="o">@</span> <span class="n">theta</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">10000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">3</span><span class="o">*</span><span class="n">x</span> <span class="o">+</span><span class="mi">4</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="c1"># +np.random.randn(n,1)</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">np</span><span class="o">.</span><span class="n">ones</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="mi">1</span><span class="p">)),</span> <span class="n">x</span><span class="p">,</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="p">]</span>
<span class="n">XT_X</span> <span class="o">=</span> <span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X</span>
<span class="n">theta_linreg</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">pinv</span><span class="p">(</span><span class="n">XT_X</span><span class="p">)</span> <span class="o">@</span> <span class="p">(</span><span class="n">X</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Own inversion&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta_linreg</span><span class="p">)</span>
<span class="c1"># Note that we request the derivative wrt third argument (theta, 2 here)</span>
<span class="n">training_gradient</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">CostOLS</span><span class="p">,</span><span class="mi">2</span><span class="p">)</span>
<span class="c1"># Define parameters for Stochastic Gradient Descent</span>
<span class="n">n_epochs</span> <span class="o">=</span> <span class="mi">50</span>
<span class="n">M</span> <span class="o">=</span> <span class="mi">5</span> <span class="c1">#size of each minibatch</span>
<span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">n</span><span class="o">/</span><span class="n">M</span><span class="p">)</span> <span class="c1">#number of minibatches</span>
<span class="c1"># Guess for unknown parameters theta</span>
<span class="n">theta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># Value for learning rate</span>
<span class="n">eta</span> <span class="o">=</span> <span class="mf">0.01</span>
<span class="c1"># Value for parameter rho</span>
<span class="n">rho</span> <span class="o">=</span> <span class="mf">0.99</span>
<span class="c1"># Including AdaGrad parameter to avoid possible division by zero</span>
<span class="n">delta</span> <span class="o">=</span> <span class="mf">1e-8</span>
<span class="k">for</span> <span class="n">epoch</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">n_epochs</span><span class="p">):</span>
<span class="n">Giter</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">shape</span><span class="o">=</span><span class="p">(</span><span class="mi">3</span><span class="p">,</span><span class="mi">3</span><span class="p">))</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">m</span><span class="p">):</span>
<span class="n">random_index</span> <span class="o">=</span> <span class="n">M</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randint</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
<span class="n">xi</span> <span class="o">=</span> <span class="n">X</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
<span class="n">yi</span> <span class="o">=</span> <span class="n">y</span><span class="p">[</span><span class="n">random_index</span><span class="p">:</span><span class="n">random_index</span><span class="o">+</span><span class="n">M</span><span class="p">]</span>
<span class="n">gradients</span> <span class="o">=</span> <span class="p">(</span><span class="mf">1.0</span><span class="o">/</span><span class="n">M</span><span class="p">)</span><span class="o">*</span><span class="n">training_gradient</span><span class="p">(</span><span class="n">yi</span><span class="p">,</span> <span class="n">xi</span><span class="p">,</span> <span class="n">theta</span><span class="p">)</span>
<span class="c1"># Previous value for the outer product of gradients</span>
<span class="n">Previous</span> <span class="o">=</span> <span class="n">Giter</span>
<span class="c1"># Accumulated gradient</span>
<span class="n">Giter</span> <span class="o">+=</span><span class="n">gradients</span> <span class="o">@</span> <span class="n">gradients</span><span class="o">.</span><span class="n">T</span>
<span class="c1"># Scaling with rho the new and the previous results</span>
<span class="n">Gnew</span> <span class="o">=</span> <span class="p">(</span><span class="n">rho</span><span class="o">*</span><span class="n">Previous</span><span class="o">+</span><span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">rho</span><span class="p">)</span><span class="o">*</span><span class="n">Giter</span><span class="p">)</span>
<span class="c1"># Taking the diagonal only and inverting</span>
<span class="n">Ginverse</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">c_</span><span class="p">[</span><span class="n">eta</span><span class="o">/</span><span class="p">(</span><span class="n">delta</span><span class="o">+</span><span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">diagonal</span><span class="p">(</span><span class="n">Gnew</span><span class="p">)))]</span>
<span class="c1"># Hadamard product</span>
<span class="n">update</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">multiply</span><span class="p">(</span><span class="n">Ginverse</span><span class="p">,</span><span class="n">gradients</span><span class="p">)</span>
<span class="n">theta</span> <span class="o">-=</span> <span class="n">update</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;theta from own RMSprop&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">theta</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>
<div class="section" id="introducing-jax">
<h2><span class="section-number">7.15. </span>Introducing <a class="reference external" href="https://jax.readthedocs.io/en/latest/">JAX</a><a class="headerlink" href="#introducing-jax" title="Permalink to this headline"></a></h2>
<p>Presently, instead of using <strong>autograd</strong>, we recommend using <a class="reference external" href="https://jax.readthedocs.io/en/latest/">JAX</a></p>
<p><strong>JAX</strong> is Autograd and <a class="reference external" href="https://www.tensorflow.org/xla">XLA (Accelerated Linear Algebra))</a>,
brought together for high-performance numerical computing and machine learning research.
It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more.</p>
<p>Here’s a simple example on how you can use <strong>JAX</strong> to compute the derivate of the logistic function.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">jax.numpy</span> <span class="k">as</span> <span class="nn">jnp</span>
<span class="kn">from</span> <span class="nn">jax</span> <span class="kn">import</span> <span class="n">grad</span><span class="p">,</span> <span class="n">jit</span><span class="p">,</span> <span class="n">vmap</span>
<span class="k">def</span> <span class="nf">sum_logistic</span><span class="p">(</span><span class="n">x</span><span class="p">):</span>
<span class="k">return</span> <span class="n">jnp</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="mf">1.0</span> <span class="o">/</span> <span class="p">(</span><span class="mf">1.0</span> <span class="o">+</span> <span class="n">jnp</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="p">)))</span>
<span class="n">x_small</span> <span class="o">=</span> <span class="n">jnp</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mf">3.</span><span class="p">)</span>
<span class="n">derivative_fn</span> <span class="o">=</span> <span class="n">grad</span><span class="p">(</span><span class="n">sum_logistic</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">derivative_fn</span><span class="p">(</span><span class="n">x_small</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>