lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update on descent methods

Browse Source
This commit is contained in:
mhjensen
2018-09-27 05:42:32 +02:00
parent a55882e5a4
commit 08f5bf245b
65 changed files with 10948 additions and 992 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Splines/html/Splines.html
+495 -24
View File
@@ -95,16 +95,16 @@ div { text-align: justify; text-justify: inter-word; }
('Steepest descent method', 2, None, '___sec20'),
('Gradient descent method', 2, None, '___sec21'),
('Final expressions', 2, None, '___sec22'),
('The Steepest descent algorithm', 2, None, '___sec23'),
('Simple codes for steepest descent and conjugate gradient '
'using a $2\\times 2$ matrix, in c++, Python code to come',
2,
None,
'___sec24'),
'___sec23'),
('The routine for the steepest descent method',
2,
None,
'___sec25'),
'___sec24'),
('Steepest descent example', 2, None, '___sec25'),
('Revisiting our first homework', 2, None, '___sec26'),
('Gradient descent example', 2, None, '___sec27'),
('The derivative of the cost/loss function', 2, None, '___sec28'),
@@ -116,13 +116,34 @@ div { text-align: justify; text-justify: inter-word; }
None,
'___sec32'),
('Gradient descent and Ridge', 2, None, '___sec33'),
('Stochastic Gradient Descent', 2, None, '___sec34'),
('Computation of gradients', 2, None, '___sec35'),
('SGD example', 2, None, '___sec36'),
('The gradient step', 2, None, '___sec37'),
('Simple example code', 2, None, '___sec38'),
('When do we stop?', 2, None, '___sec39'),
('Slightly different approach', 2, None, '___sec40')]}
('Automatic differentiation', 2, None, '___sec34'),
('Using autograd', 2, None, '___sec35'),
('Autograd with more complicated functions', 2, None, '___sec36'),
('More complicated functions using the elements of their '
'arguments directly',
2,
None,
'___sec37'),
('Functions using mathematical functions from Numpy',
2,
None,
'___sec38'),
('More autograd', 2, None, '___sec39'),
('And with loops', 2, None, '___sec40'),
('Using recursion', 2, None, '___sec41'),
('Unsupported functions', 2, None, '___sec42'),
('The syntax a.dot(b) when finding the dot product',
2,
None,
'___sec43'),
('Recommended to avoid', 2, None, '___sec44'),
('Stochastic Gradient Descent', 2, None, '___sec45'),
('Computation of gradients', 2, None, '___sec46'),
('SGD example', 2, None, '___sec47'),
('The gradient step', 2, None, '___sec48'),
('Simple example code', 2, None, '___sec49'),
('When do we stop?', 2, None, '___sec50'),
('Slightly different approach', 2, None, '___sec51')]}
end of tocinfo -->
<body>
@@ -772,12 +793,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">The Steepest descent algorithm </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
<h2 id="___sec23">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -817,7 +833,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">The routine for the steepest descent method </h2>
<h2 id="___sec24">The routine for the steepest descent method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -850,6 +866,75 @@ $$
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Steepest descent example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy.linalg</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">la</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">scipy.optimize</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">sopt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">mpl_toolkits.mplot3d</span> <span style="color: #008000; font-weight: bold">import</span> axes3d
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f</span>(x):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">0.5*</span>x[<span style="color: #666666">0</span>]<span style="color: #666666">**2</span> <span style="color: #666666">+</span> <span style="color: #666666">2.5*</span>x[<span style="color: #666666">1</span>]<span style="color: #666666">**2</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">df</span>(x):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>array([x[<span style="color: #666666">0</span>], <span style="color: #666666">5*</span>x[<span style="color: #666666">1</span>]])
fig <span style="color: #666666">=</span> pt<span style="color: #666666">.</span>figure()
ax <span style="color: #666666">=</span> fig<span style="color: #666666">.</span>gca(projection<span style="color: #666666">=</span><span style="color: #BA2121">&quot;3d&quot;</span>)
xmesh, ymesh <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mgrid[<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>,<span style="color: #666666">-2</span>:<span style="color: #666666">2</span>:<span style="color: #666666">50j</span>]
fmesh <span style="color: #666666">=</span> f(np<span style="color: #666666">.</span>array([xmesh, ymesh]))
ax<span style="color: #666666">.</span>plot_surface(xmesh, ymesh, fmesh)
</pre></div>
<p>
And then as countor plot
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&quot;equal&quot;</span>)
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh)
guesses <span style="color: #666666">=</span> [np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">2./5</span>])]
</pre></div>
<p>
Find guesses
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>x <span style="color: #666666">=</span> guesses[<span style="color: #666666">-1</span>]
s <span style="color: #666666">=</span> <span style="color: #666666">-</span>df(x)
</pre></div>
<p>
Run it!
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1d</span>(alpha):
<span style="color: #008000; font-weight: bold">return</span> f(x <span style="color: #666666">+</span> alpha<span style="color: #666666">*</span>s)
alpha_opt <span style="color: #666666">=</span> sopt<span style="color: #666666">.</span>golden(f1d)
next_guess <span style="color: #666666">=</span> x <span style="color: #666666">+</span> alpha_opt <span style="color: #666666">*</span> s
guesses<span style="color: #666666">.</span>append(next_guess)
<span style="color: #008000; font-weight: bold">print</span>(next_guess)
</pre></div>
<p>
What happened?
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pt<span style="color: #666666">.</span>axis(<span style="color: #BA2121">&quot;equal&quot;</span>)
pt<span style="color: #666666">.</span>contour(xmesh, ymesh, fmesh, <span style="color: #666666">50</span>)
it_array <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array(guesses)
pt<span style="color: #666666">.</span>plot(it_array<span style="color: #666666">.</span>T[<span style="color: #666666">0</span>], it_array<span style="color: #666666">.</span>T[<span style="color: #666666">1</span>], <span style="color: #BA2121">&quot;x-&quot;</span>)
</pre></div>
<p>
<!-- !split -->
@@ -1111,7 +1196,393 @@ beta_ridge <span style="color: #666666">=</span> np<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">Stochastic Gradient Descent </h2>
<h2 id="___sec34">Automatic differentiation </h2>
Python has tools for so-called <b>automatic differentiation</b>.
Consider the following example
$$
f(x) = \sin\left(2\pi x + x^2\right)
$$
which has the following derivative
$$
f'(x) = \cos\left(2\pi x + x^2\right)\left(2\pi + 2x\right)
$$
Using <b>autograd</b> we have
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic"># To do elementwise differentiation:</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> elementwise_grad <span style="color: #008000; font-weight: bold">as</span> egrad
<span style="color: #408080; font-style: italic"># To plot:</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f</span>(x):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f_grad_analytic</span>(x):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>)<span style="color: #666666">*</span>(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi <span style="color: #666666">+</span> <span style="color: #666666">2*</span>x)
<span style="color: #408080; font-style: italic"># Do the comparison:</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">1</span>,<span style="color: #666666">1000</span>)
f_grad <span style="color: #666666">=</span> egrad(f)
computed <span style="color: #666666">=</span> f_grad(x)
analytic <span style="color: #666666">=</span> f_grad_analytic(x)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&#39;Derivative computed from Autograd compared with the analytical derivative&#39;</span>)
plt<span style="color: #666666">.</span>plot(x,computed,label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;autograd&#39;</span>)
plt<span style="color: #666666">.</span>plot(x,analytic,label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;analytic&#39;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&#39;x&#39;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39;y&#39;</span>)
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The max absolute difference is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(np<span style="color: #666666">.</span>max(np<span style="color: #666666">.</span>abs(computed <span style="color: #666666">-</span> analytic))))
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec35">Using autograd </h2>
<p>
Here we
experiment with what kind of functions Autograd is capable
of finding the gradient of. The following Python functions are just
meant to illustrate what Autograd can do, but please feel free to
experiment with other, possibly more complicated, functions as well.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f1</span>(x):
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">**3</span> <span style="color: #666666">+</span> <span style="color: #666666">1</span>
f1_grad <span style="color: #666666">=</span> grad(f1)
<span style="color: #408080; font-style: italic"># Remember to send in float as argument to the computed gradient from Autograd!</span>
a <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
<span style="color: #408080; font-style: italic"># See the evaluated gradient at a using autograd:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> using autograd is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,f1_grad(a)))
<span style="color: #408080; font-style: italic"># Compare with the analytical derivative, that is f1&#39;(x) = 3*x**2 </span>
grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">3*</span>a<span style="color: #666666">**2</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f1 evaluated at a = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> by finding the analytic expression is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(a,grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Autograd with more complicated functions </h2>
<p>
To differentiate with respect to two (or more) arguments of a Python
function, Autograd need to know at which variable the function if
being differentiated with respect to.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f2</span>(x1,x2):
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">3*</span>x1<span style="color: #666666">**3</span> <span style="color: #666666">+</span> x2<span style="color: #666666">*</span>(x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
<span style="color: #408080; font-style: italic"># By sending the argument 0, Autograd will compute the derivative w.r.t the first variable, in this case x1</span>
f2_grad_x1 <span style="color: #666666">=</span> grad(f2,<span style="color: #666666">0</span>)
<span style="color: #408080; font-style: italic"># ... and differentiate w.r.t x2 by sending 1 as an additional arugment to grad</span>
f2_grad_x2 <span style="color: #666666">=</span> grad(f2,<span style="color: #666666">1</span>)
x1 <span style="color: #666666">=</span> <span style="color: #666666">1.0</span>
x2 <span style="color: #666666">=</span> <span style="color: #666666">3.0</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Evaluating at x1 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">, x2 = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x1,x2))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;-&quot;</span><span style="color: #666666">*30</span>)
<span style="color: #408080; font-style: italic"># Compare with the analytical derivatives:</span>
<span style="color: #408080; font-style: italic"># Derivative of f2 w.r.t x1 is: 9*x1**2 + x2:</span>
f2_grad_x1_analytical <span style="color: #666666">=</span> <span style="color: #666666">9*</span>x1<span style="color: #666666">**2</span> <span style="color: #666666">+</span> x2
<span style="color: #408080; font-style: italic"># Derivative of f2 w.r.t x2 is: x1 - 5:</span>
f2_grad_x2_analytical <span style="color: #666666">=</span> x1 <span style="color: #666666">-</span> <span style="color: #666666">5</span>
<span style="color: #408080; font-style: italic"># See the evaluated derivations:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x1: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x1(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>()
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f2 w.r.t x2: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>( f2_grad_x2(x1,x2) ))
</pre></div>
<p>
Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">More complicated functions using the elements of their arguments directly </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f3</span>(x): <span style="color: #408080; font-style: italic"># Assumes x is an array of length 5 or higher</span>
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">2*</span>x[<span style="color: #666666">0</span>] <span style="color: #666666">+</span> <span style="color: #666666">3*</span>x[<span style="color: #666666">1</span>] <span style="color: #666666">+</span> <span style="color: #666666">5*</span>x[<span style="color: #666666">2</span>] <span style="color: #666666">+</span> <span style="color: #666666">7*</span>x[<span style="color: #666666">3</span>] <span style="color: #666666">+</span> <span style="color: #666666">11*</span>x[<span style="color: #666666">4</span>]<span style="color: #666666">**2</span>
f3_grad <span style="color: #666666">=</span> grad(f3)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">0</span>,<span style="color: #666666">4</span>,<span style="color: #666666">5</span>)
<span style="color: #408080; font-style: italic"># Print the computed gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed gradient of f3 is: &quot;</span>, f3_grad(x))
<span style="color: #408080; font-style: italic"># The analytical gradient is: (2, 3, 5, 7, 22*x[4])</span>
f3_grad_analytical <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">2</span>, <span style="color: #666666">3</span>, <span style="color: #666666">5</span>, <span style="color: #666666">7</span>, <span style="color: #666666">22*</span>x[<span style="color: #666666">4</span>]])
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f3 is: &quot;</span>, f3_grad_analytical)
</pre></div>
<p>
Note that in this case, when sending an array as input argument, the
output from Autograd is another array. This is the true gradient of
the function, as opposed to the function in the previous example. By
using arrays to represent the variables, the output from Autograd
might be easier to work with, as the output is closer to what one
could expect form a gradient-evaluting function.
<p>
<!-- !split -->
<h2 id="___sec38">Functions using mathematical functions from Numpy </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f4</span>(x):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1+</span>x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>sin(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)
f4_grad <span style="color: #666666">=</span> grad(f4)
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad(x)))
<span style="color: #408080; font-style: italic"># The analytical derivative is: x/sqrt(1 + x**2) + exp(x) + cos(2*pi*x)*2*pi</span>
f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #666666">/</span>np<span style="color: #666666">.</span>sqrt(<span style="color: #666666">1</span> <span style="color: #666666">+</span> x<span style="color: #666666">**2</span>) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>exp(x) <span style="color: #666666">+</span> np<span style="color: #666666">.</span>cos(<span style="color: #666666">2*</span>np<span style="color: #666666">.</span>pi<span style="color: #666666">*</span>x)<span style="color: #666666">*2*</span>np<span style="color: #666666">.</span>pi
<span style="color: #408080; font-style: italic"># Print the analytical gradient:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical gradient of f4 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f4_grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">More autograd </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f5</span>(x):
<span style="color: #008000; font-weight: bold">if</span> x <span style="color: #666666">&gt;=</span> <span style="color: #666666">0</span>:
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">**2</span>
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">-3*</span>x <span style="color: #666666">+</span> <span style="color: #666666">1</span>
f5_grad <span style="color: #666666">=</span> grad(f5)
x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<span style="color: #408080; font-style: italic"># Print the computed derivative:</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f5 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f5_grad(x)))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">And with loops </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_for</span>(x):
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
val <span style="color: #666666">=</span> val <span style="color: #666666">+</span> x<span style="color: #666666">**</span>i
<span style="color: #008000; font-weight: bold">return</span> val
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f6_while</span>(x):
val <span style="color: #666666">=</span> <span style="color: #666666">0</span>
i <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">while</span> i <span style="color: #666666">&lt;</span> <span style="color: #666666">10</span>:
val <span style="color: #666666">=</span> val <span style="color: #666666">+</span> x<span style="color: #666666">**</span>i
i <span style="color: #666666">=</span> i <span style="color: #666666">+</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">return</span> val
f6_for_grad <span style="color: #666666">=</span> grad(f6_for)
f6_while_grad <span style="color: #666666">=</span> grad(f6_while)
x <span style="color: #666666">=</span> <span style="color: #666666">0.5</span>
<span style="color: #408080; font-style: italic"># Print the computed derivaties of f6_for and f6_while</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_for at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_for_grad(x)))
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f6_while at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_while_grad(x)))
</pre></div>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #408080; font-style: italic"># Both of the functions are implementation of the sum: sum(x**i) for i = 0, ..., 9</span>
<span style="color: #408080; font-style: italic"># The analytical derivative is: sum(i*x**(i-1)) </span>
f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">10</span>):
f6_grad_analytical <span style="color: #666666">+=</span> i<span style="color: #666666">*</span>x<span style="color: #666666">**</span>(i<span style="color: #666666">-1</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f6 at x = </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(x,f6_grad_analytical))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">Using recursion </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f7</span>(n): <span style="color: #408080; font-style: italic"># Assume that n is an integer</span>
<span style="color: #008000; font-weight: bold">if</span> n <span style="color: #666666">==</span> <span style="color: #666666">1</span> <span style="color: #AA22FF; font-weight: bold">or</span> n <span style="color: #666666">==</span> <span style="color: #666666">0</span>:
<span style="color: #008000; font-weight: bold">return</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">else</span>:
<span style="color: #008000; font-weight: bold">return</span> n<span style="color: #666666">*</span>f7(n<span style="color: #666666">-1</span>)
f7_grad <span style="color: #666666">=</span> grad(f7)
n <span style="color: #666666">=</span> <span style="color: #666666">2.0</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The computed derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad(n)))
<span style="color: #408080; font-style: italic"># The function f7 is an implementation of the factorial of n.</span>
<span style="color: #408080; font-style: italic"># By using the product rule, one can find that the derivative is:</span>
f7_grad_analytical <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">int</span>(n)<span style="color: #666666">-1</span>):
tmp <span style="color: #666666">=</span> <span style="color: #666666">1</span>
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #008000">int</span>(n)<span style="color: #666666">-1</span>):
<span style="color: #008000; font-weight: bold">if</span> k <span style="color: #666666">!=</span> i:
tmp <span style="color: #666666">*=</span> (n <span style="color: #666666">-</span> k)
f7_grad_analytical <span style="color: #666666">+=</span> tmp
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The analytical derivative of f7 at n = </span><span style="color: #BB6688; font-weight: bold">%d</span><span style="color: #BA2121"> is: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span>(n,f7_grad_analytical))
</pre></div>
<p>
Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec42">Unsupported functions </h2>
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
<p>
Assigning a value to the variable being differentiated with respect to
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f8</span>(x): <span style="color: #408080; font-style: italic"># Assume x is an array</span>
x[<span style="color: #666666">2</span>] <span style="color: #666666">=</span> <span style="color: #666666">3</span>
<span style="color: #008000; font-weight: bold">return</span> x<span style="color: #666666">*2</span>
f8_grad <span style="color: #666666">=</span> grad(f8)
x <span style="color: #666666">=</span> <span style="color: #666666">8.4</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f8 is:&quot;</span>,f8_grad(x))
</pre></div>
<p>
Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec43">The syntax a.dot(b) when finding the dot product </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9</span>(a): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
<span style="color: #008000; font-weight: bold">return</span> a<span style="color: #666666">.</span>dot(b)
f9_grad <span style="color: #666666">=</span> grad(f9)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">0.0</span>])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The derivative of f9 is:&quot;</span>,f9_grad(x))
</pre></div>
<p>
Here we are told that the 'dot' function does not belong to Autograd's
version of a Numpy array. To overcome this, an alternative syntax
which also computed the dot product can be used:
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">autograd.numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">autograd</span> <span style="color: #008000; font-weight: bold">import</span> grad
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">f9_alternative</span>(x): <span style="color: #408080; font-style: italic"># Assume a is an array with 2 elements</span>
b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">1.0</span>,<span style="color: #666666">2.0</span>])
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>dot(x,b) <span style="color: #408080; font-style: italic"># The same as x_1*b_1 + x_2*b_2</span>
f9_alternative_grad <span style="color: #666666">=</span> grad(f9_alternative)
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>array([<span style="color: #666666">3.0</span>,<span style="color: #666666">0.0</span>])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;The gradient of f9 is:&quot;</span>,f9_alternative_grad(x))
<span style="color: #408080; font-style: italic"># The analytical gradient of the dot product of vectors x and b with two elements (x_1,x_2) and (b_1, b_2) respectively</span>
<span style="color: #408080; font-style: italic"># w.r.t x is (b_1, b_2).</span>
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Recommended to avoid </h2>
The documentation recommends to avoid inplace operations such as
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>a <span style="color: #666666">+=</span> b
a <span style="color: #666666">-=</span> b
a<span style="color: #666666">*=</span> b
a <span style="color: #666666">/=</span>b
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -1129,7 +1600,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec35">Computation of gradients </h2>
<h2 id="___sec46">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -1149,7 +1620,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">SGD example </h2>
<h2 id="___sec47">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -1173,7 +1644,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">The gradient step </h2>
<h2 id="___sec48">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -1192,7 +1663,7 @@ the number of minibatches, as exemplified in the code below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec38">Simple example code </h2>
<h2 id="___sec49">Simple example code </h2>
<p>
@@ -1224,7 +1695,7 @@ all \( n \) datapoints.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">When do we stop? </h2>
<h2 id="___sec50">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -1241,7 +1712,7 @@ gave the lowest value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">Slightly different approach </h2>
<h2 id="___sec51">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the