lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating splines

Browse Source
This commit is contained in:
mhjensen
2018-10-12 04:55:15 +02:00
parent 74e29912f4
commit c71c008945
83 changed files with 14530 additions and 4991 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Splines/html/Splines.html
+268 -80
View File
@@ -94,74 +94,86 @@ div { text-align: justify; text-justify: inter-word; }
('Steepest descent method', 2, None, '___sec19'),
('Steepest descent method', 2, None, '___sec20'),
('Final expressions', 2, None, '___sec21'),
('Code examples for steepest descent', 2, None, '___sec22'),
('Simple codes for steepest descent and conjugate gradient '
'using a $2\\times 2$ matrix, in c++, Python code to come',
2,
None,
'___sec22'),
'___sec23'),
('The routine for the steepest descent method',
2,
None,
'___sec23'),
('Steepest descent example', 2, None, '___sec24'),
('Conjugate gradient', 2, None, '___sec25'),
('Revisiting our first homework', 2, None, '___sec26'),
('Gradient descent example', 2, None, '___sec27'),
('The derivative of the cost/loss function', 2, None, '___sec28'),
('The Hessian matrix', 2, None, '___sec29'),
('Simple program', 2, None, '___sec30'),
('Gradient Descent Example', 2, None, '___sec31'),
'___sec24'),
('Steepest descent example', 2, None, '___sec25'),
('Conjugate gradient', 2, None, '___sec26'),
('Revisiting our first homework', 2, None, '___sec27'),
('Gradient descent example', 2, None, '___sec28'),
('The derivative of the cost/loss function', 2, None, '___sec29'),
('The Hessian matrix', 2, None, '___sec30'),
('Simple program', 2, None, '___sec31'),
('Gradient Descent Example', 2, None, '___sec32'),
('And a corresponding example using _scikit-learn_',
2,
None,
'___sec32'),
('Gradient descent and Ridge', 2, None, '___sec33'),
('Automatic differentiation', 2, None, '___sec34'),
('Using autograd', 2, None, '___sec35'),
('Autograd with more complicated functions', 2, None, '___sec36'),
'___sec33'),
('Gradient descent and Ridge', 2, None, '___sec34'),
('Automatic differentiation', 2, None, '___sec35'),
('Using autograd', 2, None, '___sec36'),
('Autograd with more complicated functions', 2, None, '___sec37'),
('More complicated functions using the elements of their '
'arguments directly',
2,
None,
'___sec37'),
'___sec38'),
('Functions using mathematical functions from Numpy',
2,
None,
'___sec38'),
('More autograd', 2, None, '___sec39'),
('And with loops', 2, None, '___sec40'),
('Using recursion', 2, None, '___sec41'),
('Unsupported functions', 2, None, '___sec42'),
'___sec39'),
('More autograd', 2, None, '___sec40'),
('And with loops', 2, None, '___sec41'),
('Using recursion', 2, None, '___sec42'),
('Unsupported functions', 2, None, '___sec43'),
('The syntax a.dot(b) when finding the dot product',
2,
None,
'___sec43'),
('Recommended to avoid', 2, None, '___sec44'),
('Stochastic Gradient Descent', 2, None, '___sec45'),
('Computation of gradients', 2, None, '___sec46'),
('SGD example', 2, None, '___sec47'),
('The gradient step', 2, None, '___sec48'),
('Simple example code', 2, None, '___sec49'),
('When do we stop?', 2, None, '___sec50'),
('Slightly different approach', 2, None, '___sec51'),
('Program for stochastic gradient', 2, None, '___sec52'),
('Momentum based methods', 2, None, '___sec53'),
('Conjugate gradient method', 2, None, '___sec54'),
'___sec44'),
('Recommended to avoid', 2, None, '___sec45'),
('Stochastic Gradient Descent', 2, None, '___sec46'),
('Computation of gradients', 2, None, '___sec47'),
('SGD example', 2, None, '___sec48'),
('The gradient step', 2, None, '___sec49'),
('Simple example code', 2, None, '___sec50'),
('When do we stop?', 2, None, '___sec51'),
('Slightly different approach', 2, None, '___sec52'),
('Program for stochastic gradient', 2, None, '___sec53'),
('Momentum based methods', 2, None, '___sec54'),
('Conjugate gradient method', 2, None, '___sec55'),
('Conjugate gradient method', 2, None, '___sec56'),
('Conjugate gradient method', 2, None, '___sec57'),
('Conjugate gradient method and iterations', 2, None, '___sec58'),
('Conjugate gradient method', 2, None, '___sec59'),
('Conjugate gradient method', 2, None, '___sec58'),
('Conjugate gradient method and iterations', 2, None, '___sec59'),
('Conjugate gradient method', 2, None, '___sec60'),
('Conjugate gradient method', 2, None, '___sec61'),
('Conjugate gradient method', 2, None, '___sec62'),
('Simple implementation of the Conjugate gradient algorithm',
2,
None,
'___sec62'),
'___sec63'),
('Broyden–Fletcher–Goldfarb–Shanno algorithm',
2,
None,
'___sec63')]}
'___sec64'),
('Using gradient descent methods, limitations',
2,
None,
'___sec65'),
('Momentum based GD', 2, None, '___sec66'),
('More on momentum based approaches', 2, None, '___sec67'),
('Momentum parameter', 2, None, '___sec68'),
('Second moment of the gradient', 2, None, '___sec69'),
('RMS prop', 2, None, '___sec70'),
('ADAM optimizer', 2, None, '___sec71'),
('Practical tips', 2, None, '___sec72')]}
end of tocinfo -->
<body>
@@ -203,7 +215,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 6, 2018</h4></center> <!-- date -->
<center><h4>Oct 12, 2018</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -787,7 +799,12 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
<h2 id="___sec22">Code examples for steepest descent </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Simple codes for steepest descent and conjugate gradient using a \( 2\times 2 \) matrix, in c++, Python code to come </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -827,7 +844,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">The routine for the steepest descent method </h2>
<h2 id="___sec24">The routine for the steepest descent method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -863,7 +880,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Steepest descent example </h2>
<h2 id="___sec25">Steepest descent example </h2>
<p>
@@ -932,12 +949,12 @@ pt<span style="color: #666666">.</span>plot(it_array<span style="color: #666666"
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Conjugate gradient </h2>
<h2 id="___sec26">Conjugate gradient </h2>
<p>
<!-- !split -->
<h2 id="___sec26">Revisiting our first homework </h2>
<h2 id="___sec27">Revisiting our first homework </h2>
<p>
We will use linear regression as a case study for the gradient descent
@@ -970,7 +987,7 @@ $$
<p>
<!-- !split -->
<h2 id="___sec27">Gradient descent example </h2>
<h2 id="___sec28">Gradient descent example </h2>
<p>
Let \( \mathbf{y} = (y_1,\cdots,y_n)^T \), \( \mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T \) and \( \beta = (\beta_0, \beta_1)^T \)
@@ -995,7 +1012,7 @@ and we want to find \( \beta \) such that \( C(\beta) \) is minimized.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">The derivative of the cost/loss function </h2>
<h2 id="___sec29">The derivative of the cost/loss function </h2>
<p>
Computing \( \partial C(\beta) / \partial \beta_0 \) and \( \partial C(\beta) / \partial \beta_1 \) we can show that the gradient can be written as
@@ -1010,7 +1027,7 @@ where \( X \) is the design matrix defined above.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">The Hessian matrix </h2>
<h2 id="___sec30">The Hessian matrix </h2>
The Hessian matrix of \( C(\beta) \) is given by
$$
\hat{H} \equiv \begin{bmatrix}
@@ -1024,7 +1041,7 @@ This result implies that \( C(\beta) \) is a convex function since the matrix \(
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Simple program </h2>
<h2 id="___sec31">Simple program </h2>
<p>
We can now write a program that minimizes \( C(\beta) \) using the gradient descent method with a constant learning rate \( \gamma \) according to
@@ -1065,7 +1082,7 @@ beta_NE <span style="color: #666666">=</span> np<span style="color: #666666">.</
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Gradient Descent Example </h2>
<h2 id="___sec32">Gradient Descent Example </h2>
<p>
Another simple example is here
@@ -1114,7 +1131,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">And a corresponding example using <b>scikit-learn</b> </h2>
<h2 id="___sec33">And a corresponding example using <b>scikit-learn</b> </h2>
<p>
@@ -1138,7 +1155,7 @@ sgdreg<span style="color: #666666">.</span>fit(x,y<span style="color: #666666">.
<p>
<!-- !split -->
<h2 id="___sec33">Gradient descent and Ridge </h2>
<h2 id="___sec34">Gradient descent and Ridge </h2>
<p>
We have also discussed Ridge regression where the loss function contains a regularized given by the \( L_2 \) norm of \( \beta \),
@@ -1195,7 +1212,7 @@ beta_ridge <span style="color: #666666">=</span> np<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec34">Automatic differentiation </h2>
<h2 id="___sec35">Automatic differentiation </h2>
Python has tools for so-called <b>automatic differentiation</b>.
Consider the following example
$$
@@ -1250,7 +1267,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec35">Using autograd </h2>
<h2 id="___sec36">Using autograd </h2>
<p>
Here we
@@ -1283,7 +1300,7 @@ grad_analytical <span style="color: #666666">=</span> <span style="color: #66666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec36">Autograd with more complicated functions </h2>
<h2 id="___sec37">Autograd with more complicated functions </h2>
<p>
To differentiate with respect to two (or more) arguments of a Python
@@ -1333,7 +1350,7 @@ Note that the grad function will not produce the true gradient of the function.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec37">More complicated functions using the elements of their arguments directly </h2>
<h2 id="___sec38">More complicated functions using the elements of their arguments directly </h2>
<p>
@@ -1367,7 +1384,7 @@ could expect form a gradient-evaluting function.
<p>
<!-- !split -->
<h2 id="___sec38">Functions using mathematical functions from Numpy </h2>
<h2 id="___sec39">Functions using mathematical functions from Numpy </h2>
<p>
@@ -1393,7 +1410,7 @@ f4_grad_analytical <span style="color: #666666">=</span> x<span style="color: #6
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec39">More autograd </h2>
<h2 id="___sec40">More autograd </h2>
<p>
@@ -1416,7 +1433,7 @@ x <span style="color: #666666">=</span> <span style="color: #666666">2.7</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec40">And with loops </h2>
<h2 id="___sec41">And with loops </h2>
<p>
@@ -1462,7 +1479,7 @@ f6_grad_analytical <span style="color: #666666">=</span> <span style="color: #66
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec41">Using recursion </h2>
<h2 id="___sec42">Using recursion </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1500,7 +1517,7 @@ Note that if n is equal to zero or one, Autograd will give an error message. Thi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec42">Unsupported functions </h2>
<h2 id="___sec43">Unsupported functions </h2>
Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.
<p>
@@ -1526,7 +1543,7 @@ Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec43">The syntax a.dot(b) when finding the dot product </h2>
<h2 id="___sec44">The syntax a.dot(b) when finding the dot product </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1568,7 +1585,7 @@ x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>a
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec44">Recommended to avoid </h2>
<h2 id="___sec45">Recommended to avoid </h2>
The documentation recommends to avoid inplace operations such as
<p>
@@ -1581,7 +1598,7 @@ a <span style="color: #666666">/=</span>b
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec45">Stochastic Gradient Descent </h2>
<h2 id="___sec46">Stochastic Gradient Descent </h2>
<p>
Stochastic gradient descent (SGD) and variants thereof address some of
@@ -1599,7 +1616,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec46">Computation of gradients </h2>
<h2 id="___sec47">Computation of gradients </h2>
<p>
This in turn means that the gradient can be
@@ -1619,7 +1636,7 @@ minibatches. We denote these minibatches by \( B_k \) where
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec47">SGD example </h2>
<h2 id="___sec48">SGD example </h2>
As an example, suppose we have \( 10 \) data points \( (\mathbf{x}_1,\cdots, \mathbf{x}_{10}) \)
and we choose to have \( M=5 \) minibathces,
then each minibatch contains two data points. In particular we have
@@ -1643,7 +1660,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec48">The gradient step </h2>
<h2 id="___sec49">The gradient step </h2>
<p>
Thus a gradient descent step now looks like
@@ -1662,7 +1679,7 @@ the number of minibatches, as exemplified in the code below.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec49">Simple example code </h2>
<h2 id="___sec50">Simple example code </h2>
<p>
@@ -1694,7 +1711,7 @@ all \( n \) datapoints.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec50">When do we stop? </h2>
<h2 id="___sec51">When do we stop? </h2>
<p>
A natural question is when do we stop the search for a new minimum?
@@ -1711,7 +1728,7 @@ gave the lowest value.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec51">Slightly different approach </h2>
<h2 id="___sec52">Slightly different approach </h2>
<p>
Another approach is to let the step length \( \gamma_j \) depend on the
@@ -1759,7 +1776,7 @@ j <span style="color: #666666">=</span> <span style="color: #666666">0</span>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec52">Program for stochastic gradient </h2>
<h2 id="___sec53">Program for stochastic gradient </h2>
<p>
@@ -1838,12 +1855,12 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec53">Momentum based methods </h2>
<h2 id="___sec54">Momentum based methods </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec54">Conjugate gradient method </h2>
<h2 id="___sec55">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1873,7 +1890,7 @@ this inner product. Being conjugate is a symmetric relation: if \( \hat{s} \) is
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec55">Conjugate gradient method </h2>
<h2 id="___sec56">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1891,7 +1908,7 @@ which is zero unless \( i=j \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec56">Conjugate gradient method </h2>
<h2 id="___sec57">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1918,7 +1935,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec57">Conjugate gradient method </h2>
<h2 id="___sec58">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1950,7 +1967,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec58">Conjugate gradient method and iterations </h2>
<h2 id="___sec59">Conjugate gradient method and iterations </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1986,7 +2003,7 @@ instead.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec59">Conjugate gradient method </h2>
<h2 id="___sec60">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2016,7 +2033,7 @@ hence the name conjugate gradient method.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec60">Conjugate gradient method </h2>
<h2 id="___sec61">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2045,7 +2062,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec61">Conjugate gradient method </h2>
<h2 id="___sec62">Conjugate gradient method </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2083,7 +2100,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec62">Simple implementation of the Conjugate gradient algorithm </h2>
<h2 id="___sec63">Simple implementation of the Conjugate gradient algorithm </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2122,7 +2139,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec63">Broyden&#8211;Fletcher&#8211;Goldfarb&#8211;Shanno algorithm </h2>
<h2 id="___sec64">Broyden&#8211;Fletcher&#8211;Goldfarb&#8211;Shanno algorithm </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -2155,6 +2172,177 @@ over the scalar \( \alpha > 0 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec65">Using gradient descent methods, limitations </h2>
<ul>
<li> <b>Gradient descent (GD) finds local minima of our function</b>. Since the GD algorithm is deterministic, if it converges, it will converge to a local minimum of our energy function. Because in ML we are often dealing with extremely rugged landscapes with many local minima, this can lead to poor performance.</li>
<li> <b>GD is sensitive to initial conditions</b>. One consequence of the local nature of GD is that initial conditions matter. Depending on where one starts, one will end up at a different local minima. Therefore, it is very important to think about how one initializes the training process. This is true for GD as well as more complicated variants of GD.</li>
<li> <b>Gradients are computationally expensive to calculate for large datasets</b>. In many cases in statistics and ML, the energy function is a sum of terms, with one term for each data point. For example, in linear regression, \( E \propto \sum_{i=1}^n (y_i - \mathbf{w}^T\cdot\mathbf{x}_i)^2 \); for logistic regression, the square error is replaced by the cross entropy. To calculate the gradient we have to sum over <em>all</em> \( n \) data points. Doing this at every GD step becomes extremely computationally expensive. An ingenious solution to this, is to calculate the gradients using small subsets of the data called &quot;mini batches&quot;. This has the added benefit of introducing stochasticity into our algorithm.</li>
<li> <b>GD is very sensitive to choices of learning rates</b>. GD is extremely sensitive to the choice of learning rates. If the learning rate is very small, the training process take an extremely long time. For larger learning rates, GD can diverge and give poor results. Furthermore, depending on what the local landscape looks like, we have to modify the learning rates to ensure convergence. Ideally, we would <em>adaptively</em> choose the learning rates to match the landscape.</li>
<li> <b>GD treats all directions in parameter space uniformly.</b> Another major drawback of GD is that unlike Newton's method, the learning rate for GD is the same in all directions in parameter space. For this reason, the maximum learning rate is set by the behavior of the steepest direction and this can significantly slow down training. Ideally, we would like to take large steps in flat directions and small steps in steep directions. Since we are exploring rugged landscapes where curvatures change, this requires us to keep track of not only the gradient but second derivatives. The ideal scenario would be to calculate the Hessian but this proves to be too computationally expensive.</li>
<li> GD can take exponential time to escape saddle points, even with random initialization. As we mentioned, GD is extremely sensitive to initial condition since it determines the particular local minimum GD would eventually reach. However, even with a good initialization scheme, through the introduction of randomness, GD can still take exponential time to escape saddle points.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec66">Momentum based GD </h2>
<p>
The stochastic gradient descent (SGD) is almost always used with a <em>momentum</em> or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
implemented as follows
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t},
\label{_auto1}
\end{align}
$$
where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
$$
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
$$
where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\boldsymbol{\theta}_{t-1} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec67">More on momentum based approaches </h2>
<p>
Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
$$
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
$$
We can discretize this equation in the usual way to get
$$
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
$$
Rearranging this equation, we can rewrite this as
$$
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec68">Momentum parameter </h2>
Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
$$
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
$$
Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
<p>
Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
<p>
These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
<p>
In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}.
\label{_auto2}
\end{align}
$$
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec69">Second moment of the gradient </h2>
<p>
In stochastic gradient descent, with and without momentum, we still
have to specify a schedule for tuning the learning rates \( \eta_t \)
as a function of time. As discussed in the context of Newton's
method, this presents a number of dilemmas. The learning rate is
limited by the steepest direction which can change depending on the
current position in the landscape. To circumvent this problem, ideally
our algorithm would keep track of curvature and take large steps in
shallow, flat directions and small steps in steep, narrow directions.
Second-order methods accomplish this by calculating or approximating
the Hessian and normalizing the learning rate by the
curvature. However, this is very computationally expensive for
extremely large models. Ideally, we would like to be able to
adaptively change the step size to match the landscape without paying
the steep computational price of calculating or approximating
Hessians.
<p>
Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec70">RMS prop </h2>
<p>
In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\label{_auto3}\\
\mathbf{s}_t &=\beta \mathbf{s}_{t-1} +(1-\beta)\mathbf{g}_t^2 \nonumber \\
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
$$
where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec71">ADAM optimizer </h2>
<p>
A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
\label{_auto4}\\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\label{_auto5}
\end{align}
$$
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
<p>
Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
$$
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec72">Practical tips </h2>
<ul>
<li> <b>Randomize the data when making mini-batches</b>. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.</li>
<li> <b>Transform your inputs</b>. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the energy matrix is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case.</li>
<li> <b>Monitor the out-of-sample performance.</b> Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This <em>early stopping</em> significantly improves performance in many settings.</li>
<li> <b>Adaptive optimization methods don't always have good generalization.</b> Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.</li>
</ul>
Geron's text, see chapter 11, has several interesting discussions.
<!-- ------------------- end of main content --------------- -->