Adding to do issues on log reg
This commit is contained in:
mhjensen
@@ -295,7 +295,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Sep 26, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Sep 28, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
|
||||
@@ -279,8 +279,11 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec35" class="anchor">Momentum based GD </h2>
|
||||
|
||||
<p>
|
||||
The stochastic gradient descent (SGD) is almost always used with a <em>momentum</em> or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
|
||||
implemented as follows
|
||||
The stochastic gradient descent (SGD) is almost always used with a
|
||||
<em>momentum</em> or inertia term that serves as a memory of the direction we
|
||||
are moving in parameter space. This is typically implemented as
|
||||
follows
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
|
||||
@@ -289,7 +292,18 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
|
||||
<p>
|
||||
where we have introduced a momentum parameter \( \gamma \), with
|
||||
\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
|
||||
indicate the gradient is to be taken over a different mini-batch at
|
||||
each step. We call this algorithm gradient descent with momentum
|
||||
(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
|
||||
running average of recently encountered gradients and
|
||||
\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
|
||||
used in the averaging procedure. Consistent with this, when
|
||||
\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
|
||||
earlier. An equivalent way of writing the updates is
|
||||
|
||||
$$
|
||||
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
|
||||
$$
|
||||
|
||||
@@ -279,18 +279,26 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec36" class="anchor">More on momentum based approaches </h2>
|
||||
|
||||
<p>
|
||||
Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
|
||||
Let us try to get more intuition from these equations. It is helpful
|
||||
to consider a simple physical analogy with a particle of mass \( m \)
|
||||
moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
|
||||
then its motion is described by
|
||||
|
||||
$$
|
||||
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
We can discretize this equation in the usual way to get
|
||||
|
||||
$$
|
||||
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
Rearranging this equation, we can rewrite this as
|
||||
|
||||
$$
|
||||
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
|
||||
$$
|
||||
|
||||
@@ -277,22 +277,47 @@ MathJax.Hub.Config({
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec37" class="anchor">Momentum parameter </h2>
|
||||
Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
|
||||
us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
|
||||
|
||||
<p>
|
||||
Notice that this equation is identical to previous one if we identify
|
||||
the position of the particle, \( \mathbf{w} \), with the parameters
|
||||
\( \boldsymbol{\theta} \). This allows us to identify the momentum
|
||||
parameter and learning rate with the mass of the particle and the
|
||||
viscous drag as:
|
||||
|
||||
$$
|
||||
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
|
||||
$$
|
||||
|
||||
Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
<p>
|
||||
Thus, as the name suggests, the momentum parameter is proportional to
|
||||
the mass of the particle and effectively provides inertia.
|
||||
Furthermore, in the large viscosity/small learning rate limit, our
|
||||
memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
|
||||
<p>
|
||||
Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
|
||||
Why is momentum useful? SGD momentum helps the gradient descent
|
||||
algorithm gain speed in directions with persistent but small gradients
|
||||
even in the presence of stochasticity, while suppressing oscillations
|
||||
in high-curvature directions. This becomes especially important in
|
||||
situations where the landscape is shallow and flat in some directions
|
||||
and narrow and steep in others. It has been argued that first-order
|
||||
methods (with appropriate initial conditions) can perform comparable
|
||||
to more expensive second order methods, especially in the context of
|
||||
complex deep learning models.
|
||||
|
||||
<p>
|
||||
These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
These beneficial properties of momentum can sometimes become even more
|
||||
pronounced by using a slight modification of the classical momentum
|
||||
algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
|
||||
<p>
|
||||
In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
In the NAG algorithm, rather than calculating the gradient at the
|
||||
current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
|
||||
calculates the gradient at the expected value of the parameters given
|
||||
our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
|
||||
\mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
|
||||
@@ -301,6 +326,7 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
<p>
|
||||
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
|
||||
|
||||
<p>
|
||||
|
||||
@@ -296,7 +296,10 @@ the steep computational price of calculating or approximating
|
||||
Hessians.
|
||||
|
||||
<p>
|
||||
Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
|
||||
Recently, a number of methods have been introduced that accomplish
|
||||
this by tracking not only the gradient, but also the second moment of
|
||||
the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
|
||||
ADAM.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
|
||||
@@ -279,7 +279,11 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec39" class="anchor">RMS prop </h2>
|
||||
|
||||
<p>
|
||||
In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
|
||||
In RMS prop, in addition to keeping a running average of the first
|
||||
moment of the gradient, we also keep track of the second moment
|
||||
denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
|
||||
for RMS prop is given by
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
|
||||
@@ -289,7 +293,16 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
|
||||
<p>
|
||||
where \( \beta \) controls the averaging time of the second moment and is
|
||||
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
|
||||
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
|
||||
small regularization constant to prevent divergences. Multiplication
|
||||
and division by vectors is understood as an element-wise operation. It
|
||||
is clear from this formula that the learning rate is reduced in
|
||||
directions where the norm of the gradient is consistently large. This
|
||||
greatly speeds up the convergence by allowing us to use a larger
|
||||
learning rate for flat directions.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
|
||||
@@ -279,7 +279,19 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec40" class="anchor">ADAM optimizer </h2>
|
||||
|
||||
<p>
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
|
||||
average of both the first and second moment of the gradient and use
|
||||
this information to adaptively change the learning rate for different
|
||||
parameters. In addition to keeping a running average of the first and
|
||||
second moments of the gradient
|
||||
(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
|
||||
\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
|
||||
performs an additional bias correction to account for the fact that we
|
||||
are estimating the first two moments of the gradient using a running
|
||||
average (denoted by the hats in the update rule below). The update
|
||||
rule for ADAM is given by (where multiplication and division are once
|
||||
again understood to be element-wise operations below)
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
|
||||
@@ -293,10 +305,19 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
<p>
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
|
||||
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
|
||||
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
|
||||
<p>
|
||||
Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
|
||||
Like in RMSprop, the effective step size of a parameter depends on the
|
||||
magnitude of its gradient squared. To understand this better, let us
|
||||
rewrite this expression in terms of the variance
|
||||
\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
|
||||
(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
|
||||
update rule for this parameter is given by
|
||||
|
||||
$$
|
||||
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
|
||||
$$
|
||||
|
||||
@@ -295,7 +295,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Sep 26, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Sep 28, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
|
||||
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>Sep 26, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Sep 28, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -1187,8 +1187,11 @@ plt.show()
|
||||
<h2 id="___sec35">Momentum based GD </h2>
|
||||
|
||||
<p>
|
||||
The stochastic gradient descent (SGD) is almost always used with a <em>momentum</em> or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
|
||||
implemented as follows
|
||||
The stochastic gradient descent (SGD) is almost always used with a
|
||||
<em>momentum</em> or inertia term that serves as a memory of the direction we
|
||||
are moving in parameter space. This is typically implemented as
|
||||
follows
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align}
|
||||
@@ -1199,7 +1202,18 @@ $$
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
|
||||
<p>
|
||||
where we have introduced a momentum parameter \( \gamma \), with
|
||||
\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
|
||||
indicate the gradient is to be taken over a different mini-batch at
|
||||
each step. We call this algorithm gradient descent with momentum
|
||||
(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
|
||||
running average of recently encountered gradients and
|
||||
\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
|
||||
used in the averaging procedure. Consistent with this, when
|
||||
\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
|
||||
earlier. An equivalent way of writing the updates is
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
|
||||
@@ -1214,22 +1228,30 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
|
||||
<h2 id="___sec36">More on momentum based approaches </h2>
|
||||
|
||||
<p>
|
||||
Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
|
||||
Let us try to get more intuition from these equations. It is helpful
|
||||
to consider a simple physical analogy with a particle of mass \( m \)
|
||||
moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
|
||||
then its motion is described by
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
We can discretize this equation in the usual way to get
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
Rearranging this equation, we can rewrite this as
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
|
||||
@@ -1240,24 +1262,49 @@ $$
|
||||
|
||||
<section>
|
||||
<h2 id="___sec37">Momentum parameter </h2>
|
||||
Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
|
||||
us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
|
||||
|
||||
<p>
|
||||
Notice that this equation is identical to previous one if we identify
|
||||
the position of the particle, \( \mathbf{w} \), with the parameters
|
||||
\( \boldsymbol{\theta} \). This allows us to identify the momentum
|
||||
parameter and learning rate with the mass of the particle and the
|
||||
viscous drag as:
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
<p>
|
||||
Thus, as the name suggests, the momentum parameter is proportional to
|
||||
the mass of the particle and effectively provides inertia.
|
||||
Furthermore, in the large viscosity/small learning rate limit, our
|
||||
memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
|
||||
<p>
|
||||
Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
|
||||
Why is momentum useful? SGD momentum helps the gradient descent
|
||||
algorithm gain speed in directions with persistent but small gradients
|
||||
even in the presence of stochasticity, while suppressing oscillations
|
||||
in high-curvature directions. This becomes especially important in
|
||||
situations where the landscape is shallow and flat in some directions
|
||||
and narrow and steep in others. It has been argued that first-order
|
||||
methods (with appropriate initial conditions) can perform comparable
|
||||
to more expensive second order methods, especially in the context of
|
||||
complex deep learning models.
|
||||
|
||||
<p>
|
||||
These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
These beneficial properties of momentum can sometimes become even more
|
||||
pronounced by using a slight modification of the classical momentum
|
||||
algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
|
||||
<p>
|
||||
In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
In the NAG algorithm, rather than calculating the gradient at the
|
||||
current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
|
||||
calculates the gradient at the expected value of the parameters given
|
||||
our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
|
||||
\mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align}
|
||||
@@ -1268,6 +1315,7 @@ $$
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
|
||||
</section>
|
||||
|
||||
@@ -1293,7 +1341,10 @@ the steep computational price of calculating or approximating
|
||||
Hessians.
|
||||
|
||||
<p>
|
||||
Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
|
||||
Recently, a number of methods have been introduced that accomplish
|
||||
this by tracking not only the gradient, but also the second moment of
|
||||
the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
|
||||
ADAM.
|
||||
</section>
|
||||
|
||||
|
||||
@@ -1301,7 +1352,11 @@ Recently, a number of methods have been introduced that accomplish this by track
|
||||
<h2 id="___sec39">RMS prop </h2>
|
||||
|
||||
<p>
|
||||
In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
|
||||
In RMS prop, in addition to keeping a running average of the first
|
||||
moment of the gradient, we also keep track of the second moment
|
||||
denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
|
||||
for RMS prop is given by
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align}
|
||||
@@ -1313,7 +1368,16 @@ $$
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
|
||||
<p>
|
||||
where \( \beta \) controls the averaging time of the second moment and is
|
||||
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
|
||||
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
|
||||
small regularization constant to prevent divergences. Multiplication
|
||||
and division by vectors is understood as an element-wise operation. It
|
||||
is clear from this formula that the learning rate is reduced in
|
||||
directions where the norm of the gradient is consistently large. This
|
||||
greatly speeds up the convergence by allowing us to use a larger
|
||||
learning rate for flat directions.
|
||||
</section>
|
||||
|
||||
|
||||
@@ -1321,7 +1385,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica
|
||||
<h2 id="___sec40">ADAM optimizer </h2>
|
||||
|
||||
<p>
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
|
||||
average of both the first and second moment of the gradient and use
|
||||
this information to adaptively change the learning rate for different
|
||||
parameters. In addition to keeping a running average of the first and
|
||||
second moments of the gradient
|
||||
(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
|
||||
\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
|
||||
performs an additional bias correction to account for the fact that we
|
||||
are estimating the first two moments of the gradient using a running
|
||||
average (denoted by the hats in the update rule below). The update
|
||||
rule for ADAM is given by (where multiplication and division are once
|
||||
again understood to be element-wise operations below)
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align}
|
||||
@@ -1337,10 +1413,19 @@ $$
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
<p>
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
|
||||
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
|
||||
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
|
||||
<p>
|
||||
Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
|
||||
Like in RMSprop, the effective step size of a parameter depends on the
|
||||
magnitude of its gradient squared. To understand this better, let us
|
||||
rewrite this expression in terms of the variance
|
||||
\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
|
||||
(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
|
||||
update rule for this parameter is given by
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
|
||||
|
||||
@@ -213,7 +213,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Sep 26, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Sep 28, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1175,8 +1175,11 @@ plt.show()
|
||||
<h2 id="___sec35">Momentum based GD </h2>
|
||||
|
||||
<p>
|
||||
The stochastic gradient descent (SGD) is almost always used with a <em>momentum</em> or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
|
||||
implemented as follows
|
||||
The stochastic gradient descent (SGD) is almost always used with a
|
||||
<em>momentum</em> or inertia term that serves as a memory of the direction we
|
||||
are moving in parameter space. This is typically implemented as
|
||||
follows
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
|
||||
@@ -1185,7 +1188,18 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
|
||||
<p>
|
||||
where we have introduced a momentum parameter \( \gamma \), with
|
||||
\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
|
||||
indicate the gradient is to be taken over a different mini-batch at
|
||||
each step. We call this algorithm gradient descent with momentum
|
||||
(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
|
||||
running average of recently encountered gradients and
|
||||
\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
|
||||
used in the averaging procedure. Consistent with this, when
|
||||
\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
|
||||
earlier. An equivalent way of writing the updates is
|
||||
|
||||
$$
|
||||
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
|
||||
$$
|
||||
@@ -1198,18 +1212,26 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
|
||||
<h2 id="___sec36">More on momentum based approaches </h2>
|
||||
|
||||
<p>
|
||||
Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
|
||||
Let us try to get more intuition from these equations. It is helpful
|
||||
to consider a simple physical analogy with a particle of mass \( m \)
|
||||
moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
|
||||
then its motion is described by
|
||||
|
||||
$$
|
||||
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
We can discretize this equation in the usual way to get
|
||||
|
||||
$$
|
||||
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
Rearranging this equation, we can rewrite this as
|
||||
|
||||
$$
|
||||
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
|
||||
$$
|
||||
@@ -1218,22 +1240,47 @@ $$
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec37">Momentum parameter </h2>
|
||||
Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
|
||||
us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
|
||||
|
||||
<p>
|
||||
Notice that this equation is identical to previous one if we identify
|
||||
the position of the particle, \( \mathbf{w} \), with the parameters
|
||||
\( \boldsymbol{\theta} \). This allows us to identify the momentum
|
||||
parameter and learning rate with the mass of the particle and the
|
||||
viscous drag as:
|
||||
|
||||
$$
|
||||
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
|
||||
$$
|
||||
|
||||
Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
<p>
|
||||
Thus, as the name suggests, the momentum parameter is proportional to
|
||||
the mass of the particle and effectively provides inertia.
|
||||
Furthermore, in the large viscosity/small learning rate limit, our
|
||||
memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
|
||||
<p>
|
||||
Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
|
||||
Why is momentum useful? SGD momentum helps the gradient descent
|
||||
algorithm gain speed in directions with persistent but small gradients
|
||||
even in the presence of stochasticity, while suppressing oscillations
|
||||
in high-curvature directions. This becomes especially important in
|
||||
situations where the landscape is shallow and flat in some directions
|
||||
and narrow and steep in others. It has been argued that first-order
|
||||
methods (with appropriate initial conditions) can perform comparable
|
||||
to more expensive second order methods, especially in the context of
|
||||
complex deep learning models.
|
||||
|
||||
<p>
|
||||
These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
These beneficial properties of momentum can sometimes become even more
|
||||
pronounced by using a slight modification of the classical momentum
|
||||
algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
|
||||
<p>
|
||||
In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
In the NAG algorithm, rather than calculating the gradient at the
|
||||
current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
|
||||
calculates the gradient at the expected value of the parameters given
|
||||
our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
|
||||
\mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
|
||||
@@ -1242,6 +1289,7 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
<p>
|
||||
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
|
||||
|
||||
<p>
|
||||
@@ -1267,7 +1315,10 @@ the steep computational price of calculating or approximating
|
||||
Hessians.
|
||||
|
||||
<p>
|
||||
Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
|
||||
Recently, a number of methods have been introduced that accomplish
|
||||
this by tracking not only the gradient, but also the second moment of
|
||||
the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
|
||||
ADAM.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1275,7 +1326,11 @@ Recently, a number of methods have been introduced that accomplish this by track
|
||||
<h2 id="___sec39">RMS prop </h2>
|
||||
|
||||
<p>
|
||||
In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
|
||||
In RMS prop, in addition to keeping a running average of the first
|
||||
moment of the gradient, we also keep track of the second moment
|
||||
denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
|
||||
for RMS prop is given by
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
|
||||
@@ -1285,7 +1340,16 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
|
||||
<p>
|
||||
where \( \beta \) controls the averaging time of the second moment and is
|
||||
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
|
||||
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
|
||||
small regularization constant to prevent divergences. Multiplication
|
||||
and division by vectors is understood as an element-wise operation. It
|
||||
is clear from this formula that the learning rate is reduced in
|
||||
directions where the norm of the gradient is consistently large. This
|
||||
greatly speeds up the convergence by allowing us to use a larger
|
||||
learning rate for flat directions.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1293,7 +1357,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica
|
||||
<h2 id="___sec40">ADAM optimizer </h2>
|
||||
|
||||
<p>
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
|
||||
average of both the first and second moment of the gradient and use
|
||||
this information to adaptively change the learning rate for different
|
||||
parameters. In addition to keeping a running average of the first and
|
||||
second moments of the gradient
|
||||
(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
|
||||
\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
|
||||
performs an additional bias correction to account for the fact that we
|
||||
are estimating the first two moments of the gradient using a running
|
||||
average (denoted by the hats in the update rule below). The update
|
||||
rule for ADAM is given by (where multiplication and division are once
|
||||
again understood to be element-wise operations below)
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
|
||||
@@ -1307,10 +1383,19 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
<p>
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
|
||||
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
|
||||
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
|
||||
<p>
|
||||
Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
|
||||
Like in RMSprop, the effective step size of a parameter depends on the
|
||||
magnitude of its gradient squared. To understand this better, let us
|
||||
rewrite this expression in terms of the variance
|
||||
\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
|
||||
(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
|
||||
update rule for this parameter is given by
|
||||
|
||||
$$
|
||||
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
|
||||
$$
|
||||
|
||||
@@ -218,7 +218,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Sep 26, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Sep 28, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1180,8 +1180,11 @@ plt<span style="color: #666666">.</span>show()
|
||||
<h2 id="___sec35">Momentum based GD </h2>
|
||||
|
||||
<p>
|
||||
The stochastic gradient descent (SGD) is almost always used with a <em>momentum</em> or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
|
||||
implemented as follows
|
||||
The stochastic gradient descent (SGD) is almost always used with a
|
||||
<em>momentum</em> or inertia term that serves as a memory of the direction we
|
||||
are moving in parameter space. This is typically implemented as
|
||||
follows
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
|
||||
@@ -1190,7 +1193,18 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
|
||||
<p>
|
||||
where we have introduced a momentum parameter \( \gamma \), with
|
||||
\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
|
||||
indicate the gradient is to be taken over a different mini-batch at
|
||||
each step. We call this algorithm gradient descent with momentum
|
||||
(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
|
||||
running average of recently encountered gradients and
|
||||
\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
|
||||
used in the averaging procedure. Consistent with this, when
|
||||
\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
|
||||
earlier. An equivalent way of writing the updates is
|
||||
|
||||
$$
|
||||
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
|
||||
$$
|
||||
@@ -1203,18 +1217,26 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
|
||||
<h2 id="___sec36">More on momentum based approaches </h2>
|
||||
|
||||
<p>
|
||||
Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
|
||||
Let us try to get more intuition from these equations. It is helpful
|
||||
to consider a simple physical analogy with a particle of mass \( m \)
|
||||
moving in a viscous medium with drag coefficient \( \mu \) and potential
|
||||
\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
|
||||
then its motion is described by
|
||||
|
||||
$$
|
||||
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
We can discretize this equation in the usual way to get
|
||||
|
||||
$$
|
||||
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
|
||||
$$
|
||||
|
||||
<p>
|
||||
Rearranging this equation, we can rewrite this as
|
||||
|
||||
$$
|
||||
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
|
||||
$$
|
||||
@@ -1223,22 +1245,47 @@ $$
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec37">Momentum parameter </h2>
|
||||
Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
|
||||
us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
|
||||
|
||||
<p>
|
||||
Notice that this equation is identical to previous one if we identify
|
||||
the position of the particle, \( \mathbf{w} \), with the parameters
|
||||
\( \boldsymbol{\theta} \). This allows us to identify the momentum
|
||||
parameter and learning rate with the mass of the particle and the
|
||||
viscous drag as:
|
||||
|
||||
$$
|
||||
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
|
||||
$$
|
||||
|
||||
Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
<p>
|
||||
Thus, as the name suggests, the momentum parameter is proportional to
|
||||
the mass of the particle and effectively provides inertia.
|
||||
Furthermore, in the large viscosity/small learning rate limit, our
|
||||
memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
|
||||
|
||||
<p>
|
||||
Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
|
||||
Why is momentum useful? SGD momentum helps the gradient descent
|
||||
algorithm gain speed in directions with persistent but small gradients
|
||||
even in the presence of stochasticity, while suppressing oscillations
|
||||
in high-curvature directions. This becomes especially important in
|
||||
situations where the landscape is shallow and flat in some directions
|
||||
and narrow and steep in others. It has been argued that first-order
|
||||
methods (with appropriate initial conditions) can perform comparable
|
||||
to more expensive second order methods, especially in the context of
|
||||
complex deep learning models.
|
||||
|
||||
<p>
|
||||
These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
These beneficial properties of momentum can sometimes become even more
|
||||
pronounced by using a slight modification of the classical momentum
|
||||
algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
|
||||
<p>
|
||||
In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
In the NAG algorithm, rather than calculating the gradient at the
|
||||
current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
|
||||
calculates the gradient at the expected value of the parameters given
|
||||
our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
|
||||
\mathbf{v}_{t-1}) \). This yields the NAG update rule
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
|
||||
@@ -1247,6 +1294,7 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
<p>
|
||||
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
|
||||
|
||||
<p>
|
||||
@@ -1272,7 +1320,10 @@ the steep computational price of calculating or approximating
|
||||
Hessians.
|
||||
|
||||
<p>
|
||||
Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
|
||||
Recently, a number of methods have been introduced that accomplish
|
||||
this by tracking not only the gradient, but also the second moment of
|
||||
the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
|
||||
ADAM.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1280,7 +1331,11 @@ Recently, a number of methods have been introduced that accomplish this by track
|
||||
<h2 id="___sec39">RMS prop </h2>
|
||||
|
||||
<p>
|
||||
In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
|
||||
In RMS prop, in addition to keeping a running average of the first
|
||||
moment of the gradient, we also keep track of the second moment
|
||||
denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
|
||||
for RMS prop is given by
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
|
||||
@@ -1290,7 +1345,16 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
|
||||
<p>
|
||||
where \( \beta \) controls the averaging time of the second moment and is
|
||||
typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
|
||||
typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
|
||||
small regularization constant to prevent divergences. Multiplication
|
||||
and division by vectors is understood as an element-wise operation. It
|
||||
is clear from this formula that the learning rate is reduced in
|
||||
directions where the norm of the gradient is consistently large. This
|
||||
greatly speeds up the convergence by allowing us to use a larger
|
||||
learning rate for flat directions.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1298,7 +1362,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica
|
||||
<h2 id="___sec40">ADAM optimizer </h2>
|
||||
|
||||
<p>
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
|
||||
average of both the first and second moment of the gradient and use
|
||||
this information to adaptively change the learning rate for different
|
||||
parameters. In addition to keeping a running average of the first and
|
||||
second moments of the gradient
|
||||
(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
|
||||
\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
|
||||
performs an additional bias correction to account for the fact that we
|
||||
are estimating the first two moments of the gradient using a running
|
||||
average (denoted by the hats in the update rule below). The update
|
||||
rule for ADAM is given by (where multiplication and division are once
|
||||
again understood to be element-wise operations below)
|
||||
|
||||
$$
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
|
||||
@@ -1312,10 +1388,19 @@ $$
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
<p>
|
||||
where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
|
||||
second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
|
||||
respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
|
||||
|
||||
<p>
|
||||
Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
|
||||
Like in RMSprop, the effective step size of a parameter depends on the
|
||||
magnitude of its gradient squared. To understand this better, let us
|
||||
rewrite this expression in terms of the variance
|
||||
\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
|
||||
(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
|
||||
update rule for this parameter is given by
|
||||
|
||||
$$
|
||||
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
|
||||
$$
|
||||
|
||||
+188
-309
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
@@ -450,7 +450,8 @@ if __name__ == "__main__":
|
||||
|
||||
!split
|
||||
===== The Credit Card example =====
|
||||
Here we use the the "credit card data":"https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients". More text to come.
|
||||
Here we use the the "credit card data":"https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients".
|
||||
The data are from an extensive database from Taiwan and include more than ten predictors.
|
||||
|
||||
!bc pycod
|
||||
import pandas as pd
|
||||
|
||||
+101
-17
@@ -881,15 +881,29 @@ _Challenge_: try to write a similar code for a Logistic Regression case.
|
||||
!split
|
||||
===== Momentum based GD =====
|
||||
|
||||
The stochastic gradient descent (SGD) is almost always used with a *momentum* or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
|
||||
implemented as follows
|
||||
The stochastic gradient descent (SGD) is almost always used with a
|
||||
*momentum* or inertia term that serves as a memory of the direction we
|
||||
are moving in parameter space. This is typically implemented as
|
||||
follows
|
||||
|
||||
!bt
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
|
||||
\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t},
|
||||
\end{align}
|
||||
!et
|
||||
where we have introduced a momentum parameter $\gamma$, with $0\le\gamma\le 1$, and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that $\mathbf{v}_t$ is a running average of recently encountered gradients and $(1-\gamma)^{-1}$ sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when $\gamma=0$, this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
|
||||
|
||||
where we have introduced a momentum parameter $\gamma$, with
|
||||
$0\le\gamma\le 1$, and for brevity we dropped the explicit notation to
|
||||
indicate the gradient is to be taken over a different mini-batch at
|
||||
each step. We call this algorithm gradient descent with momentum
|
||||
(GDM). From these equations, it is clear that $\mathbf{v}_t$ is a
|
||||
running average of recently encountered gradients and
|
||||
$(1-\gamma)^{-1}$ sets the characteristic time scale for the memory
|
||||
used in the averaging procedure. Consistent with this, when
|
||||
$\gamma=0$, this just reduces down to ordinary SGD as discussed
|
||||
earlier. An equivalent way of writing the updates is
|
||||
|
||||
!bt
|
||||
\[
|
||||
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
|
||||
@@ -900,20 +914,28 @@ where we have defined $\Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\bo
|
||||
!split
|
||||
===== More on momentum based approaches =====
|
||||
|
||||
Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass $m$ moving in a viscous medium with drag coefficient $\mu$ and potential
|
||||
$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$, then its motion is described by
|
||||
Let us try to get more intuition from these equations. It is helpful
|
||||
to consider a simple physical analogy with a particle of mass $m$
|
||||
moving in a viscous medium with drag coefficient $\mu$ and potential
|
||||
$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$,
|
||||
then its motion is described by
|
||||
|
||||
!bt
|
||||
\[
|
||||
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
|
||||
\]
|
||||
!et
|
||||
|
||||
We can discretize this equation in the usual way to get
|
||||
|
||||
!bt
|
||||
\[
|
||||
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
|
||||
\]
|
||||
!et
|
||||
|
||||
Rearranging this equation, we can rewrite this as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
|
||||
@@ -922,26 +944,51 @@ Rearranging this equation, we can rewrite this as
|
||||
|
||||
!split
|
||||
===== Momentum parameter =====
|
||||
Notice that this equation is identical to previous one if we identify the position of the particle, $\mathbf{w}$, with the parameters $\boldsymbol{\theta}$. This allows
|
||||
us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
|
||||
|
||||
Notice that this equation is identical to previous one if we identify
|
||||
the position of the particle, $\mathbf{w}$, with the parameters
|
||||
$\boldsymbol{\theta}$. This allows us to identify the momentum
|
||||
parameter and learning rate with the mass of the particle and the
|
||||
viscous drag as:
|
||||
|
||||
!bt
|
||||
\[
|
||||
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
|
||||
\]
|
||||
!et
|
||||
Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$.
|
||||
|
||||
Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
|
||||
Thus, as the name suggests, the momentum parameter is proportional to
|
||||
the mass of the particle and effectively provides inertia.
|
||||
Furthermore, in the large viscosity/small learning rate limit, our
|
||||
memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$.
|
||||
|
||||
These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
Why is momentum useful? SGD momentum helps the gradient descent
|
||||
algorithm gain speed in directions with persistent but small gradients
|
||||
even in the presence of stochasticity, while suppressing oscillations
|
||||
in high-curvature directions. This becomes especially important in
|
||||
situations where the landscape is shallow and flat in some directions
|
||||
and narrow and steep in others. It has been argued that first-order
|
||||
methods (with appropriate initial conditions) can perform comparable
|
||||
to more expensive second order methods, especially in the context of
|
||||
complex deep learning models.
|
||||
|
||||
These beneficial properties of momentum can sometimes become even more
|
||||
pronounced by using a slight modification of the classical momentum
|
||||
algorithm called Nesterov Accelerated Gradient (NAG).
|
||||
|
||||
In the NAG algorithm, rather than calculating the gradient at the
|
||||
current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one
|
||||
calculates the gradient at the expected value of the parameters given
|
||||
our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma
|
||||
\mathbf{v}_{t-1})$. This yields the NAG update rule
|
||||
|
||||
In the NAG algorithm, rather than calculating the gradient at the current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one calculates the gradient at the expected value of the parameters given our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1})$. This yields the NAG update rule
|
||||
!bt
|
||||
\begin{align}
|
||||
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
|
||||
\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}.
|
||||
\end{align}
|
||||
!et
|
||||
|
||||
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\gamma$.
|
||||
|
||||
|
||||
@@ -965,12 +1012,19 @@ adaptively change the step size to match the landscape without paying
|
||||
the steep computational price of calculating or approximating
|
||||
Hessians.
|
||||
|
||||
Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
|
||||
Recently, a number of methods have been introduced that accomplish
|
||||
this by tracking not only the gradient, but also the second moment of
|
||||
the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
|
||||
ADAM.
|
||||
|
||||
!split
|
||||
===== RMS prop =====
|
||||
|
||||
In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule for RMS prop is given by
|
||||
In RMS prop, in addition to keeping a running average of the first
|
||||
moment of the gradient, we also keep track of the second moment
|
||||
denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule
|
||||
for RMS prop is given by
|
||||
|
||||
!bt
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
|
||||
@@ -978,13 +1032,34 @@ In RMS prop, in addition to keeping a running average of the first moment of the
|
||||
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
|
||||
\end{align}
|
||||
!et
|
||||
where $\beta$ controls the averaging time of the second moment and is typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
|
||||
|
||||
where $\beta$ controls the averaging time of the second moment and is
|
||||
typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate
|
||||
typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a
|
||||
small regularization constant to prevent divergences. Multiplication
|
||||
and division by vectors is understood as an element-wise operation. It
|
||||
is clear from this formula that the learning rate is reduced in
|
||||
directions where the norm of the gradient is consistently large. This
|
||||
greatly speeds up the convergence by allowing us to use a larger
|
||||
learning rate for flat directions.
|
||||
|
||||
|
||||
!split
|
||||
===== ADAM optimizer =====
|
||||
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and $\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
|
||||
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
|
||||
average of both the first and second moment of the gradient and use
|
||||
this information to adaptively change the learning rate for different
|
||||
parameters. In addition to keeping a running average of the first and
|
||||
second moments of the gradient
|
||||
(i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and
|
||||
$\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM
|
||||
performs an additional bias correction to account for the fact that we
|
||||
are estimating the first two moments of the gradient using a running
|
||||
average (denoted by the hats in the update rule below). The update
|
||||
rule for ADAM is given by (where multiplication and division are once
|
||||
again understood to be element-wise operations below)
|
||||
|
||||
!bt
|
||||
\begin{align}
|
||||
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
|
||||
@@ -995,9 +1070,18 @@ A related algorithm is the ADAM optimizer. In ADAM, we keep a running average o
|
||||
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\
|
||||
\end{align}
|
||||
!et
|
||||
where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
|
||||
|
||||
Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance $\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The update rule for this parameter is given by
|
||||
where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and
|
||||
second moment and are typically taken to be $0.9$ and $0.99$
|
||||
respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
|
||||
|
||||
Like in RMSprop, the effective step size of a parameter depends on the
|
||||
magnitude of its gradient squared. To understand this better, let us
|
||||
rewrite this expression in terms of the variance
|
||||
$\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
|
||||
(\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The
|
||||
update rule for this parameter is given by
|
||||
|
||||
!bt
|
||||
\[
|
||||
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
|
||||
|
||||
Reference in New Issue
Block a user