-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
-implemented as follows
+The stochastic gradient descent (SGD) is almost always used with a
+momentum or inertia term that serves as a memory of the direction we
+are moving in parameter space. This is typically implemented as
+follows
+
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
@@ -289,7 +292,18 @@ $$
\end{align}
$$
-where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
+
+where we have introduced a momentum parameter \( \gamma \), with
+\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
+indicate the gradient is to be taken over a different mini-batch at
+each step. We call this algorithm gradient descent with momentum
+(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
+running average of recently encountered gradients and
+\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
+used in the averaging procedure. Consistent with this, when
+\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
+earlier. An equivalent way of writing the updates is
+
$$
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
$$
diff --git a/doc/pub/Splines/html/._Splines-bs037.html b/doc/pub/Splines/html/._Splines-bs037.html
index 6bc14f3ca..fee482924 100644
--- a/doc/pub/Splines/html/._Splines-bs037.html
+++ b/doc/pub/Splines/html/._Splines-bs037.html
@@ -279,18 +279,26 @@ MathJax.Hub.Config({
More on momentum based approaches
-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
-\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
+Let us try to get more intuition from these equations. It is helpful
+to consider a simple physical analogy with a particle of mass \( m \)
+moving in a viscous medium with drag coefficient \( \mu \) and potential
+\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
+then its motion is described by
+
$$
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
$$
+
We can discretize this equation in the usual way to get
+
$$
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
$$
+
Rearranging this equation, we can rewrite this as
+
$$
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
$$
diff --git a/doc/pub/Splines/html/._Splines-bs038.html b/doc/pub/Splines/html/._Splines-bs038.html
index 6b884699c..4cc1b9d05 100644
--- a/doc/pub/Splines/html/._Splines-bs038.html
+++ b/doc/pub/Splines/html/._Splines-bs038.html
@@ -277,22 +277,47 @@ MathJax.Hub.Config({
Momentum parameter
-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
-us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
+
+
+Notice that this equation is identical to previous one if we identify
+the position of the particle, \( \mathbf{w} \), with the parameters
+\( \boldsymbol{\theta} \). This allows us to identify the momentum
+parameter and learning rate with the mass of the particle and the
+viscous drag as:
+
$$
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
$$
-Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
+
+Thus, as the name suggests, the momentum parameter is proportional to
+the mass of the particle and effectively provides inertia.
+Furthermore, in the large viscosity/small learning rate limit, our
+memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
+Why is momentum useful? SGD momentum helps the gradient descent
+algorithm gain speed in directions with persistent but small gradients
+even in the presence of stochasticity, while suppressing oscillations
+in high-curvature directions. This becomes especially important in
+situations where the landscape is shallow and flat in some directions
+and narrow and steep in others. It has been argued that first-order
+methods (with appropriate initial conditions) can perform comparable
+to more expensive second order methods, especially in the context of
+complex deep learning models.
-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
+These beneficial properties of momentum can sometimes become even more
+pronounced by using a slight modification of the classical momentum
+algorithm called Nesterov Accelerated Gradient (NAG).
-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
+In the NAG algorithm, rather than calculating the gradient at the
+current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
+calculates the gradient at the expected value of the parameters given
+our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
+\mathbf{v}_{t-1}) \). This yields the NAG update rule
+
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
@@ -301,6 +326,7 @@ $$
\end{align}
$$
+
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
diff --git a/doc/pub/Splines/html/._Splines-bs039.html b/doc/pub/Splines/html/._Splines-bs039.html
index 976c5d5e4..b4201bb3d 100644
--- a/doc/pub/Splines/html/._Splines-bs039.html
+++ b/doc/pub/Splines/html/._Splines-bs039.html
@@ -296,7 +296,10 @@ the steep computational price of calculating or approximating
Hessians.
-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
+Recently, a number of methods have been introduced that accomplish
+this by tracking not only the gradient, but also the second moment of
+the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
+ADAM.
-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
+In RMS prop, in addition to keeping a running average of the first
+moment of the gradient, we also keep track of the second moment
+denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
+for RMS prop is given by
+
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
@@ -289,7 +293,16 @@ $$
\end{align}
$$
-where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
+
+where \( \beta \) controls the averaging time of the second moment and is
+typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
+typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
+small regularization constant to prevent divergences. Multiplication
+and division by vectors is understood as an element-wise operation. It
+is clear from this formula that the learning rate is reduced in
+directions where the norm of the gradient is consistently large. This
+greatly speeds up the convergence by allowing us to use a larger
+learning rate for flat directions.
-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
+A related algorithm is the ADAM optimizer. In ADAM, we keep a running
+average of both the first and second moment of the gradient and use
+this information to adaptively change the learning rate for different
+parameters. In addition to keeping a running average of the first and
+second moments of the gradient
+(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
+\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
+performs an additional bias correction to account for the fact that we
+are estimating the first two moments of the gradient using a running
+average (denoted by the hats in the update rule below). The update
+rule for ADAM is given by (where multiplication and division are once
+again understood to be element-wise operations below)
+
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
@@ -293,10 +305,19 @@ $$
\end{align}
$$
-where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
+
+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
+second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
+respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
+Like in RMSprop, the effective step size of a parameter depends on the
+magnitude of its gradient squared. To understand this better, let us
+rewrite this expression in terms of the variance
+\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
+(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
+update rule for this parameter is given by
+
$$
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
$$
diff --git a/doc/pub/Splines/html/Splines-bs.html b/doc/pub/Splines/html/Splines-bs.html
index 02f35b586..74abae7d1 100644
--- a/doc/pub/Splines/html/Splines-bs.html
+++ b/doc/pub/Splines/html/Splines-bs.html
@@ -295,7 +295,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Sep 26, 2019
+
Sep 28, 2019
@@ -1187,8 +1187,11 @@ plt.show()
Momentum based GD
-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
-implemented as follows
+The stochastic gradient descent (SGD) is almost always used with a
+momentum or inertia term that serves as a memory of the direction we
+are moving in parameter space. This is typically implemented as
+follows
+
$$
\begin{align}
@@ -1199,7 +1202,18 @@ $$
$$
-where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
+
+where we have introduced a momentum parameter \( \gamma \), with
+\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
+indicate the gradient is to be taken over a different mini-batch at
+each step. We call this algorithm gradient descent with momentum
+(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
+running average of recently encountered gradients and
+\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
+used in the averaging procedure. Consistent with this, when
+\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
+earlier. An equivalent way of writing the updates is
+
$$
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
@@ -1214,22 +1228,30 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
More on momentum based approaches
-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
-\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
+Let us try to get more intuition from these equations. It is helpful
+to consider a simple physical analogy with a particle of mass \( m \)
+moving in a viscous medium with drag coefficient \( \mu \) and potential
+\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
+then its motion is described by
+
-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
-us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
+
+
+Notice that this equation is identical to previous one if we identify
+the position of the particle, \( \mathbf{w} \), with the parameters
+\( \boldsymbol{\theta} \). This allows us to identify the momentum
+parameter and learning rate with the mass of the particle and the
+viscous drag as:
+
$$
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
$$
-Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
+
+Thus, as the name suggests, the momentum parameter is proportional to
+the mass of the particle and effectively provides inertia.
+Furthermore, in the large viscosity/small learning rate limit, our
+memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
+Why is momentum useful? SGD momentum helps the gradient descent
+algorithm gain speed in directions with persistent but small gradients
+even in the presence of stochasticity, while suppressing oscillations
+in high-curvature directions. This becomes especially important in
+situations where the landscape is shallow and flat in some directions
+and narrow and steep in others. It has been argued that first-order
+methods (with appropriate initial conditions) can perform comparable
+to more expensive second order methods, especially in the context of
+complex deep learning models.
-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
+These beneficial properties of momentum can sometimes become even more
+pronounced by using a slight modification of the classical momentum
+algorithm called Nesterov Accelerated Gradient (NAG).
-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
+In the NAG algorithm, rather than calculating the gradient at the
+current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
+calculates the gradient at the expected value of the parameters given
+our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
+\mathbf{v}_{t-1}) \). This yields the NAG update rule
+
$$
\begin{align}
@@ -1268,6 +1315,7 @@ $$
$$
+
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
@@ -1293,7 +1341,10 @@ the steep computational price of calculating or approximating
Hessians.
-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
+Recently, a number of methods have been introduced that accomplish
+this by tracking not only the gradient, but also the second moment of
+the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
+ADAM.
@@ -1301,7 +1352,11 @@ Recently, a number of methods have been introduced that accomplish this by track
RMS prop
-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
+In RMS prop, in addition to keeping a running average of the first
+moment of the gradient, we also keep track of the second moment
+denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
+for RMS prop is given by
+
$$
\begin{align}
@@ -1313,7 +1368,16 @@ $$
$$
-where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
+
+where \( \beta \) controls the averaging time of the second moment and is
+typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
+typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
+small regularization constant to prevent divergences. Multiplication
+and division by vectors is understood as an element-wise operation. It
+is clear from this formula that the learning rate is reduced in
+directions where the norm of the gradient is consistently large. This
+greatly speeds up the convergence by allowing us to use a larger
+learning rate for flat directions.
@@ -1321,7 +1385,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica
ADAM optimizer
-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
+A related algorithm is the ADAM optimizer. In ADAM, we keep a running
+average of both the first and second moment of the gradient and use
+this information to adaptively change the learning rate for different
+parameters. In addition to keeping a running average of the first and
+second moments of the gradient
+(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
+\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
+performs an additional bias correction to account for the fact that we
+are estimating the first two moments of the gradient using a running
+average (denoted by the hats in the update rule below). The update
+rule for ADAM is given by (where multiplication and division are once
+again understood to be element-wise operations below)
+
$$
\begin{align}
@@ -1337,10 +1413,19 @@ $$
$$
-where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
+
+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
+second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
+respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
+Like in RMSprop, the effective step size of a parameter depends on the
+magnitude of its gradient squared. To understand this better, let us
+rewrite this expression in terms of the variance
+\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
+(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
+update rule for this parameter is given by
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Sep 26, 2019
+
Sep 28, 2019
@@ -1175,8 +1175,11 @@ plt.show()
Momentum based GD
-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
-implemented as follows
+The stochastic gradient descent (SGD) is almost always used with a
+momentum or inertia term that serves as a memory of the direction we
+are moving in parameter space. This is typically implemented as
+follows
+
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
@@ -1185,7 +1188,18 @@ $$
\end{align}
$$
-where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
+
+where we have introduced a momentum parameter \( \gamma \), with
+\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
+indicate the gradient is to be taken over a different mini-batch at
+each step. We call this algorithm gradient descent with momentum
+(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
+running average of recently encountered gradients and
+\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
+used in the averaging procedure. Consistent with this, when
+\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
+earlier. An equivalent way of writing the updates is
+
$$
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
$$
@@ -1198,18 +1212,26 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
More on momentum based approaches
-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
-\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
+Let us try to get more intuition from these equations. It is helpful
+to consider a simple physical analogy with a particle of mass \( m \)
+moving in a viscous medium with drag coefficient \( \mu \) and potential
+\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
+then its motion is described by
+
$$
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
$$
+
We can discretize this equation in the usual way to get
+
$$
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
$$
+
Rearranging this equation, we can rewrite this as
+
$$
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
$$
@@ -1218,22 +1240,47 @@ $$
Momentum parameter
-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
-us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
+
+
+Notice that this equation is identical to previous one if we identify
+the position of the particle, \( \mathbf{w} \), with the parameters
+\( \boldsymbol{\theta} \). This allows us to identify the momentum
+parameter and learning rate with the mass of the particle and the
+viscous drag as:
+
$$
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
$$
-Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
+
+Thus, as the name suggests, the momentum parameter is proportional to
+the mass of the particle and effectively provides inertia.
+Furthermore, in the large viscosity/small learning rate limit, our
+memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
+Why is momentum useful? SGD momentum helps the gradient descent
+algorithm gain speed in directions with persistent but small gradients
+even in the presence of stochasticity, while suppressing oscillations
+in high-curvature directions. This becomes especially important in
+situations where the landscape is shallow and flat in some directions
+and narrow and steep in others. It has been argued that first-order
+methods (with appropriate initial conditions) can perform comparable
+to more expensive second order methods, especially in the context of
+complex deep learning models.
-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
+These beneficial properties of momentum can sometimes become even more
+pronounced by using a slight modification of the classical momentum
+algorithm called Nesterov Accelerated Gradient (NAG).
-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
+In the NAG algorithm, rather than calculating the gradient at the
+current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
+calculates the gradient at the expected value of the parameters given
+our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
+\mathbf{v}_{t-1}) \). This yields the NAG update rule
+
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
@@ -1242,6 +1289,7 @@ $$
\end{align}
$$
+
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
@@ -1267,7 +1315,10 @@ the steep computational price of calculating or approximating
Hessians.
-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
+Recently, a number of methods have been introduced that accomplish
+this by tracking not only the gradient, but also the second moment of
+the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
+ADAM.
@@ -1275,7 +1326,11 @@ Recently, a number of methods have been introduced that accomplish this by track
RMS prop
-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
+In RMS prop, in addition to keeping a running average of the first
+moment of the gradient, we also keep track of the second moment
+denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
+for RMS prop is given by
+
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
@@ -1285,7 +1340,16 @@ $$
\end{align}
$$
-where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
+
+where \( \beta \) controls the averaging time of the second moment and is
+typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
+typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
+small regularization constant to prevent divergences. Multiplication
+and division by vectors is understood as an element-wise operation. It
+is clear from this formula that the learning rate is reduced in
+directions where the norm of the gradient is consistently large. This
+greatly speeds up the convergence by allowing us to use a larger
+learning rate for flat directions.
@@ -1293,7 +1357,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica
ADAM optimizer
-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
+A related algorithm is the ADAM optimizer. In ADAM, we keep a running
+average of both the first and second moment of the gradient and use
+this information to adaptively change the learning rate for different
+parameters. In addition to keeping a running average of the first and
+second moments of the gradient
+(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
+\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
+performs an additional bias correction to account for the fact that we
+are estimating the first two moments of the gradient using a running
+average (denoted by the hats in the update rule below). The update
+rule for ADAM is given by (where multiplication and division are once
+again understood to be element-wise operations below)
+
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
@@ -1307,10 +1383,19 @@ $$
\end{align}
$$
-where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
+
+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
+second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
+respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
+Like in RMSprop, the effective step size of a parameter depends on the
+magnitude of its gradient squared. To understand this better, let us
+rewrite this expression in terms of the variance
+\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
+(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
+update rule for this parameter is given by
+
$$
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
$$
diff --git a/doc/pub/Splines/html/Splines.html b/doc/pub/Splines/html/Splines.html
index 3bcf473b9..2ec8fb04b 100644
--- a/doc/pub/Splines/html/Splines.html
+++ b/doc/pub/Splines/html/Splines.html
@@ -218,7 +218,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Sep 26, 2019
+
Sep 28, 2019
@@ -1180,8 +1180,11 @@ plt.show()
Momentum based GD
-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
-implemented as follows
+The stochastic gradient descent (SGD) is almost always used with a
+momentum or inertia term that serves as a memory of the direction we
+are moving in parameter space. This is typically implemented as
+follows
+
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
@@ -1190,7 +1193,18 @@ $$
\end{align}
$$
-where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
+
+where we have introduced a momentum parameter \( \gamma \), with
+\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to
+indicate the gradient is to be taken over a different mini-batch at
+each step. We call this algorithm gradient descent with momentum
+(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a
+running average of recently encountered gradients and
+\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory
+used in the averaging procedure. Consistent with this, when
+\( \gamma=0 \), this just reduces down to ordinary SGD as discussed
+earlier. An equivalent way of writing the updates is
+
$$
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
$$
@@ -1203,18 +1217,26 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\
More on momentum based approaches
-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential
-\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by
+Let us try to get more intuition from these equations. It is helpful
+to consider a simple physical analogy with a particle of mass \( m \)
+moving in a viscous medium with drag coefficient \( \mu \) and potential
+\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \),
+then its motion is described by
+
$$
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
$$
+
We can discretize this equation in the usual way to get
+
$$
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
$$
+
Rearranging this equation, we can rewrite this as
+
$$
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
$$
@@ -1223,22 +1245,47 @@ $$
Momentum parameter
-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows
-us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
+
+
+Notice that this equation is identical to previous one if we identify
+the position of the particle, \( \mathbf{w} \), with the parameters
+\( \boldsymbol{\theta} \). This allows us to identify the momentum
+parameter and learning rate with the mass of the particle and the
+viscous drag as:
+
$$
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
$$
-Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
+
+Thus, as the name suggests, the momentum parameter is proportional to
+the mass of the particle and effectively provides inertia.
+Furthermore, in the large viscosity/small learning rate limit, our
+memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).
-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
+Why is momentum useful? SGD momentum helps the gradient descent
+algorithm gain speed in directions with persistent but small gradients
+even in the presence of stochasticity, while suppressing oscillations
+in high-curvature directions. This becomes especially important in
+situations where the landscape is shallow and flat in some directions
+and narrow and steep in others. It has been argued that first-order
+methods (with appropriate initial conditions) can perform comparable
+to more expensive second order methods, especially in the context of
+complex deep learning models.
-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
+These beneficial properties of momentum can sometimes become even more
+pronounced by using a slight modification of the classical momentum
+algorithm called Nesterov Accelerated Gradient (NAG).
-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule
+In the NAG algorithm, rather than calculating the gradient at the
+current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one
+calculates the gradient at the expected value of the parameters given
+our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma
+\mathbf{v}_{t-1}) \). This yields the NAG update rule
+
$$
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
@@ -1247,6 +1294,7 @@ $$
\end{align}
$$
+
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).
@@ -1272,7 +1320,10 @@ the steep computational price of calculating or approximating
Hessians.
-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
+Recently, a number of methods have been introduced that accomplish
+this by tracking not only the gradient, but also the second moment of
+the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
+ADAM.
@@ -1280,7 +1331,11 @@ Recently, a number of methods have been introduced that accomplish this by track
RMS prop
-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by
+In RMS prop, in addition to keeping a running average of the first
+moment of the gradient, we also keep track of the second moment
+denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule
+for RMS prop is given by
+
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
@@ -1290,7 +1345,16 @@ $$
\end{align}
$$
-where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
+
+where \( \beta \) controls the averaging time of the second moment and is
+typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate
+typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a
+small regularization constant to prevent divergences. Multiplication
+and division by vectors is understood as an element-wise operation. It
+is clear from this formula that the learning rate is reduced in
+directions where the norm of the gradient is consistently large. This
+greatly speeds up the convergence by allowing us to use a larger
+learning rate for flat directions.
@@ -1298,7 +1362,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica
ADAM optimizer
-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
+A related algorithm is the ADAM optimizer. In ADAM, we keep a running
+average of both the first and second moment of the gradient and use
+this information to adaptively change the learning rate for different
+parameters. In addition to keeping a running average of the first and
+second moments of the gradient
+(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
+\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
+performs an additional bias correction to account for the fact that we
+are estimating the first two moments of the gradient using a running
+average (denoted by the hats in the update rule below). The update
+rule for ADAM is given by (where multiplication and division are once
+again understood to be element-wise operations below)
+
$$
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
@@ -1312,10 +1388,19 @@ $$
\end{align}
$$
-where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
+
+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and
+second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
+respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by
+Like in RMSprop, the effective step size of a parameter depends on the
+magnitude of its gradient squared. To understand this better, let us
+rewrite this expression in terms of the variance
+\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
+(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
+update rule for this parameter is given by
+
$$
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
$$
diff --git a/doc/pub/Splines/ipynb/Splines.ipynb b/doc/pub/Splines/ipynb/Splines.ipynb
index 63e448d1e..688dd3851 100644
--- a/doc/pub/Splines/ipynb/Splines.ipynb
+++ b/doc/pub/Splines/ipynb/Splines.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 26, 2019**\n",
+ "Date: **Sep 28, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -724,30 +724,11 @@
},
{
"cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[[3.7347544 ]\n",
- " [3.28652545]]\n",
- "[[3.7347544 ]\n",
- " [3.28652545]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmYFPW1//H3mWEdRYUBFIEZXFE0xgWjUWMwiVGMN5pdM+4aLphrNjWJ8kuMJsQkZjOLcQvRyMRoNNckXkncBlAQcERQdpVNFmUXBBxg5vz+qBpshu6e6unqbebzep5+6K71VE1Tp+t7vlVl7o6IiEhrygodgIiIlAYlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlD8s7MlpjZJ8L3N5rZvQWKY5iZLS/EujsKM7vPzH5U6DgkHkoYshszu8DMppnZFjNbHb6/2swsF+tz9x+7+1XZLsfMBpmZm1mnOOIqNB1opRgpYcguZnYtcDtwG3AAsD8wEjgV6JJinvK8BSgiBaWEIQCY2b7ALcDV7v6Iu2/2wMvuXuPuDeF095nZH8zsCTPbApxhZp8ys5fNbJOZvWlmP2ix7IvNbKmZrTOz0S3G/cDMxiV8PtnMppjZRjObZWbDEsZNMLMfmtlkM9tsZk+aWe9w9KTw341m9q6ZfTjJNnYP499gZnOBE1uMP9DMHjWzNWa22My+ljDuQ2ZWH27j22b2y4RxpyXE/KaZXRYO72pmPzezZeE8d5pZ93DcMDNbbmbXhmdyq8zs8nDcCKAG+Ha4Lf9K8Tc7wsyeMrP1ZrbAzL4YDu9iZjPN7Jrwc3m4z76fsC0vhPGuMrPfmVmXhOV6eFb5Wriff2hmh4TbuMnMHm6ePmE7bjSztWFzY02yeMPpzw1j2xgu75hU00oRcne99AI4G9gJdGpluvuAdwjOOsqAbsAw4APh52OAt4Hzw+mHAO8CpwNdgV+G6/lEOP4HwLjwfX9gHXBOuKwzw899wvETgDeAw4Hu4eefhOMGAZ4ufuAnwHNAL2AgMBtYHo4rA14Cvk9wNnUwsAg4Kxz/AnBx+H5v4OTwfTWwGbgQ6AxUAseG434F/DNcXw/gX8Ct4bhh4X64JZzvHGAr0DNhP/8ozbbsBbwJXA50Ao4D1gJDwvFHAxuAI4HRwFSgPBx3AnByON8gYB7wjYRlO/APYB/gKKABeCbcJ/sCc4FLW2zHL8O/70eBLcDgltsRxrgaOAkoBy4FlgBdC/391yvaS2cY0qw3sNbddzYPSPjVvM3MTk+Y9h/uPtndm9z9PXef4O6vhp9fAR4kOHAAfB543N0neXCW8j2gKUUMFwFPuPsT4bKeAuoJDqbN/uTuC919G/AwcGwG2/hFYIy7r3f3N4HfJIw7kSAx3eLu2919EXAPcEE4fgdwqJn1dvd33X1qOPzLwNPu/qC773D3de4+M6z5jAC+Ga5vM/DjhOU1L/OWcL4nCBLr4Ijbci6wxN3/5O473f1l4FHgCwDuPhv4EfAYcB1BsmsMx73k7lPD+ZYAd/H+36vZz9x9k7vPIUisT7r7Ind/BxhPcPBP9D13b3D3icD/hfu6pRHAXe4+zd0b3f1+gmR0csRtlgJTwpBm64DeiUVjdz/F3fcLxyV+V95MnNHMTjKzurAp5x2CukdzU9GBidO7+5ZweclUA18Ik9RGM9sInAb0S5jmrYT3Wwl+7Ue1WyzA0hbrPrDFum8kqOMAXElwZjPfzF40s3PD4QMJznpa6gNUAC8lLO/f4fBm6xITdIbbUw2c1CLeGoLaU7P7w+mecPfXmgea2eFm9riZvWVmmwgSWW9293bC+21JPifGuSH8uzZbSrCvk8V8bYuYB6aYVoqQEoY0e4Hg1955EaZteYvjvxA0vQx0932BO4HmXlWrCA4KAJhZBUGzTTJvAg+4+34Jr73c/SdtiCmZ3WIBqlqse3GLdfdw93MA3P01d78Q6Av8FHjEzJqbhQ5Jsq61BAfWoxKWt6+7R00IrW3Pm8DEFvHu7e6jEqa5A3gcOMvMTksY/gdgPnCYu+9DkBiz6QXXM9wXzaqAlSliHtMi5gp3fzCLdUseKWEIAO6+EbgZuMPMPm9mPcyszMyOJWgvT6cHsN7d3zOzDxE00zR7BDg3LAx3IWizT/W9Gwf8l5mdFRZqu4VF1QERNmENQVPXwWmmeRi4wcx6hsu8JmHcdGCzmX0nLI6Xm9nRZnYigJldZGZ93L0J2BjO0wTUAp8wsy+aWSczqzSzY8Pp7gF+ZWZ9w2X0N7OzImwLBL/o023L48DhFnQo6By+TjSzI8N1XUxQq7gM+Bpwv5k1J6sewCbgXTM7Ahi15+IzdnNYbP8IQXPZ35JMcw8wMjwjNTPby4IOEz1iWL/kgRKG7OLuPwO+BXyb4ID1NkH79neAKWlmvRq4xcw2ExSNH05Y5hzgqwRnIasICrFJL5YL6wrnEfziXUPwi/R6InxP3X0rMAaYHDZ3JGsXv5mguWQx8CTwQML8jQQHumPD8WuBewmKvBB0CphjZu8SdD2+wN23ufsyghrLtcB6YCbwwXCe7wCvA1PDpp+niV6j+CMwJNyWx5Js72bgkwQ1kZUETXU/BbqaWRXwa+CSsN7yF4Ja0K/C2a8jSOqbCQ7iD0WMKZW3CP6uKwkS6Eh3n58k5nrgK8DvwulfJ0hoUiLMXQ9QEpG2saDb8zh3j3IWKCVOZxgiIhKJEoaIiESiJikREYlEZxgiIhJJSd3Zs3fv3j5o0KBChyEiUlJeeumlte7ep/Up0yuphDFo0CDq6+sLHYaISEkxs6WtT9U6NUmJiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISSc4ThpmNNbPVZjY7ybhrzczNrOUD6EVEpMjk4wzjPoLHW+7GzAYSPGJyWR5iEBGRLOU8Ybj7JIJnHbf0K4JnR+uBHCIiJaAgNQwzOw9Y4e6zIkw7wszqzax+zZo1eYhORESSyXvCMLMK4Ebg+1Gmd/e73X2ouw/t0yfr27mLiEgbFeIM4xDgIGCWmS0BBgAzzOyAAsQiIiIR5f0BSu7+KtC3+XOYNIa6+9p8xyIiItHlo1vtg8ALwGAzW25mV+Z6nSIiEr+cn2G4+4WtjB+U6xhERCR7utJbREQiUcIQEZFIlDBERHKpthYGDYKysuDf2tpCR9Rmee8lJSLSYdTWwogRsHVr8Hnp0uAzQE1N4eJqI51hiIjkyujR7yeLZlu3BsNLkBKGiEiuLEtxb9VUw4ucEoaISK5UVWU2vMgpYYiI5MqYMVBRsfuwiopgeAlSwhARyZWaGrj7bqiuBrPg37vvLsmCN6iXlIhIbtXUlGyCaElnGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESS84RhZmPNbLWZzU4YdpuZzTezV8zsf81sv1zHISIi2cnHGcZ9wNkthj0FHO3uxwALgRvyEIeIiGQh5wnD3ScB61sMe9Ldd4YfpwIDch2HiIhkpxhqGFcA41ONNLMRZlZvZvVr1qzJY1giIpKooAnDzEYDO4HaVNO4+93uPtTdh/bp0yd/wYmIyG4K9jwMM7sMOBf4uLt7oeIQEZFoCnKGYWZnA98GPu3uWwsRg4hIu1NbC4MGQVlZ8G9tysabNslHt9oHgReAwWa23MyuBH4H9ACeMrOZZnZnruMQEYkkxwfdnKmthREjYOlScA/+HTEi1vitlFqDhg4d6vX19YUOQ0Taq+aD7taEho+KitJ4DvegQUGSaKm6Glu69CV3H5rtKoqhl5SISHEYPXr3ZAHB59GjCxNPJpYty2x4GyhhiIg0y8NBNydqa4MmtGSqqmJbjRKGiEizVAfXGA+6sWtuRmts3HNcRQWMGRPbqpQwRESajRkTHGQTxXzQjV2yZjSA8vLYay9KGCIizWpqgoNsdTWYBf8We8E7VXNZU1PscRfswj0RkaJUU1PcCaKlqqrkvaNy0IymMwwRkVKWx2Y0JQwRaV9K9cK7tspjM5qapESk/Wh54V3z1c5QWs1MmcpTM5rOMESk/SjlC+9KgBKGiLQfpXrhXYlQwhCR+BWqjlCKF96VECUMEYlXHu6amlIpXnhXQpQwRCRehawjlOKFdyVEtzcXkXiVlQVnFi2ZBVcfS96ZmW5vLiJFSHWEdksJQ0Tila6O0NEuqmtndOGeiMSruV4wenTQnbWq6v2ic0e8qK6ANizeyNO/nRfb8lTDEJH8SPMIUZYsyXc076ut3TO5lWgCa9rZxMyHFjD+vrcZP60XUzcPoZFOQDw1DJ1hiEh+FONFde3gViLr39jAk7fP499PNPHvxYfzdtORwJGcUDGXG057nuEXVXLqyHjWlfMzDDMbC5wLrHb3o8NhvYCHgEHAEuCL7r6htWXpDEOkhBXjGUYxxtSKpp1NzPjLfMbfv5rx0yuZ9u4Qmiinl63nkwPnM/ysJs762mD2P7rPrnlKqZfUfcDZLYZ9F3jG3Q8Dngk/i0h7VowX1RXjWU8Saxes4y9fncwlhzxPvy7rOPHSIdz07Ons9DIeGnwT2/fqyVqv5MFlp3LJ389n/1lP5iSOnCcMd58ErG8x+Dzg/vD9/cD5uY5DRAosXxfVZdITq0i7ADftbGL6n+Zw8xkTOHnv2fQ9oic1d5zKE4uP5BNVC3lg5GTenrue6XfN5POLb6Pzlo1Y88zr1sFFF8HVV8cfmLvn/EXQ9DQ74fPGhPeW+DnJvCOAeqC+qqrKRURSGjfOvaLCPbh0MHhVVATD45i+eZ7qanez4N9002Yw/+q5a/yBkc97zaDnvbetcXA3Gv2kvV71Hwyr82ljZ/vOhp27L6u6evfYE19mu5YN1Hscx/I4FtLqStIkjPDzhijLOeGEEzL7w4hIx5LqAFpdnXqeTBJAWxJMK/M3gT9c/iU3Gh3c+9hqv+ig57z26ud9zfy16ZdnljphJGx3XAkjL91qzWwQ8Li/X/ReAAxz91Vm1g+Y4O6DW1uOit4iklaub0uSZZG88cCBlK9avsfwJuCxIaOp+s6XOf7LR1DWKWK1IFU8zcLtLqWidzL/BC4N318K/KNAcYhIe5LrmkSGRfLG7Y1MuetVvveRCQzday62akXS6cqAz24Zx9BLhkRPFhB0GOjSJfX4mGsxOU8YZvYg8AIw2MyWm9mVwE+AM83sNeAT4WcRkfRaK2jnuidWhIT01iurue+q57mgagp9um3i1JEf4MfPf4Ru5TvZ1K1v6mW3pWdWTQ2MHQt77bXnuFz0QIujXStfL9UwRDqwqPWDbIvSGcbQVFHh8y79sd94Sp0f133urlEHlL3llx06yR/6xmRfv2jD+/Onqjukq7NEjS3FdlNKNYy4qIYh0oEVy0V2tbXsvO67lL+1grXlfbmh8Yf8ka9Qzk5O2WcOw0/ewPAr+vHBLxyOldme8199Ndx55+61loqKnD63o9RrGCLSUbX1jrUFvMhux9YdTPrtLG748ASOvWoond96kzKaONZfxg87gr996wXWLtnCpHc+yA3/GcaxXxqcPFkA3HEHPPBAaT7kKY7TlHy91CQlUuKy6Zbali6zWVj+4kq/55JJ/rn+U3wfNjq4d2K739J9jG/str83mXlTVVX6azxy1TSWIUrpOoy4XkoYIgmK6IAUWTYH/WyvgUi2vIT9t2Ps/V73q5f92x+q8w90W7BrFf3LVvpVgyf6o9e/4Ft+c0/0OkqcsWZJCUOkIyuyA1JkqQq+ZtHmjytJJtl/W+juFzLOO9PgZ+w3w392Tp2/+veF3tTY9P58URNens+GWhNXwlDRW6QUFUsBOFMFjnv7u9t5/q45HHfjcHpuf3uP8Vv3OYDGeQvpcWCP5AuIemFgkT3XXEVvkY6sRO6yuocCPL516eTl3PnlSZzfbxqVPRr4+HXHse/21Umnrdj8dupkAdEvDCzSmxpmSwlDioOe9ZyZUj0gpbpjLQQPLlq6NPhl3vwgozZ8Dxo2NfD0z2Zw7dAJDOn6BoNOG8CoB09n1toDufiol/nHjdNgwIDkM7e2/6JeGFiMt3KPQxztWvl6qYbRTpVqe3whtbd9lmWb/6KJy/w/J4/2t8sO8EbMF1PtF3Ofn9mr3n95Xp3P+783dq9FZLP/otZRiqhTAip6S7tRZAXCklFEB6SspbvjahLbNmzz//y43r9x3AQf3OUNv5Bx/i4trsDu3j23tykvIUoY0n5k23NGSl95efLvQHn5rklef2aJ//bzE/ycPtO9O1sc3Luyzc+qfNHf6b5/YX905OgZGXGJK2GohiGFV6rt8RKfxsakg72xka8fO5HDuyzm0I9Xc80jH2Xhxj5cdcyL/N8PXmT9mib+ffsC9tm2Z48nYM9OALmoldXWZld/yXb+fIoj6+TrpTOMdqq9tcdL5lI0Sy6m2rux1Yf3me6/+dwEf+3pJbvPl+y7k+oMI47vWbIzgWybVPPQJIuapKRd6UDtyUm11+1vZbu2rNni//eD6f6nAaN9C913O2A2WFef+flbfOu6ramXn+4RpS2TQbYH5lQJJ9X6ozap5qFJVglDpL1or2dYKW4FvuL6X/qvzp/gn6x80buyzcG9O1v8Zz1u9ncqgns0RU6a6R5R2nL+bA/MqRJOqvqLzjCUMERiF8cBoxjPUNI0M4H7EV1e928eX+f/+XG9b9uwLdZ1JN132e7ndMkp5ud8x/2DQQlDpL2I4/5KRXSG0tTY5HP/9bo3kXy7mjBfNHFZtIW1lggz2fZU044aFS3Zpks4HaSXVMGTQCYvJQzJWDH+8m6pBIqmrdm8arP/48apPnLIRK8uf9PBfTFZxpWqoF1ZufvfMZO/cctpR43KPuEU43eqBSUMkdaUyn/wbOMswHUsTY1NPvux1/y2T9X5x3q+5J1pcHDfm01+3gFT/c4vT/TV3/9NdtuVSUG7rTJNtuPGBQkrVfIqUkoYIq0pgl/ekWVzJpSn7dy0YpP/73en+ogjJnpVeBYB7kd3XejXn1jnz/x8hjdsbth9pmy2K13NIK7tyzTZlsqPkBbaRcIAvgnMAWYDDwLd0k2vhCEZKfUryDO5Z1EODmJNjU3+yiML/KfD63zYfjO8E9sd3Hvwjn+m3wt+98UTfdnUFVmtI610Zxhx/R0zTbal9CMkQcknDKA/sBjoHn5+GLgs3TxKGJKREv3P7e6ZJ4GYajXvvPmOP3r9C37V4Ik+oHzFrlUf022+f+ekOq/71ct7nkXkSiYX5cW5juYfGsn2Y4n+CGkvCeNNoBfQCXgc+GS6eZQwOrBUB8R0B8pSaj5ouR2J7eQ5THZNjU0+86H5futZdX76vi/vOovYh43+uf5T/N5LJ/nyF1fGus6MtKwZ5OLvmHi1dsuEEPfFf/mQ5P9EyScMD5LG14F3gTVAbYppRgD1QH1VVVWMe1VKRrrukK0lhFLoJdXaL+mYf8luWLLR//atKX7FYZP8wLKVuxZ9bPd5fsOH63zib2b69i3bY9iwGOXj7xglGRT7j5AU8fWGRV7KCQPoCTwL9AE6A48BF6WbR2cYHVSurrAtFq211We5bU2NTT7jL/N8zJl1fto+M72cHQ7u+7LRvzBgio+9fJKvfPmttsVeCgk5qqjNTcW8zSm+S8dAg5d4wvgC8MeEz5cAd6SbRwmjg2qtt0yJtSfvIer2ZfBLdv2iDf7QNyb7ZYdO8gPK3tq1iOO7z/XRp9b5c7+f5Tu27cgu7mL/tZ1MuoN9KTQ3tSbFd+kEcC/xhHFS2EOqAjDgfuCadPMoYXRQHfUMo7Iy8i/Zxh2NXv/AXP/hx+v8lB6zvIydDu49bb1/aeBkv++q53zVrLfzE3ex7v/WElwpJsCW2usZhgdJ42Zgftit9gGga7rplTA6qGxqGPmOsy1NFW08UK17fb0/eM1kv+Tg53z/srd3zTq0Yo5/7yN1PvnOV7I/i0in1HoMRa1RFGtzUxTttYbRlpcSRgfWll5S+Y4vk+6ZyeZvZTsadzT69Pvm+M1n1PnJe7+y6yyil63zC6uf9z//93O+8Sd/aHvSynS+PPXkik2pJbi2KmQvKeAp4INxrCzblxKGFK3WCtdtPPNZM3+t1179vF900HPex1YHxzca/UN7veo3fbTOX7jnVd/ZsDOYuK1NKm2Zb9w4986d99zOLl2K91d5qTWhxSiuhGHBslIzs+OBXwBLgBvdfVXaGXJo6NChXl9fX6jVi6RWVhYcftKproYlS9JO0ri9kfoH5jH+gbWMr+/Ni1uG4JTR29ZyVvUChg+HT14zmD5H9t5z5kGDgsd7ZrretsyXap7KSli7NvW6Cqn5Uahbt74/rKIC7r4bamoKF1cemNlL7j406wVFzSzA54BZwE2EV2fn+6UzDCkaUS+0a63pY9w439lvgDdhvrp8f/8Kd+46izh571f85jPqfNrY2e+fRaTT1iaXtsyXyYOLikmxNGHmGfm8cI+gF9PRwEhgLbAcuDiOADJ5KWFIJLk+KCRrwuncOWiOSZcwysuDBNGw06fc9Yo/PHi0b6XbbtO8Z1198pk3+dqF6zKPq61NLm2Zr7UmuBK5i2tHkbeEAUwGVhLUMn4InAscCvwWuDuOIKK+lDCkVfnoGtlaN9jEgneL13vW1b/CXQ7uS6hq2wE+rm3P5JYYUdYVU91G4pfPhHEUBLWOJOPmxRFE1JcShrQqH4XNCE04O//0Z28sK0s63ZpO+/tfvzY5eHZ1W5qQ0snmDreZ9OpqXkZrzXAdoKBcCuJKGK0WvdMxs4PdfVGbF5AhFb2lVamKz2bQ1BTPOlIUfHf2G8C4cx5k/JNlPLl8COu8F2WkiaWtReo4xLXuVMtpFud+lzaLq+hdls3M+UwWIpFUVWU2vC3GjAl61yTYRjcuWfUTLv/jaTy34mA+c+irbOvRN30sSZZDRUUwPNeWLctseCrJtiFRnPtdCi6rhCFSdHJ8EF454y3GPl3F7V2uZxkDacJYShW3df8ex5zVn5f/uoAVO/Zn7MKPsNcffpE+lpqaoEtndXXwS7y6On9dPONKrM3bUFm557h8JT/JnzjatfL1Ug1DIomxl9T2Ldt9wq9f9u+cVOfHdJu/q2m+f9lKv/LwSf7IdVN849KNeYklVtl0Dij2q+5lD7SH52Fk+lLC6KDiPhC1srw3p6/0ey6Z5J898AXfh40O7p3Y7sP2m+E/HV7ns/62wJsam7KLoRi0Zb+2hxv0dUBxJYysit75pqJ3BxT31blJlufdK5j7mRu5/7VTGf9Kf2Y3HAbAgPKVDD/sdYZ/ugsfv2YI+wzYJ9utKX2FLNRLm8VV9FbCkOIW9wEqxfKWUM3hLOQjPWcz/JRNnH1lf44671CszDJfR3uWj15oErui6CUleVBbGxzkysqCf2trCx1RfsXUm6dhUwPP3DYDX5p8vmqWsW5FA8+sP57rHh/G0Z85LLtk0V7/bvnohSZFSwmjmDU3nyxdGvyqW7o0+NxeDj5RZHGAWjp5OXd+eRLn9ZtG5b47+MS3j2cZA5NOa9VV9DiwRzaRvq89/90K2RVYCi+OQki+Xh2u6N2Bb8e8SwZF1u333Odb9zvAmzBfzoF+IeMc3Ad1Wuajjpro//x/03zb7/9YuFuH5OLvVoieSeoNVXJQL6kOoKM88KU1aQ5QiyYu899/aYLf1uNmf5fuu+2n7eVdffl1v9yzR1OuD3j5+rupx5JEFFfCUNG7mGVT8K2thdGjg7b+qqqgyaAd3PP/vY3vMfH3sxn/yBbGz6li4Y6DAFjOAPqzYs8ZCtF7J189idRjSSJS0bsjaGt7cTtrQ3/j2aX87gsT+VTfF+nVs4mz/99Q7pr5IQ7edy23f3YiC/69mANtZfKZM73VRRzy1c4f1+09RKKK4zQlX68O1yTl3rbmkxKvfWxdt9WfuHm6X3PMBD+08+Jd4R/aebFfc8wEf+Lm6b5lzZbdZyq2bc5HO3+xbbMULdpDDQPYD3gEmA/MAz6cbvoOmTDaothqHxEOngufXOy3f3aCn917undjq4N7d7b4OX2m+28/P8Ffe3pJ6+voaO35HXGbpU3aS8K4H7gqfN8F2C/d9EoYERX6l2digqisDJ5G1+Kg9t4fxvrjN033r35ggh/SacmuUYd3XuRfP3aC//tHL/rWdVvbvt5MftWXcq+fUo5d8iauhFGworeZ7QvMBA72iEF0uKJ3WxXyYffJ1p3EGnrRl3VUsIWP9Z3D8DO2cfbIgzh4WJ4vACvkvhLJk7iK3oU8uzgWmA7cB7wM3AvslWS6EUA9UF9VVRVjzm3nCvXLs7VnPYevJvBXvvhD37ZhW2bLj3u7Cn02JpIHtIMzjKHAVOBUd59mZrcDm9z9e6nm0RlGcfMmh07lWNTvVLLun+m6A+fibED3RpIOoD10q10OLHf3aeHnR4DjCxiPJKqthd69gwOnWfA+Sbfcd996l3/cOI2RQyZxUJcVLPUMmpQSu382r++ii1J3Bx49es+mrq1bg+FtpXsjiURWsITh7m8Bb5rZ4HDQx4G5hYonJ0r1BnS1tXD55bBu3fvD1q2DK67Ax9Uy5x+v8/NzJ/DxXjPo1a8L5996ErXzjuO4vst545SLaeraPdp6ysqCV+/ee66vWWJCiHLdQab7XPdGEokujnattr4I6hj1wCvAY0DPdNOXVC+pUu7ymKYO8Sb9d308uutCv/7EOn/2FzO8YXPD+/M31xki1DIivZq7A7dWb2jrPldPI2nnaA/dajN9FX3CSDzwlJenP7gVsaZU13GAN2J+zyWTfNnUFa0vKK6k0bzPRo1KPn7UqPTrK4F9LpJLcSUM3RokLi1vx9HYmHy6Ir1twzvL3uHR61/gnn7fp9FTfy3Kqqu46v6PMPCkA1tfaKrmnkwkNg898UTyaZqH61YZIjmlhBGXZAXZZIqkmOpNzsyHFvCTsyfw0f1m0ru6gkd/voiat35OJ1Ikuy5dMmvbr6kJejBVVweF8+rq9z9HUVm5ew+o1hKCCtgiuRXHaUq+XkXdJJWmGadYahgblmz0h785xS8/bJL3K1u1K6xju8/zGz5c59t69ksVZj3JAAAO+ElEQVQde2VlfLEnqzV06RKsI10dIVc1DJF2DtUwYpZt4TPVway8vGDF1MYdjf7SuLk+5sw6P22fmV7ODgf3/WyDf3HgZP/Tlc/5ypffen+GfN6Dqi37O0pCUAFbZA9KGHGK45dpkfy6Xb9og//1a5P90kOe8/3L3t4VyvHd5/roU+v8+Ttm+Y5tO5LPXApFYyUEkYwpYcQprgNlPg9m4bqazPy9yn7+9yGj/ZQes7yMnQ7uvWydX1A12e//ynO+atbb0ZdZBElPROIVV8LQE/eg5G4PsfkXd9P9u1+j086GXcO2UMGtnW+i7OQPMfyiSk7qOpOym76X+RP32umT+kQ6spK/+WBbXnk/w8hl/SGDs5HGHY0+bexsv/mMOj9571d8CVXpz4hGjdqzHqEzBZEOCzVJxShZU0wuezhFaPpZM3+tjxv1vNcMet5725qg9kyjf2ivV72JNMXpceNSF6+LqRYhInkTV8JQk1SzxKaYsrLkF94lu7tqWwwaFFzg10JDr37c+oG/ML6+Ny9uGYJTRh9bw1mDFjB8uHHm/wymz5G9U86/6/qGZOOgbU1saqISKXlxNUkpYSST65pGiuU3YXRiJyftPYfhJ65j+KV9OaHmCMo6tbi+Mt1tvi++OHnskHnC08OFRNqF9nB78+KVoyuGG7c3MuWuV9nYpW/S8dt69GXNwo28sPkDfP/ZYZx46ZA9kwWkvoK6piZ1jGaZ34E1F7cTF5GSpYSRTDa3vG5xe+13fnon93/leS6omkKfbps4deQH+GrDbWyj2x7L3+sPv6DysF7RYqypCc4WmpqCf5t/8SeL3QxGjsz8rED3ZhKRBEoYyaT7BZ9ObS2eeAPCpUvp9N1r+c+9S5m44hDOO2Q2D31jCr99/Vy6j7s38+VH1T3heRSVlfDAA3DHHZkvR/dmEpFEcVTO8/Uq1ntJrXz5LR97+SRfXb5/0t5JDb0P9MYdjbkPJO4L7/J5IZ+u4BbJGdStto1iODBt37LdJ/12pt/w4To/tvu8XcfSxnTdXfMhF9eT5ONArivMRXIqroTRsXpJZdHrZ0X9Kv79u9cZ/3Qnnl4xhHfYl3J2cuq+sxl+8kaGX9GPY67/JJasfT+u7ritSdW7K1Ex9nJK1004H/tNpJ3Tld5tkcE9o7Zv2e4Tfv2yf+ekOj+m2/xdk/YvW+lXDZ7oj17/gm9cunH3mXL1Sznqr/yoT7grtgv48nmXXJEOCDVJtUErB6Y3p6/0uy+e6J/p94L34B0H905s92H7zfCfDq/zVx5Z4E2NTenXEXcTTiZJKMoV68V4IC6Fu+SKlLC4EkbHapJK0fSxocv+nG7PMbvhMAAGlq9g+GFvMPz8rnzsq0eyz4B92r7ObGXaXJPPK9bjogsERXKq3Vy4Z2blZvaymT2e85WNGUNT1+67DdpCBV/b/lP6Vmzmtk9NYPZjr7N0+4HcNe90zr/1pMImC8j8WojE6zPuv7/t15PkU1u7MYtIXhU8YQBfB+blauENmxp45rYZXDd0AkddeRIXNdzDEqppwljXaX/mf/rb3LHiszyz/niue3wYR513KFZmuQpndy0u8qO2ds9psrkWopQOxKkuRBSRotGpkCs3swHAp4AxwLfiWu6S55cz/veLGF/XlWffPootHE8XGji912yGntWfbV95BjvnYCrLjMq4Vpqpls0wS5cGn2H3g+WYMcmba6KeJdTU6OArIvGIoxDS1hfwCHACMAx4PMU0I4B6oL6qqippQWfbhm3+5K31/s3j6/yILq/vqpke1GmpX330BP/X96b55lWb21ovSi2bAncmhV5d1CYiWaDUi95mdi5wjrtfbWbDgOvc/dx08yQWvRdNWMb4OxYzfmJ36lYfxVb2oivv8dHK2Qz/yBaGjxjI4WcdlLvmpdpauOIK2L599+GjRkW7DUeJPeVPREpXyd/e3MxuBS4GdgLdgH2Av7v7RanmObz/ED+n7x8YP6eKhTsOAuCQTksZfuQShn+ugmFfPYqK3hWpZo9X796wbl3ycePGtd4MpIvVRCRPSj5h7BZExDMMs6HejecY1ns2w0/fyvCR1Rx25qC8xJgkmNTjohz01ZVURPIkroRR0KJ3pg47cCuzXoXuvU4sdCjpRbn9d3NS0NPsRKREFMUZRlR5e+JeFOmapNSsJCJFpN1cuFeybr89KFy31KVL8V0YJyISAyWMtqqpgT//OXhAUbPKShg7Vs1KItIulVQNo+joojgR6UB0hiEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikXSshBHl+RMiIpJUx+lWG/X5EyIiklTHOcMYPXr3G/1B8Hn06MLEIyJSYjpOwsj02dgiIrKbjpMwsnk2toiIdKCEMWZM8LyJRJk8G1tEpIPrOAmjpiZ4OFF1dfDwo+pqPaxIRCQDHaeXFOhmgSIiWeg4ZxgiIpIVJQwREYlECUNERCJRwhARkUgKljDMbKCZ1ZnZXDObY2ZfL1QsIiLSukL2ktoJXOvuM8ysB/CSmT3l7nMLGJOIiKRQsDMMd1/l7jPC95uBeUD/QsUjIiLpFUUNw8wGAccB05KMG2Fm9WZWv2bNmnyHJiIioYInDDPbG3gU+Ia7b2o53t3vdveh7j60T58++Q9QRESAAicMM+tMkCxq3f3vhYxFRETSK2QvKQP+CMxz918WKg4REYmmkGcYpwIXAx8zs5nh65wCxiMiImkUrFutuz8PWKHWLyIimSl40VtEREqDEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJAVNGGZ2tpktMLPXzey7hYxFRETSK1jCMLNy4PfAcGAIcKGZDSlUPCIikl4hzzA+BLzu7ovcfTvwV+C8AsYjIiJpdCrguvsDbyZ8Xg6c1HIiMxsBjAg/NpjZ7DzElq3ewNpCBxGB4oxPKcQIijNupRLn4DgWUsiEEYm73w3cDWBm9e4+tMAhtUpxxqsU4iyFGEFxxq2U4oxjOYVskloBDEz4PCAcJiIiRaiQCeNF4DAzO8jMugAXAP8sYDwiIpJGwZqk3H2nmf0P8B+gHBjr7nName3u3EcWC8UZr1KIsxRiBMUZtw4Vp7l7HMsREZF2Tld6i4hIJEoYIiISSdEkjNZuE2JmXc3soXD8NDMblDDuhnD4AjM7q4AxfsvM5prZK2b2jJlVJ4xrNLOZ4Sunxf0IcV5mZmsS4rkqYdylZvZa+Lq0wHH+KiHGhWa2MWFcXvanmY01s9Wprv+xwG/CbXjFzI5PGJfPfdlanDVhfK+a2RQz+2DCuCXh8Jlxdb/MIs5hZvZOwt/2+wnj8nYroQhxXp8Q4+zw+9grHJeX/WlmA82sLjzmzDGzryeZJt7vp7sX/EVQ9H4DOBjoAswChrSY5mrgzvD9BcBD4fsh4fRdgYPC5ZQXKMYzgIrw/ajmGMPP7xbRvrwM+F2SeXsBi8J/e4bvexYqzhbTX0PQMSLf+/N04Hhgdorx5wDjAQNOBqble19GjPOU5vUT3I5nWsK4JUDvItmfw4DHs/2+5DrOFtP+F/Bsvvcn0A84PnzfA1iY5P96rN/PYjnDiHKbkPOA+8P3jwAfNzMLh//V3RvcfTHweri8vMfo7nXuvjX8OJXg2pJ8y+aWK2cBT7n7enffADwFnF0kcV4IPJijWFJy90nA+jSTnAf82QNTgf3MrB/53ZetxunuU8I4oHDfzSj7M5W83koowzgL9d1c5e4zwvebgXkEd9BIFOv3s1gSRrLbhLTc8F3TuPtO4B2gMuK8+Yox0ZUEmb1ZNzOrN7OpZnZ+DuJrFjXOz4WnqI+YWfMFlPnalxmtK2zaOwh4NmFwvvZna1JtRz73ZaZafjcdeNLMXrLgVjyF9mEzm2Vm483sqHBYUe5PM6sgONA+mjA47/vTgib644BpLUbF+v0s+luDlCIzuwgYCnw0YXC1u68ws4OBZ83sVXd/ozAR8i/gQXdvMLP/Jjhz+1iBYoniAuARd29MGFZM+7NkmNkZBAnjtITBp4X7si/wlJnND39hF8IMgr/tu2Z2DvAYcFiBYoniv4DJ7p54NpLX/WlmexMkrG+4+6ZcrQeK5wwjym1Cdk1jZp2AfYF1EefNV4yY2SeA0cCn3b2hebi7rwj/XQRMIPg1kAutxunu6xJiuxc4Ieq8+YwzwQW0OOXP4/5sTartKLpb35jZMQR/7/PcfV3z8IR9uRr4X3LTpBuJu29y93fD908Anc2sN0W4P0Ppvps5359m1pkgWdS6+9+TTBLv9zPXhZmIxZtOBEWXg3i/oHVUi2m+yu5F74fD90exe9F7EbkpekeJ8TiCwtxhLYb3BLqG73sDr5Gjgl3EOPslvP8MMNXfL4QtDuPtGb7vVag4w+mOICgiWiH2Z7iOQaQu0n6K3YuK0/O9LyPGWUVQ3zulxfC9gB4J76cAZxcwzgOa/9YEB9pl4b6N9H3JV5zh+H0J6hx7FWJ/hvvlz8Cv00wT6/czZzu7DRt/DkGV/w1gdDjsFoJf6gDdgL+FX/rpwMEJ844O51sADC9gjE8DbwMzw9c/w+GnAK+GX/JXgSsLvC9vBeaE8dQBRyTMe0W4j18HLi9knOHnHwA/aTFf3vYnwa/HVcAOgnbeK4GRwMhwvBE8COyNMJahBdqXrcV5L7Ah4btZHw4/ONyPs8LvxOgCx/k/Cd/NqSQkuGTfl0LFGU5zGUGHm8T58rY/CZoVHXgl4e96Ti6/n7o1iIiIRFIsNQwRESlyShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEIZKF8HkEZ4bvf2Rmvy10TCK5opsPimTnJuCW8EZzxwGfLnA8IjmjK71FsmRmE4G9gWEePJdApF1Sk5RIFszsAwRPPtuuZCHtnRKGSBuFTy6rJXiq2btmlrMn6okUAyUMkTYIn7T2d+Bad58H/JCgniHSbqmGISIikegMQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCSS/w+0DgVy+oMosQAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 1,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
@@ -802,27 +783,11 @@
},
{
"cell_type": "code",
- "execution_count": 25,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[[4.]\n",
- " [3.]]\n",
- "[3.86998544] [3.11177547]\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:130: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.\n",
- " DeprecationWarning)\n"
- ]
- }
- ],
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"# Importing various packages\n",
"from random import random, seed\n",
@@ -831,12 +796,12 @@
"from sklearn.linear_model import SGDRegressor\n",
"\n",
"x = 2*np.random.rand(100,1)\n",
- "y = 4+3*x#+np.random.randn(100,1)\n",
+ "y = 4+3*x+np.random.randn(100,1)\n",
"\n",
"xb = np.c_[np.ones((100,1)), x]\n",
"beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
"print(beta_linreg)\n",
- "sgdreg = SGDRegressor(n_iter = 100, penalty=None, eta0=0.01)\n",
+ "sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
"sgdreg.fit(x,y.ravel())\n",
"print(sgdreg.intercept_, sgdreg.coef_)"
]
@@ -903,30 +868,11 @@
},
{
"cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[[4.26950679]\n",
- " [2.93950818]]\n",
- "[[4.16449871]\n",
- " [3.01998132]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X+8HHV97/HXO7/QAxHISawIyTlgqVYsokTFH7dSFcUUi7daSwgUWiQFqtVq1WruVfwBetui9dYiDUiJnCMWUVvlihUVS/0B9qAgPy1CcgKKJPwmYIEkn/vHzObMbvbH7O7szu7J+/l4TLI7Mzvz3dk585nvj/l+FRGYmZm1MqfsBJiZ2XBwwDAzs1wcMMzMLBcHDDMzy8UBw8zMcnHAMDOzXBwwdnGSNkh6Vfr6fZLOKykdh0u6s4x97yokXSDpIx1+9qWSbpW0RdLri05bnf2tkvSNJsu/I+nNvU6HVXPAGGCSjpF0taRHJG1KX58mSb3YX0ScGRFd/xFKGpcUkuYVka6ydXOhnUU+BHwqIvaIiH/pdmPpMX08DUD3Sbpc0rMqyyNiMiJe3e1+rFgOGANK0juBTwJ/AzwN+DXgFOClwIIGn5nbtwTarmYMuLGTDza5cfjriNgD2Bf4OfCZDtNmfeKAMYAk7UlyR3daRFwSEQ9H4scRsSoiHkvXu0DSpyV9TdIjwO9I+l1JP5b0kKQ7JJ1es+3jJU1LulfSmpplp0uayLw/TNL3JT0g6TpJh2eWfUfShyV9T9LDkr4haXG6+Mr0/wfSO8gX1/mOT07Tf7+km4AX1Cx/uqQvStosab2kP88se6GkqfQ73i3p45llL8uk+Q5JJ6bzd5P0t5I2pp85R9KT02WHS7pT0jvTnNxdkv44XbYaWAW8O/0uX23wmz0rvUu+T9JPJb0pnb9A0rWS3pq+n5ses/dnvssP0vTeJelTkhZkthtprvLW9Dh/WNIz0u/4kKSLK+tnvsf7JN2jpLhxVb30pusflabtgXR7BzdY7zbgAOCr6THYLf19vpJ+359JOjmz/umSLpE0Iekh4MRGaQCIiF8BFwOHZLZxoqTvZt4fIekWSQ9K+hSgzLK5ks5Kv/N6SW9RJocraU9Jn0mP788lfUS+uepMRHgasAk4EtgKzGux3gXAgyS5jjnAk4DDgd9K3x8M3A28Pl3/2cAW4LeB3YCPp/t5Vbr8dGAifb0vcC+wIt3WEen7Jeny7wC3Ab8BPDl9/7F02TgQzdIPfAz4D2ARsBS4AbgzXTYHuAZ4P0lu6gDgduA16fIfAMenr/cADktfjwEPAyuB+cAocEi67BPAV9L9LQS+Cnw0XXZ4ehw+lH5uBfAosHfmOH+kyXfZHbgD+GNgHvA84B7g2eny5wD3A78JrAGuAuamyw4FDks/Nw7cDLw9s+0A/hV4CnAQ8BjwrfSY7AncBJxQ8z0+nv6+LwceAZ5Z+z3SNG4CXgTMBU4ANgC7NfiOG0jPk/T9lcDZJOfcIcBm4BWZ8+gJ4PXpb/nkBufuRzLH70LguszyE4Hvpq8Xp7/rG9Pf5y/S7/nmdPkp6XHYD9gb+CaZ8w/4MvCP6X6eCvwQ+NOy/86HcSo9AZ7q/ChwHPDLmnnfBx4AfgX8djrvAuCzLbb1d8An0tfvBz6fWbY78Dj1A8Z7gAtrtvVvmYvTd4D/lVl2GvD19PU4rQPG7cCRmfermQkYLwI21qz/XuCf0tdXAh8EFtdZ58t19iWSC+czMvNeDKxPXx+eHtd5meWbmAlEOy5uDb7LHwL/UTPvH4EPZN6/E/gpSeA4sMm23p79DulxfGnm/TXAezLvzwL+LvM9tgK7Z5ZfDPzv2u8BfBr4cM2+fwq8vEG6NmTOk6XANmBhZvlHgQsy59GVLc7LC4D/JjmntwPrgYMzy09kJmD8EXBVze95JzMB49tkAgDwqsr5R1KU+xiZoEVyQ3FFt3+nu+LkIqnBdC+wWJmy34h4SUTslS7L/m53ZD8o6UWSrkiLch4kufuqFBU9Pbt+RDySbq+eMeAP0uKKByQ9ALwM2Cezzi8zrx8ludvPqyotwHTNvp9es+/3kfzxA5xEkrO5RdJ/Sjoqnb+UJNdTawkwAlyT2d7X0/kV90bE1g6/zxjwopr0riKpe6pYl673tYi4tTJT0m9IulTSL9PimzOZ+b0q7s68/lWd99l03p/+rhXTJMe6XprfWZPmpQ3WrfV04L6IeLhmP/tm3t9Ba3+bntPjJN/jmU32lz1vo2b7tedS9vUYSa7krsz3/EeSnIa1yQFjMP2A5K7o6Bzr1nY3/DmSopelEbEncA4z5b13kVwUAJA0QlJsU88dJDmMvTLT7hHxsQ7SVE9VWoBlNfteX7PvhRGxAiAibo2IlSR/9P8HuERSpVjoGXX2dQ/JBemgzPb2jKTCNY9W3+cO4N9r0rtHRJyaWeds4FLgNZJelpn/aeAWklzHU0gCYzet4PZOj0XFMuAXDdJ8Rk2aRyLiohz7+AWwSNLCmv38PPM+dzfYEbEReBvwyUq9Uo3a81ZUnzt3kRRHVWSX3UHyt7Q48z2fEhEH5U2fzXDAGEAR8QBJkcvZkt4oaaGkOZIOISlGamYhyd3ff0t6IXBsZtklwFFKKoYXkJTZNzoHJoDXSXpNWqn4pLRSdb8G62dtJilmOKDJOhcD75W0d7rNt2aW/RB4WNJ7lFSOz5X0HEkvAJB0nKQlEbGdpEiDdH+TwKskvUnSPEmjkg5J1zsX+ISkp6bb2FfSa3J8F0ju6Jt9l0uB31DSoGB+Or1A0m+m+zqepK7iRODPgXWSKsFqIfAQsEVJs9JTd9582z6opLL9fwBHAV+os865wClpjlSSdlfSYGJhnXWrRMQdJEWkH03Pi4NJcn0TzT/ZdJuXkwSi1XUW/z/gIEm/n+a6/5zq3NvFwNvS33QvkuLUynbvAr4BnCXpKenf0TMkvbzTtO7KHDAGVET8NfAO4N0kF6y7SbLS7yH5Y23kNOBDkh4mqbO4OLPNG4E/I8mF3EVSnl73Ybn0onA0yR3vZpI7tXeR45yJiEeBM4DvpcUAh9VZ7YMkxRjrSf6gL8x8fhvJhe6QdPk9wHkklbyQNAq4UdIWkqbHx0TEr9I71RUk9QX3AdcCz00/8x7gZ8BVadHPN2lcBFLrM8Cz0++y0zMIadHMq4FjSC56vyTJ+ewmaRlJPdIfRcSWiPgcMEVSCQ/wlyRB/WGSi/g/50xTI78k+V1/QRJAT4mIW+qkeQo4GfhUuv7PaNGaqcZKkqKkX5BUKn8gIr7ZTcJJmpC/W9JuNWm9B/gDkoYS9wIHAt/LrHIuyTn0E+DHwNdI6nK2pcv/iKTxxE0k3/USqotWLSclxYFmNuyUNHueiIg8ucBZS9JrgXMiYqzstMw2zmGY2VBLiy1XpMWQ+wIfIMn1WMEcMMxs2ImkiPN+kiKpm0mKY61gLpIyM7NcnMMwM7Nchqo30cWLF8f4+HjZyTAzGyrXXHPNPRGxpPWazQ1VwBgfH2dqaqrsZJiZDRVJ063Xas1FUmZmlosDhpmZ5eKAYWZmuThgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpaLA4aZmeXS84Ah6XxJmyTdUGfZOyWFpNpB783MbMD0I4dxAcmQmlUkLSUZ1nJjH9JgZmZd6nnAiIgrScZXrvUJkvGqPSCHmdkQKKUOQ9LRwM8j4roc666WNCVpavPmzX1InZmZ1dP3gCFpBHgfOYdQjIi1EbE8IpYvWdJ1d+5mZtahMnIYzwD2B66TtAHYD/iRpKeVkBYzM8up7wMoRcT1wFMr79OgsTwi7ul3WszMLL9+NKu9CPgB8ExJd0o6qdf7NDOz4vU8hxERK1ssH+91GszMrHt+0tvMzHJxwDAzs1wcMMzMLBcHDDMzy8UBw8zMcnHAMDOzXBwwzMwsFwcMMzPLxQHDzMxyccAwM7NcHDDMzCwXBwwzM8vFAcPMzHJxwDAzs1wcMMzMLBcHDDMzy8UBw8zMcnHAMDOzXBwwzMwsFwcMMzPLpecBQ9L5kjZJuiEz728k3SLpJ5K+LGmvXqfDzMy6048cxgXAkTXzLgeeExEHA/8FvLcP6TAzsy70PGBExJXAfTXzvhERW9O3VwH79TodZmbWnUGow/gT4LJGCyWtljQlaWrz5s19TJaZmWWVGjAkrQG2ApON1omItRGxPCKWL1mypH+JMzOzKvPK2rGkE4GjgFdGRJSVDjMzy6eUgCHpSODdwMsj4tEy0mBmZu3pR7Pai4AfAM+UdKekk4BPAQuByyVdK+mcXqfDzMy6049WUisjYp+ImB8R+0XEZyLi1yNiaUQckk6n9DodZmZDZ3ISxsdhzpzk/8mG1b19UVodhpmZNTE5CatXw6Npqf30dPIeYNWqUpI0CM1qzcys1po1M8Gi4tFHk/klccAwMxtEGze2N78PHDDMzAbRsmXtze8DBwwzmx0GrIK4a2ecASMj1fNGRpL5JXHAMLPhV6kgnp6GiJkK4mEOGqtWwdq1MDYGUvL/2rWlVXiDA4aZzQYDWEFciFWrYMMG2L49+X/VqlJzUm5Wa2bDbwAriHui5Ka2zmGY2fAbwArinig5J+WAYWbDbwAriHuiUY5pejopnjrttJ4WV7lIysyGX6U4Zs2a5KK6bFkSLEqsIO6JZcuS4FDP9DR8+tPV7yvFVQXRMPUsvnz58piamio7GWZm5aitw8hjbAxNT18TEcu73b2LpMzMytROq6dsU9u8Cqz4d8Aws+E3rA/tdfL8SKWpbd6gUWDFvwOGmQ23Xj601+tA1E2rp3oV/bUKrvh3wDCz4darpqb9eHq8m+dH6j0JfuqpPX0y3JXeZjbc5sxJLui1pOQJ6U6Nj9dvkTQ2lhQJFaEf+wAkudLbzKxnD+314+nxIXt+xAHDzIZbry66/Xh6fAA7GGzGAcPMhluvLrr9uvuv18Fgt9atS46FREjdby/V84Ah6XxJmyTdkJm3SNLlkm5N/9+71+kws1msFxfdYbr7//rXdwSIy/RadOIJiOBYJphmjEPh0CJ2048cxgXAkTXz/gr4VkQcCHwrfW9m1p5eNnudnBzcrkauvXZHgLhaL0KvPRIRiGAFlwGwkknOZTXjNOhKpAM9DxgRcSVwX83so4F16et1wOt7nQ4zm2U6bfaaJ8gM2oBMDzxQVcS04Xmv51gmEMFhXF33I+ctWcPutNGFSB4R0fMJGAduyLx/IPNa2fd1PrsamAKmli1bFmZmERExNhaRXM6rp7Gxxp+ZmIgYGalef2Qkmd/ttouW7vMqXhgrmYgtVKd7CyOxkokds04/vebz0o51D4WIAq7lfXkOQ9I4cGlEPCd9/0BE7JVZfn9EtKzH8HMYZrZDJ89f5H3uoVfPdkDjoq60cvo2DuDXua3qI+sZr1+01Ox5jcx3XQ5MRXRd+11WK6m7Je0DkP6/qaR0mNmw6qTZa95nK3rVpLZOUVccdxzHanJHHURtsABYRgfPhOTpOqRNZQWMrwAnpK9PAP61pHSY2bDqpNlr3kDQiya1r341HHfcTt2YCPgsJ7CNOaxnnJXM1JNUypTmjHUQwDrp2baVIsq1mk3ARcBdwBPAncBJwChJ66hbgW8Ci/Js69BDD+2uTNDMZpeJiaReQUr+r62LqLd+njqMTrZd67LLIiC2w45dbUNRt26kVXraSXedtC+G26OI63kRG+nX5IBhZl3rNhA08vDDOy7mjWLBesZaB4xGlet5010nuDwftkUB12B3Pmhm+Q3yswllSCuqRb7r6CPnTjLythwj5nVTuV6nYn/YK73NbFhUnluQ4Pjji382YZgGP0qfhUj/21FR3chtt1VnHUbeXPP0+Ny59T/YTeV6kZ0j1nDAMLPGTjttJkjAzk1Nux13YtAekKvVZoD45jerA8QBB9RZKduNybp1xVeuF9k5Yg0HDDOrb3ISzjmn/vMIWd3c0fZq8KNOvfrVdQPESiZZz/hOLZnOOisNDhOTxNg4rzyizVxSL/qrqtPCK6DLh0cqGxqAyuy8kyu9zeroVSVuo6edi3z6WQ1aDUnFfIdWrriiaSU1RN2nrKtaKOVpwdSr36gRt5JywLA29OMPtN8XgUZpaKe5ZTsaXcyL3FejoDQ62ptj+8QTLQNE7dSym5BWy3v5G+UETIUDhlkd/fgDHYCLQEQU2+dRbQAcHW1+JS3iQl7vOM6fHzFvXvW8BQs631e7AaJWs8DZ7BhVckkD0C+VA4ZZI/34Ax2Ai0BEFFek0+jCvWDBzts99dRiv0NtoNpjj8YX5zwgVnBp5wGiVt6iuUbnQtnFblFcwHClt80+/RiLuR/7yKOoPo/qVT4/8QQsXFhdIXvhhXD22Z2lNa8tW+rPv/fe+vMlTtfpVRXVX+N3G26+9srdUid9MmVbOvVjqNd+KSLq9GtyDsNy2ZVyGBMTO+cCOim+KesuuF7OplV24A1viC9zdHE5iLzpbCenUVvh3az4sg91YbhIyqyBXakOY2IiKTqqLUpqNx1lBcA2LsKbGO1fgOgmve1069HL8yizz4PhsXDAMGtgV2klVdSFvqwA2CBns73O+02MVg0Y1JcAUatVjqjdY9arQF2TzqIGUCo9CLQzOWCY1SiyKKmMANjggrmJ0djE6E6BYwsl5eKyx+XUU2fej44mU6fHrFdFgTXHdahG3CuKOx80q5F3BLlBI+14gvpcVleNPf0II5zMWs5kTfujzBWt0nVJtkHAyEj3T2NX9Or3qxkx0J0PmllvBvrphWx3G5rp3fUiVnEya9nAGNsRGxhj6z+s5XOxinENQEu0Xndd0u7vl7ejxl61wCoim9KvyUVSVphBqH8oyiB+l3e9K3eDou9+t8E2BqElWj9aj3UxzkXTwZ9ch+GAYQUYlBZOs8ndd+cOEJ/8ZM5tDsLv1Iug1WmAbzctbiXlgGEFGIQ7117qU46jVWBYyUSsZywZlrTTdJSdeyo6aHWzvS5yO/g5DLMODUBXDT3Tw7vyvDkI6G06+q7IoNXNzUoXny0qYLiVlO16hrVlUR4Ffje10aZmp8vIbD7G3ahpvbRDniFZu2ixJemaiFjeQYqrlNpKStJfSLpR0g2SLpL0pDLTM+sM09CX/TQsLYs60UUfV1WtmFoEi9rb3CLTMat1069ULwZbalcR2ZROJmBfYD3w5PT9xcCJzT7jIqk2zKYigV4ou2y8V9ootjhwn4fyFzH1MB2la+dc6Pa8KenvkmGvw0gDxh3AImAecCnw6mafccBowzD9wVpxmlyQ3vpnW3sXINpIx0Dpoqlqx9+phJuVoQ8YyXfgbcAWYDMw2WCd1cAUMLVs2bJCD+KsNpsrdq259IK0XYr1jNXtf6nwANEkHQOdi2vnxmqIb8KKChilVXpL2hv4IvCHwAPAF4BLImKi0Wdc6d0GVzrucm66CQ46KN+68cRWmDevtwkaBu1UQndTYV2y2VDp/SpgfURsjogngC8BLykxPbNLo4rdFStcEd5PPWx48OCD1ZXUzYJF/Mu/Vt0WO1ik2qmEnk0DIXWozICxEThM0ogkAa8Ebi4xPbNLvRYVJ5wA69YlOY+I5P/Vqx00eqXSDLKg4x1RHSD22qvJuqOLqwPE0Ud39h1mu3ZazM3m1nV5FVGu1ekEfBC4BbgBuBDYrdn6rvTu0hCXwQ6lTo93puw/bx3E48wbjjqDespOdz9bSZWE2VDp3e7kgNElV4T3V6PjXQkadS42K5lIxnzIrLuFkZ2Cxi94WvUHh6VVUq1hTfeQKSpgtCySknS5pOf2MpdjfeIy2M51UhfR7LimxVPHarKqmOlM1lSNDQGwO49yPidVRZx94q7q7bXbDfegPNTZ6+7DrVitIgrwfOAK4J+AfYqIUp1OzmF0adjv5soqDuj0uLUazhNiPWNVs7bRYS6wndzjIJ0HzvX2Bf0ukgLeAFwHfID06ex+Tw4YBRjSMthSL3Jd1P1UemytHWq0Mm1DARFv4AsRZ57Z+b6G9XmCQUrLLNbXgAEIeA5wCnAPcCdwfBEJaGdywJjFWgWyMi8sbdwFN8pMrCdn+hsFxuw40vWOTzsBdZDu6gcptzOL9S1gAN8DfgFcDnwYOAr4deDvgbVFJCLv5IAxS01MRCxYUH3RWLCg+qJR5kWuSbBqUtpUNa1kYueZzbqgyAaHU0/Nd1HNm3sctLv6Yc31DpF+BoyDIHkivM6ym4tIRN7JAWOWGh2tfwEbHZ1Zp8yLXJ274Hotl2qn5J+a7fRjpLUOvo/v6me3vtdh1P0wHFBEIvJODhizVNOrbqqki9zChbEjh1AZPa7RsxE7BYii9CJ35bv6XUpRAaOrJ70j4vZuPm+WW5/GAjj55OqnqR9+GFYyyZmsYRkb2cgyLmUFZ7KGQMQ+TycmJokguYz3Qi+aQ69alfQptn178n8/x1SwoVXqAEpmAIyO5pvfg4vchRdWB4jzzqtevpJJzmU140wzh2Ccad7Cpxkn7djxrrt6371Kkf2CDcrzFzacisim9GtykdQsNTERMX9+dXHL/Pk9KSa54YbmJWA7FTHlnRrVJxRV9NNpRXjtNlx3sUtiEOow+j05YMxiPSpTf/DBDgJEVrPuPVrVJ3R7gW52TDqpCB+01lHWN0UFjNLGw+iEx8OwViKS0pZc66KZDzXSaFyRWvXGGelmTJJKT7fZbjNGRmbqbToZm2GIx3Ow7syG8TCslsuXq+U8Htk6iGbB4nHmJxXVW7cl183KPXYz9eoPajXq4nrjxvrrN5qf1aqPpU4qwt2XmHWriGxKv6ZZXSTl8uVqTY5H3iKmX/LU5MVll3Wfltr6g14/INeqKW0n54vPsV0WrsOYZVy+XK3B8ajtrC87XcULkxfj42WnPtFpNx8R+c6HTup9/PzFLskBY9C1+4c5SP37lAwa99pa6awPIs5h9cyyohR9Qe2mmw/nBqwgDhiDrJM/9kHNYfThjrTe127YWV/RASKrHxfpdn5n5wasIA4Yg6yTi/8g3lH2KE156h9WMhGP8OT+Ho9+BG3nJK0ERQUMt5LqhU5ax/Sp64uWsi2TTjihkNHQsq2YpObrBiIQn3vwdYxMnJscB4C5c2f23avWY920asrLLZVsiDlg9EKei0K9JqPddH1RRJPcStv/6enkvnfbtvrrtbiAjo62HyDivM8Qwcw991e/mgSH6elkI5W0pEOb9iRo9ONi3qibj3rNcs0GTRHZlE4nYC/gEuAW4Gbgxc3WH5oiqVZFOUUX9RS1vUZFMi2KaFauzPex5GxLX+y1V+My+hxDm7Ys3uuk7L9fxYKum7A+YzbUYQDrgDenrxcAezVbf2gCRkTx3To0U9T28nSDMTISlx2f/1mIuhXVExP1x8CoXJzzBK5GZf697I7DbEgVFTBK6xpE0p7AtSRjauRKxKzpGqRRFw2QlNlv3JgUg5xxRr5iqaK6fGjQlcX2OXNh+3Y2soz3cQYX0ThNO7rbgPppqtflRVbl+7c6JRp1r9FNdxxms9Rs6Bpkf2Az8E+SfizpPEm7164kabWkKUlTmzdv7n8qe6FRmbg0U3/QTll9o+0tWtReuuqUrz/CCMdtX8dctrM/G3YKFjvqIFByjc/e29dTr8uLrEqwbKZZmX8/Kq7NdlFlBox5wPOBT0fE84BHgL+qXSki1kbE8ohYvmTJkn6nsTfqVXxKO19k87ZIOuMMWLBg5/kPPZQr4DzySFpBfdwqjn10LRsYYztiA2OczNqqIFEVIDbe0TpA1Gp14a7krOodH2jdesytkMx6psyAcSdwZ0Rcnb6/hCSAzH71mtA2uuDmuTNetQoWLtx5/hNP1A04ETMtmI7VJJv3GGcbc1jPOAD7s2FHjuJzHDcTIP73+6sDxNKlbXzpVLMLdyXnUO/4XHhhss9WrcfcCsmsd4qoCOl0Av4DeGb6+nTgb5qtP1SV3u3qtOK6VSVxWjlcb9FKJmIL1RXE23eqtS5YoxZQo6PFVTC74tqsCrPkwb23ApOSfgIcApxZcnrKs2JFe/Oh+rmJBjbEsobPQpzJGnanuj5BMJPj6UWDiHq5h4kJuOee4h5S9HjVZj3hAZQGRSete1oM7vMIIzvVQVzHwRzM9c3T4gF1zGaV2dBKyrLabN0jwfbpxvUbW5nLyazl9/lyVUX1wduunck9VLrdqOUKYjOrwwFjULRo3VOvP6aNNL6wz2Mbn+M43vjD91TXFmSHpHMFsZm1wQFjUDR4BuLY6TMa1kG8jzNoWKBYqYd4wQsa73NQOjw0s6Ewr+wEWOKV56/i1x5NKqKXsbHhU9V7cT/3U/NAXu0zHO3kElatcoAws1ycwyjJe99bXcT07W/DRayqegaiEiyydRD3s2jnFrIXXuhcgpn1nANGn3zhC9UB4mMfa7xu1dPUaOcAUcvNSHuniG7jzWYJB4weuf766gDxpjc1XnenALF0afJsQq+ehbB8ascH6eVYHGZDwAGjW+kdaMyZwwaNc6wmkeDggxt/ZKcA8YY3VI80dMcdvjANgnodJXYw4qDZbOGA0aFf/Srph+mR45I7UEUwzjTnspqVVF/odwoQ78k0dZ2YgC99qfOOB6133POtWRUHjJy2bq0uYhoZqd+1xu48Wt1hH4LnPre6DiJbgbFmTXcdD1qiUV1DN3UQ7vnWrIqb1TZQ+4xbPctockEvortvX5jyqR2UqVLX8L3vwbp1O8+HfA0Dzjhj58Ge/GCj7cJmXw6jizvKRYtmchDNgsU25hCIOY0emxsdzZ/eZoMp+cKUT6O6hrVru6uD8IONZlWGK2Dcd1/zYNBmq5bly6uLme6/v/5uH2d+VRHTHCKpxGgnMDTSaLCgU07xhSmvRrm0bdvaW78eN1k222G4Asb0dPNg0KJVyxvfWB0grrmm/m62sHtVgJjP1plxpivTk56UBLB6Gs2vp9FgQWefnX8bu7pGubS5c9tb38yaGq6AUdvldm3xQoM7x5ieZrvm8LdfHN+pBRPAZhZXBYjdeRS+8Y3Wo8sVVSnqu9juNOpEcfVqd65oVqDhChj1VILE5CTbVf/rCJjDTLPXTYxWBYjF3Asf+lB1gDjiiNb7bqe3Vz8x3DuN6hrOPtt1EGZFKmIe8AbIAAAKPElEQVTYvn5Nh9YZZ/TehWN1hxptOb3kJfnGNmw13Gee4UBPPTVZnt3/yIiHDjWzvqCgIVqHa8S9uXNjKlMsVRlR7kzWME7jked2kndEudrmmpDkINq5S52chOOPr9/MttloemZmBdk1R9wbG+PePcbYjtjAGCezlqXcyVg7wQLy1zEU0TWEH8wzs1liuB7cW7SIRScezYMfOItxpvkcxzVff3Q0af7a6YNXRXQN4QfzzGyWKD2HIWmupB9LujTX+i8+jL3mPwrveAds2jTTH1O9yudPfrK7Ss8iWkH5wTwzmyVKDxjA24Cbc699xBHw+ONw1lmwZEkyr9kTud00WS1izGs/mNeaW5CZDYcias47nYD9gG8BrwAubbX+oYceWlSjgfzytILqxzZmq4mJpMWYW5CZ9QyzoZWUpEuAjwILgb+MiKPqrLMaWA2wbNmyQ6en26zgtsE2Pp48tV/LLcjMCjP0raQkHQVsiogGHXQkImJtRCyPiOVLKkVQNnt4zAmzoVFmHcZLgd+TtAH4PPAKSRMlpsfK4DEnzIZGaQEjIt4bEftFxDhwDPDtiGjRTtZmnSIaFphZXwxCKynblVsJecwJs6ExXF2DLF8eU1NTZSejWEV0P2Jm1sTQV3pbqojuR8zM+sABo2xuJWRmQ8IBo2xuJWRmQ8IBo2xuJWRmQ8IBo2xuJWRmQ2K4ujefrSqdJJqZDTDnMMzMLBcHjDx25QfrzMxSLpJqpfbBuunp5D24GMnMdinOYbTiB+vMzAAHjNb8YJ2ZGeCA0ZofrDMzAxwwWvODdWZmgANGa36wzswMcCupfPxgnZmZcxhmZpaPA4aZmeXigGFmZrk4YJiZWS4OGGZmlktpAUPSUklXSLpJ0o2S3lZWWszMrLUym9VuBd4ZET+StBC4RtLlEXFTiWkyM7MGSsthRMRdEfGj9PXDwM3AvmWlx8zMmhuIOgxJ48DzgKvrLFstaUrS1ObNm/udNDMzS5UeMCTtAXwReHtEPFS7PCLWRsTyiFi+ZMmS/ifQzMyAkgOGpPkkwWIyIr5UZlrMzKy5MltJCfgMcHNEfLysdJiZWT5l5jBeChwPvELStem0osT0mJlZE6U1q42I7wIqa/9mZtae0iu9zcxsODhgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpaLA4aZmeXigGFmZrk4YJiZWS4OGGZmlosDhpmZ5eKAYWZmuThgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpaLA4aZmeVSasCQdKSkn0r6maS/KjMtZmbWXGkBQ9Jc4B+A1wLPBlZKenZZ6TEzs+bKzGG8EPhZRNweEY8DnweOLjE9ZmbWxLwS970vcEfm/Z3Ai2pXkrQaWJ2+fUzSDX1IW7cWA/eUnYgcnM7iDEMaweks2rCk85lFbKTMgJFLRKwF1gJImoqI5SUnqSWns1jDkM5hSCM4nUUbpnQWsZ0yi6R+DizNvN8vnWdmZgOozIDxn8CBkvaXtAA4BvhKiekxM7MmSiuSioitkt4C/BswFzg/Im5s8bG1vU9ZIZzOYg1DOochjeB0Fm2XSqcioojtmJnZLOcnvc3MLBcHDDMzy2VgAkarbkIk7Sbpn9PlV0sazyx7bzr/p5JeU2Ia3yHpJkk/kfQtSWOZZdskXZtOPa3cz5HOEyVtzqTnzZllJ0i6NZ1OKDmdn8ik8b8kPZBZ1pfjKel8SZsaPf+jxP9Nv8NPJD0/s6yfx7JVOlel6bte0vclPTezbEM6/9qiml92kc7DJT2Y+W3fn1nWt66EcqTzXZk03pCej4vSZX05npKWSroivebcKOltddYp9vyMiNInkkrv24ADgAXAdcCza9Y5DTgnfX0M8M/p62en6+8G7J9uZ25JafwdYCR9fWoljen7LQN0LE8EPlXns4uA29P/905f711WOmvWfytJw4h+H8/fBp4P3NBg+QrgMkDAYcDV/T6WOdP5ksr+SbrjuTqzbAOweECO5+HApd2eL71OZ826rwO+3e/jCewDPD99vRD4rzp/64Wen4OSw8jTTcjRwLr09SXAKyUpnf/5iHgsItYDP0u31/c0RsQVEfFo+vYqkmdL+q2bLldeA1weEfdFxP3A5cCRA5LOlcBFPUpLQxFxJXBfk1WOBj4biauAvSTtQ3+PZct0RsT303RAeedmnuPZSF+7EmoznWWdm3dFxI/S1w8DN5P0oJFV6Pk5KAGjXjchtV98xzoRsRV4EBjN+dl+pTHrJJLIXvEkSVOSrpL0+h6kryJvOt+QZlEvkVR5gLJfx7KtfaVFe/sD387M7tfxbKXR9+jnsWxX7bkZwDckXaOkK56yvVjSdZIuk3RQOm8gj6ekEZIL7Rczs/t+PJUU0T8PuLpmUaHn58B3DTKMJB0HLAdenpk9FhE/l3QA8G1J10fEbeWkkK8CF0XEY5L+lCTn9oqS0pLHMcAlEbEtM2+QjufQkPQ7JAHjZZnZL0uP5VOByyXdkt5hl+FHJL/tFkkrgH8BDiwpLXm8DvheRGRzI309npL2IAlYb4+Ih3q1HxicHEaebkJ2rCNpHrAncG/Oz/YrjUh6FbAG+L2IeKwyPyJ+nv5/O/AdkruBXmiZzoi4N5O284BD8362n+nMOIaaLH8fj2crjb7HwHV9I+lgkt/76Ii4tzI/cyw3AV+mN0W6uUTEQxGxJX39NWC+pMUM4PFMNTs3e348Jc0nCRaTEfGlOqsUe372umImZ+XNPJJKl/2ZqdA6qGadP6O60vvi9PVBVFd6305vKr3zpPF5JBVzB9bM3xvYLX29GLiVHlXY5UznPpnX/xO4KmYqwtan6d07fb2orHSm6z2LpBJRZRzPdB/jNK6k/V2qKxV/2O9jmTOdy0jq915SM393YGHm9feBI0tM59MqvzXJhXZjemxznS/9Sme6fE+Seo7dyzie6XH5LPB3TdYp9Pzs2cHu4MuvIKnlvw1Yk877EMmdOsCTgC+kJ/0PgQMyn12Tfu6nwGtLTOM3gbuBa9PpK+n8lwDXpyf59cBJJR/LjwI3pum5AnhW5rN/kh7jnwF/XGY60/enAx+r+VzfjifJ3eNdwBMk5bwnAacAp6TLRTIQ2G1pWpaXdCxbpfM84P7MuTmVzj8gPY7XpefEmpLT+ZbMuXkVmQBX73wpK53pOieSNLjJfq5vx5OkWDGAn2R+1xW9PD/dNYiZmeUyKHUYZmY24BwwzMwsFwcMMzPLxQHDzMxyccAwM7NcHDDMzCwXBwwzM8vFAcOsC+l4BEekrz8i6e/LTpNZr7jzQbPufAD4UNrR3POA3ys5PWY94ye9zbok6d+BPYDDIxmXwGxWcpGUWRck/RbJyGePO1jYbOeAYdahdOSySZJRzbZI6tmIemaDwAHDrAPpSGtfAt4ZETcDHyapzzCbtVyHYWZmuTiHYWZmuThgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpbL/wfPdkIFtJMFQAAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"from random import random, seed\n",
"import numpy as np\n",
@@ -1102,8 +1048,10 @@
},
{
"cell_type": "code",
- "execution_count": 5,
- "metadata": {},
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import numpy as np \n",
@@ -1164,17 +1112,11 @@
},
{
"cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "gamma_j after 500 epochs: 9.97108e-05\n"
- ]
- }
- ],
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import numpy as np \n",
"\n",
@@ -1211,37 +1153,11 @@
},
{
"cell_type": "code",
- "execution_count": 27,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Own inversion\n",
- "[[4.13137049]\n",
- " [2.85144795]]\n",
- "sgdreg from scikit\n",
- "[4.16307568] [2.89603874]\n",
- "theta frm own gd\n",
- "[[4.13137049]\n",
- " [2.85144795]]\n",
- "theta from own sdg\n",
- "[[4.12146611]\n",
- " [2.87377643]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcHHWd//HXZ5LJMSQkZBKSEJhJAiEcbiAwCwrIBFHAyOmyu2LkdhFcWXU9fiKKKKKuroKiPjACIiSCLt7uegAmE64EhiPhSgiBHIQjk5uQe+bz+6NqQqfTR3VPdVd3z/v5ePRjuqurqz9V3VOf/h71/Zq7IyIikk9d0gGIiEh1UMIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMKRmmdkUM3sl6TjiZmZuZgclHYf0PkoYUlZmttTMtpjZJjN73cxuN7NBScclIvkpYUgSznD3QcCRwGTgqoTj6ZXMrG/SMUh1UcKQxLj768BfCRIHAGb2ATN70sw2mtkKM7s25bmxYXXMhWa23MxWm9nVKc8PDEss68zsOeAfU9/PzA41s9lmtt7MnjWzM1Oeu93Mfmxmfw5LPw+Z2SgzuzHc3kIzm5xtX8K4LjezxeH2f2RmFj53rZnNyLAffcPHs83s62b2cPjefzSzRjObGR6Hx8xsbNpbTjWzl8Jj8B0zq0vZ/iVm9nwY91/NrDktzn83s8XAYgvcYGarwvd62szeke+zk95JCUMSY2b7A+8HXkxZ/BZwATAU+ABwhZmdnfbSE4CJwMnANWZ2aLj8K8CB4e1U4MKU96oH/gj8DdgXuBKYaWYTU7b7L8CXgOHANuAR4Inw8T3A9/Ls0ukESWpSuK1T86yf6kPA+cCYMP5HgJ8Bw4Dnw31LdQ7QAhwFnAVcEu7nWcAXgQ8CI4AHgLvSXns2cCxwGHAKcCJwMDAkjHtNAXFLL6KEIUn4nZm9CawAVpFyMnT32e7+tLt3ufsCgpNda9rrv+ruW9x9PjAfOCJc/i/A9e6+1t1XAD9Iec07gUHAt9x9u7v/HfgTcF7KOr9198fdfSvwW2Cru9/h7p3ALwmqz3L5lruvd/flwCxSSk4R/Mzdl7j7BuDPwBJ3v8/ddwL/k+G9/yvcz+XAjSn7cTnwTXd/PnztN4AjU0sZ4fNr3X0LsAMYDBwCWPi61wqIW3oRJQxJwtnuPhiYQnCiGt79hJkda2azzKzDzDYQnACHp73+9ZT7mwkSAcB+BEmo27KU+/sBK9y9K+35MSmP30i5vyXD43yN89niiqLQ907fz/3C+83A98NqsfXAWsDYfT93vTZMnD8EfgSsMrPpZrZ3AXFLL6KEIYlx9zbgduC/Uxb/AvgDcIC7DwFuJjjhRfEacEDK46aU+68CB6TW9YfPryww7GK8BTSkPB4VwzbT9/PV8P4K4GPuPjTlNtDdH05Zf7chqt39B+5+NEEV1cHA52KIT2qQEoYk7UbgfWbWXa00GFjr7lvN7BjgwwVs61fAVWa2T9g+cmXKc/MIfvV/3szqzWwKcAZwd4/3IL+ngBPNrMnMhhBPr7DPhft5APBJgiozCBLsVWZ2OICZDTGzf862ETP7x7BUV0+Q2LYCXdnWl95NCUMS5e4dwB3ANeGijwNfC9s4riFIAlF9laB65mWCxu07U95nO0GCeD+wGvgxcIG7L+zpPuTj7vcSnNAXAI8TtJ301O/DbT0F/C9wa/hevwX+C7jbzDYCzxDsczZ7Az8F1hEcuzXAd2KIT2qQaQIlERGJQiUMERGJRAlDREQiUcIQEZFIlDBERCSSqhp8bPjw4T527NikwxARqSqPP/74ancf0dPtVFXCGDt2LO3t7UmHISJSVcxsWf618lOVlIiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpGUPGGY2W1mtsrMnsnw3GfMzM1seKnjEBGRnilHCeN24LT0heFcxKcAy8sQg4iI9FDJE4a7zwHWZnjqBuDzgOaIFRGpAom0YZjZWcBKd58fYd3LzKzdzNo7OjrKEJ2IiGRS9oRhZg3AF4Froqzv7tPdvcXdW0aM6PFw7iIiUqQkShgHAuOA+Wa2FNgfeMLMRiUQi4iIRFT2CZTc/Wlg3+7HYdJocffV5Y5FRESiK0e32ruAR4CJZvaKmV1a6vcUEZH4lbyE4e7n5Xl+bKljEBGRntOV3iIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkJU8YZnabma0ys2dSln3HzBaa2QIz+62ZDS11HCIi0jPlKGHcDpyWtuxe4B3uPgl4AbiqDHGIiEgPlDxhuPscYG3asr+5+87w4Vxg/1LHISIiPVMJbRiXAH/O9qSZXWZm7WbW3tHRUcawREQkVaIJw8yuBnYCM7Ot4+7T3b3F3VtGjBhRvuBERGQ3fZN6YzO7CDgdONndPak4REQkmkQShpmdBnweaHX3zUnEICIihSlHt9q7gEeAiWb2ipldCvwQGAzca2ZPmdnNpY5DRER6puQlDHc/L8PiW0v9viIiEq9K6CUlIiJVQAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhCR6jZzJowdC3V1wd+ZWUcaSmZ7NSSxoUFERHps5ky47DLYHA4YsWxZ8Bhg2rTkt1djrJqGcWppafH29vakwxCRSjF2bHBST9fcDEuXJr+9CmFmj7t7S0+3oyopEaley5cXtrzc26sxShgiUr2amgpbXu7t1RglDBGpXtdfDw0Nuy9raAiWV8L2aowShohUr2nTYPr0oI3BLPg7fXrxDdRxb6/GqNFbRKTGqdFbRETKSglDREQiUcIQEZFIlDBERCQSJQwRkXKr0vGqNJaUiEg5VfF4VSUvYZjZbWa2ysyeSVk2zMzuNbPF4d99Sh2HiJRIlf5ajlUhx+Dqq99OFt02bw6WV7hyVEndDpyWtuwLwP3uPgG4P3wsItWm+9fysmXg/vav5WpOGoUmwEKPQRWPV1XyhOHuc4C1aYvPAn4e3v85cHap4xCREqjiX8sZFZMAsx2DCy/MnHSyjUtVV1fxiTapRu+R7v5aeP91YGS2Fc3sMjNrN7P2jo6O8kQnUi2Srg6qpF/LcRyLYhJgtn3t7MycdDKNV9W9fqWXzty95DdgLPBMyuP1ac+vi7Kdo48+2kUkNGOGe0ODe3BaCm4NDcHycmlu3v39u2/NzeWLwT2+Y2GWeX/Msr8m2zHIdUxmzHDv06ekx277W9v94Z8s8G+eOsuBdo/hXJ5UCeMNMxsNEP5dlVAcItWrEqqDKmV017iORTHDm0+dGgxUmE9qSWTaNOjqyr9eAaWmbRu38eCPF/D1987mfY2PM3Sv7Rz3sX/gqr9OyR9bVHFknXw39ixhfAf4Qnj/C8C3o2xHJQyRFMX8Gi6FGTOCX8Vmwd9ylnC6xXUsCi2pZFo/SgnDPXvJpLExUixb1m3xWTc86ddOmeUnDX3CB7B512qTBiz0KyfN9ns++7Cveq4jthJGOZLFXcBrwA7gFeBSoJGgd9Ri4D5gWJRtKWGIpKiU6qBKEOexKCQBZnvf9ASWKenMmOHer9+er62vfzuGDNte13+knzjkSe/PluCt6PTJA5/zT02e7b+7aq6veXHtHmHGlTA0vLlItUq/AAyC6qDeOH9DUseiri44jWfS3BxULzU1BVV0meIYPhzWrMn4Wl++HMuw7S6MYxueYcphq2h9fwMnXDqRoc1DcoYZ1/DmutJbpFp1n4Cuvjr/ianWJXUsmpqCXlDpmpth6dL8r1+bfsVBoGvZclZwAM1k6IE1Zn8ee+Uw4LCCQo2DxpISqWbTpgUnpq6u4G8tJ4t8DcBRjkXc3ZB70Oi/ftkGNu+d+YqCVTaKeRPOp7N+wO5PmFG3ckVyV9THUa9VrpvaMKTiVUIDcC2Ko9tspm3U1weNzD35vCJ+5mteXOu/u2quf2rybJ888Dk3Ov08Zvgmdo+pa+DAt7eR2pYRpV0kC9SGIVJh1KZQOmPH9qzqJ9c2UsX4eXU8v5o5ty2m7W/baXthFE9vnYBTxwC28K6hC2k9cgOtZw3luIYn6feNr+auSuvh/sfVhqGEIRKXOE5qSZs5szLbRLI1Lptlv54h6jbSFfl5vfFMB223Lqbtvh20Ld6PZ7dNAKCBtzhu2EJaJ79J69n7cMwFh9B/7/6FbbyH+69Gb5FKU0nDZBSjkofdzta4nOuCuqjbSBfx83r1iddp+9lLtN2/k7YlY1i4/UBgBIN4k+MbFzGtdTZT/qmRoz88kX6Djo4eZyGxF7L/cYijXqtcN7VhSEWr9usiosafRDtNqdowCvi8ls9d6Xde/qB/dGKbT6h/adfqe7Pep4541L89dZbPu+0Z37FlRzz7nC/29P3P8blQLRfuxXlTwpCKVgljO/VElKulk9zHOBJV6jYaG/e8cC5lX15+YIXf/tEH/OIJc3xc32W7Vhlq6/zMUXP9u2fO8vY7n/Od23bGuZfRYk/f/zyfixKGSCWq5l5SUUoY1V6KShd+Xl1mvn3fMT7r3V/288c/4E19VuzatWG2xs8Z/YjfeM5sf/LuheVLEDni3eP7ledzUcIQkXhFKT1kK4VAZSfHtBNt150zfNFfXvLp57f5h5sf9DF1r+7ajRG2ys8d87DfdO5sX3DPIu/c0Zl09IFcn0+uz8XMJ8E2j+EcrF5SIvK2fL2kcnVNrdQuxDNn4pddhqV0d97MQD7KT7mLaYyqe4PWMUtoPX4nreftx6GnH4jVRRh9ttxy9cKDnA36LUC7e893Ko6sU66bShgiCcvXcNyTqqkYq/M6d3T6gnsW+U3nzvZVfUZmjHVjw0hf9JeXvKuzq/iY45Rv/3O1MeX5XI4G9xjOwYkngUJuShjSa1VS28gVV2RPGMUOrd7DxvTOHZ3+5N0L/cZzZvs5ox/xRlu9azOdVMgw8LlE2f987Uep3xElDCUM6aUqrfdVrhnmii1hFNiYvnPbTm+/8zn/7pmz/MxRc32ordv1knF9l/lFB83x2z/6gL/8wIrqaKiPEmMh34O07SlhiPQWlXbCK0XDd54uvTu27PB5tz3j3546y6eOeNT3Zv2uVQ6qf9k/OrHN77z8QV8+d2WwvQK6z1aEqBNARS1ppiWXuBKGrvQWqXSVdgV5tquOGxuLb/DOss0N/fflX4e389CaiWzicAAO6beE8w6dT+vJfWm9eDz7HTWWYFLPUPoV62vWQH19EN/atZU15Em3qFdyT5sWPe6BA3cdg07Y2cMIAQ0NIlL5KmVYiG7XX595kMXvf7/oTe740lep+/jl9Nmxddeyt2jgiq3f5RUfwgXveILW99Zz4sUHMWrSgcCB2TeWaX7vHTtg0CBYvbroGEsq2zEtZm70DINg1sU1lUUcxZRy3VQlJTWlyOqFiqhS6WEjfKb5qM9jhr9Ms3divqZ+pM+d+hVf9VxH4bFVylznhYqrY0OGKkxVSYlUs0IG+quBmfU2r97MIz9bSNsfN9K2YCjzNhzCNo7E6OLIgYu4fPKjtJ56EIMveYK6CcMYBhxb7JtVWoksqkKqm3IpZVVlHFmnXDeVMKRmVFpDdiEilHjefO1N/+s32v2Lx83y4wfP93q2ObjXsdNbGp71zxw9y//45Xm+bun6ROKLtI1K6cZcqBKWMBJNAMCngWeBZ4C7gAG51lfCkKr+R06V7yKsJEaDjfqeWZLd5iGj/P8dO8uP3etp78t2B/dp3OGv2mjvwnzzkFH+1g9+Wvp9KXR/Mr220qoAC5Eh/qOg06s5YQBjgJeBgeHjXwEX5XqNEkYvV+3/yKmylTAaG8u/j4Ue1yzJrgv8I9zuxw2e7188bpbP/+frgulGM2231EmxJ9uv5tJft7T9Hw4veQ0kjBXAMILeWn8CTsn1GiWMXq4W/pG7ZTtJNzaWfx8jHNfU+ahX2ujM65M2H3VSSbGnPyyqtdE8B2phtFrgk8AmoAOYmWWdy4B2oL2pqSnWgyhVptb+kTP9Ck5iH7OWGMyvnDTbJw1Y6Eang/sANvt1A7/u2+r6Z00auxJNrgv8SpkUe/rDopZ+mITiShjx9M0tgpntA5wFjAP2A/Yys4+kr+fu0929xd1bRowYUe4wpZJk6+VS6b1fspk2LZg7uqsr+DttWjL7mGXby2ji1gUt7Nuwia+dPIc5N81n/YY6vrT5avrdcWv27XX30ik05rh69+S70HHmzGDk17q64O/Mmbuvd/31wTUQqYq9JqLWxJF1irkB/wzcmvL4AuDHuV6jKqlerpbaMLLJNupoY2Os+7ny8df8F594yD92aJt/ps/3fBO7v+e2uv6+6JJv+rY3t2XfSJTB8JKodssVV9TvUK10rghR7VVSBN2snwUaAAN+DlyZ6zVKGFJr/8juvuc+XXFF5pNq1OSY4Rilzkd9UP3LuzbZPR/1HyZd7VsbR3tXagz5jnOx80yXOvHn2n4NVjdFUfUJI9gHvgosJOhWeyfQP9f6ShiSmFIlqrh/hWfY3lsM9POY4RDMR33GyDzzURdyQi/2uCTVS6oW2sGKOHY1kTAKvSlhSGwK+acr5S/iXEOFZ7ulxdzV2eUv3r/Ub71ojnf0zTxZ0IaBI6PPR13Lv8Krfd+K/C6WLWEA9wJHxPFmPb0pYUgsCv2nK+VJptCeRCm37X36+w8br9ltPupYJguqhV/h2VR7O1iR38W4EkaUXlL/D7jRzH5mZqNjaWkXiVu+ni+pMo1munlzsDyTUg4vnq0nUWPjnj110tR3buPMNbdxwpiX+fGH5vDs71/Emg4o7H0KWTfXNgo5/kmaNi2Yd7y5GcyCv5U4D3k2SQ91HzWzAP8EzAe+Qnh1drlvKmHUiLjrr2O6UjnrL+hSljAyxN7V0ODL/uO//S/HXO2r+oz0Tsy7spQyujJNsBPHOEqFbKPaf7VXk4RLGFGThQHvAC4HVgOvAOfHEUAhNyWMGlCKk0uh/0SFrl/iE2LnHXf6tuH7eRfmb9SN8n/j5l1v09RnhZ8//gHfuFfmtomMMceRkGMYW6pq2gWqSRW0YTwEvErQlnEdcDpwEHATMD2OIKLelDBqQClOLoWWGIr5p4uxVBRlPuqfXRrOR92TmMulXG0etdiluhiV3EsKOBywLM89H0cQUW9KGDWgFCeXQpJQal/8Pn12/1uik1C++agvPXjO7vNRZ1OpJ8xylDAqOWFWgbJWSWV9MYyPI4ioNyWMGlCKk0shV+9muoo65pPQ9re2+8M/WeDfOGWWn9r4mA9i467NT+y3xD92aJv/4hMP+SuPvZp9fyoxMWRTjpO5qr16pCISRrlvShg1oFQnlygn2ajXPBR4Etq6Yas/8KP5ft3Js/y9w9q9gU27NnV4/xf84++Y7b/81EP+2vw3ou1HNf6SLnWSq+WuvmUQV8KwYFvVoaWlxdvb25MOQ3pq5sziphst9nXd6uqC00w+ZsGAgFlsXb+VubcvpO3362l7agiPrD+ErQwEYNKARbQe/Dqtp/TjxEsmMOLQ4dHjg6BLaqbpRZubgwEKeysdlx4xs8fdvaXH21HCkKqQPgc2BNcpFNKHPttJJ13aSSjzfNQDds1H3XrIKlpPHcC7L5lA44RhBe3WHrIltTxJrObF8fn3YnEljMSrmQq5qUqqQiRRxx5HHfaMGe719bmroxoafMuPbs07H/UfvjTP1760rjL3s1ZVW9tOBSGmKqm+Pc9d0quk/9Jbtix4DKX9pRfXFa5muz/u25euQXtj69exvt++3LDz03zz389nJ/X0YSctey3k08c8zJQP7MXxl0xk7/0PAw4rahciuf76zL+kNRdD8P1SaSJRqpKSwiRVlxzH+2bZxjKaGMsy6tnOPw5eyJR/WEvr6YM57uKJDBo1qCdRF6enbTUiaeKqkkpsxj2pUkmNZZNpFjQzmDo170vXLlnH7784D1+WOcYmVnDft59gfcdOHto4iesfmsIpVx2dTLKAzDPxlVLc40BVy7hSUrg46rXKdVMbRgUoZx17ep31ySfv2b0yQ5fTVc91+D2ffXiP+aiXcoDaB9LF3Y23WrsF1zjUrVYSUa7eKpnexyxjD6LO0fvz63/9JW337aBt8X48u21CEBZvcdywhbROfpPWs/fhnQOeov6TH1dPm1RxVzGq+2tFUi8pSU45eqsUMLFQJ+bgPoiNfmrjY/6NU2b5QzcvyDwfdSX2tEkyprgviNMFdhUJlTCkpkW9yA7YMGAki358P0edN5G+A6qs41/S1xeohNErqNFbatbSB19h0177ZnxujxTS0MCQW77LMRcfXn3JAgqfzClumToTFNqNN7WRe9MmqK/v2fakYilhlJp6jOTkXc6Svy/jtosf4IIDH6S57yuMe/f+XLbpu2wOh9vYtW5DA3bFFcnNllaKzzLpGdR6OgNddwlp2bKgRLhmTbCdxsbqnNFOcoujXqvYGzAUuAdYCDwPvCvX+nu0YVRifXQq9RjZQ1dnly/6y0v+k2lt/uHmB3ebj3qErfJzxzzsN5072xfcs8g777izcj7fUn2W1X5ld7XH30tQC20YZvZz4AF3v8XM+gEN7r4+2/q7tWEkXfcbhepz8S7n+T8toe2uV2l7qC9tKw/k9a6RAIysW8WUMS/SevxOWs/bj0NPPxCrszxbTEipPstq+B7norGvqkLV95IChgAvk2Vypky33UoY1fDLphf2GOnc0ekL7lnkN507288d87Dva6t27faYulf9w80P+k+mtfnC/1viXZ1d5Q+w2FJpKT/LnpaUkyxpV8P/ocRWwkgyYRwJPArcDjwJ3ALslWG9y4B2oL2pqentI5DEybjQf8xa/2eaMcO7mpq8y8w3DBzp3xtyrTfa6l272T0f9S0XzvHF9y0NEkSSJ7eeVCtV6meZdLVn0u8vkdRCwmgBdgLHho+/D1yX6zWJljCKnQe6xv6Zuuej/tNRV/sWBuy2b28x0G/e98t7zkfdLenj0ZPvTNKxZ1MJiazS2xKlJhLGKGBpyuN3A/+b6zW7JYxy/wMX+49Z5f9M2eajfpnm6McjdR7tJE9uPS2VVuJn2QurPaVwVZ8wgn3gAWBieP9a4Du51i95L6lc26uGKrAYZJqP+jxm+Ms0eyfmq/uO9Ife9xXvynY8YM99yDePdpzHMNcxq4Rf43GrxX2S2NVKwjgybJ9YAPwO2CfX+iUdGiRfiaUaqsCKsHXDVp9z01MZ56M+rP9iv3XMl3xH3/57xtHYmP3kH+UkXYpjmO+YVWq1Uk/U4j5J7GoiYRR6K2nCyJcQqqUKLI8t67b4rBue9GunzPKThj7hA9i8a9OTBiz0KyfN9ns++7Cveq4jdxyNjdlLXakx5iqJxH0MoxyzSqxW6qla3CeJlRJG3KJUOZXzHzOmeN7qeMvv+/bj/uV3z/IThzzp/dgabIZOnzzwOf/U5Nn+2y/M9dUvrCk8jihVTPnaLuI8hqrPF8koroRRhYPvlEhTU+YLs5qa3r5fziki88WTZarUrRu2MWfDEbT96U3anh7Go28ewg6Ooo5OjmpYxJVHP0Lr+xs44ZKD2WfcocChPYsj3zHLNuVoKS5Mi/IZikjx4sg65bol2oZRbkW2qSylycG9Dzv82L2e9s8fM8v/99pHff2y9fHHEfWYlatkVmmfoUiFQFVSJVBpdcEZ4lm3dL3/8cvzvIvM1S9dmP/l64/5xpUbSxpHpOeSUGnxiFSAuBJGbc+HMXNmMEz08uVBtcT111fH+Dwp1i5Zx5xbXqDtr1tpW7gvT22ZiFPHUpppJsOIpkmOU1UDx1ukFmk+jHzSh10O6/gTH148zxDZHc+v5tefe4T/OKKNIwYuYvhBQzjnW8dy85PH8EF+w9r++9FFHQfsswn69dt920nOO1Cpx1tEYlO7JYxKHCk2w8ikXQMGMm/K55nxykm7zUc9kM0ct89Cphy1Mft81PX1sPfesHZtMr/oU0sUdXXQ2bnnOr1oZF6RShVXCaN2E0YlDrucJYktpZl38AwnNC6k9ehNtH6wkZZpE+k3qF/e1yZ2Qs40LHcmGuZaJHFxJYza7VZbIV0sV8x7lbbbX6ZtVic/WbY8Yx1gM8tZ91Z/6htyfJ5Jz8yWLtPUopmoS6tIzajdNow45iouwtIHX+Hn//YgF094gPH1y2l6536cf/Px3PPCJFbXjcz4GutTR/2g/rmn/cx24k3qhBwlUWkuZ5GaUrsJo6dzFUfQPR/1rRftPh/1RbecwB+WHM4RI17lhrPbeOIXC1m9dTD73vHfeyYxCOr+8zUUJ5QAs8qWqPr00VzOIjWqdtswSsC7nMX3LmX2nStoe7APbSvGs7JrNAAjrIMT93uR1nduZ8p5ozn8rIOo65shH3c3FGeqLuuWrV2ikrqtVvvUoiK9iBq9yyDffNStY15kSjHzUedrMK6WhuJKSmAikpUSRgl07ezi2d+/SNvdr9H2SD/mvHoQq3wEAGPqXqP1gJdoPaGT1mn7c/Cp46IniHTZejx1U1dUEYmRekllUuAv3q6dXSz49WLafvk6bXP7M+f1Cazxg4GDOaDPSk4dt4jWdy+k9fwmDjypCasbHU+cuRqM1VAsIhWqdhJGltFbgV1Jo3N7J0/96gXa/ucNZs8byAOrJrLeJwITGdd3OWcc+BytrUbrBc2MPWF/rG5MaWLN1uW3Tx+1AYhIxaqdKqks1TzbGkfz/WN+QVv7XjzYcTAbGQLAQfVLaR23gtaTjNYLx9H0rhIlh0zUYCwiZaQ2jHRZruzuwuhDFxP7vUTr+FdofU8fWi8ez5iWmKqXiqUGYxEpE7VhANs2buPROxbS9rt1/JuNZKS/vsc6Wwbvy2sPrmLUpPHA+PIHmU05J2MSEYlBVV24553O7Buf4qsnzeY9+zzJ0CFdnHjlEXz5/in83U9ij/KFGXt95IOMmrRv9o3mGT224lRbvCJSMxKvkjKzPkA7sNLdT8+1bp0d7c7jGF0cMfBdEDEsAAAKRklEQVQFphzyBq2nDuDk4fMZfM2nM1/XkKttoNraEqotXhGpCDXThmFm/wm0AHvnSxij9proP/3PO8P5qIe+/USx1zVU2giw+VRbvCJSEWpiAiUz2x/4AHBLlPX3P3QwZ1x3zO7JAvIPhJfp+ZkzsyeZXMknSZU2Yq2I9CpJt2HcCHweyDoOhpldZmbtZtbe0dGReaV8I7amP99dtZP9TSuzbaDSRqwVkV4lsYRhZqcDq9z98Vzruft0d29x95YRI0ZkXinTSK7dMl05nW8uB/dgnUpTaSPWikivkmQJ43jgTDNbCtwNvMfMZuR8xdq1mXsIpQ5lDsEV05B9iO0oVTilquYptpdT93Ubmzfn3z8RkRJIvNEbwMymAJ/N1+jd0qePt6eO4lpsD6F8jeRQmobkYns5qXeUiPRATTR6Fyx9yO/Nm4urOspVhQWlq+bJVBUWZR+KfZ2ISIwqooQRVYuZ7zEwSLFzR6QOzTFsWLBs7drSDtORZfiSvPtQ7OtERNDQIG8rtodQEkNzZBulNkovr2JeJyISo+qqkqpLCzfJHkLFNF4X28tJvaNEpAJUV8Jobg5uZsn2EOpuhF62LKgq6p57I1/SSO3NVcg+FPs6EZEYVVcbRkuLt3/608kPC64hOkSkitTMWFKFaBk/3tvfeCP57qVqhBaRKtI7u9WuXFkZ3Us1RIeI9ELVlTC2b8+8vNyD76kRWkR6oepKGP36ZV5e7l/2aoQWkV6ouq7DGDMGMrVhJPHLXlOsikgvU10ljGHD9MteRCQh1VXCAP2yFxFJSHWVMEREJDFKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkiSUMMzvAzGaZ2XNm9qyZfTKpWEREJL8khwbZCXzG3Z8ws8HA42Z2r7s/l2BMIiKSRWIlDHd/zd2fCO+/CTwPjEkqHhERya0i2jDMbCwwGZiX4bnLzKzdzNo7OjrKHZqIiIQSTxhmNgj4NfApd9+Y/ry7T3f3FndvGTFiRPkDFBERIOGEYWb1BMliprv/JslYREQktyR7SRlwK/C8u38vqThERCSaJEsYxwPnA+8xs6fC29QE4xERkRwS61br7g8CltT7i4hIYRJv9BYRkeqghCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISSaIJw8xOM7NFZvaimX0hyVhERCS3xBKGmfUBfgS8HzgMOM/MDksqHhERyS3JEsYxwIvu/pK7bwfuBs5KMB4REcmhb4LvPQZYkfL4FeDY9JXM7DLgsvDhNjN7pgyx9dRwYHXSQUSgOONTDTGC4oxbtcQ5MY6NJJkwInH36cB0ADNrd/eWhEPKS3HGqxrirIYYQXHGrZrijGM7SVZJrQQOSHm8f7hMREQqUJIJ4zFggpmNM7N+wIeAPyQYj4iI5JBYlZS77zSzTwB/BfoAt7n7s3leNr30kcVCccarGuKshhhBccatV8Vp7h7HdkREpMbpSm8REYlECUNERCKpmISRb5gQM+tvZr8Mn59nZmNTnrsqXL7IzE5NMMb/NLPnzGyBmd1vZs0pz3Wa2VPhraSN+xHivMjMOlLi+WjKcxea2eLwdmHCcd6QEuMLZrY+5bmyHE8zu83MVmW7/scCPwj3YYGZHZXyXDmPZb44p4XxPW1mD5vZESnPLQ2XPxVX98sexDnFzDakfLbXpDxXtqGEIsT5uZQYnwm/j8PC58pyPM3sADObFZ5znjWzT2ZYJ97vp7snfiNo9F4CjAf6AfOBw9LW+Thwc3j/Q8Avw/uHhev3B8aF2+mTUIwnAQ3h/Su6Ywwfb6qgY3kR8MMMrx0GvBT+3Se8v09ScaatfyVBx4hyH88TgaOAZ7I8PxX4M2DAO4F55T6WEeM8rvv9CYbjmZfy3FJgeIUczynAn3r6fSl1nGnrngH8vdzHExgNHBXeHwy8kOF/PdbvZ6WUMKIME3IW8PPw/j3AyWZm4fK73X2bu78MvBhur+wxuvssd98cPpxLcG1JufVkyJVTgXvdfa27rwPuBU6rkDjPA+4qUSxZufscYG2OVc4C7vDAXGComY2mvMcyb5zu/nAYByT33YxyPLMp61BCBcaZ1HfzNXd/Irz/JvA8wQgaqWL9flZKwsg0TEj6ju9ax913AhuAxoivLVeMqS4lyOzdBphZu5nNNbOzSxBft6hx/lNYRL3HzLovoCzXsSzovcKqvXHA31MWl+t45pNtP8p5LAuV/t104G9m9rgFQ/Ek7V1mNt/M/mxmh4fLKvJ4mlkDwYn21ymLy348LaiinwzMS3sq1u9nxQ8NUo3M7CNAC9CasrjZ3Vea2Xjg72b2tLsvSSZC/gjc5e7bzOxjBCW39yQUSxQfAu5x986UZZV0PKuGmZ1EkDBOSFl8Qngs9wXuNbOF4S/sJDxB8NluMrOpwO+ACQnFEsUZwEPunloaKevxNLNBBAnrU+6+sVTvA5VTwogyTMiudcysLzAEWBPxteWKETN7L3A1cKa7b+te7u4rw78vAbMJfg2UQt443X1NSmy3AEdHfW0540zxIdKK/GU8nvlk24+KG/rGzCYRfN5nufua7uUpx3IV8FtKU6UbibtvdPdN4f3/A+rNbDgVeDxDub6bJT+eZlZPkCxmuvtvMqwS7/ez1A0zERtv+hI0uozj7Qatw9PW+Xd2b/T+VXj/cHZv9H6J0jR6R4lxMkHD3IS05fsA/cP7w4HFlKjBLmKco1PunwPM9bcbwl4O490nvD8sqTjD9Q4haES0JI5n+B5jyd5I+wF2b1R8tNzHMmKcTQTte8elLd8LGJxy/2HgtATjHNX9WROcaJeHxzbS96VccYbPDyFo59grieMZHpc7gBtzrBPr97NkB7uInZ9K0Mq/BLg6XPY1gl/qAAOA/wm/9I8C41Nee3X4ukXA+xOM8T7gDeCp8PaHcPlxwNPhl/xp4NKEj+U3gWfDeGYBh6S89pLwGL8IXJxknOHja4Fvpb2ubMeT4Nfja8AOgnreS4HLgcvD541gIrAlYSwtCR3LfHHeAqxL+W62h8vHh8dxfviduDrhOD+R8t2cS0qCy/R9SSrOcJ2LCDrcpL6ubMeToFrRgQUpn+vUUn4/NTSIiIhEUiltGCIiUuGUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDpAfC+QjeF97/upndlHRMIqWiwQdFeuYrwNfCgeYmA2cmHI9IyehKb5EeMrM2YBAwxYN5CURqkqqkRHrAzP6BYOaz7UoWUuuUMESKFM5cNpNgVrNNZlayGfVEKoEShkgRwpnWfgN8xt2fB64jaM8QqVlqwxARkUhUwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJ5P8DY8IBC9SCZV8AAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"# Importing various packages\n",
"from math import exp, sqrt\n",
@@ -1318,8 +1234,10 @@
"\n",
"## Momentum based GD\n",
"\n",
- "The stochastic gradient descent (SGD) is almost always used with a *momentum* or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically \n",
- "implemented as follows"
+ "The stochastic gradient descent (SGD) is almost always used with a\n",
+ "*momentum* or inertia term that serves as a memory of the direction we\n",
+ "are moving in parameter space. This is typically implemented as\n",
+ "follows"
]
},
{
@@ -1350,7 +1268,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "where we have introduced a momentum parameter $\\gamma$, with $0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a running average of recently encountered gradients and $(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when $\\gamma=0$, this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is"
+ "where we have introduced a momentum parameter $\\gamma$, with\n",
+ "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
+ "indicate the gradient is to be taken over a different mini-batch at\n",
+ "each step. We call this algorithm gradient descent with momentum\n",
+ "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n",
+ "running average of recently encountered gradients and\n",
+ "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n",
+ "used in the averaging procedure. Consistent with this, when\n",
+ "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n",
+ "earlier. An equivalent way of writing the updates is"
]
},
{
@@ -1370,8 +1297,11 @@
"\n",
"## More on momentum based approaches\n",
"\n",
- "Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass $m$ moving in a viscous medium with drag coefficient $\\mu$ and potential \n",
- "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$, then its motion is described by"
+ "Let us try to get more intuition from these equations. It is helpful\n",
+ "to consider a simple physical analogy with a particle of mass $m$\n",
+ "moving in a viscous medium with drag coefficient $\\mu$ and potential\n",
+ "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$,\n",
+ "then its motion is described by"
]
},
{
@@ -1420,8 +1350,12 @@
"metadata": {},
"source": [
"## Momentum parameter\n",
- "Notice that this equation is identical to previous one if we identify the position of the particle, $\\mathbf{w}$, with the parameters $\\boldsymbol{\\theta}$. This allows\n",
- "us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:"
+ "\n",
+ "Notice that this equation is identical to previous one if we identify\n",
+ "the position of the particle, $\\mathbf{w}$, with the parameters\n",
+ "$\\boldsymbol{\\theta}$. This allows us to identify the momentum\n",
+ "parameter and learning rate with the mass of the particle and the\n",
+ "viscous drag as:"
]
},
{
@@ -1437,13 +1371,30 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n",
+ "Thus, as the name suggests, the momentum parameter is proportional to\n",
+ "the mass of the particle and effectively provides inertia.\n",
+ "Furthermore, in the large viscosity/small learning rate limit, our\n",
+ "memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n",
"\n",
- "Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. \n",
+ "Why is momentum useful? SGD momentum helps the gradient descent\n",
+ "algorithm gain speed in directions with persistent but small gradients\n",
+ "even in the presence of stochasticity, while suppressing oscillations\n",
+ "in high-curvature directions. This becomes especially important in\n",
+ "situations where the landscape is shallow and flat in some directions\n",
+ "and narrow and steep in others. It has been argued that first-order\n",
+ "methods (with appropriate initial conditions) can perform comparable\n",
+ "to more expensive second order methods, especially in the context of\n",
+ "complex deep learning models.\n",
"\n",
- "These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). \n",
+ "These beneficial properties of momentum can sometimes become even more\n",
+ "pronounced by using a slight modification of the classical momentum\n",
+ "algorithm called Nesterov Accelerated Gradient (NAG).\n",
"\n",
- "In the NAG algorithm, rather than calculating the gradient at the current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one calculates the gradient at the expected value of the parameters given our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1})$. This yields the NAG update rule"
+ "In the NAG algorithm, rather than calculating the gradient at the\n",
+ "current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one\n",
+ "calculates the gradient at the expected value of the parameters given\n",
+ "our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma\n",
+ "\\mathbf{v}_{t-1})$. This yields the NAG update rule"
]
},
{
@@ -1496,11 +1447,17 @@
"the steep computational price of calculating or approximating\n",
"Hessians.\n",
"\n",
- "Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.\n",
+ "Recently, a number of methods have been introduced that accomplish\n",
+ "this by tracking not only the gradient, but also the second moment of\n",
+ "the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and\n",
+ "ADAM.\n",
"\n",
"## RMS prop\n",
"\n",
- "In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule for RMS prop is given by"
+ "In RMS prop, in addition to keeping a running average of the first\n",
+ "moment of the gradient, we also keep track of the second moment\n",
+ "denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule\n",
+ "for RMS prop is given by"
]
},
{
@@ -1540,12 +1497,31 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "where $\\beta$ controls the averaging time of the second moment and is typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.\n",
+ "where $\\beta$ controls the averaging time of the second moment and is\n",
+ "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n",
+ "typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a\n",
+ "small regularization constant to prevent divergences. Multiplication\n",
+ "and division by vectors is understood as an element-wise operation. It\n",
+ "is clear from this formula that the learning rate is reduced in\n",
+ "directions where the norm of the gradient is consistently large. This\n",
+ "greatly speeds up the convergence by allowing us to use a larger\n",
+ "learning rate for flat directions.\n",
"\n",
"\n",
"## ADAM optimizer\n",
"\n",
- "A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)"
+ "A related algorithm is the ADAM optimizer. In ADAM, we keep a running\n",
+ "average of both the first and second moment of the gradient and use\n",
+ "this information to adaptively change the learning rate for different\n",
+ "parameters. In addition to keeping a running average of the first and\n",
+ "second moments of the gradient\n",
+ "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n",
+ "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n",
+ "performs an additional bias correction to account for the fact that we\n",
+ "are estimating the first two moments of the gradient using a running\n",
+ "average (denoted by the hats in the update rule below). The update\n",
+ "rule for ADAM is given by (where multiplication and division are once\n",
+ "again understood to be element-wise operations below)"
]
},
{
@@ -1626,9 +1602,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n",
+ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n",
+ "second moment and are typically taken to be $0.9$ and $0.99$\n",
+ "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n",
"\n",
- "Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance $\\boldsymbol{\\sigma}_t^2 = \\hat{\\mathbf{s}}_t - (\\hat{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The update rule for this parameter is given by"
+ "Like in RMSprop, the effective step size of a parameter depends on the\n",
+ "magnitude of its gradient squared. To understand this better, let us\n",
+ "rewrite this expression in terms of the variance\n",
+ "$\\boldsymbol{\\sigma}_t^2 = \\hat{\\mathbf{s}}_t -\n",
+ "(\\hat{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n",
+ "update rule for this parameter is given by"
]
},
{
@@ -1725,27 +1708,11 @@
},
{
"cell_type": "code",
- "execution_count": 28,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd0AAAEWCAYAAAAjEk0ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xd8FHX+x/HXZ1MIgUBC7wnSayAECBbkrKgo6qmoWLAhVu7On56np2Kvd/besKCIFBty2MFCgACBUKVLqKGHElL2+/tjBlxjKkn2u+XzfDz28djdmZ15T/1M2xkxxqCUUkqpmuexHUAppZQKF1p0lVJKKT/RoquUUkr5iRZdpZRSyk+06CqllFJ+okVXKaWU8hMrRVdEThCRFTXU7btE5I2a6HaoE5GxIvJQGc0fEpHtIrLFn7lClYgYEWlvO4ctIjJGRN6vYjeGi8hXZTQfJCLZVexHwE+nqozL6lofi8g6ETmlgu2OEJGfqtCvV0TknqP9fTndXiIig2qi21CBouuOyIMikisiu0XkFxEZJSJHXbCNMT8aYzod7e99sv1pgTLGPGKMubaq3Q5GNblyEJE2wG1AV2NMs5roRwVztBURr4i8XMnfVWkhV4HJGDPOGHPa4c9VXQZE5AcRCen1R/FxVF3rY38yxowyxjxY1e6UtKNhjOlmjPmhqt0uTUUL59nGmDggEXgM+Cfw5tH0UEQij+Z3yro2wA5jzLaSGvpxul4B7AKGiUgtP/WzTOII61M1ulwrfxGRCNsZqsQYU+YLWAecUuy7foAX6O5+rgU8BfwGbAVeAWq7zQYB2TiFegvw3uHv3Ob/BCYW6/6zwHPu+6uAZUAusAa43v2+DnDQzbHPfbUAxgDvu+1MA24u1u2FwPnu+87A18BOYAVwURnjoQHwNrAJZ6X/iU+z64BVbnc+A1r4NDPAjcBKdxgeBNoBvwB7gQlAdLFxdRew3R33w3269QNwrc/nEcBP7vuZbr/2u+NimPv9ECAT2O32s6fP73sD891cHwHjgYdKGPZTio3rsUCS279r3Ok+0233HGCJ278fgC7F5qXbgUVuzjeBpu50ygW+ARLKmAYCrAZuwJnPLvBpdjhPZPHxBXQB8oAiN/9ut3l94F0gB1gP/BvwuM0igP+402EtcLNv991uPwz87I6b9pQyr/rkuR3YjDMPXe12r30Az29nAQvc320AxpQwvotP/zS3X7txlrVBPr9pC8xwc30NvIC7rJYw/DOAv7rvj3P7dZb7+WQgsyLLgM8w3gZsc8f/VaX082F3Hslzf/+Czzgd5Y7T3cCLgPj87mp3uu8CpgOJZczDH+OsB/e4ebv5NBvrdnuqO45mA+2KrRc3uNNjHnCCT7Mx/L7emwrcUqy/i4DzyhpHPu22BibjLBc7fMZDO+A797vtwDggvqxa4dOsIc68uheYgzNf/uTTvNR1sTteXga+dHOf4n73kNt8GTDEp/1IN3tKWeMcGAkUAPnuuPjcdzhw6slBoEGxdeZ2IKqy0/5IN8ptoZQRibOg3eC+f9odoQ2AOOBz4FGfBbsQeBynONfmj0U3ETgAxPms7DYDaT4LfjucFe6JbrspPt3OLpZrDL/PfFcAP/s064qz0NTCKdobcFaUkT4js2sp42EqTmFKAKKAE93vT3J/l+J293ncFZDPAvspUA/oBhwCvgWOwVnpLwWuLDau/ut260ScmayTbxHx6fYI/jjj/mEl7g7TNqC/O16vdKdnLSAap9D83R2eC3BmwD8V3ZLGNb+vdN91x2VtoKOb91S3m3fgFIfDK/l1QDpOoW3pZpvv5ozBWaDvK2NePMEdfwnueP68hDx/KroljSv3u3fdaRPn/v5X4Bq32Sh32rRy+/cNfy66v7nTNNId3rLm1cE4Gwrd3fH1QfHpFYDz2yCgB84RsZ5u/nPLmP4tcVbIZ7q/OdX93Nj9zSyffg3EKSylFd0HgOfd93fhbGw97tPs2QouA4eH8QF3PJ7pTpcSN+4otoz5dPMLIB7niE8OMNhtNhRnHu/izgf/Bn4pYx6+Gmd+qwU8g7vxYH4vLjtwdmoicYraeJ/ml+EUr0icjYgtQEwJ672LgNk+v0t2uxtdxjg6vD6OwNlYetqdrjHA8W6z9u40rQU0xilgz5RXK9xm43E2+OrgLAMb+X1jqcx1sTte9uBsfHncTGP5vejeC4zz6ddZwLJKjPOHimU9Mhw466TrfJo9CbxyNNP+SDfKbaH0opsO3I2zgtnPH7fIBgBrfSZo/uGZo5QV+E/AFe77U4HVZeT5BBhdUndKmPni3GyJ7ueHgbfc98OAH4v99lVKWOkDzXH28v60oOLsrT3h87kuTvFK8pnBj/NpPg/4p8/n/+DOuPy+gqjj03wCcE9JKwTKX+G8DDxYLO8KnJXrQJy9KN8t9l+Kz4BlTLMkt3/H+Hx3DzDB57MHZ+Ea5DMv+e5JTQJe9vl8Cz57dCVkeONwc3ceKwCaFMtToaKLs3LJx2cjC7ge+MFnYbvep9kp/LnoPlDOsuM7r74FPObTrGPx6RVo81sJ/X4GeLqM6f9P4L1iv5mOs7HXpoR+fUDpRfdkYJH7/n84RyzS3c8z+P1oVfHpWlJBOVhsvtiGu1FfQn+PzDPFunl8sXF0p/t+Gu6Gms88f4CK7PE4RdwA9d3PY4E3fJqfCSwv4/e7gGT3/Rh+X+/FuM06uJ+fAl4qZxwdLroDcDYqIiuQ/1xggc/ndZRcKyLcebSzz3eP8HvRLXNd7I6Xd4s1H8vvRbc9zgZcrPt5HHBvJcZ5WUX3WuA7973gbBwMrMq0r8p5qJY4hwIaA7HAPPdCq904C0ljn3ZzjDF5ZXTrA+AS9/2l7mcAROQMEUkXkZ1ut88EGlUkoDEmF2eP4WL3q0twJgg4e9j9D2d2uz0cKOkiodbATmPMrhKatcDZYzzcz304W5UtfdrZ6vP+YAmf6/p83mWM2e/zeb3bj6ORCNxWbBhbu91rAWw07tzi06/K2uDzvvi48LrNj3ZcHCEitYELcaefMWYWzp7mpUeRGZx5KIo/DvN6n6wt+OOw+b4v8bty5tXi3StrXAfE/CYi/UXkexHJEZE9OHv/xZc932FKBC4sNr8dj7MR0aKUfpVmFtBRRJoCvXD2qFuLSCOcPcGZZfy2uB3GmEKfzwcoZT4rg+8V+76/TwSe9RnenTgr55bFfo+IRIjIYyKyWkT24qzc4Y/jtLT+ICL/JyLLRGSP26/6lLAudNe1HwGXudcaXIJzWq8iWgPri42vw/1vKiLjRWSjm//9kvpfgsY4e4Klzf8VWReXtPwBYIxZhXOI92wRicU5xfWBm7ki47wsk4ABItIcZ0fFC/zok7tC097XURVdEenrdvgnnMMAB3GOk8e7r/rGGN+Z2pTUHR8fA4NEpBXOeYfDI6wWzkA/BTQ1xsTjHNeXCnYX4EPgEhEZgLMF+L37/QZghk/meGNMXWPMDSV0YwPQQETiS2i2CWfk42aug3MIaGMFspUkwe3GYW3cfoCz1x7r06y8q4g3AA8XG8ZYY8yHOIfwW4qI+LTf5ijy+k6D4uNCcBbiox0Xvs7DOWT6kohscf+21BJnLwqccQOlj5/i88p2nK3vRJ/v2vhk3YxzaPmw1iVkOtLNCsyrm4t1o6xxHSjz2wc4p41aG2Pq41yrIcV+7zteN+Ds6frOb3WMMY/hDH9J/SqRMeYAzl76aGCxMSYf50jMP3COhG2v9JBWTEXWKb424BwR8R3m2saYX0po91KcQ5Kn4BTMJPf74uP0T0TkBJzTNRfhHAGJxznkWtpv38EpXCcDB9yN1IoOT5tSLox7BGf89DDG1MM53F1udpw950JKn/8rsi4ub7p8iLNxMRRY6hZiKH+cl9ldd8P3K5y98UtxDvcf/k1lpv0RlSq6IlJPRIbgHJ9/3xiT5e7NvA48LSJN3PZaisjpFe2uMSYH57DO2ziHpZe5jaJxjsPnAIUicgZwms9PtwINRaR+GZ3/Emcl9QDwkZsXnHM0HUXkchGJcl99RaRLCfk24xxKeElEEtx2B7qNPwSuEpFe7or3EZzzKesqOvwluF9Eot0FbQjORgk4F0SdLyKx7iX/1xT73Vacc3eHvQ6McvdYRETqiMhZIhKHsydRCNzqDs/5OHsQVTEBOEtEThaRKJzzTodwVpZVdSXOIdoeOHs+vXDO8SSLSA93HtqIs3UfISJX45xfPWwr0EpEogGMMUVu3odFJE5EEnFW6If/6zgBGO3Oy/E4h07LUt68OgEYISJd3a3x+0rrUADNb3E4e9x5ItKP8o8qvI+zt3G6Ow1ixPlbXytjzHogw6dfxwNnl9O9GTgXsM1wP/9Q7HNJii8DlVXZ378C/EtEugGISH0RubCUduNwlocdOBuHj1SiP3E4y2sOECki9+JshJbILbJenNMJxfdyyxrGOTgbSI+564sYETnOJ8M+YI+ItMS5MLBc7rI2GRjjrru68vvGMlRiXVyG8TjL2w34HCml/HFeken9Ac71QRcU63Zlpv0RFS26n4tILk5lvxvnYoirfJr/E+eEcrq7C/8NUNn/fX2AszVyZKDcw8O34qywduEs9J/5NF+OsxJa4+7i/+kwrDHmEM4EL6nbp+Ecet6Ec1jn8MVeJbkcZ89oOc45ob+53fkG51zmJJyZtR2/H84+GltwhnUTzqHUUe5wgnNxQz7OjPIOvx8qP2wM8I47Li4yxmTgXOn6gtvNVTjnwHD3HM53P+/E2ZKbXIXcGGNW4Gz9Po+zJ3k2zt/N8qvSXXcBPxnnXOQWn9c8nFMZhxfg63BWBDtwLiLyLfbf4VxVvUVEDu8l3YKzh7wG56jNBziFHZwNlq9wrvpcgLPxVohzdWtJw17evDoN55zodzjT4btyBjsQ5rcbgQfcZf9ed9hKZYzZgLNXcRdOcdiAMz0Or2cuxbmobyfORse75WSbgbPSnFnK55KMwWcZKKf7JXkWuEBEdonIc+W1bIyZgrPeGO+u+xYDZ5TS+rs4h1U34lzQll6JXNNx5vVf3W7kUcYhV5/+9eD3DcnDxlDKOHIL5Nk450l/w7nye5jb+H6cC/j24Jy2q8z64macQ+VbcM6jvu3Tz8qui//E3VCdBRyLc2j9sPLG+ZtAV3dcfFJK5z8DOgBbjDELffpZmWl/hPzxlJ6ySZy7oLxvjGlVXrvKv9w911eMMYnlthwkdH4LbSJyBTDSGHO87Szqd2H9h36lSiMitUXkTBGJdPe07wOm2M6lVEW4pzBuBF6znUX9kRZdpUomOIfTduEcXl6Gc4hVqYDmXk+Tg3Ma6oNyWld+poeXlVJKKT/RPV2llFLKT/Qm5WVo1KiRSUpKsh1DKaWCyrx587YbYxqX32b40aJbhqSkJDIyMmzHUEqpoCIiR3N3u7Cgh5eVUkopP9Giq5RSSvmJFl2llFLKT7ToKqWUUn6iRVcppZTyEy26SimllJ+EXdEVkb+LyBIRWSwiH4pIjO1MSimlwkNY/U/XvXH9rUBXY8xBEZmA8zipsVaDKaXCivF62bRuBVtXpJO/ayMmby8SFYOnbiPiE5Np0yWVmNp1bMdUNSCsiq4rEqgtIgU4DzXeZDmPUipMrFk8m20/vkXbrV/Tkh20LKmlhXDw02jm1+2P9LyQnicPJyIyHFfVoSmspqQxZqOIPIXzcOaDwFfGmK982xGRkcBIgDZt2vg/pFIq5KzM/JED0x8g+eAcWplIltTpx7qkv9Cg4wAat+pA3foNOJR3gF1bs9m2KoOCVTNot/07Gs36kez0R9nc61ZSzx6FeMLujGDICaunDIlIAjAJGAbsBj4GJhpj3i+p/dTUVKO3gVRKHa09u7azfNzt9M2Zwh6JY3nScLqe/XfqN2xa7m+LCgtZ+PV71Mt4nvZFq1kW1ZXYv75IYucUPySvGhGZZ4xJtZ0jEIXbZtMpwFpjTI4xpgCYDBxrOZNSKgQtz/iWg8/2JzVnCnOaXEDE3zIZMOKxChVcgIjISFLOuIpj7prL3J4P0KxgA40/HMycSc9gvN4aTq9qSrgV3d+ANBGJFREBTsZ5OLlSSlWb9A8fpt3nF+KVCFYP/ZS0m96gXnzDo+qWJyKCvuePpmjkz6yJ6UK/rPuY89I1FBbkV3Nq5Q9hVXSNMbOBicB8IAtn+F+zGkopFTK8RUWkvzSStBVPsLhOf+rc+gsdU06slm43apFIl9u/Jb3ZcPpvn8zi/57NgX17qqXbyn/CqugCGGPuM8Z0NsZ0N8Zcbow5ZDuTUir4FRbks+CZC0jb9hHpTS4i+bYvqJ/QqFr7EREZSdqol5jd9W56HJjN2ufOJu/Avmrth6pZYVd0lVKquhUVFpL5/CX0yf2OWcfcSv9Rr+KJiKix/vW/6A4W9HmMLocW8etzQzmUd6DG+qWqlxZdpZSqAm9REfNevJLUvd+QnnQTA6540C9/7Uk9ZxTzku+nZ14GS164GG9RUY33U1WdFl2llKqC2W/9g367vmBWq6tJG/GIX/vd9/zRpLf/Gyn7ZjD77f/za7/V0dGiq5RSR2nupy8xYONY5jQ4m7Sr/2MlQ/9L72NOwhAGZL9FxmcvW8mgKk6LrlJKHYXls78ief49LK7Vi96j3rR2tyjxeOh9w1ssie5Jt3n3snbpXCs5VMVo0VVKqUravuU3Gk27jq2eJrQeOYGo6FpW80RF16Lp1ePYL7F4Jo5gf+5uq3lU6bToKqVUJRQVFrLl7cupYw5QeOG7Fb7DVE1r1KwNW055ntZFG1n6xkjbcVQptOgqpVQlzHnvbrofyiSr59207drXdpw/6H78OcxufRV990xnwVcl3lJeWaZFVymlKmjprGn0W/cqGfVOoe95t9qOU6I+lz/K6oi2tPnlLnblbLYdRxWjRVcppSpgf+5u4r+6lc2epnS+9o2AfcxedK0YOPcV6pl9rHn3RttxVDGBOdcopVSAWfzO32nmzSF38HPUrZdgO06Z2vVIIyPpOvrkfkfm1x/YjqN8aNFVSqlyLP7pM/pvn8ycphfRpf/ptuNUSOrwB1jnaUPTn+/l4P5c23GUS4uuUkqVYX/ubhp8exsbpAXJV9q5AcbRiIquxf5THqc5OWR+cI/tOMqlRVcppcqQ9f6dtDDb2Df4GWrXibMdp1K6HXsmGfVOpU/2e2xYudB2HIUWXaWUKtXapXNJ3fIRcxKGBM1h5eKSLv0vh4hi56TbbEdRQKTtAKFoeca3FH79AFFFeUR58zDiId8TQ35kHHl1WmLi21CnTS8Sew6kXnxD23GVUiUwXi8HP/k7+ySWDpc+ZTvOUWvUrA3pHUaRtuppFv/4Kd1PGGo7UljTolsTvF4iiw6RH1GbA9ENEOMlquggdfNzaHtwMfW274dV4P1WWBuRyJbmg2jQeygdUwYF7N8QlAo3GZ+/Qt/8LOb0GEO/xs1tx6mSXn+9nc2Pv0/MD/fjPXZIjT7rV5VNjDG2M/iViMQDbwDdAQNcbYyZVVK7qampJiMjo9oz7Nmxld+WpLNv9S/U2/wLnQ4tJlK8ZEtzNrS9kI6nX0/Dpq2qvb9KqYrZs2s7hc+msD2yOR3+9UtIFKmMz18ldd4dzO39KH2H1uz/d0VknjEmtUZ7EqTCsei+A/xojHlDRKKBWGNMiXcHr6miW9yeHVtZMXMCdZeOp2vBYvJMFAubDCXxnH/RrHX7Gu+/UuqP0l+5kX6bP2DN+VNpn3yc7TjVwltUxOpH+hFXtJv4OxYSE1u3xvqlRbd0YXUsU0TqAwOBNwGMMfmlFVx/qt+wKf3Ou4Wud//M+ktmsKjBaaRsm0KDN/qR/tJ17Nm13XZEpcLGprXLSdn8ERkJZ4RMwQXwRESQf9IDNGM7mZOftB0nbIVV0QXaAjnA2yKyQETeEJE6vi2IyEgRyRCRjJycHL8HTOzUi36jP2D71bPIbHgm/bZ+jPfZ3syZ9DTeoiK/51Eq3Gye/C8KiSDpwkdsR6l23Y47i6xaKXRc9RYH9u2xHScshVvRjQRSgJeNMb2B/cCdvi0YY14zxqQaY1IbN25sIyMAzRM70e/W91lz/lS2RLWmX9YYlj0+iE3rVljLpFSoW5HxHX1yv2NR68to0rKt7Tg1IuqUu2nAXhZNDt4rsoNZuBXdbCDbGDPb/TwRpwgHrPbJx9H5Xz8xt+cDJB5aSb23T2Tu5GcxXq/taEqFFOP14p1+N9uJp/tFoXsHp859T2FRTCqd1ozVh91bEFZF1xizBdggIp3cr04GllqMVCHi8dD3/NHsvWoG62t1oO+ie8l49hLyDuyzHU2pkJH59Xt0KVjK6m63BvwDDaoq+pS7SGAvi/Tcrt+FVdF13QKME5FFQC8gaE7ctEjqRJd//sCsNiPps3s6G/9zAhvXLLMdS6mgV1RYSIPZT7De05o+595iO06N65x6Moti+tJl7Vj27d1lO05YCbuia4zJdM/Z9jTGnGuMCao5zhMRwYCrnyTrxNdoXLSV2HdPZfncb2zHUiqozZ/6GonebHb0+z8io6Jtx/GL6FP+RTz7WPz5c7ajhJWwK7qhIvmki9h72Vfsl7okfXEx86e/ZzuSUkGpIP8QzTOfZXXEMfQ69XLbcfymc+rJLInuwTErx5J/KM92nLChRTeItWrfndgbvmV9VDt6/XILsyc8YTuSUkFnwecv0cpsIXfAHSFx56nKKBwwmibsZOG0N2xHCRtadINcgyYtSfzHtyyqk0b/pQ+T/sGDtiMpFTQO5R2gTdaL/BrZkeSThtmO43c9T/wraz1JNF70it4HwE+06IaAmNi6dB39CfPrnEDar0+R/t69tiMpFRQyP32eZuSQd/ydYfmwEfF42NFrFEneDSz6foLtOGEh/OayEBVdK4YeoycxL+4vpK1+lvR3Q/d/hkpVh7yD+2m77BWWRXWlx8DzbMexJnnw1WyhMbVm6wVV/qBFN4RERdci+dYJzIs7ibQ1zzFn4n9tR1IqYC384mWasJPCE8JzL/ewqOharOt0FV0Kluo/IfwgfOe0EBUZFU3PW8azMKYvfbIeYP60t21HUirgFBbk03rpa/wa2ZHux59tO451PYbcRK6pzf6ZL9qOEvK06IagqOhadLxlCr9Gd6F7+m1kzfzUdiSlAkrmtDdpYbZyoP/fw3ov97A6cfEsaTaUnntnsG3jWttxQprObSGqdp04Wtz4ORsjWpP07fWsXzbPdiSlAoK3qIhGmS+y1pNIzzC8Yrk0rU+/lQi8rJ6m53ZrkhbdEFY/oRG1r5rEIalF1IRL2Llto+1ISlm38NsPSfJuYEfvm8Luf7llaXlMNxbVSaNT9kTyDu63HSdkadENcc1at2fnOe/QwLuTba9foAuTCmvG66XOnGfJlmb0GnyV7TgBJ3LADTRgL1nT9VqQmqJFNwx0TBnEkv5P0rlgKYtfvlIfC6jC1uKfPqVj4a9s6nZ92NxjuTK6HXc26zytSch6S9cTNUSLbpjoc+ZVzEocRerer5n90aO24yhlheen/7KNBiQPGWU7SkASj4etXUbQvmg1K/TvQzVCi24YSbvyURbEHkuf5f9h+eyvbMdRyq9+nT+DbvmLWNNhBLViYm3HCVg9zriOvdQh95c3bUcJSVp0w4h4PBxz3Xts9TSmwbTr2b5lg+1ISvlN7vdPk2tq021I6D8vtypi69Znyzkf0H2kPgShJmjRDTP1ExqR/9d3iTP72PrWpRQW5NuOpFSN27x+Bcl7Z7Ck+fnE1W9gO07A65gyiNp14mzHCElhV3RFJEJEFojIF7az2HJM9/4s7j2GbvmLmPvOP23HUarGrf/yvxiEpLP+YTuKCnNhV3SB0cAy2yFs63vuTcyNP4N+G95m6axptuMoVWP27t5B9y2fsrD+X2jWur3tOCrMhVXRFZFWwFmAnqwAulz9Mps9zWgw/Wb27MyxHUepGrH0i+eoKweJP/nvtqMoFV5FF3gGuAPQP6ABdeslcODsV2lodrH6rav1f3kq5BTkH6LtqvdYEt2T9snH246jVPgUXREZAmwzxpR5E2IRGSkiGSKSkZMT+nt/HVNOJKPdTaTsm0nGJ8/bjqNUtVo4fSxN2UFB/5tsR1EKCKOiCxwHnCMi64DxwEki8n7xlowxrxljUo0xqY0bN/Z3Riv6Dx/D4lq96LbwYTauCfvT3SpEGK+X+MxX+c3Tkp6DLrQdRykgjIquMeZfxphWxpgk4GLgO2PMZZZjBQRPRASNLnuTIjzsHn8d3qIi25GUqrJls6fTvmg1m7tcow82UAEjbIquKluz1u1Z1vNOuuVnMffjJ2zHUarK8n56iT3UoeeZI21HUeqIsCy6xpgfjDFDbOcINH3Pu5VFMX3psexpNq5ZYjuOUkdty4ZV9Nz3E8uanas3eVABJSyLriqZeDw0vexVivCwZ/z1ephZBa21/3sewdBm8GjbUZT6Ay266g+atmrH8uR/0TU/izkTHrcdR6lKyzu4n84bJ7OozgBaJHWyHUepP9Ciq/4k9dxbWBjTl57Ln2Hz+hW24yhVKVnT3yaBvUQN0Mf3qcCjRVf9iXg8NL30ZQC2jb9Fb5qhgobxeonPept1ntZ0O+5s23GU+hMtuqpEzdp0YFHHm0k+OJsF09+xHUepClkx7zs6FK1ia+crEI+u3lTg0blSlSr1ojtZFdGO1rPvZ+/uHbbjKFWufTNfYi+xdD9D/yakApMWXVWqyKhozJBnaGB2s+z9/7MdR6kybd+0nuS9P7C0ydnUiYu3HUepEmnRVWXq0Hsgc5tcQN+cKazI+M52HKVKtfJ/LxCBl1an3Wo7ilKl0qKrytX98ifZLglEffl3CgvybcdR6k/yD+XR4bcJZMX2pVX77rbjKFUqLbqqXHXrJZCddh/HeNcxb9J/bMdR6k8Wff0ujdgN/fRcrgpsWnRVhfQ+7QoW1+pFl+XPs3PbRttxlPqDOgvfJlua02Pg+bajKFUmLbqqQsTjIe7c/1Lb5LFq/D9tx1HqiLVLZtOlYCnZ7S/RpwmpgKdFV1VYYpc+zGt2Iak7vmDlgpm24ygFwLbvX+GQiaLz6dfbjqJUubToqkrpeskj7JT6eKferg9EUNYd2LeHrjnTyKo/iPhGzWzHUar4KFM+AAAgAElEQVRcWnRVpdSLb8ia5P+jU+Fy5n3+su04Kswtnv42cXKQusfpBVQqOGjRVZWWes6NrIjsTNvMJ/VOVcqqhGXvs9aTSKe+p9iOolSFaNFVleaJiMBz1pM0MHtYOv4e23FUmFqZ+SMdCleyreMlep9lFTTCak4VkdYi8r2ILBWRJSKiT7g+Sh16D2RewmBSNn/EprXLbcdRYWjXzFc5YGrRZbAeWlbBI6yKLlAI3GaM6QqkATeJSFfLmYJW4oWPUkgEmyfpX4iUf+Xu2Un3HV+xOOFk6sU3tB1HqQoLq6JrjNlsjJnvvs8FlgEt7aYKXk1atmVhmyvos+8Hls/52nYcFUaWTn+DWDlE/EB9UL0KLmFVdH2JSBLQG5htN0lwSx52DzkkIF/drQ+7V35hvF6aLB/Hqoh2dOh1gu04SlVKWBZdEakLTAL+ZozZW6zZSBHJEJGMnJwcOwGDSGzd+qxN/gedClcwb9qbtuOoMLBi3ne09a5jR+fhegGVCjphN8eKSBROwR1njJlcvLkx5jVjTKoxJrVx48b+DxiE+px9I6sj2tJy7uPkHdxvO44Kcbk/vc4+U5tup19tO4pSlRZWRVdEBHgTWGaM+a/tPKEiIjKSg4Pupzk5ZH78mO04KoTt2bGVHru/ZUmj06lbL8F2HKUqLayKLnAccDlwkohkuq8zbYcKBd1PGMrC2v3ptvp1dmzNth1Hhahl018jRgpoNOgG21GUOiphVXSNMT8ZY8QY09MY08t9fWk7V6iIH/oYtTnEqon32Y6iQpDxemm+ajwrIjvTrkea7ThKHZWwKrqqZiV2TmF+w7PovW0KG9cssx1HhZil6f8j0ZvN3u6X246i1FHToquqVdJfH8SLh82f/Nt2FBVi8ma9zl7q0OO0EbajKHXUtOiqatWkZVsWtLiElD3fsjor3XYcFSJ2bM2mx94ZLG1yFjGxdW3HUeqoadFV1a7rhfeyT2LZN1X3dlX1WDn9VaKliOYn6QVUKrhp0VXVrn6Dxiw95hqS8+ay5Be9Tk1VjbeoiNZrP2JpdA8SO6fYjqNUlWjRVTWi1wX/ZBsNiPxujN4eUlXJkp8/o6XZyoGeV9iOolSVadFVNSImti7retxKp8IVLPh6nO04KogVzH6TXdSjxymX2Y6iVJVp0VU1JuWcm1jvaUXD2Y9RWJBvO44KQjmb1tFz38+saHY2tWJibcdRqsq06KoaExkVzY7+d5LozWb+Zy/ZjqOC0KrpLxMpXlqdcqPtKEpVCy26qkb1PnU4KyI7k5T1LHkH9tmOo4JIUWEhbddPJKtWb1q17247jlLVQouuqlHi8VD4l3tpwk4yp+gzJlTFZc2YSDO2U9B7hO0oSlUbLbqqxnU77iyyavWm48rX2Z+723YcFSwy3mY78fQ46RLbSZSqNlp0lV9En3ovDdjLoslP2I6igsDm9SvoeWA2K1ueR1R0LdtxlKo2WnSVX3RKPYnM2AF0WzuWPbu2246jAty6r18BIOk0vQOVCi0BWXRF5BYR0SdUh5i4wfdSj/0sm/SI7SgqgBXkH6J99hSyYvvSPLGT7ThKVauALLpAU2CuiEwQkcEiIrYDqapr1/NY5tc9kR4bxrErZ7PtOCpALf5+PI3Zhelzte0oSlW7gCy6xph/Ax2AN4ERwEoReURE2lkNpqqs4ZAxxHCIFZMetB1FBajI+WPZQiN6DLrQdhSlql1AFl0AY4wBtrivQiABmCgiVboSx91zXiEiq0TkzmqIqiohsXMK8+NPo9fmCWzftN52HBVgNq5ZQo9D81mbeAERkZG24yhV7QKy6IrIaBGZBzwB/Az0MMbcAPQB/lqF7kYALwJnAF2BS0SkazVEVpXQYuj9ROBl9eT7bUdRAea3r1+i0Hhof7peQKVCU0AWXaABcL4x5nRjzMfGmAIAY4wXGFKF7vYDVhlj1hhj8oHxwNCqx1WV0fKYLsxvNITeOZ+wef0K23FUgDiUd4DOmz8jq+6xNG6RZDuOUjUiIIuuMeY+Y0yJxx6NMcuq0OmWwAafz9nud0eIyEgRyRCRjJycnCr0SpUl6bz7MHjY8Inu7SpH1jfvk8BeIvtdYzuKUjUmIIuuTcaY14wxqcaY1MaNG9uOE7KatmrHgqbnkbJzGhtWZdmOowJA7KJ32ShN6Xa8HnxSoSvciu5GoLXP51bud8qC9uffSz5RbP1sjO0oyrL1y+fTNT+LDW2H4YmIsB1HqRoTbkV3LtBBRNqKSDRwMfCZ5Uxhq1Gz1ixsOYyUPd+ydulc23GURZu/e5l8E0GH06+3HUWpGhVWRdcYUwjcDEwHlgETjDFL7KYKb13/+m/2E8PuqWNsR1GWHNyfS9dtU8mqdyINm7ayHUepGhVWRRfAGPOlMaajMaadMeZh23nCXf2GTVmceDm99//EyswfbcdRFmR9NZZ67CdmwHW2oyhV48Ku6KrA0+38O9lNXQ5Mf8B2FGVB/SXvs97Tiq5pg21HUarGadFV1tWLb8iytiNIPjiH5XO/sR1H+dHqRb/QqXA5m9tfjHh0daRCn87lKiAk//UOdlCfwm/0nszhZPuMV8kzUXQ5faTtKEr5hRZdFRBi69ZnZcfr6H4okyU/T7UdR/nBvr276L79f2TFn0z9hk1tx1HKL7ToqoDR67x/sI0GeH54GOP12o6jatiS6W9RR/KIO14voFLhQ4uuChgxteuwtusNdClYQtbMKbbjqBpkvF4aLh/HGk8SnfqcZDuOUn6jRVcFlN5Db2Uzjan906O6txvCVmbOpH3RanI6D9cLqFRY0bldBZToWjFsSL6VDoUryfzmA9txVA3ZM/MVDphadD39WttRlPIrLboq4KQMGcUGaUH99CfxFhXZjqOq2a6czfTc9Q1Zjc4grn4D23GU8istuirgREZFszXlbxzjXceC6WNtx1HVbMW0l6glBTQ5+WbbUZTyOy26KiD1PuMa1nna0CjjvxQVFtqOo6pJUWEhiWs+ZEl0T9p27Ws7jlJ+p0VXBaSIyEh29ruNRG8286e+ZjuOqiZZP3xMc3I41Ptq21GUskKLrgpYvU69nFUR7WiR+QwF+Ydsx1HVwJPxOttoQI+TL7UdRSkrtOiqgOWJiGD/cXfQ0mxlwWcv2o6jqmjDyoX0zJvH6sSLiIquZTuOUlZo0VUBreegi1gR2Zk2i1/kUN4B23FUFWz8+gXnQfVn3GQ7ilLWaNFVAU08HvIH3kUztpP5ybO246ijtD93N922fs6ieoNo1KyN7ThKWaNFVwW87sefzZLoHrRb/goH9+fajqOOwuL/vUGcHKTuCTfYjqKUVWFTdEXkSRFZLiKLRGSKiMTbzqQqRjwe5KR/04jdLJzylO04qpKM10uTZe+yOuIYOqWebDuOUlaFTdEFvga6G2N6Ar8C/7KcR1VC17TBLIrpQ6dVb7Jv7y7bcVQlLJs9nbbe9ezoeqXeZ1mFvbBZAowxXxljDt9lIR1oZTOPqryY0+4lgVyyJj9uO4qqhLyfX2YPdegx+BrbUZSyLmyKbjFXA9NKaiAiI0UkQ0QycnJy/BxLlaVjyiAWxB5Lt3XvsGenTptgsDV7NT1zf2RZ03OoXSfOdhylrAupoisi34jI4hJeQ33auRsoBMaV1A1jzGvGmFRjTGrjxo39FV1VUL0z7qMeB1g6+RHbUVQFrJn6DIKhzRl/tx1FqYAQaTtAdTLGnFJWcxEZAQwBTjbGGL+EUtWqXY805k8/kZ4bPmBXzh0kNG5uO5IqxYF9e+i6eRIL655ASlIn23GUCgghtadbFhEZDNwBnGOM0bssBLGGQ8YQwyFWTHrQdhRVhqwvX6U++4kdeIvtKEoFjLApusALQBzwtYhkisgrtgOpo5PYOYX58aeSvPljtm9abzuOKoG3qIgWy95mZWQHOvUt8wCUUmElbIquMaa9Maa1MaaX+xplO5M6es3Pvo8oClk95QHbUVQJsmZ8TGuziT3J1+nfhJTyoUuDCkqt2ndnfoMz6b3tE7ZsWGU7jiomYvbLbKMByaePsB1FqYCiRVcFrTbnjQFg/ZT77QZRf7B2yWy6H8pkddtL9WlCShWjRVcFrWZtOrCg8Tmk7JjKxjXLbMdRrpyvn+GgiabrkFttR1Eq4GjRVUHtmPPvowgPmz+5x3YUBezYmk3yrq9Z1OhM6jdsajuOUgFHi64Kao1bJLGgxTBS9nzD6qx023HC3q9Tn6WWFNDsNL0ZhlIl0aKrgl7XC8eQK7Hsn3q37Shh7eD+XDr/Np6FtfuT2KmX7ThKBSQtuiro1W/QmGXtR9IzL4PFP35qO07YWvT5CySwl6gT/2E7ilIBS4uuCgm9/no7W2hMzA/34y0qsh0n7BTkH6LN8jdZHtmFLv1Osx1HqYClRVeFhJjaddjQ+x+0L1rN/Glv2o4Tdhb+722ak0Ne/1v1ZhhKlUGXDhUyUs4ayRpPEi3mPcWhPL29tr8Yr5cGmS+zztOanicNsx1HqYCmRVeFjIjISPadcA8tzFYWTHnadpywsWjGJI7xrmNbj+vxRETYjqNUQNOiq0JKjxPPZ3GtXnRa8Qq5e3bajhMWImc9y1Ya0uvM62xHUSrgadFVIUU8Hmqd8RAJ7GXxBH0YQk1bkfEd3fKzWNvhSqJrxdiOo1TA06KrQk6HXicwL+4kemWPI2fTOttxQtqB755kL3Xofrbe8lGpitCiq0JSs/MeIYIi1k74l+0oIWvVwp/pfeAXlrS5jLr1EmzHUSooaNFVIanlMV2Y3/xiUndNY2Xmj7bjhKS9Xz3KXmLpet4dtqMoFTS06KqQ1fXiB9ktcRRO/SfG67UdJ6SsXTKblP0/sqT1pdRPaGQ7jlJBI+yKrojcJiJGRHRNEeLqxTdkZbe/0aVgCfP/947tOCFl57RH2Gdq0/Vc3ctVqjLCquiKSGvgNOA321mUf6SeN9q5Ycach8k7uN92nJCwftk8eufOIKvVMH18n1KVFFZFF3gauAMwtoMo/4iIjOTAyQ/RnBwyP3rYdpyQkPPlw+QRTZfz9CI1pSorbIquiAwFNhpjFpbT3kgRyRCRjJycHD+lUzWp+3Fns6DO8fRc+wbbN623HSeorV8+n5S937GwxTDiGzWzHUepoBNSRVdEvhGRxSW8hgJ3AfeW1w1jzGvGmFRjTGrjxo1rPrTyiyZ/fYJIilg74Z+2owS1HZ/fx0Fq0em8O21HUSoohVTRNcacYozpXvwFrAHaAgtFZB3QCpgvIrqpHiZaHtONeS0upu/uaSzP+NZ2nKC0csFMUvbPZFGby2nQpKXtOEoFpZAquqUxxmQZY5oYY5KMMUlANpBijNliOZryo56XPsw2GhA17f8oLMi3HSfo5P3vPnYRR/cL7rIdRamgFRZFVymAOnHxZPe/l3ZFa8iY+JTtOEFl8c+f0+PQfFZ0uI64+g1sx1EqaIVl0XX3eLfbzqH8r/fpV7IoJpVuy5/Ti6oqyHi9RH3/oPMkofP/z3YcpYJaWBZdFb7E4yHhgmeoRQHrxv/DdpygkPnNB3QqXMH6HrcQU7uO7ThKBTUtuirstG7fg/mtryR17zcs/vlz23ECWkH+IRrNepjfPC1JOecm23GUCnpadFVY6nXpA2ySpsR9eyf5h/JsxwlY8yY+SWuziZ3H3UtkVLTtOEoFPS26KizFxNYl54SHSPRmM29cuX/fDkt7dmyly68vkVUrheS/XGQ7jlIhQYuuClvJJ13EvLiT6LP+DdYunWs7TsBZNv5u6poD1B36BOLRVYVS1UGXJBXW2l7+AvulDgWTb6SosNB2nIDx26+Z9Nk2mYxG59C2a1/bcZQKGVp0VVhr0KQlq1PvoWPhr8zVByIcsWvKHeQRTfuLHrEdRamQokVXhb0+Z15LZuwAkn99gexVi23HsW7BV++TfHA2SzpcT8OmrWzHUSqkRNoOEGwKCgrIzs4mL0+veD1aMTExtGrViqioKNtRAOe/uy2Gv0zhawPYM2EULf45A09EhO1YVuzP3U3zX+5jrSeRPhfp7R6Vqm5adCspOzubuLg4kpKSEBHbcYKOMYYdO3aQnZ1N27Ztbcc5oknLtsxN/hd9F91L+viHSBt+n+1IVmSNu4s0trP89JeJiq5lO45SIUcPL1dSXl4eDRs21IJ7lESEhg0bBuSRgtRzb2FB7HGk/PocaxbPth3H79YumU3q5g+ZkzCEzv1Psx1HqZCkRfcoaMGtmkAdf+LxkDjidfZKXWTySPIO7rcdyW+8RUUc+uRv5EpdOg7/j+04SoUsLbpK+WjQpCUbBz5JW+86MsfeZjuO38z56BE6FyxlVa87iW+kj5lWqqZo0Q0Dn3zyCUuXLvV7f8eMGcNTTwXfI/SST7qI2Q3PJW3rh2TNmGw7To3bsHIhySueI7N2Gqnn3GA7jlIhTYtuGKjOolsYJjeQ6Hn1C6z1JNLq+9FszV5tO06NKSosZP+EURRIFC0vf1XvPKVUDdOrl6vg/s+XsHTT3mrtZtcW9bjv7G7ltnfuueeyYcMG8vLyGD16NCNHjqRu3brs27cPgIkTJ/LFF18wcuRIPvvsM2bMmMFDDz3EpEmTyM3NZdSoURw4cIB27drx1ltvkZCQwNy5c7nmmmvweDyceuqpTJs2jcWLFzN27FgmT57Mvn37KCoqYurUqQwdOpRdu3ZRUFDAQw89xNChQwF4+OGHeeedd2jSpAmtW7emT58+1Tp+/KV2nTgiLn6X6HGns/Wdy2hw+w8heTXv3PEPkVawlIyUx0htkWQ7jlIhTzdrg9Rbb73FvHnzyMjI4LnnnmPHjh0ltnfsscdyzjnn8OSTT5KZmUm7du244oorePzxx1m0aBE9evTg/vvvB+Cqq67i1VdfJTMzk4hi/1OdP38+EydOZMaMGcTExDBlyhTmz5/P999/z2233YYxhnnz5jF+/HgyMzP58ssvmTs3uO9n3KZjL5b3e5jOBUuZ9+Zo23Gq3crMH0lZ+RwLYo+jz5DrbcdRKiyE1Z6uiNwC3AQUAVONMXdUpXsV2SOtKc899xxTpkwBYMOGDaxcubJCv9uzZw+7d+/mxBNPBODKK6/kwgsvZPfu3eTm5jJgwAAALr30Ur744osjvzv11FNp0KAB4PzX9q677mLmzJl4PB42btzI1q1b+fHHHznvvPOIjY0F4Jxzzqm24bWlz1nXMnvtz6Rt/ZAF0wfQ+/QrbUeqFvv27qL2p9exS+Jpe/VbelhZKT8Jm6IrIn8BhgLJxphDItLEdqaj9cMPP/DNN98wa9YsYmNjGTRoEHl5eX/4K051/w+2Tp06R96PGzeOnJwc5s2bR1RUFElJSQH5v9vq0uvaF/n1qcV0+uV2VjfvQLuex9qOVCXG62X5G9fS27uFFWeMp6terayU34TT5u0NwGPGmEMAxphtlvMctT179pCQkEBsbCzLly8nPT0dgKZNm7Js2TK8Xu+RvWCAuLg4cnNzAahfvz4JCQn8+OOPALz33nuceOKJxMfHExcXx+zZzk0hxo8fX2b/mzRpQlRUFN9//z3r168HYODAgXzyySccPHiQ3NxcPv/88xoZfn+rFRNLg6snkit1qTP5crZv+c12pCqZO+U5Uvd+w5zEkXRNG2w7jlJhJZyKbkfgBBGZLSIzRKTE55WJyEgRyRCRjJycHD9HrJjBgwdTWFhIly5duPPOO0lLSwPgscceY8iQIRx77LE0b978SPsXX3wxTz75JL1792b16tW888473H777fTs2ZPMzEzuvdd5iPubb77JddddR69evdi/fz/169cvsf/Dhw8nIyODHj168O6779K5c2cAUlJSGDZsGMnJyZxxxhn07Rs6j4Rr1CKR3PPeo57JZccbF5B3YJ/tSEdl+Zyv6bXoAbJqpdDvCn2CkFL+JsYY2xmqjYh8A5R0rOxu4GHge+BWoC/wEXCMKWMEpKammoyMjD98t2zZMrp06VJtmQPJvn37qFu3LuAU8M2bN/Pss8/WSL+CdTzOn/4eKbNuZn7dgST/bQoRkcFzhmZr9moi3jiJPIkh7uaZ1G/Y1HYkFaJEZJ4xJtV2jkAUPGuMCjDGnFJaMxG5AZjsFtk5IuIFGgGBuTtrwdSpU3n00UcpLCwkMTGRsWPH2o4UcFJOv5z07WtIW/lf5rw0gr43vxsUFyEd3J/L3rHDaG4OsX/YZC24SlkS+GuL6vMJ8BcAEekIRAPbrSYKMMOGDSMzM5PFixczdepUGjdubDtSQEobfh+zWo6g387PSX8j8P9KVJB/iF9fOJ9jClax6oSnSewSnP+dVioUhFPRfQs4RkQWA+OBK8s6tKxUWdKueZrZDYcyYNO7zHrrDozXaztSibxFRWS+eBnJB+eQ0f0eep1yie1ISoW1sCm6xph8Y8xlxpjuxpgUY8x3tjOp4CUeD6k3vMXc+oMZ8NurpL8xOuAKr/F6mfvKSPru+YpZiaPof2H4PMBBqUAVNkVXqeoWERlJn1s/YHbDcxmw6V1mvzwSb1GR7ViAs4c758UR9M+ZSHrTS0i78lHbkZRSaNFVqko8ERH0u+lt0psMIy3nYzKfPpeD+3OtZirIP8S854fTf8enzGpxBf2vfykoLvZSKhzokhjGxo4dy80331xuO5s2bTry+dprr7XymMBAJh4P/Ue9QnqHf9Ar90c2PH0S2zett5Jlz67tLP/P6fTdPY1ZbUaSdu2zWnCVCiC6NKoyFS+6b7zxBl27drWYKDCJx0Pa8PtYeNwLtCpYD68NJGvmlPJ/WI02rFzI7udPpFPeIuYkP8iAq5/UgqtUgAmp/+n63bQ7YUtW9XazWQ8447FyWyvt0X6jR4/miy++oHbt2nz66ac0bdqUzz//nIceeoj8/HwaNmzIuHHjaNr09/9p5ubm0rNnT3799VeioqLYu3cvycnJPPHEE2RkZDB8+HBq167NrFmzOOOMM3jqqadITU3lf//7H3fddRdFRUU0atSIb7/9tnrHRRDqfdplrGvdGT4eQbdvr2LW8u/offljxNSuU/6Pj5Lxepn76Qt0z3yIQxLNqsHj6DfgjBrrn1Lq6OlmcJAq6dF++/fvJy0tjYULFzJw4EBef/11AI4//njS09NZsGABF198MU888cQfuhUXF8egQYOYOnUq4Nx3+fzzz+fCCy8kNTWVcePGkZmZSe3atY/8Jicnh+uuu45JkyaxcOFCPv74Y/8NfIBL6pJK09tmkdHgLAZsepecJ1JZ/NNnNdKvzetXkPnUWfRbeA9ra3Ui/9oZdNWCq1TA0j3dqqjAHmlNKenRftHR0QwZMgSAPn368PXXXwOQnZ3NsGHD2Lx5M/n5+bRt2/ZP3bv22mt54oknOPfcc3n77bePFOzSpKenM3DgwCPdOvzYP+WoXSeOfqPHkTVzCgnf30n3by5nwS/HEX/WfbTt1r/K3d+7ewdLJj9O8vqxxGNIb3crfS+9L6huS6lUONI93SDk+2i/hQsX0rt3b/Ly8oiKijryeL+IiAgKCwsBuOWWW7j55pvJysri1VdfLfExfMcddxzr1q3jhx9+oKioiO7du/t1mEJVj4Hn0ej2ecxqcz3t988nccLpZD4xmIXfTaDInT6VsXHNMma9Phqe6c6A315lRd2+7L76J9KueFALrlJBQJfSIFTao/3Kar9ly5YAvPPOO6W2d8UVV3DppZdyzz33HPnO97GAvtLS0rjxxhtZu3Ytbdu2ZefOnbq3W4qY2LoMuPoJ9uy4jTlTHqd99iQazbyOHTP/j9Xxx+Fp/xcad+hHq3bd/1Q4d2/fwqaVC9iz7HviN/9Il4KlNDfCwjrHEnf6XfROPt7SUCmljoYW3SA0ePBgXnnlFbp06UKnTp2OPNqvNGPGjOHCCy8kISGBk046ibVr15bY3vDhw/n3v//NJZf8fqvAESNGMGrUqCMXUh3WuHFjXnvtNc4//3y8Xi9NmjQ5cjhblax+w6akXftf8g89wvzvPsQs/ZTOu3+gXsaXkAGFxsMOieOgxBJpCojlAPEcIB7wGmFN5DGkJ91E21OuoXerdrYHRyl1FELq0X7VLdwe7Tdx4kQ+/fRT3nvvvRrvVyiPx8ooyD/EbysWsGPVXIq2r8JzcCeRBfvwRtTCGxmLSUiidvOOJPYcRHyjkp5aqVTg0Uf7lU73dBXgnPedNm0aX375pe0oYSUquhbteqTRrkfZRyuUUqFBi64C4Pnnn7cdQSmlQp5evXwU9JB81ej4U0qFKy26lRQTE8OOHTu0cBwlYww7duwgJibGdhSllPI7PbxcSa1atSI7O5ucnBzbUYJWTEwMrVq1sh1DKaX8TotuJUVFRZV4RyellFKqPGFzeFlEeolIuohkikiGiPSznUkppVR4CZuiCzwB3G+M6QXc635WSiml/Caciq4B6rnv6wObymhXKaWUqnZhc0cqEekCTAcEZ2PjWGPM+hLaGwmMdD92AlYcZS8bAduP8rfBSoc5POgwh4eqDHOiMaZxdYYJFSFVdEXkG6Cke+XdDZwMzDDGTBKRi4CRxphTajBLRrjdBk2HOTzoMIeHcBxmfwipq5fLKqIi8i4w2v34MfCGX0IppZRSrnA6p7sJONF9fxKw0mIWpZRSYSik9nTLcR3wrIhEAnn8ft62prxWw90PRDrM4UGHOTyE4zDXuJA6p6uUUkoFsnA6vKyUUkpZpUVXKaWU8hMtulUkIoNFZIWIrBKRO0toXktEPnKbzxaRJP+nrF4VGOZ/iMhSEVkkIt+KSKKNnNWpvGH2ae+vImJEJOj/alGRYRaRi9xpvUREPvB3xupWgXm7jYh8LyIL3Pn7TBs5q4uIvCUi20RkcSnNRUSec8fHIhFJ8XfGkGOM0ddRvoAIYDVwDBANLAS6FmvnRuAV9/3FwEe2c/thmP8CxLrvbwiHYXbbiwNmAulAqu3cfpjOHYAFQIL7uYnt3H4Y5teAG9z3XYF1tnNXcZgHAoccVloAAAOHSURBVCnA4lKanwlMw7mpUBow23bmYH/pnm7V9ANWGWPWGGPygfHA0GLtDAXecd9PBE4WEfFjxupW7jAbY743xhxwP6YDwf4cv4pMZ4AHgcdxro4PdhUZ5uuAF40xuwCMMdv8nLG6VWSYQ+p2ssaYmcDOMloZCrxrHOlAvIg090+60KRFt2paAht8Pme735XYjjGmENgDNPRLuppRkWH2dQ3OlnIwK3eY3cNurY0xU/0ZrAZVZDp3BDqKyM/uE7wG+y1dzajIMI8BLhORbOBL4Bb/RLOmssu7Kkc4/U9X+ZmIXAak8vtNSUKSiHiA/wIjLEfxt0icQ8yDcI5mzBSRHsaY3VZT1axLgLHGmP+IyADgPRHpbozx2g6mgoPu6VbNRqC1z+dW7ncltuPemKM+sMMv6WpGRYYZETkF557X5xhjDvkpW00pb5jjgO7ADyKyDufc12dBfjFVRaZzNvCZMabAGLMW+BWnCAerigzzNcAEAGPMLCAG58EAoapCy7uqOC26VTMX6CAibUUkGudCqc+KtfMZcKX7/gLgO+NeoRCkyh1mEekNvIpTcIP9PB+UM8zGmD3GmEbGmCRjTBLOeexzjDEZduJWi4rM25/g7OUiIo1wDjev8WfIalaRYf4N5+Eph59cFgPk+DWlf30GXOFexZwG7DHGbLYdKpjp4eUqMMYUisjNOI8MjADeMsYsEZEHgAxjzGfAmziHoFbhXLBwsb3EVVfBYX4SqAt87F4z9psx5hxroauogsMcUio4zNOB00RkKVAE3G6MCdqjOBUc5tuA10Xk7zgXVY0I5o1oEfkQZ8OpkXue+j4gCsAY8wrOeeszgVXAAeAqO0lDh94GUimllPITPbyslFJK+YkWXaWUUspPtOgqpZRSfqJFVymllPITLbpKKaWUn2jRVUoppf6/vTu0qSgIogB6JxgIQaGQeCwGSSFU8MsgoROCpgAkhjbogpBBvOd5ajebnFPBdTezm8wMonQBYBClCwupqvv9rul5VV3ud2zvZucCjrEcAxZTVc/Z1g9eJPnu7pfJkYCDlC4sZt8L/JXtbu9Dd/9OjgQc5HkZ1nOdbbf1VbaJF1iESRcWU1XvSd6S3Ca56e7T5EjAQa4MwUKq6inJT3e/VtVZks+qeuzuj9nZgP+ZdAFgEH+6ADCI0gWAQZQuAAyidAFgEKULAIMoXQAYROkCwCB/RA0LwsxVL1sAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The max absolute difference is: 1.77636e-15\n"
- ]
- }
- ],
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"\n",
@@ -1799,18 +1766,11 @@
},
{
"cell_type": "code",
- "execution_count": 29,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The gradient of f1 evaluated at a = 1 using autograd is: 3\n",
- "The gradient of f1 evaluated at a = 1 by finding the analytic expression is: 3\n"
- ]
- }
- ],
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -1844,23 +1804,11 @@
},
{
"cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Evaluating at x1 = 1, x2 = 3\n",
- "------------------------------\n",
- "The derivative of f2 w.r.t x1: 12\n",
- "The analytical derivative of f2 w.r.t x1: 12\n",
- "\n",
- "The derivative of f2 w.r.t x2: -4\n",
- "The analytical derivative of f2 w.r.t x2: -4\n"
- ]
- }
- ],
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -1909,18 +1857,11 @@
},
{
"cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The computed gradient of f3 is: [ 2. 3. 5. 7. 88.]\n",
- "The analytical gradient of f3 is: [ 2. 3. 5. 7. 88.]\n"
- ]
- }
- ],
+ "execution_count": 10,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -1958,18 +1899,11 @@
},
{
"cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The computed derivative of f4 at x = 2.7 is: 13.8759\n",
- "The analytical gradient of f4 at x = 2.7 is: 13.8759\n"
- ]
- }
- ],
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -1999,17 +1933,11 @@
},
{
"cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The computed derivative of f5 at x = 2.7 is: 5.4\n"
- ]
- }
- ],
+ "execution_count": 12,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -2067,17 +1995,11 @@
},
{
"cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The analytical derivative of f6 at x = 2.7 is: 37732.5\n"
- ]
- }
- ],
+ "execution_count": 13,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -2099,18 +2021,11 @@
},
{
"cell_type": "code",
- "execution_count": 18,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The computed derivative of f7 at n = 2 is: 1\n",
- "The analytical derivative of f7 at n = 2 is: 1\n"
- ]
- }
- ],
+ "execution_count": 14,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -2155,27 +2070,11 @@
},
{
"cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [
- {
- "ename": "TypeError",
- "evalue": "'ArrayBox' object does not support item assignment",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m8.4\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"The derivative of f8 is:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf8_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36mnary_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0margnum\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0munary_operator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munary_f\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnary_op_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnary_op_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 21\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_operator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/differential_operators.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0marguments\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbut\u001b[0m \u001b[0mreturns\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgradient\u001b[0m \u001b[0minstead\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m should be scalar-valued. The gradient has the same type as the argument.\"\"\"\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0mvjp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mans\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 25\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mans\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m raise TypeError(\"Grad only applies to real scalar-output functions. \"\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/core.py\u001b[0m in \u001b[0;36mmake_vjp\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVJPNode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_root\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_node\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/tracer.py\u001b[0m in \u001b[0;36mtrace\u001b[0;34m(start_node, fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtrace_stack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_node\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_box\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_box\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mstart_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_node\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36munary_f\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0msubargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubvals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msubargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m\u001b[0m in \u001b[0;36mf8\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mautograd\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mf8\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# Assume x is an array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mTypeError\u001b[0m: 'ArrayBox' object does not support item assignment"
- ]
- }
- ],
+ "execution_count": 15,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -2201,27 +2100,11 @@
},
{
"cell_type": "code",
- "execution_count": 20,
- "metadata": {},
- "outputs": [
- {
- "ename": "AttributeError",
- "evalue": "'ArrayBox' object has no attribute 'dot'",
- "output_type": "error",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0.0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"The derivative of f9 is:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf9_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36mnary_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0margnum\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0munary_operator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munary_f\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnary_op_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnary_op_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 21\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_operator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/differential_operators.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0marguments\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbut\u001b[0m \u001b[0mreturns\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgradient\u001b[0m \u001b[0minstead\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m should be scalar-valued. The gradient has the same type as the argument.\"\"\"\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0mvjp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mans\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 25\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mans\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m raise TypeError(\"Grad only applies to real scalar-output functions. \"\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/core.py\u001b[0m in \u001b[0;36mmake_vjp\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVJPNode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_root\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_node\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/tracer.py\u001b[0m in \u001b[0;36mtrace\u001b[0;34m(start_node, fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtrace_stack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_node\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_box\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_box\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mstart_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_node\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36munary_f\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0msubargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubvals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msubargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;32m\u001b[0m in \u001b[0;36mf9\u001b[0;34m(a)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mf9\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# Assume a is an array with 2 elements\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2.0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mf9_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf9\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
- "\u001b[0;31mAttributeError\u001b[0m: 'ArrayBox' object has no attribute 'dot'"
- ]
- }
- ],
+ "execution_count": 16,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import autograd.numpy as np\n",
"from autograd import grad\n",
@@ -2248,7 +2131,9 @@
{
"cell_type": "code",
"execution_count": 17,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -2278,7 +2163,9 @@
{
"cell_type": "code",
"execution_count": 18,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"a += b\n",
@@ -2599,7 +2486,9 @@
{
"cell_type": "code",
"execution_count": 19,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -2634,7 +2523,9 @@
{
"cell_type": "code",
"execution_count": 20,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"pt.axis(\"equal\")\n",
@@ -2652,7 +2543,9 @@
{
"cell_type": "code",
"execution_count": 21,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"x = guesses[-1]\n",
@@ -2669,7 +2562,9 @@
{
"cell_type": "code",
"execution_count": 22,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"def f1d(alpha):\n",
@@ -2691,7 +2586,9 @@
{
"cell_type": "code",
"execution_count": 23,
- "metadata": {},
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"pt.axis(\"equal\")\n",
@@ -3130,25 +3027,7 @@
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.4"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
diff --git a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz
index 134512084..72d439fbf 100644
Binary files a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz and b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz differ
diff --git a/doc/pub/Splines/pdf/Splines-minted.pdf b/doc/pub/Splines/pdf/Splines-minted.pdf
index 7c47294d4..61d97dce6 100644
Binary files a/doc/pub/Splines/pdf/Splines-minted.pdf and b/doc/pub/Splines/pdf/Splines-minted.pdf differ
diff --git a/doc/src/LogisticRegression/LogReg.do.txt b/doc/src/LogisticRegression/LogReg.do.txt
index f2fe7576e..4aa899374 100644
--- a/doc/src/LogisticRegression/LogReg.do.txt
+++ b/doc/src/LogisticRegression/LogReg.do.txt
@@ -450,7 +450,8 @@ if __name__ == "__main__":
!split
===== The Credit Card example =====
-Here we use the the "credit card data":"https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients". More text to come.
+Here we use the the "credit card data":"https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients".
+The data are from an extensive database from Taiwan and include more than ten predictors.
!bc pycod
import pandas as pd
diff --git a/doc/src/Splines/Splines.do.txt b/doc/src/Splines/Splines.do.txt
index 35bbeb89a..0c30f2a66 100644
--- a/doc/src/Splines/Splines.do.txt
+++ b/doc/src/Splines/Splines.do.txt
@@ -881,15 +881,29 @@ _Challenge_: try to write a similar code for a Logistic Regression case.
!split
===== Momentum based GD =====
-The stochastic gradient descent (SGD) is almost always used with a *momentum* or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically
-implemented as follows
+The stochastic gradient descent (SGD) is almost always used with a
+*momentum* or inertia term that serves as a memory of the direction we
+are moving in parameter space. This is typically implemented as
+follows
+
!bt
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\
\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t},
\end{align}
!et
-where we have introduced a momentum parameter $\gamma$, with $0\le\gamma\le 1$, and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that $\mathbf{v}_t$ is a running average of recently encountered gradients and $(1-\gamma)^{-1}$ sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when $\gamma=0$, this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is
+
+where we have introduced a momentum parameter $\gamma$, with
+$0\le\gamma\le 1$, and for brevity we dropped the explicit notation to
+indicate the gradient is to be taken over a different mini-batch at
+each step. We call this algorithm gradient descent with momentum
+(GDM). From these equations, it is clear that $\mathbf{v}_t$ is a
+running average of recently encountered gradients and
+$(1-\gamma)^{-1}$ sets the characteristic time scale for the memory
+used in the averaging procedure. Consistent with this, when
+$\gamma=0$, this just reduces down to ordinary SGD as discussed
+earlier. An equivalent way of writing the updates is
+
!bt
\[
\Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t),
@@ -900,20 +914,28 @@ where we have defined $\Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\bo
!split
===== More on momentum based approaches =====
-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass $m$ moving in a viscous medium with drag coefficient $\mu$ and potential
-$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$, then its motion is described by
+Let us try to get more intuition from these equations. It is helpful
+to consider a simple physical analogy with a particle of mass $m$
+moving in a viscous medium with drag coefficient $\mu$ and potential
+$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$,
+then its motion is described by
+
!bt
\[
m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}).
\]
!et
+
We can discretize this equation in the usual way to get
+
!bt
\[
m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}).
\]
!et
+
Rearranging this equation, we can rewrite this as
+
!bt
\[
\Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t.
@@ -922,26 +944,51 @@ Rearranging this equation, we can rewrite this as
!split
===== Momentum parameter =====
-Notice that this equation is identical to previous one if we identify the position of the particle, $\mathbf{w}$, with the parameters $\boldsymbol{\theta}$. This allows
-us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:
+
+Notice that this equation is identical to previous one if we identify
+the position of the particle, $\mathbf{w}$, with the parameters
+$\boldsymbol{\theta}$. This allows us to identify the momentum
+parameter and learning rate with the mass of the particle and the
+viscous drag as:
+
!bt
\[
\gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}.
\]
!et
-Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$.
-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models.
+Thus, as the name suggests, the momentum parameter is proportional to
+the mass of the particle and effectively provides inertia.
+Furthermore, in the large viscosity/small learning rate limit, our
+memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$.
-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG).
+Why is momentum useful? SGD momentum helps the gradient descent
+algorithm gain speed in directions with persistent but small gradients
+even in the presence of stochasticity, while suppressing oscillations
+in high-curvature directions. This becomes especially important in
+situations where the landscape is shallow and flat in some directions
+and narrow and steep in others. It has been argued that first-order
+methods (with appropriate initial conditions) can perform comparable
+to more expensive second order methods, especially in the context of
+complex deep learning models.
+
+These beneficial properties of momentum can sometimes become even more
+pronounced by using a slight modification of the classical momentum
+algorithm called Nesterov Accelerated Gradient (NAG).
+
+In the NAG algorithm, rather than calculating the gradient at the
+current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one
+calculates the gradient at the expected value of the parameters given
+our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma
+\mathbf{v}_{t-1})$. This yields the NAG update rule
-In the NAG algorithm, rather than calculating the gradient at the current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one calculates the gradient at the expected value of the parameters given our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1})$. This yields the NAG update rule
!bt
\begin{align}
\mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\
\boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}.
\end{align}
!et
+
One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\gamma$.
@@ -965,12 +1012,19 @@ adaptively change the step size to match the landscape without paying
the steep computational price of calculating or approximating
Hessians.
-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.
+Recently, a number of methods have been introduced that accomplish
+this by tracking not only the gradient, but also the second moment of
+the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and
+ADAM.
!split
===== RMS prop =====
-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule for RMS prop is given by
+In RMS prop, in addition to keeping a running average of the first
+moment of the gradient, we also keep track of the second moment
+denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule
+for RMS prop is given by
+
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
@@ -978,13 +1032,34 @@ In RMS prop, in addition to keeping a running average of the first moment of the
\boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber
\end{align}
!et
-where $\beta$ controls the averaging time of the second moment and is typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.
+
+where $\beta$ controls the averaging time of the second moment and is
+typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate
+typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a
+small regularization constant to prevent divergences. Multiplication
+and division by vectors is understood as an element-wise operation. It
+is clear from this formula that the learning rate is reduced in
+directions where the norm of the gradient is consistently large. This
+greatly speeds up the convergence by allowing us to use a larger
+learning rate for flat directions.
!split
===== ADAM optimizer =====
-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and $\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)
+A related algorithm is the ADAM optimizer. In ADAM, we keep a running
+average of both the first and second moment of the gradient and use
+this information to adaptively change the learning rate for different
+parameters. In addition to keeping a running average of the first and
+second moments of the gradient
+(i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and
+$\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM
+performs an additional bias correction to account for the fact that we
+are estimating the first two moments of the gradient using a running
+average (denoted by the hats in the update rule below). The update
+rule for ADAM is given by (where multiplication and division are once
+again understood to be element-wise operations below)
+
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
@@ -995,9 +1070,18 @@ A related algorithm is the ADAM optimizer. In ADAM, we keep a running average o
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\end{align}
!et
-where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance $\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The update rule for this parameter is given by
+where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and
+second moment and are typically taken to be $0.9$ and $0.99$
+respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
+
+Like in RMSprop, the effective step size of a parameter depends on the
+magnitude of its gradient squared. To understand this better, let us
+rewrite this expression in terms of the variance
+$\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
+(\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The
+update rule for this parameter is given by
+
!bt
\[
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.