From c3af27517e1d30c1cfa2fd8402305355a42f6144 Mon Sep 17 00:00:00 2001 From: mhjensen Date: Sat, 28 Sep 2019 19:51:45 +0200 Subject: [PATCH] Adding to do issues on log reg --- doc/pub/Splines/html/._Splines-bs000.html | 2 +- doc/pub/Splines/html/._Splines-bs036.html | 20 +- doc/pub/Splines/html/._Splines-bs037.html | 12 +- doc/pub/Splines/html/._Splines-bs038.html | 38 +- doc/pub/Splines/html/._Splines-bs039.html | 5 +- doc/pub/Splines/html/._Splines-bs040.html | 17 +- doc/pub/Splines/html/._Splines-bs041.html | 27 +- doc/pub/Splines/html/Splines-bs.html | 2 +- doc/pub/Splines/html/Splines-reveal.html | 121 ++++- doc/pub/Splines/html/Splines-solarized.html | 121 ++++- doc/pub/Splines/html/Splines.html | 121 ++++- doc/pub/Splines/ipynb/Splines.ipynb | 497 +++++++----------- .../Splines/ipynb/ipynb-Splines-src.tar.gz | Bin 211 -> 211 bytes doc/pub/Splines/pdf/Splines-minted.pdf | Bin 358263 -> 358328 bytes doc/src/LogisticRegression/LogReg.do.txt | 3 +- doc/src/Splines/Splines.do.txt | 118 ++++- 16 files changed, 704 insertions(+), 400 deletions(-) diff --git a/doc/pub/Splines/html/._Splines-bs000.html b/doc/pub/Splines/html/._Splines-bs000.html index 02f35b586..74abae7d1 100644 --- a/doc/pub/Splines/html/._Splines-bs000.html +++ b/doc/pub/Splines/html/._Splines-bs000.html @@ -295,7 +295,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 26, 2019

+

Sep 28, 2019


diff --git a/doc/pub/Splines/html/._Splines-bs036.html b/doc/pub/Splines/html/._Splines-bs036.html index 1eee01c60..139a36876 100644 --- a/doc/pub/Splines/html/._Splines-bs036.html +++ b/doc/pub/Splines/html/._Splines-bs036.html @@ -279,8 +279,11 @@ MathJax.Hub.Config({

Momentum based GD

-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically -implemented as follows +The stochastic gradient descent (SGD) is almost always used with a +momentum or inertia term that serves as a memory of the direction we +are moving in parameter space. This is typically implemented as +follows + $$ \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\ @@ -289,7 +292,18 @@ $$ \end{align} $$ -where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +

+where we have introduced a momentum parameter \( \gamma \), with +\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to +indicate the gradient is to be taken over a different mini-batch at +each step. We call this algorithm gradient descent with momentum +(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a +running average of recently encountered gradients and +\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory +used in the averaging procedure. Consistent with this, when +\( \gamma=0 \), this just reduces down to ordinary SGD as discussed +earlier. An equivalent way of writing the updates is + $$ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), $$ diff --git a/doc/pub/Splines/html/._Splines-bs037.html b/doc/pub/Splines/html/._Splines-bs037.html index 6bc14f3ca..fee482924 100644 --- a/doc/pub/Splines/html/._Splines-bs037.html +++ b/doc/pub/Splines/html/._Splines-bs037.html @@ -279,18 +279,26 @@ MathJax.Hub.Config({

More on momentum based approaches

-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential -\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by +Let us try to get more intuition from these equations. It is helpful +to consider a simple physical analogy with a particle of mass \( m \) +moving in a viscous medium with drag coefficient \( \mu \) and potential +\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), +then its motion is described by + $$ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). $$ +

We can discretize this equation in the usual way to get + $$ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). $$ +

Rearranging this equation, we can rewrite this as + $$ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. $$ diff --git a/doc/pub/Splines/html/._Splines-bs038.html b/doc/pub/Splines/html/._Splines-bs038.html index 6b884699c..4cc1b9d05 100644 --- a/doc/pub/Splines/html/._Splines-bs038.html +++ b/doc/pub/Splines/html/._Splines-bs038.html @@ -277,22 +277,47 @@ MathJax.Hub.Config({

Momentum parameter

-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows -us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: + +

+Notice that this equation is identical to previous one if we identify +the position of the particle, \( \mathbf{w} \), with the parameters +\( \boldsymbol{\theta} \). This allows us to identify the momentum +parameter and learning rate with the mass of the particle and the +viscous drag as: + $$ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. $$ -Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \). +

+Thus, as the name suggests, the momentum parameter is proportional to +the mass of the particle and effectively provides inertia. +Furthermore, in the large viscosity/small learning rate limit, our +memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).

-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +Why is momentum useful? SGD momentum helps the gradient descent +algorithm gain speed in directions with persistent but small gradients +even in the presence of stochasticity, while suppressing oscillations +in high-curvature directions. This becomes especially important in +situations where the landscape is shallow and flat in some directions +and narrow and steep in others. It has been argued that first-order +methods (with appropriate initial conditions) can perform comparable +to more expensive second order methods, especially in the context of +complex deep learning models.

-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +These beneficial properties of momentum can sometimes become even more +pronounced by using a slight modification of the classical momentum +algorithm called Nesterov Accelerated Gradient (NAG).

-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule +In the NAG algorithm, rather than calculating the gradient at the +current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one +calculates the gradient at the expected value of the parameters given +our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma +\mathbf{v}_{t-1}) \). This yields the NAG update rule + $$ \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\ @@ -301,6 +326,7 @@ $$ \end{align} $$ +

One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).

diff --git a/doc/pub/Splines/html/._Splines-bs039.html b/doc/pub/Splines/html/._Splines-bs039.html index 976c5d5e4..b4201bb3d 100644 --- a/doc/pub/Splines/html/._Splines-bs039.html +++ b/doc/pub/Splines/html/._Splines-bs039.html @@ -296,7 +296,10 @@ the steep computational price of calculating or approximating Hessians.

-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +Recently, a number of methods have been introduced that accomplish +this by tracking not only the gradient, but also the second moment of +the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and +ADAM.

diff --git a/doc/pub/Splines/html/._Splines-bs040.html b/doc/pub/Splines/html/._Splines-bs040.html index 4ca347247..eb50aa535 100644 --- a/doc/pub/Splines/html/._Splines-bs040.html +++ b/doc/pub/Splines/html/._Splines-bs040.html @@ -279,7 +279,11 @@ MathJax.Hub.Config({

RMS prop

-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by +In RMS prop, in addition to keeping a running average of the first +moment of the gradient, we also keep track of the second moment +denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule +for RMS prop is given by + $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) @@ -289,7 +293,16 @@ $$ \end{align} $$ -where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. +

+where \( \beta \) controls the averaging time of the second moment and is +typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a +small regularization constant to prevent divergences. Multiplication +and division by vectors is understood as an element-wise operation. It +is clear from this formula that the learning rate is reduced in +directions where the norm of the gradient is consistently large. This +greatly speeds up the convergence by allowing us to use a larger +learning rate for flat directions.

diff --git a/doc/pub/Splines/html/._Splines-bs041.html b/doc/pub/Splines/html/._Splines-bs041.html index 7bf93b4b6..ed4deaeb5 100644 --- a/doc/pub/Splines/html/._Splines-bs041.html +++ b/doc/pub/Splines/html/._Splines-bs041.html @@ -279,7 +279,19 @@ MathJax.Hub.Config({

ADAM optimizer

-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +A related algorithm is the ADAM optimizer. In ADAM, we keep a running +average of both the first and second moment of the gradient and use +this information to adaptively change the learning rate for different +parameters. In addition to keeping a running average of the first and +second moments of the gradient +(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and +\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM +performs an additional bias correction to account for the fact that we +are estimating the first two moments of the gradient using a running +average (denoted by the hats in the update rule below). The update +rule for ADAM is given by (where multiplication and division are once +again understood to be element-wise operations below) + $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) @@ -293,10 +305,19 @@ $$ \end{align} $$ -where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. +

+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) +respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by +Like in RMSprop, the effective step size of a parameter depends on the +magnitude of its gradient squared. To understand this better, let us +rewrite this expression in terms of the variance +\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - +(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +update rule for this parameter is given by + $$ \Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$ diff --git a/doc/pub/Splines/html/Splines-bs.html b/doc/pub/Splines/html/Splines-bs.html index 02f35b586..74abae7d1 100644 --- a/doc/pub/Splines/html/Splines-bs.html +++ b/doc/pub/Splines/html/Splines-bs.html @@ -295,7 +295,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 26, 2019

+

Sep 28, 2019


diff --git a/doc/pub/Splines/html/Splines-reveal.html b/doc/pub/Splines/html/Splines-reveal.html index 66ff38f9e..c23723298 100644 --- a/doc/pub/Splines/html/Splines-reveal.html +++ b/doc/pub/Splines/html/Splines-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

 
-

Sep 26, 2019

+

Sep 28, 2019


@@ -1187,8 +1187,11 @@ plt.show()

Momentum based GD

-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically -implemented as follows +The stochastic gradient descent (SGD) is almost always used with a +momentum or inertia term that serves as a memory of the direction we +are moving in parameter space. This is typically implemented as +follows +

 
$$ \begin{align} @@ -1199,7 +1202,18 @@ $$ $$

 
-where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +

+where we have introduced a momentum parameter \( \gamma \), with +\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to +indicate the gradient is to be taken over a different mini-batch at +each step. We call this algorithm gradient descent with momentum +(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a +running average of recently encountered gradients and +\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory +used in the averaging procedure. Consistent with this, when +\( \gamma=0 \), this just reduces down to ordinary SGD as discussed +earlier. An equivalent way of writing the updates is +

 
$$ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), @@ -1214,22 +1228,30 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\

More on momentum based approaches

-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential -\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by +Let us try to get more intuition from these equations. It is helpful +to consider a simple physical analogy with a particle of mass \( m \) +moving in a viscous medium with drag coefficient \( \mu \) and potential +\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), +then its motion is described by +

 
$$ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). $$

 
+

We can discretize this equation in the usual way to get +

 
$$ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). $$

 
+

Rearranging this equation, we can rewrite this as +

 
$$ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. @@ -1240,24 +1262,49 @@ $$

Momentum parameter

-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows -us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: + +

+Notice that this equation is identical to previous one if we identify +the position of the particle, \( \mathbf{w} \), with the parameters +\( \boldsymbol{\theta} \). This allows us to identify the momentum +parameter and learning rate with the mass of the particle and the +viscous drag as: +

 
$$ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. $$

 
-Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \). +

+Thus, as the name suggests, the momentum parameter is proportional to +the mass of the particle and effectively provides inertia. +Furthermore, in the large viscosity/small learning rate limit, our +memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).

-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +Why is momentum useful? SGD momentum helps the gradient descent +algorithm gain speed in directions with persistent but small gradients +even in the presence of stochasticity, while suppressing oscillations +in high-curvature directions. This becomes especially important in +situations where the landscape is shallow and flat in some directions +and narrow and steep in others. It has been argued that first-order +methods (with appropriate initial conditions) can perform comparable +to more expensive second order methods, especially in the context of +complex deep learning models.

-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +These beneficial properties of momentum can sometimes become even more +pronounced by using a slight modification of the classical momentum +algorithm called Nesterov Accelerated Gradient (NAG).

-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule +In the NAG algorithm, rather than calculating the gradient at the +current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one +calculates the gradient at the expected value of the parameters given +our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma +\mathbf{v}_{t-1}) \). This yields the NAG update rule +

 
$$ \begin{align} @@ -1268,6 +1315,7 @@ $$ $$

 
+

One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).

@@ -1293,7 +1341,10 @@ the steep computational price of calculating or approximating Hessians.

-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +Recently, a number of methods have been introduced that accomplish +this by tracking not only the gradient, but also the second moment of +the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and +ADAM. @@ -1301,7 +1352,11 @@ Recently, a number of methods have been introduced that accomplish this by track

RMS prop

-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by +In RMS prop, in addition to keeping a running average of the first +moment of the gradient, we also keep track of the second moment +denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule +for RMS prop is given by +

 
$$ \begin{align} @@ -1313,7 +1368,16 @@ $$ $$

 
-where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. +

+where \( \beta \) controls the averaging time of the second moment and is +typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a +small regularization constant to prevent divergences. Multiplication +and division by vectors is understood as an element-wise operation. It +is clear from this formula that the learning rate is reduced in +directions where the norm of the gradient is consistently large. This +greatly speeds up the convergence by allowing us to use a larger +learning rate for flat directions. @@ -1321,7 +1385,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica

ADAM optimizer

-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +A related algorithm is the ADAM optimizer. In ADAM, we keep a running +average of both the first and second moment of the gradient and use +this information to adaptively change the learning rate for different +parameters. In addition to keeping a running average of the first and +second moments of the gradient +(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and +\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM +performs an additional bias correction to account for the fact that we +are estimating the first two moments of the gradient using a running +average (denoted by the hats in the update rule below). The update +rule for ADAM is given by (where multiplication and division are once +again understood to be element-wise operations below) +

 
$$ \begin{align} @@ -1337,10 +1413,19 @@ $$ $$

 
-where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. +

+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) +respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by +Like in RMSprop, the effective step size of a parameter depends on the +magnitude of its gradient squared. To understand this better, let us +rewrite this expression in terms of the variance +\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - +(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +update rule for this parameter is given by +

 
$$ \Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. diff --git a/doc/pub/Splines/html/Splines-solarized.html b/doc/pub/Splines/html/Splines-solarized.html index ebd7ee050..37c4d214d 100644 --- a/doc/pub/Splines/html/Splines-solarized.html +++ b/doc/pub/Splines/html/Splines-solarized.html @@ -213,7 +213,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 26, 2019

+

Sep 28, 2019












@@ -1175,8 +1175,11 @@ plt.show()

Momentum based GD

-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically -implemented as follows +The stochastic gradient descent (SGD) is almost always used with a +momentum or inertia term that serves as a memory of the direction we +are moving in parameter space. This is typically implemented as +follows + $$ \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\ @@ -1185,7 +1188,18 @@ $$ \end{align} $$ -where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +

+where we have introduced a momentum parameter \( \gamma \), with +\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to +indicate the gradient is to be taken over a different mini-batch at +each step. We call this algorithm gradient descent with momentum +(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a +running average of recently encountered gradients and +\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory +used in the averaging procedure. Consistent with this, when +\( \gamma=0 \), this just reduces down to ordinary SGD as discussed +earlier. An equivalent way of writing the updates is + $$ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), $$ @@ -1198,18 +1212,26 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\

More on momentum based approaches

-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential -\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by +Let us try to get more intuition from these equations. It is helpful +to consider a simple physical analogy with a particle of mass \( m \) +moving in a viscous medium with drag coefficient \( \mu \) and potential +\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), +then its motion is described by + $$ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). $$ +

We can discretize this equation in the usual way to get + $$ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). $$ +

Rearranging this equation, we can rewrite this as + $$ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. $$ @@ -1218,22 +1240,47 @@ $$









Momentum parameter

-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows -us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: + +

+Notice that this equation is identical to previous one if we identify +the position of the particle, \( \mathbf{w} \), with the parameters +\( \boldsymbol{\theta} \). This allows us to identify the momentum +parameter and learning rate with the mass of the particle and the +viscous drag as: + $$ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. $$ -Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \). +

+Thus, as the name suggests, the momentum parameter is proportional to +the mass of the particle and effectively provides inertia. +Furthermore, in the large viscosity/small learning rate limit, our +memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).

-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +Why is momentum useful? SGD momentum helps the gradient descent +algorithm gain speed in directions with persistent but small gradients +even in the presence of stochasticity, while suppressing oscillations +in high-curvature directions. This becomes especially important in +situations where the landscape is shallow and flat in some directions +and narrow and steep in others. It has been argued that first-order +methods (with appropriate initial conditions) can perform comparable +to more expensive second order methods, especially in the context of +complex deep learning models.

-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +These beneficial properties of momentum can sometimes become even more +pronounced by using a slight modification of the classical momentum +algorithm called Nesterov Accelerated Gradient (NAG).

-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule +In the NAG algorithm, rather than calculating the gradient at the +current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one +calculates the gradient at the expected value of the parameters given +our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma +\mathbf{v}_{t-1}) \). This yields the NAG update rule + $$ \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\ @@ -1242,6 +1289,7 @@ $$ \end{align} $$ +

One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).

@@ -1267,7 +1315,10 @@ the steep computational price of calculating or approximating Hessians.

-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +Recently, a number of methods have been introduced that accomplish +this by tracking not only the gradient, but also the second moment of +the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and +ADAM.











@@ -1275,7 +1326,11 @@ Recently, a number of methods have been introduced that accomplish this by track

RMS prop

-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by +In RMS prop, in addition to keeping a running average of the first +moment of the gradient, we also keep track of the second moment +denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule +for RMS prop is given by + $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) @@ -1285,7 +1340,16 @@ $$ \end{align} $$ -where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. +

+where \( \beta \) controls the averaging time of the second moment and is +typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a +small regularization constant to prevent divergences. Multiplication +and division by vectors is understood as an element-wise operation. It +is clear from this formula that the learning rate is reduced in +directions where the norm of the gradient is consistently large. This +greatly speeds up the convergence by allowing us to use a larger +learning rate for flat directions.











@@ -1293,7 +1357,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica

ADAM optimizer

-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +A related algorithm is the ADAM optimizer. In ADAM, we keep a running +average of both the first and second moment of the gradient and use +this information to adaptively change the learning rate for different +parameters. In addition to keeping a running average of the first and +second moments of the gradient +(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and +\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM +performs an additional bias correction to account for the fact that we +are estimating the first two moments of the gradient using a running +average (denoted by the hats in the update rule below). The update +rule for ADAM is given by (where multiplication and division are once +again understood to be element-wise operations below) + $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) @@ -1307,10 +1383,19 @@ $$ \end{align} $$ -where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. +

+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) +respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by +Like in RMSprop, the effective step size of a parameter depends on the +magnitude of its gradient squared. To understand this better, let us +rewrite this expression in terms of the variance +\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - +(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +update rule for this parameter is given by + $$ \Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$ diff --git a/doc/pub/Splines/html/Splines.html b/doc/pub/Splines/html/Splines.html index 3bcf473b9..2ec8fb04b 100644 --- a/doc/pub/Splines/html/Splines.html +++ b/doc/pub/Splines/html/Splines.html @@ -218,7 +218,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 26, 2019

+

Sep 28, 2019












@@ -1180,8 +1180,11 @@ plt.show()

Momentum based GD

-The stochastic gradient descent (SGD) is almost always used with a momentum or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically -implemented as follows +The stochastic gradient descent (SGD) is almost always used with a +momentum or inertia term that serves as a memory of the direction we +are moving in parameter space. This is typically implemented as +follows + $$ \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\ @@ -1190,7 +1193,18 @@ $$ \end{align} $$ -where we have introduced a momentum parameter \( \gamma \), with \( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a running average of recently encountered gradients and \( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when \( \gamma=0 \), this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is +

+where we have introduced a momentum parameter \( \gamma \), with +\( 0\le\gamma\le 1 \), and for brevity we dropped the explicit notation to +indicate the gradient is to be taken over a different mini-batch at +each step. We call this algorithm gradient descent with momentum +(GDM). From these equations, it is clear that \( \mathbf{v}_t \) is a +running average of recently encountered gradients and +\( (1-\gamma)^{-1} \) sets the characteristic time scale for the memory +used in the averaging procedure. Consistent with this, when +\( \gamma=0 \), this just reduces down to ordinary SGD as discussed +earlier. An equivalent way of writing the updates is + $$ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), $$ @@ -1203,18 +1217,26 @@ where we have defined \( \Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\

More on momentum based approaches

-Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass \( m \) moving in a viscous medium with drag coefficient \( \mu \) and potential -\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), then its motion is described by +Let us try to get more intuition from these equations. It is helpful +to consider a simple physical analogy with a particle of mass \( m \) +moving in a viscous medium with drag coefficient \( \mu \) and potential +\( E(\mathbf{w}) \). If we denote the particle's position by \( \mathbf{w} \), +then its motion is described by + $$ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). $$ +

We can discretize this equation in the usual way to get + $$ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). $$ +

Rearranging this equation, we can rewrite this as + $$ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. $$ @@ -1223,22 +1245,47 @@ $$









Momentum parameter

-Notice that this equation is identical to previous one if we identify the position of the particle, \( \mathbf{w} \), with the parameters \( \boldsymbol{\theta} \). This allows -us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: + +

+Notice that this equation is identical to previous one if we identify +the position of the particle, \( \mathbf{w} \), with the parameters +\( \boldsymbol{\theta} \). This allows us to identify the momentum +parameter and learning rate with the mass of the particle and the +viscous drag as: + $$ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. $$ -Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \). +

+Thus, as the name suggests, the momentum parameter is proportional to +the mass of the particle and effectively provides inertia. +Furthermore, in the large viscosity/small learning rate limit, our +memory time scales as \( (1-\gamma)^{-1} \approx m/(\mu \Delta t) \).

-Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +Why is momentum useful? SGD momentum helps the gradient descent +algorithm gain speed in directions with persistent but small gradients +even in the presence of stochasticity, while suppressing oscillations +in high-curvature directions. This becomes especially important in +situations where the landscape is shallow and flat in some directions +and narrow and steep in others. It has been argued that first-order +methods (with appropriate initial conditions) can perform comparable +to more expensive second order methods, especially in the context of +complex deep learning models.

-These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +These beneficial properties of momentum can sometimes become even more +pronounced by using a slight modification of the classical momentum +algorithm called Nesterov Accelerated Gradient (NAG).

-In the NAG algorithm, rather than calculating the gradient at the current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one calculates the gradient at the expected value of the parameters given our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \). This yields the NAG update rule +In the NAG algorithm, rather than calculating the gradient at the +current parameters, \( \nabla_\theta E(\boldsymbol{\theta}_t) \), one +calculates the gradient at the expected value of the parameters given +our current momentum, \( \nabla_\theta E(\boldsymbol{\theta}_t +\gamma +\mathbf{v}_{t-1}) \). This yields the NAG update rule + $$ \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\ @@ -1247,6 +1294,7 @@ $$ \end{align} $$ +

One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of \( \gamma \).

@@ -1272,7 +1320,10 @@ the steep computational price of calculating or approximating Hessians.

-Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +Recently, a number of methods have been introduced that accomplish +this by tracking not only the gradient, but also the second moment of +the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and +ADAM.











@@ -1280,7 +1331,11 @@ Recently, a number of methods have been introduced that accomplish this by track

RMS prop

-In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule for RMS prop is given by +In RMS prop, in addition to keeping a running average of the first +moment of the gradient, we also keep track of the second moment +denoted by \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2] \). The update rule +for RMS prop is given by + $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) @@ -1290,7 +1345,16 @@ $$ \end{align} $$ -where \( \beta \) controls the averaging time of the second moment and is typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. +

+where \( \beta \) controls the averaging time of the second moment and is +typically taken to be about \( \beta=0.9 \), \( \eta_t \) is a learning rate +typically chosen to be \( 10^{-3} \), and \( \epsilon\sim 10^{-8} \) is a +small regularization constant to prevent divergences. Multiplication +and division by vectors is understood as an element-wise operation. It +is clear from this formula that the learning rate is reduced in +directions where the norm of the gradient is consistently large. This +greatly speeds up the convergence by allowing us to use a larger +learning rate for flat directions.











@@ -1298,7 +1362,19 @@ where \( \beta \) controls the averaging time of the second moment and is typica

ADAM optimizer

-A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +A related algorithm is the ADAM optimizer. In ADAM, we keep a running +average of both the first and second moment of the gradient and use +this information to adaptively change the learning rate for different +parameters. In addition to keeping a running average of the first and +second moments of the gradient +(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and +\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM +performs an additional bias correction to account for the fact that we +are estimating the first two moments of the gradient using a running +average (denoted by the hats in the update rule below). The update +rule for ADAM is given by (where multiplication and division are once +again understood to be element-wise operations below) + $$ \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) @@ -1312,10 +1388,19 @@ $$ \end{align} $$ -where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. +

+where \( \beta_1 \) and \( \beta_2 \) set the memory lifetime of the first and +second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) +respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.

-Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance \( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The update rule for this parameter is given by +Like in RMSprop, the effective step size of a parameter depends on the +magnitude of its gradient squared. To understand this better, let us +rewrite this expression in terms of the variance +\( \boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - +(\hat{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The +update rule for this parameter is given by + $$ \Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. $$ diff --git a/doc/pub/Splines/ipynb/Splines.ipynb b/doc/pub/Splines/ipynb/Splines.ipynb index 63e448d1e..688dd3851 100644 --- a/doc/pub/Splines/ipynb/Splines.ipynb +++ b/doc/pub/Splines/ipynb/Splines.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 26, 2019**\n", + "Date: **Sep 28, 2019**\n", "\n", "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -724,30 +724,11 @@ }, { "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[[3.7347544 ]\n", - " [3.28652545]]\n", - "[[3.7347544 ]\n", - " [3.28652545]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmYFPW1//H3mWEdRYUBFIEZXFE0xgWjUWMwiVGMN5pdM+4aLphrNjWJ8kuMJsQkZjOLcQvRyMRoNNckXkncBlAQcERQdpVNFmUXBBxg5vz+qBpshu6e6unqbebzep5+6K71VE1Tp+t7vlVl7o6IiEhrygodgIiIlAYlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlD8s7MlpjZJ8L3N5rZvQWKY5iZLS/EujsKM7vPzH5U6DgkHkoYshszu8DMppnZFjNbHb6/2swsF+tz9x+7+1XZLsfMBpmZm1mnOOIqNB1opRgpYcguZnYtcDtwG3AAsD8wEjgV6JJinvK8BSgiBaWEIQCY2b7ALcDV7v6Iu2/2wMvuXuPuDeF095nZH8zsCTPbApxhZp8ys5fNbJOZvWlmP2ix7IvNbKmZrTOz0S3G/cDMxiV8PtnMppjZRjObZWbDEsZNMLMfmtlkM9tsZk+aWe9w9KTw341m9q6ZfTjJNnYP499gZnOBE1uMP9DMHjWzNWa22My+ljDuQ2ZWH27j22b2y4RxpyXE/KaZXRYO72pmPzezZeE8d5pZ93DcMDNbbmbXhmdyq8zs8nDcCKAG+Ha4Lf9K8Tc7wsyeMrP1ZrbAzL4YDu9iZjPN7Jrwc3m4z76fsC0vhPGuMrPfmVmXhOV6eFb5Wriff2hmh4TbuMnMHm6ePmE7bjSztWFzY02yeMPpzw1j2xgu75hU00oRcne99AI4G9gJdGpluvuAdwjOOsqAbsAw4APh52OAt4Hzw+mHAO8CpwNdgV+G6/lEOP4HwLjwfX9gHXBOuKwzw899wvETgDeAw4Hu4eefhOMGAZ4ufuAnwHNAL2AgMBtYHo4rA14Cvk9wNnUwsAg4Kxz/AnBx+H5v4OTwfTWwGbgQ6AxUAseG434F/DNcXw/gX8Ct4bhh4X64JZzvHGAr0DNhP/8ozbbsBbwJXA50Ao4D1gJDwvFHAxuAI4HRwFSgPBx3AnByON8gYB7wjYRlO/APYB/gKKABeCbcJ/sCc4FLW2zHL8O/70eBLcDgltsRxrgaOAkoBy4FlgBdC/391yvaS2cY0qw3sNbddzYPSPjVvM3MTk+Y9h/uPtndm9z9PXef4O6vhp9fAR4kOHAAfB543N0neXCW8j2gKUUMFwFPuPsT4bKeAuoJDqbN/uTuC919G/AwcGwG2/hFYIy7r3f3N4HfJIw7kSAx3eLu2919EXAPcEE4fgdwqJn1dvd33X1qOPzLwNPu/qC773D3de4+M6z5jAC+Ga5vM/DjhOU1L/OWcL4nCBLr4Ijbci6wxN3/5O473f1l4FHgCwDuPhv4EfAYcB1BsmsMx73k7lPD+ZYAd/H+36vZz9x9k7vPIUisT7r7Ind/BxhPcPBP9D13b3D3icD/hfu6pRHAXe4+zd0b3f1+gmR0csRtlgJTwpBm64DeiUVjdz/F3fcLxyV+V95MnNHMTjKzurAp5x2CukdzU9GBidO7+5ZweclUA18Ik9RGM9sInAb0S5jmrYT3Wwl+7Ue1WyzA0hbrPrDFum8kqOMAXElwZjPfzF40s3PD4QMJznpa6gNUAC8lLO/f4fBm6xITdIbbUw2c1CLeGoLaU7P7w+mecPfXmgea2eFm9riZvWVmmwgSWW9293bC+21JPifGuSH8uzZbSrCvk8V8bYuYB6aYVoqQEoY0e4Hg1955EaZteYvjvxA0vQx0932BO4HmXlWrCA4KAJhZBUGzTTJvAg+4+34Jr73c/SdtiCmZ3WIBqlqse3GLdfdw93MA3P01d78Q6Av8FHjEzJqbhQ5Jsq61BAfWoxKWt6+7R00IrW3Pm8DEFvHu7e6jEqa5A3gcOMvMTksY/gdgPnCYu+9DkBiz6QXXM9wXzaqAlSliHtMi5gp3fzCLdUseKWEIAO6+EbgZuMPMPm9mPcyszMyOJWgvT6cHsN7d3zOzDxE00zR7BDg3LAx3IWizT/W9Gwf8l5mdFRZqu4VF1QERNmENQVPXwWmmeRi4wcx6hsu8JmHcdGCzmX0nLI6Xm9nRZnYigJldZGZ93L0J2BjO0wTUAp8wsy+aWSczqzSzY8Pp7gF+ZWZ9w2X0N7OzImwLBL/o023L48DhFnQo6By+TjSzI8N1XUxQq7gM+Bpwv5k1J6sewCbgXTM7Ahi15+IzdnNYbP8IQXPZ35JMcw8wMjwjNTPby4IOEz1iWL/kgRKG7OLuPwO+BXyb4ID1NkH79neAKWlmvRq4xcw2ExSNH05Y5hzgqwRnIasICrFJL5YL6wrnEfziXUPwi/R6InxP3X0rMAaYHDZ3JGsXv5mguWQx8CTwQML8jQQHumPD8WuBewmKvBB0CphjZu8SdD2+wN23ufsyghrLtcB6YCbwwXCe7wCvA1PDpp+niV6j+CMwJNyWx5Js72bgkwQ1kZUETXU/BbqaWRXwa+CSsN7yF4Ja0K/C2a8jSOqbCQ7iD0WMKZW3CP6uKwkS6Eh3n58k5nrgK8DvwulfJ0hoUiLMXQ9QEpG2saDb8zh3j3IWKCVOZxgiIhKJEoaIiESiJikREYlEZxgiIhJJSd3Zs3fv3j5o0KBChyEiUlJeeumlte7ep/Up0yuphDFo0CDq6+sLHYaISEkxs6WtT9U6NUmJiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISSc4ThpmNNbPVZjY7ybhrzczNrOUD6EVEpMjk4wzjPoLHW+7GzAYSPGJyWR5iEBGRLOU8Ybj7JIJnHbf0K4JnR+uBHCIiJaAgNQwzOw9Y4e6zIkw7wszqzax+zZo1eYhORESSyXvCMLMK4Ebg+1Gmd/e73X2ouw/t0yfr27mLiEgbFeIM4xDgIGCWmS0BBgAzzOyAAsQiIiIR5f0BSu7+KtC3+XOYNIa6+9p8xyIiItHlo1vtg8ALwGAzW25mV+Z6nSIiEr+cn2G4+4WtjB+U6xhERCR7utJbREQiUcIQEZFIlDBERHKpthYGDYKysuDf2tpCR9Rmee8lJSLSYdTWwogRsHVr8Hnp0uAzQE1N4eJqI51hiIjkyujR7yeLZlu3BsNLkBKGiEiuLEtxb9VUw4ucEoaISK5UVWU2vMgpYYiI5MqYMVBRsfuwiopgeAlSwhARyZWaGrj7bqiuBrPg37vvLsmCN6iXlIhIbtXUlGyCaElnGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESS84RhZmPNbLWZzU4YdpuZzTezV8zsf81sv1zHISIi2cnHGcZ9wNkthj0FHO3uxwALgRvyEIeIiGQh5wnD3ScB61sMe9Ldd4YfpwIDch2HiIhkpxhqGFcA41ONNLMRZlZvZvVr1qzJY1giIpKooAnDzEYDO4HaVNO4+93uPtTdh/bp0yd/wYmIyG4K9jwMM7sMOBf4uLt7oeIQEZFoCnKGYWZnA98GPu3uWwsRg4hIu1NbC4MGQVlZ8G9tysabNslHt9oHgReAwWa23MyuBH4H9ACeMrOZZnZnruMQEYkkxwfdnKmthREjYOlScA/+HTEi1vitlFqDhg4d6vX19YUOQ0Taq+aD7taEho+KitJ4DvegQUGSaKm6Glu69CV3H5rtKoqhl5SISHEYPXr3ZAHB59GjCxNPJpYty2x4GyhhiIg0y8NBNydqa4MmtGSqqmJbjRKGiEizVAfXGA+6sWtuRmts3HNcRQWMGRPbqpQwRESajRkTHGQTxXzQjV2yZjSA8vLYay9KGCIizWpqgoNsdTWYBf8We8E7VXNZU1PscRfswj0RkaJUU1PcCaKlqqrkvaNy0IymMwwRkVKWx2Y0JQwRaV9K9cK7tspjM5qapESk/Wh54V3z1c5QWs1MmcpTM5rOMESk/SjlC+9KgBKGiLQfpXrhXYlQwhCR+BWqjlCKF96VECUMEYlXHu6amlIpXnhXQpQwRCRehawjlOKFdyVEtzcXkXiVlQVnFi2ZBVcfS96ZmW5vLiJFSHWEdksJQ0Tila6O0NEuqmtndOGeiMSruV4wenTQnbWq6v2ic0e8qK6ANizeyNO/nRfb8lTDEJH8SPMIUZYsyXc076ut3TO5lWgCa9rZxMyHFjD+vrcZP60XUzcPoZFOQDw1DJ1hiEh+FONFde3gViLr39jAk7fP499PNPHvxYfzdtORwJGcUDGXG057nuEXVXLqyHjWlfMzDDMbC5wLrHb3o8NhvYCHgEHAEuCL7r6htWXpDEOkhBXjGUYxxtSKpp1NzPjLfMbfv5rx0yuZ9u4Qmiinl63nkwPnM/ysJs762mD2P7rPrnlKqZfUfcDZLYZ9F3jG3Q8Dngk/i0h7VowX1RXjWU8Saxes4y9fncwlhzxPvy7rOPHSIdz07Ons9DIeGnwT2/fqyVqv5MFlp3LJ389n/1lP5iSOnCcMd58ErG8x+Dzg/vD9/cD5uY5DRAosXxfVZdITq0i7ADftbGL6n+Zw8xkTOHnv2fQ9oic1d5zKE4uP5BNVC3lg5GTenrue6XfN5POLb6Pzlo1Y88zr1sFFF8HVV8cfmLvn/EXQ9DQ74fPGhPeW+DnJvCOAeqC+qqrKRURSGjfOvaLCPbh0MHhVVATD45i+eZ7qanez4N9002Yw/+q5a/yBkc97zaDnvbetcXA3Gv2kvV71Hwyr82ljZ/vOhp27L6u6evfYE19mu5YN1Hscx/I4FtLqStIkjPDzhijLOeGEEzL7w4hIx5LqAFpdnXqeTBJAWxJMK/M3gT9c/iU3Gh3c+9hqv+ig57z26ud9zfy16ZdnljphJGx3XAkjL91qzWwQ8Li/X/ReAAxz91Vm1g+Y4O6DW1uOit4iklaub0uSZZG88cCBlK9avsfwJuCxIaOp+s6XOf7LR1DWKWK1IFU8zcLtLqWidzL/BC4N318K/KNAcYhIe5LrmkSGRfLG7Y1MuetVvveRCQzday62akXS6cqAz24Zx9BLhkRPFhB0GOjSJfX4mGsxOU8YZvYg8AIw2MyWm9mVwE+AM83sNeAT4WcRkfRaK2jnuidWhIT01iurue+q57mgagp9um3i1JEf4MfPf4Ru5TvZ1K1v6mW3pWdWTQ2MHQt77bXnuFz0QIujXStfL9UwRDqwqPWDbIvSGcbQVFHh8y79sd94Sp0f133urlEHlL3llx06yR/6xmRfv2jD+/Onqjukq7NEjS3FdlNKNYy4qIYh0oEVy0V2tbXsvO67lL+1grXlfbmh8Yf8ka9Qzk5O2WcOw0/ewPAr+vHBLxyOldme8199Ndx55+61loqKnD63o9RrGCLSUbX1jrUFvMhux9YdTPrtLG748ASOvWoond96kzKaONZfxg87gr996wXWLtnCpHc+yA3/GcaxXxqcPFkA3HEHPPBAaT7kKY7TlHy91CQlUuKy6Zbali6zWVj+4kq/55JJ/rn+U3wfNjq4d2K739J9jG/str83mXlTVVX6azxy1TSWIUrpOoy4XkoYIgmK6IAUWTYH/WyvgUi2vIT9t2Ps/V73q5f92x+q8w90W7BrFf3LVvpVgyf6o9e/4Ft+c0/0OkqcsWZJCUOkIyuyA1JkqQq+ZtHmjytJJtl/W+juFzLOO9PgZ+w3w392Tp2/+veF3tTY9P58URNens+GWhNXwlDRW6QUFUsBOFMFjnv7u9t5/q45HHfjcHpuf3uP8Vv3OYDGeQvpcWCP5AuIemFgkT3XXEVvkY6sRO6yuocCPL516eTl3PnlSZzfbxqVPRr4+HXHse/21Umnrdj8dupkAdEvDCzSmxpmSwlDioOe9ZyZUj0gpbpjLQQPLlq6NPhl3vwgozZ8Dxo2NfD0z2Zw7dAJDOn6BoNOG8CoB09n1toDufiol/nHjdNgwIDkM7e2/6JeGFiMt3KPQxztWvl6qYbRTpVqe3whtbd9lmWb/6KJy/w/J4/2t8sO8EbMF1PtF3Ofn9mr3n95Xp3P+783dq9FZLP/otZRiqhTAip6S7tRZAXCklFEB6SspbvjahLbNmzz//y43r9x3AQf3OUNv5Bx/i4trsDu3j23tykvIUoY0n5k23NGSl95efLvQHn5rklef2aJ//bzE/ycPtO9O1sc3Luyzc+qfNHf6b5/YX905OgZGXGJK2GohiGFV6rt8RKfxsakg72xka8fO5HDuyzm0I9Xc80jH2Xhxj5cdcyL/N8PXmT9mib+ffsC9tm2Z48nYM9OALmoldXWZld/yXb+fIoj6+TrpTOMdqq9tcdL5lI0Sy6m2rux1Yf3me6/+dwEf+3pJbvPl+y7k+oMI47vWbIzgWybVPPQJIuapKRd6UDtyUm11+1vZbu2rNni//eD6f6nAaN9C913O2A2WFef+flbfOu6ramXn+4RpS2TQbYH5lQJJ9X6ozap5qFJVglDpL1or2dYKW4FvuL6X/qvzp/gn6x80buyzcG9O1v8Zz1u9ncqgns0RU6a6R5R2nL+bA/MqRJOqvqLzjCUMERiF8cBoxjPUNI0M4H7EV1e928eX+f/+XG9b9uwLdZ1JN132e7ndMkp5ud8x/2DQQlDpL2I4/5KRXSG0tTY5HP/9bo3kXy7mjBfNHFZtIW1lggz2fZU044aFS3Zpks4HaSXVMGTQCYvJQzJWDH+8m6pBIqmrdm8arP/48apPnLIRK8uf9PBfTFZxpWqoF1ZufvfMZO/cctpR43KPuEU43eqBSUMkdaUyn/wbOMswHUsTY1NPvux1/y2T9X5x3q+5J1pcHDfm01+3gFT/c4vT/TV3/9NdtuVSUG7rTJNtuPGBQkrVfIqUkoYIq0pgl/ekWVzJpSn7dy0YpP/73en+ogjJnpVeBYB7kd3XejXn1jnz/x8hjdsbth9pmy2K13NIK7tyzTZlsqPkBbaRcIAvgnMAWYDDwLd0k2vhCEZKfUryDO5Z1EODmJNjU3+yiML/KfD63zYfjO8E9sd3Hvwjn+m3wt+98UTfdnUFVmtI610Zxhx/R0zTbal9CMkQcknDKA/sBjoHn5+GLgs3TxKGJKREv3P7e6ZJ4GYajXvvPmOP3r9C37V4Ik+oHzFrlUf022+f+ekOq/71ct7nkXkSiYX5cW5juYfGsn2Y4n+CGkvCeNNoBfQCXgc+GS6eZQwOrBUB8R0B8pSaj5ouR2J7eQ5THZNjU0+86H5futZdX76vi/vOovYh43+uf5T/N5LJ/nyF1fGus6MtKwZ5OLvmHi1dsuEEPfFf/mQ5P9EyScMD5LG14F3gTVAbYppRgD1QH1VVVWMe1VKRrrukK0lhFLoJdXaL+mYf8luWLLR//atKX7FYZP8wLKVuxZ9bPd5fsOH63zib2b69i3bY9iwGOXj7xglGRT7j5AU8fWGRV7KCQPoCTwL9AE6A48BF6WbR2cYHVSurrAtFq211We5bU2NTT7jL/N8zJl1fto+M72cHQ7u+7LRvzBgio+9fJKvfPmttsVeCgk5qqjNTcW8zSm+S8dAg5d4wvgC8MeEz5cAd6SbRwmjg2qtt0yJtSfvIer2ZfBLdv2iDf7QNyb7ZYdO8gPK3tq1iOO7z/XRp9b5c7+f5Tu27cgu7mL/tZ1MuoN9KTQ3tSbFd+kEcC/xhHFS2EOqAjDgfuCadPMoYXRQHfUMo7Iy8i/Zxh2NXv/AXP/hx+v8lB6zvIydDu49bb1/aeBkv++q53zVrLfzE3ex7v/WElwpJsCW2usZhgdJ42Zgftit9gGga7rplTA6qGxqGPmOsy1NFW08UK17fb0/eM1kv+Tg53z/srd3zTq0Yo5/7yN1PvnOV7I/i0in1HoMRa1RFGtzUxTttYbRlpcSRgfWll5S+Y4vk+6ZyeZvZTsadzT69Pvm+M1n1PnJe7+y6yyil63zC6uf9z//93O+8Sd/aHvSynS+PPXkik2pJbi2KmQvKeAp4INxrCzblxKGFK3WCtdtPPNZM3+t1179vF900HPex1YHxzca/UN7veo3fbTOX7jnVd/ZsDOYuK1NKm2Zb9w4986d99zOLl2K91d5qTWhxSiuhGHBslIzs+OBXwBLgBvdfVXaGXJo6NChXl9fX6jVi6RWVhYcftKproYlS9JO0ri9kfoH5jH+gbWMr+/Ni1uG4JTR29ZyVvUChg+HT14zmD5H9t5z5kGDgsd7ZrretsyXap7KSli7NvW6Cqn5Uahbt74/rKIC7r4bamoKF1cemNlL7j406wVFzSzA54BZwE2EV2fn+6UzDCkaUS+0a63pY9w439lvgDdhvrp8f/8Kd+46izh571f85jPqfNrY2e+fRaTT1iaXtsyXyYOLikmxNGHmGfm8cI+gF9PRwEhgLbAcuDiOADJ5KWFIJLk+KCRrwuncOWiOSZcwysuDBNGw06fc9Yo/PHi0b6XbbtO8Z1198pk3+dqF6zKPq61NLm2Zr7UmuBK5i2tHkbeEAUwGVhLUMn4InAscCvwWuDuOIKK+lDCkVfnoGtlaN9jEgneL13vW1b/CXQ7uS6hq2wE+rm3P5JYYUdYVU91G4pfPhHEUBLWOJOPmxRFE1JcShrQqH4XNCE04O//0Z28sK0s63ZpO+/tfvzY5eHZ1W5qQ0snmDreZ9OpqXkZrzXAdoKBcCuJKGK0WvdMxs4PdfVGbF5AhFb2lVamKz2bQ1BTPOlIUfHf2G8C4cx5k/JNlPLl8COu8F2WkiaWtReo4xLXuVMtpFud+lzaLq+hdls3M+UwWIpFUVWU2vC3GjAl61yTYRjcuWfUTLv/jaTy34mA+c+irbOvRN30sSZZDRUUwPNeWLctseCrJtiFRnPtdCi6rhCFSdHJ8EF454y3GPl3F7V2uZxkDacJYShW3df8ex5zVn5f/uoAVO/Zn7MKPsNcffpE+lpqaoEtndXXwS7y6On9dPONKrM3bUFm557h8JT/JnzjatfL1Ug1DIomxl9T2Ldt9wq9f9u+cVOfHdJu/q2m+f9lKv/LwSf7IdVN849KNeYklVtl0Dij2q+5lD7SH52Fk+lLC6KDiPhC1srw3p6/0ey6Z5J898AXfh40O7p3Y7sP2m+E/HV7ns/62wJsam7KLoRi0Zb+2hxv0dUBxJYysit75pqJ3BxT31blJlufdK5j7mRu5/7VTGf9Kf2Y3HAbAgPKVDD/sdYZ/ugsfv2YI+wzYJ9utKX2FLNRLm8VV9FbCkOIW9wEqxfKWUM3hLOQjPWcz/JRNnH1lf44671CszDJfR3uWj15oErui6CUleVBbGxzkysqCf2trCx1RfsXUm6dhUwPP3DYDX5p8vmqWsW5FA8+sP57rHh/G0Z85LLtk0V7/bvnohSZFSwmjmDU3nyxdGvyqW7o0+NxeDj5RZHGAWjp5OXd+eRLn9ZtG5b47+MS3j2cZA5NOa9VV9DiwRzaRvq89/90K2RVYCi+OQki+Xh2u6N2Bb8e8SwZF1u333Odb9zvAmzBfzoF+IeMc3Ad1Wuajjpro//x/03zb7/9YuFuH5OLvVoieSeoNVXJQL6kOoKM88KU1aQ5QiyYu899/aYLf1uNmf5fuu+2n7eVdffl1v9yzR1OuD3j5+rupx5JEFFfCUNG7mGVT8K2thdGjg7b+qqqgyaAd3PP/vY3vMfH3sxn/yBbGz6li4Y6DAFjOAPqzYs8ZCtF7J189idRjSSJS0bsjaGt7cTtrQ3/j2aX87gsT+VTfF+nVs4mz/99Q7pr5IQ7edy23f3YiC/69mANtZfKZM73VRRzy1c4f1+09RKKK4zQlX68O1yTl3rbmkxKvfWxdt9WfuHm6X3PMBD+08+Jd4R/aebFfc8wEf+Lm6b5lzZbdZyq2bc5HO3+xbbMULdpDDQPYD3gEmA/MAz6cbvoOmTDaothqHxEOngufXOy3f3aCn917undjq4N7d7b4OX2m+28/P8Ffe3pJ6+voaO35HXGbpU3aS8K4H7gqfN8F2C/d9EoYERX6l2digqisDJ5G1+Kg9t4fxvrjN033r35ggh/SacmuUYd3XuRfP3aC//tHL/rWdVvbvt5MftWXcq+fUo5d8iauhFGworeZ7QvMBA72iEF0uKJ3WxXyYffJ1p3EGnrRl3VUsIWP9Z3D8DO2cfbIgzh4WJ4vACvkvhLJk7iK3oU8uzgWmA7cB7wM3AvslWS6EUA9UF9VVRVjzm3nCvXLs7VnPYevJvBXvvhD37ZhW2bLj3u7Cn02JpIHtIMzjKHAVOBUd59mZrcDm9z9e6nm0RlGcfMmh07lWNTvVLLun+m6A+fibED3RpIOoD10q10OLHf3aeHnR4DjCxiPJKqthd69gwOnWfA+Sbfcd996l3/cOI2RQyZxUJcVLPUMmpQSu382r++ii1J3Bx49es+mrq1bg+FtpXsjiURWsITh7m8Bb5rZ4HDQx4G5hYonJ0r1BnS1tXD55bBu3fvD1q2DK67Ax9Uy5x+v8/NzJ/DxXjPo1a8L5996ErXzjuO4vst545SLaeraPdp6ysqCV+/ee66vWWJCiHLdQab7XPdGEokujnattr4I6hj1wCvAY0DPdNOXVC+pUu7ymKYO8Sb9d308uutCv/7EOn/2FzO8YXPD+/M31xki1DIivZq7A7dWb2jrPldPI2nnaA/dajN9FX3CSDzwlJenP7gVsaZU13GAN2J+zyWTfNnUFa0vKK6k0bzPRo1KPn7UqPTrK4F9LpJLcSUM3RokLi1vx9HYmHy6Ir1twzvL3uHR61/gnn7fp9FTfy3Kqqu46v6PMPCkA1tfaKrmnkwkNg898UTyaZqH61YZIjmlhBGXZAXZZIqkmOpNzsyHFvCTsyfw0f1m0ru6gkd/voiat35OJ1Ikuy5dMmvbr6kJejBVVweF8+rq9z9HUVm5ew+o1hKCCtgiuRXHaUq+XkXdJJWmGadYahgblmz0h785xS8/bJL3K1u1K6xju8/zGz5c59t69ksVZj3JAAAO+ElEQVQde2VlfLEnqzV06RKsI10dIVc1DJF2DtUwYpZt4TPVway8vGDF1MYdjf7SuLk+5sw6P22fmV7ODgf3/WyDf3HgZP/Tlc/5ypffen+GfN6Dqi37O0pCUAFbZA9KGHGK45dpkfy6Xb9og//1a5P90kOe8/3L3t4VyvHd5/roU+v8+Ttm+Y5tO5LPXApFYyUEkYwpYcQprgNlPg9m4bqazPy9yn7+9yGj/ZQes7yMnQ7uvWydX1A12e//ynO+atbb0ZdZBElPROIVV8LQE/eg5G4PsfkXd9P9u1+j086GXcO2UMGtnW+i7OQPMfyiSk7qOpOym76X+RP32umT+kQ6spK/+WBbXnk/w8hl/SGDs5HGHY0+bexsv/mMOj9571d8CVXpz4hGjdqzHqEzBZEOCzVJxShZU0wuezhFaPpZM3+tjxv1vNcMet5725qg9kyjf2ivV72JNMXpceNSF6+LqRYhInkTV8JQk1SzxKaYsrLkF94lu7tqWwwaFFzg10JDr37c+oG/ML6+Ny9uGYJTRh9bw1mDFjB8uHHm/wymz5G9U86/6/qGZOOgbU1saqISKXlxNUkpYSST65pGiuU3YXRiJyftPYfhJ65j+KV9OaHmCMo6tbi+Mt1tvi++OHnskHnC08OFRNqF9nB78+KVoyuGG7c3MuWuV9nYpW/S8dt69GXNwo28sPkDfP/ZYZx46ZA9kwWkvoK6piZ1jGaZ34E1F7cTF5GSpYSRTDa3vG5xe+13fnon93/leS6omkKfbps4deQH+GrDbWyj2x7L3+sPv6DysF7RYqypCc4WmpqCf5t/8SeL3QxGjsz8rED3ZhKRBEoYyaT7BZ9ObS2eeAPCpUvp9N1r+c+9S5m44hDOO2Q2D31jCr99/Vy6j7s38+VH1T3heRSVlfDAA3DHHZkvR/dmEpFEcVTO8/Uq1ntJrXz5LR97+SRfXb5/0t5JDb0P9MYdjbkPJO4L7/J5IZ+u4BbJGdStto1iODBt37LdJ/12pt/w4To/tvu8XcfSxnTdXfMhF9eT5ONArivMRXIqroTRsXpJZdHrZ0X9Kv79u9cZ/3Qnnl4xhHfYl3J2cuq+sxl+8kaGX9GPY67/JJasfT+u7ritSdW7K1Ex9nJK1004H/tNpJ3Tld5tkcE9o7Zv2e4Tfv2yf+ekOj+m2/xdk/YvW+lXDZ7oj17/gm9cunH3mXL1Sznqr/yoT7grtgv48nmXXJEOCDVJtUErB6Y3p6/0uy+e6J/p94L34B0H905s92H7zfCfDq/zVx5Z4E2NTenXEXcTTiZJKMoV68V4IC6Fu+SKlLC4EkbHapJK0fSxocv+nG7PMbvhMAAGlq9g+GFvMPz8rnzsq0eyz4B92r7ObGXaXJPPK9bjogsERXKq3Vy4Z2blZvaymT2e85WNGUNT1+67DdpCBV/b/lP6Vmzmtk9NYPZjr7N0+4HcNe90zr/1pMImC8j8WojE6zPuv7/t15PkU1u7MYtIXhU8YQBfB+blauENmxp45rYZXDd0AkddeRIXNdzDEqppwljXaX/mf/rb3LHiszyz/niue3wYR513KFZmuQpndy0u8qO2ds9psrkWopQOxKkuRBSRotGpkCs3swHAp4AxwLfiWu6S55cz/veLGF/XlWffPootHE8XGji912yGntWfbV95BjvnYCrLjMq4Vpqpls0wS5cGn2H3g+WYMcmba6KeJdTU6OArIvGIoxDS1hfwCHACMAx4PMU0I4B6oL6qqippQWfbhm3+5K31/s3j6/yILq/vqpke1GmpX330BP/X96b55lWb21ovSi2bAncmhV5d1CYiWaDUi95mdi5wjrtfbWbDgOvc/dx08yQWvRdNWMb4OxYzfmJ36lYfxVb2oivv8dHK2Qz/yBaGjxjI4WcdlLvmpdpauOIK2L599+GjRkW7DUeJPeVPREpXyd/e3MxuBS4GdgLdgH2Av7v7RanmObz/ED+n7x8YP6eKhTsOAuCQTksZfuQShn+ugmFfPYqK3hWpZo9X796wbl3ycePGtd4MpIvVRCRPSj5h7BZExDMMs6HejecY1ns2w0/fyvCR1Rx25qC8xJgkmNTjohz01ZVURPIkroRR0KJ3pg47cCuzXoXuvU4sdCjpRbn9d3NS0NPsRKREFMUZRlR5e+JeFOmapNSsJCJFpN1cuFeybr89KFy31KVL8V0YJyISAyWMtqqpgT//OXhAUbPKShg7Vs1KItIulVQNo+joojgR6UB0hiEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikXSshBHl+RMiIpJUx+lWG/X5EyIiklTHOcMYPXr3G/1B8Hn06MLEIyJSYjpOwsj02dgiIrKbjpMwsnk2toiIdKCEMWZM8LyJRJk8G1tEpIPrOAmjpiZ4OFF1dfDwo+pqPaxIRCQDHaeXFOhmgSIiWeg4ZxgiIpIVJQwREYlECUNERCJRwhARkUgKljDMbKCZ1ZnZXDObY2ZfL1QsIiLSukL2ktoJXOvuM8ysB/CSmT3l7nMLGJOIiKRQsDMMd1/l7jPC95uBeUD/QsUjIiLpFUUNw8wGAccB05KMG2Fm9WZWv2bNmnyHJiIioYInDDPbG3gU+Ia7b2o53t3vdveh7j60T58++Q9QRESAAicMM+tMkCxq3f3vhYxFRETSK2QvKQP+CMxz918WKg4REYmmkGcYpwIXAx8zs5nh65wCxiMiImkUrFutuz8PWKHWLyIimSl40VtEREqDEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJAVNGGZ2tpktMLPXzey7hYxFRETSK1jCMLNy4PfAcGAIcKGZDSlUPCIikl4hzzA+BLzu7ovcfTvwV+C8AsYjIiJpdCrguvsDbyZ8Xg6c1HIiMxsBjAg/NpjZ7DzElq3ewNpCBxGB4oxPKcQIijNupRLn4DgWUsiEEYm73w3cDWBm9e4+tMAhtUpxxqsU4iyFGEFxxq2U4oxjOYVskloBDEz4PCAcJiIiRaiQCeNF4DAzO8jMugAXAP8sYDwiIpJGwZqk3H2nmf0P8B+gHBjr7nName3u3EcWC8UZr1KIsxRiBMUZtw4Vp7l7HMsREZF2Tld6i4hIJEoYIiISSdEkjNZuE2JmXc3soXD8NDMblDDuhnD4AjM7q4AxfsvM5prZK2b2jJlVJ4xrNLOZ4Sunxf0IcV5mZmsS4rkqYdylZvZa+Lq0wHH+KiHGhWa2MWFcXvanmY01s9Wprv+xwG/CbXjFzI5PGJfPfdlanDVhfK+a2RQz+2DCuCXh8Jlxdb/MIs5hZvZOwt/2+wnj8nYroQhxXp8Q4+zw+9grHJeX/WlmA82sLjzmzDGzryeZJt7vp7sX/EVQ9H4DOBjoAswChrSY5mrgzvD9BcBD4fsh4fRdgYPC5ZQXKMYzgIrw/ajmGMPP7xbRvrwM+F2SeXsBi8J/e4bvexYqzhbTX0PQMSLf+/N04Hhgdorx5wDjAQNOBqble19GjPOU5vUT3I5nWsK4JUDvItmfw4DHs/2+5DrOFtP+F/Bsvvcn0A84PnzfA1iY5P96rN/PYjnDiHKbkPOA+8P3jwAfNzMLh//V3RvcfTHweri8vMfo7nXuvjX8OJXg2pJ8y+aWK2cBT7n7enffADwFnF0kcV4IPJijWFJy90nA+jSTnAf82QNTgf3MrB/53ZetxunuU8I4oHDfzSj7M5W83koowzgL9d1c5e4zwvebgXkEd9BIFOv3s1gSRrLbhLTc8F3TuPtO4B2gMuK8+Yox0ZUEmb1ZNzOrN7OpZnZ+DuJrFjXOz4WnqI+YWfMFlPnalxmtK2zaOwh4NmFwvvZna1JtRz73ZaZafjcdeNLMXrLgVjyF9mEzm2Vm483sqHBYUe5PM6sgONA+mjA47/vTgib644BpLUbF+v0s+luDlCIzuwgYCnw0YXC1u68ws4OBZ83sVXd/ozAR8i/gQXdvMLP/Jjhz+1iBYoniAuARd29MGFZM+7NkmNkZBAnjtITBp4X7si/wlJnND39hF8IMgr/tu2Z2DvAYcFiBYoniv4DJ7p54NpLX/WlmexMkrG+4+6ZcrQeK5wwjym1Cdk1jZp2AfYF1EefNV4yY2SeA0cCn3b2hebi7rwj/XQRMIPg1kAutxunu6xJiuxc4Ieq8+YwzwQW0OOXP4/5sTartKLpb35jZMQR/7/PcfV3z8IR9uRr4X3LTpBuJu29y93fD908Anc2sN0W4P0Ppvps5359m1pkgWdS6+9+TTBLv9zPXhZmIxZtOBEWXg3i/oHVUi2m+yu5F74fD90exe9F7EbkpekeJ8TiCwtxhLYb3BLqG73sDr5Gjgl3EOPslvP8MMNXfL4QtDuPtGb7vVag4w+mOICgiWiH2Z7iOQaQu0n6K3YuK0/O9LyPGWUVQ3zulxfC9gB4J76cAZxcwzgOa/9YEB9pl4b6N9H3JV5zh+H0J6hx7FWJ/hvvlz8Cv00wT6/czZzu7DRt/DkGV/w1gdDjsFoJf6gDdgL+FX/rpwMEJ844O51sADC9gjE8DbwMzw9c/w+GnAK+GX/JXgSsLvC9vBeaE8dQBRyTMe0W4j18HLi9knOHnHwA/aTFf3vYnwa/HVcAOgnbeK4GRwMhwvBE8COyNMJahBdqXrcV5L7Ah4btZHw4/ONyPs8LvxOgCx/k/Cd/NqSQkuGTfl0LFGU5zGUGHm8T58rY/CZoVHXgl4e96Ti6/n7o1iIiIRFIsNQwRESlyShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEIZKF8HkEZ4bvf2Rmvy10TCK5opsPimTnJuCW8EZzxwGfLnA8IjmjK71FsmRmE4G9gWEePJdApF1Sk5RIFszsAwRPPtuuZCHtnRKGSBuFTy6rJXiq2btmlrMn6okUAyUMkTYIn7T2d+Bad58H/JCgniHSbqmGISIikegMQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCSS/w+0DgVy+oMosQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "

" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -802,27 +783,11 @@ }, { "cell_type": "code", - "execution_count": 25, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[[4.]\n", - " [3.]]\n", - "[3.86998544] [3.11177547]\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/usr/local/lib/python3.7/site-packages/sklearn/linear_model/stochastic_gradient.py:130: DeprecationWarning: n_iter parameter is deprecated in 0.19 and will be removed in 0.21. Use max_iter and tol instead.\n", - " DeprecationWarning)\n" - ] - } - ], + "execution_count": 2, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "# Importing various packages\n", "from random import random, seed\n", @@ -831,12 +796,12 @@ "from sklearn.linear_model import SGDRegressor\n", "\n", "x = 2*np.random.rand(100,1)\n", - "y = 4+3*x#+np.random.randn(100,1)\n", + "y = 4+3*x+np.random.randn(100,1)\n", "\n", "xb = np.c_[np.ones((100,1)), x]\n", "beta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n", "print(beta_linreg)\n", - "sgdreg = SGDRegressor(n_iter = 100, penalty=None, eta0=0.01)\n", + "sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n", "sgdreg.fit(x,y.ravel())\n", "print(sgdreg.intercept_, sgdreg.coef_)" ] @@ -903,30 +868,11 @@ }, { "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[[4.26950679]\n", - " [2.93950818]]\n", - "[[4.16449871]\n", - " [3.01998132]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3X+8HHV97/HXO7/QAxHISawIyTlgqVYsokTFH7dSFcUUi7daSwgUWiQFqtVq1WruVfwBetui9dYiDUiJnCMWUVvlihUVS/0B9qAgPy1CcgKKJPwmYIEkn/vHzObMbvbH7O7szu7J+/l4TLI7Mzvz3dk585nvj/l+FRGYmZm1MqfsBJiZ2XBwwDAzs1wcMMzMLBcHDDMzy8UBw8zMcnHAMDOzXBwwdnGSNkh6Vfr6fZLOKykdh0u6s4x97yokXSDpIx1+9qWSbpW0RdLri05bnf2tkvSNJsu/I+nNvU6HVXPAGGCSjpF0taRHJG1KX58mSb3YX0ScGRFd/xFKGpcUkuYVka6ydXOhnUU+BHwqIvaIiH/pdmPpMX08DUD3Sbpc0rMqyyNiMiJe3e1+rFgOGANK0juBTwJ/AzwN+DXgFOClwIIGn5nbtwTarmYMuLGTDza5cfjriNgD2Bf4OfCZDtNmfeKAMYAk7UlyR3daRFwSEQ9H4scRsSoiHkvXu0DSpyV9TdIjwO9I+l1JP5b0kKQ7JJ1es+3jJU1LulfSmpplp0uayLw/TNL3JT0g6TpJh2eWfUfShyV9T9LDkr4haXG6+Mr0/wfSO8gX1/mOT07Tf7+km4AX1Cx/uqQvStosab2kP88se6GkqfQ73i3p45llL8uk+Q5JJ6bzd5P0t5I2pp85R9KT02WHS7pT0jvTnNxdkv44XbYaWAW8O/0uX23wmz0rvUu+T9JPJb0pnb9A0rWS3pq+n5ses/dnvssP0vTeJelTkhZkthtprvLW9Dh/WNIz0u/4kKSLK+tnvsf7JN2jpLhxVb30pusflabtgXR7BzdY7zbgAOCr6THYLf19vpJ+359JOjmz/umSLpE0Iekh4MRGaQCIiF8BFwOHZLZxoqTvZt4fIekWSQ9K+hSgzLK5ks5Kv/N6SW9RJocraU9Jn0mP788lfUS+uepMRHgasAk4EtgKzGux3gXAgyS5jjnAk4DDgd9K3x8M3A28Pl3/2cAW4LeB3YCPp/t5Vbr8dGAifb0vcC+wIt3WEen7Jeny7wC3Ab8BPDl9/7F02TgQzdIPfAz4D2ARsBS4AbgzXTYHuAZ4P0lu6gDgduA16fIfAMenr/cADktfjwEPAyuB+cAocEi67BPAV9L9LQS+Cnw0XXZ4ehw+lH5uBfAosHfmOH+kyXfZHbgD+GNgHvA84B7g2eny5wD3A78JrAGuAuamyw4FDks/Nw7cDLw9s+0A/hV4CnAQ8BjwrfSY7AncBJxQ8z0+nv6+LwceAZ5Z+z3SNG4CXgTMBU4ANgC7NfiOG0jPk/T9lcDZJOfcIcBm4BWZ8+gJ4PXpb/nkBufuRzLH70LguszyE4Hvpq8Xp7/rG9Pf5y/S7/nmdPkp6XHYD9gb+CaZ8w/4MvCP6X6eCvwQ+NOy/86HcSo9AZ7q/ChwHPDLmnnfBx4AfgX8djrvAuCzLbb1d8An0tfvBz6fWbY78Dj1A8Z7gAtrtvVvmYvTd4D/lVl2GvD19PU4rQPG7cCRmfermQkYLwI21qz/XuCf0tdXAh8EFtdZ58t19iWSC+czMvNeDKxPXx+eHtd5meWbmAlEOy5uDb7LHwL/UTPvH4EPZN6/E/gpSeA4sMm23p79DulxfGnm/TXAezLvzwL+LvM9tgK7Z5ZfDPzv2u8BfBr4cM2+fwq8vEG6NmTOk6XANmBhZvlHgQsy59GVLc7LC4D/JjmntwPrgYMzy09kJmD8EXBVze95JzMB49tkAgDwqsr5R1KU+xiZoEVyQ3FFt3+nu+LkIqnBdC+wWJmy34h4SUTslS7L/m53ZD8o6UWSrkiLch4kufuqFBU9Pbt+RDySbq+eMeAP0uKKByQ9ALwM2Cezzi8zrx8ludvPqyotwHTNvp9es+/3kfzxA5xEkrO5RdJ/Sjoqnb+UJNdTawkwAlyT2d7X0/kV90bE1g6/zxjwopr0riKpe6pYl673tYi4tTJT0m9IulTSL9PimzOZ+b0q7s68/lWd99l03p/+rhXTJMe6XprfWZPmpQ3WrfV04L6IeLhmP/tm3t9Ba3+bntPjJN/jmU32lz1vo2b7tedS9vUYSa7krsz3/EeSnIa1yQFjMP2A5K7o6Bzr1nY3/DmSopelEbEncA4z5b13kVwUAJA0QlJsU88dJDmMvTLT7hHxsQ7SVE9VWoBlNfteX7PvhRGxAiAibo2IlSR/9P8HuERSpVjoGXX2dQ/JBemgzPb2jKTCNY9W3+cO4N9r0rtHRJyaWeds4FLgNZJelpn/aeAWklzHU0gCYzet4PZOj0XFMuAXDdJ8Rk2aRyLiohz7+AWwSNLCmv38PPM+dzfYEbEReBvwyUq9Uo3a81ZUnzt3kRRHVWSX3UHyt7Q48z2fEhEH5U2fzXDAGEAR8QBJkcvZkt4oaaGkOZIOISlGamYhyd3ff0t6IXBsZtklwFFKKoYXkJTZNzoHJoDXSXpNWqn4pLRSdb8G62dtJilmOKDJOhcD75W0d7rNt2aW/RB4WNJ7lFSOz5X0HEkvAJB0nKQlEbGdpEiDdH+TwKskvUnSPEmjkg5J1zsX+ISkp6bb2FfSa3J8F0ju6Jt9l0uB31DSoGB+Or1A0m+m+zqepK7iRODPgXWSKsFqIfAQsEVJs9JTd9582z6opLL9fwBHAV+os865wClpjlSSdlfSYGJhnXWrRMQdJEWkH03Pi4NJcn0TzT/ZdJuXkwSi1XUW/z/gIEm/n+a6/5zq3NvFwNvS33QvkuLUynbvAr4BnCXpKenf0TMkvbzTtO7KHDAGVET8NfAO4N0kF6y7SbLS7yH5Y23kNOBDkh4mqbO4OLPNG4E/I8mF3EVSnl73Ybn0onA0yR3vZpI7tXeR45yJiEeBM4DvpcUAh9VZ7YMkxRjrSf6gL8x8fhvJhe6QdPk9wHkklbyQNAq4UdIWkqbHx0TEr9I71RUk9QX3AdcCz00/8x7gZ8BVadHPN2lcBFLrM8Cz0++y0zMIadHMq4FjSC56vyTJ+ewmaRlJPdIfRcSWiPgcMEVSCQ/wlyRB/WGSi/g/50xTI78k+V1/QRJAT4mIW+qkeQo4GfhUuv7PaNGaqcZKkqKkX5BUKn8gIr7ZTcJJmpC/W9JuNWm9B/gDkoYS9wIHAt/LrHIuyTn0E+DHwNdI6nK2pcv/iKTxxE0k3/USqotWLSclxYFmNuyUNHueiIg8ucBZS9JrgXMiYqzstMw2zmGY2VBLiy1XpMWQ+wIfIMn1WMEcMMxs2ImkiPN+kiKpm0mKY61gLpIyM7NcnMMwM7Nchqo30cWLF8f4+HjZyTAzGyrXXHPNPRGxpPWazQ1VwBgfH2dqaqrsZJiZDRVJ063Xas1FUmZmlosDhpmZ5eKAYWZmuThgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpaLA4aZmeXS84Ah6XxJmyTdUGfZOyWFpNpB783MbMD0I4dxAcmQmlUkLSUZ1nJjH9JgZmZd6nnAiIgrScZXrvUJkvGqPSCHmdkQKKUOQ9LRwM8j4roc666WNCVpavPmzX1InZmZ1dP3gCFpBHgfOYdQjIi1EbE8IpYvWdJ1d+5mZtahMnIYzwD2B66TtAHYD/iRpKeVkBYzM8up7wMoRcT1wFMr79OgsTwi7ul3WszMLL9+NKu9CPgB8ExJd0o6qdf7NDOz4vU8hxERK1ssH+91GszMrHt+0tvMzHJxwDAzs1wcMMzMLBcHDDMzy8UBw8zMcnHAMDOzXBwwzMwsFwcMMzPLxQHDzMxyccAwM7NcHDDMzCwXBwwzM8vFAcPMzHJxwDAzs1wcMMzMLBcHDDMzy8UBw8zMcnHAMDOzXBwwzMwsFwcMMzPLpecBQ9L5kjZJuiEz728k3SLpJ5K+LGmvXqfDzMy6048cxgXAkTXzLgeeExEHA/8FvLcP6TAzsy70PGBExJXAfTXzvhERW9O3VwH79TodZmbWnUGow/gT4LJGCyWtljQlaWrz5s19TJaZmWWVGjAkrQG2ApON1omItRGxPCKWL1mypH+JMzOzKvPK2rGkE4GjgFdGRJSVDjMzy6eUgCHpSODdwMsj4tEy0mBmZu3pR7Pai4AfAM+UdKekk4BPAQuByyVdK+mcXqfDzMy6049WUisjYp+ImB8R+0XEZyLi1yNiaUQckk6n9DodZmZDZ3ISxsdhzpzk/8mG1b19UVodhpmZNTE5CatXw6Npqf30dPIeYNWqUpI0CM1qzcys1po1M8Gi4tFHk/klccAwMxtEGze2N78PHDDMzAbRsmXtze8DBwwzmx0GrIK4a2ecASMj1fNGRpL5JXHAMLPhV6kgnp6GiJkK4mEOGqtWwdq1MDYGUvL/2rWlVXiDA4aZzQYDWEFciFWrYMMG2L49+X/VqlJzUm5Wa2bDbwAriHui5Ka2zmGY2fAbwArinig5J+WAYWbDbwAriHuiUY5pejopnjrttJ4WV7lIysyGX6U4Zs2a5KK6bFkSLEqsIO6JZcuS4FDP9DR8+tPV7yvFVQXRMPUsvnz58piamio7GWZm5aitw8hjbAxNT18TEcu73b2LpMzMytROq6dsU9u8Cqz4d8Aws+E3rA/tdfL8SKWpbd6gUWDFvwOGmQ23Xj601+tA1E2rp3oV/bUKrvh3wDCz4darpqb9eHq8m+dH6j0JfuqpPX0y3JXeZjbc5sxJLui1pOQJ6U6Nj9dvkTQ2lhQJFaEf+wAkudLbzKxnD+314+nxIXt+xAHDzIZbry66/Xh6fAA7GGzGAcPMhluvLrr9uvuv18Fgt9atS46FREjdby/V84Ah6XxJmyTdkJm3SNLlkm5N/9+71+kws1msFxfdYbr7//rXdwSIy/RadOIJiOBYJphmjEPh0CJ2048cxgXAkTXz/gr4VkQcCHwrfW9m1p5eNnudnBzcrkauvXZHgLhaL0KvPRIRiGAFlwGwkknOZTXjNOhKpAM9DxgRcSVwX83so4F16et1wOt7nQ4zm2U6bfaaJ8gM2oBMDzxQVcS04Xmv51gmEMFhXF33I+ctWcPutNGFSB4R0fMJGAduyLx/IPNa2fd1PrsamAKmli1bFmZmERExNhaRXM6rp7Gxxp+ZmIgYGalef2Qkmd/ttouW7vMqXhgrmYgtVKd7CyOxkokds04/vebz0o51D4WIAq7lfXkOQ9I4cGlEPCd9/0BE7JVZfn9EtKzH8HMYZrZDJ89f5H3uoVfPdkDjoq60cvo2DuDXua3qI+sZr1+01Ox5jcx3XQ5MRXRd+11WK6m7Je0DkP6/qaR0mNmw6qTZa95nK3rVpLZOUVccdxzHanJHHURtsABYRgfPhOTpOqRNZQWMrwAnpK9PAP61pHSY2bDqpNlr3kDQiya1r341HHfcTt2YCPgsJ7CNOaxnnJXM1JNUypTmjHUQwDrp2baVIsq1mk3ARcBdwBPAncBJwChJ66hbgW8Ci/Js69BDD+2uTNDMZpeJiaReQUr+r62LqLd+njqMTrZd67LLIiC2w45dbUNRt26kVXraSXedtC+G26OI63kRG+nX5IBhZl3rNhA08vDDOy7mjWLBesZaB4xGlet5010nuDwftkUB12B3Pmhm+Q3yswllSCuqRb7r6CPnTjLythwj5nVTuV6nYn/YK73NbFhUnluQ4Pjji382YZgGP0qfhUj/21FR3chtt1VnHUbeXPP0+Ny59T/YTeV6kZ0j1nDAMLPGTjttJkjAzk1Nux13YtAekKvVZoD45jerA8QBB9RZKduNybp1xVeuF9k5Yg0HDDOrb3ISzjmn/vMIWd3c0fZq8KNOvfrVdQPESiZZz/hOLZnOOisNDhOTxNg4rzyizVxSL/qrqtPCK6DLh0cqGxqAyuy8kyu9zeroVSVuo6edi3z6WQ1aDUnFfIdWrriiaSU1RN2nrKtaKOVpwdSr36gRt5JywLA29OMPtN8XgUZpaKe5ZTsaXcyL3FejoDQ62ptj+8QTLQNE7dSym5BWy3v5G+UETIUDhlkd/fgDHYCLQEQU2+dRbQAcHW1+JS3iQl7vOM6fHzFvXvW8BQs631e7AaJWs8DZ7BhVckkD0C+VA4ZZI/34Ax2Ai0BEFFek0+jCvWDBzts99dRiv0NtoNpjj8YX5zwgVnBp5wGiVt6iuUbnQtnFblFcwHClt80+/RiLuR/7yKOoPo/qVT4/8QQsXFhdIXvhhXD22Z2lNa8tW+rPv/fe+vMlTtfpVRXVX+N3G26+9srdUid9MmVbOvVjqNd+KSLq9GtyDsNy2ZVyGBMTO+cCOim+KesuuF7OplV24A1viC9zdHE5iLzpbCenUVvh3az4sg91YbhIyqyBXakOY2IiKTqqLUpqNx1lBcA2LsKbGO1fgOgmve1069HL8yizz4PhsXDAMGtgV2klVdSFvqwA2CBns73O+02MVg0Y1JcAUatVjqjdY9arQF2TzqIGUCo9CLQzOWCY1SiyKKmMANjggrmJ0djE6E6BYwsl5eKyx+XUU2fej44mU6fHrFdFgTXHdahG3CuKOx80q5F3BLlBI+14gvpcVleNPf0II5zMWs5kTfujzBWt0nVJtkHAyEj3T2NX9Or3qxkx0J0PmllvBvrphWx3G5rp3fUiVnEya9nAGNsRGxhj6z+s5XOxinENQEu0Xndd0u7vl7ejxl61wCoim9KvyUVSVphBqH8oyiB+l3e9K3eDou9+t8E2BqElWj9aj3UxzkXTwZ9ch+GAYQUYlBZOs8ndd+cOEJ/8ZM5tDsLv1Iug1WmAbzctbiXlgGEFGIQ7117qU46jVWBYyUSsZywZlrTTdJSdeyo6aHWzvS5yO/g5DLMODUBXDT3Tw7vyvDkI6G06+q7IoNXNzUoXny0qYLiVlO16hrVlUR4Ffje10aZmp8vIbD7G3ahpvbRDniFZu2ixJemaiFjeQYqrlNpKStJfSLpR0g2SLpL0pDLTM+sM09CX/TQsLYs60UUfV1WtmFoEi9rb3CLTMat1069ULwZbalcR2ZROJmBfYD3w5PT9xcCJzT7jIqk2zKYigV4ou2y8V9ootjhwn4fyFzH1MB2la+dc6Pa8KenvkmGvw0gDxh3AImAecCnw6mafccBowzD9wVpxmlyQ3vpnW3sXINpIx0Dpoqlqx9+phJuVoQ8YyXfgbcAWYDMw2WCd1cAUMLVs2bJCD+KsNpsrdq259IK0XYr1jNXtf6nwANEkHQOdi2vnxmqIb8KKChilVXpL2hv4IvCHwAPAF4BLImKi0Wdc6d0GVzrucm66CQ46KN+68cRWmDevtwkaBu1UQndTYV2y2VDp/SpgfURsjogngC8BLykxPbNLo4rdFStcEd5PPWx48OCD1ZXUzYJF/Mu/Vt0WO1ik2qmEnk0DIXWozICxEThM0ogkAa8Ebi4xPbNLvRYVJ5wA69YlOY+I5P/Vqx00eqXSDLKg4x1RHSD22qvJuqOLqwPE0Ud39h1mu3ZazM3m1nV5FVGu1ekEfBC4BbgBuBDYrdn6rvTu0hCXwQ6lTo93puw/bx3E48wbjjqDespOdz9bSZWE2VDp3e7kgNElV4T3V6PjXQkadS42K5lIxnzIrLuFkZ2Cxi94WvUHh6VVUq1hTfeQKSpgtCySknS5pOf2MpdjfeIy2M51UhfR7LimxVPHarKqmOlM1lSNDQGwO49yPidVRZx94q7q7bXbDfegPNTZ6+7DrVitIgrwfOAK4J+AfYqIUp1OzmF0adjv5soqDuj0uLUazhNiPWNVs7bRYS6wndzjIJ0HzvX2Bf0ukgLeAFwHfID06ex+Tw4YBRjSMthSL3Jd1P1UemytHWq0Mm1DARFv4AsRZ57Z+b6G9XmCQUrLLNbXgAEIeA5wCnAPcCdwfBEJaGdywJjFWgWyMi8sbdwFN8pMrCdn+hsFxuw40vWOTzsBdZDu6gcptzOL9S1gAN8DfgFcDnwYOAr4deDvgbVFJCLv5IAxS01MRCxYUH3RWLCg+qJR5kWuSbBqUtpUNa1kYueZzbqgyAaHU0/Nd1HNm3sctLv6Yc31DpF+BoyDIHkivM6ym4tIRN7JAWOWGh2tfwEbHZ1Zp8yLXJ274Hotl2qn5J+a7fRjpLUOvo/v6me3vtdh1P0wHFBEIvJODhizVNOrbqqki9zChbEjh1AZPa7RsxE7BYii9CJ35bv6XUpRAaOrJ70j4vZuPm+WW5/GAjj55OqnqR9+GFYyyZmsYRkb2cgyLmUFZ7KGQMQ+TycmJokguYz3Qi+aQ69alfQptn178n8/x1SwoVXqAEpmAIyO5pvfg4vchRdWB4jzzqtevpJJzmU140wzh2Ccad7Cpxkn7djxrrt6371Kkf2CDcrzFzacisim9GtykdQsNTERMX9+dXHL/Pk9KSa54YbmJWA7FTHlnRrVJxRV9NNpRXjtNlx3sUtiEOow+j05YMxiPSpTf/DBDgJEVrPuPVrVJ3R7gW52TDqpCB+01lHWN0UFjNLGw+iEx8OwViKS0pZc66KZDzXSaFyRWvXGGelmTJJKT7fZbjNGRmbqbToZm2GIx3Ow7syG8TCslsuXq+U8Htk6iGbB4nHmJxXVW7cl183KPXYz9eoPajXq4nrjxvrrN5qf1aqPpU4qwt2XmHWriGxKv6ZZXSTl8uVqTY5H3iKmX/LU5MVll3Wfltr6g14/INeqKW0n54vPsV0WrsOYZVy+XK3B8ajtrC87XcULkxfj42WnPtFpNx8R+c6HTup9/PzFLskBY9C1+4c5SP37lAwa99pa6awPIs5h9cyyohR9Qe2mmw/nBqwgDhiDrJM/9kHNYfThjrTe127YWV/RASKrHxfpdn5n5wasIA4Yg6yTi/8g3lH2KE156h9WMhGP8OT+Ho9+BG3nJK0ERQUMt5LqhU5ax/Sp64uWsi2TTjihkNHQsq2YpObrBiIQn3vwdYxMnJscB4C5c2f23avWY920asrLLZVsiDlg9EKei0K9JqPddH1RRJPcStv/6enkvnfbtvrrtbiAjo62HyDivM8Qwcw991e/mgSH6elkI5W0pEOb9iRo9ONi3qibj3rNcs0GTRHZlE4nYC/gEuAW4Gbgxc3WH5oiqVZFOUUX9RS1vUZFMi2KaFauzPex5GxLX+y1V+My+hxDm7Ys3uuk7L9fxYKum7A+YzbUYQDrgDenrxcAezVbf2gCRkTx3To0U9T28nSDMTISlx2f/1mIuhXVExP1x8CoXJzzBK5GZf697I7DbEgVFTBK6xpE0p7AtSRjauRKxKzpGqRRFw2QlNlv3JgUg5xxRr5iqaK6fGjQlcX2OXNh+3Y2soz3cQYX0ThNO7rbgPppqtflRVbl+7c6JRp1r9FNdxxms9Rs6Bpkf2Az8E+SfizpPEm7164kabWkKUlTmzdv7n8qe6FRmbg0U3/QTll9o+0tWtReuuqUrz/CCMdtX8dctrM/G3YKFjvqIFByjc/e29dTr8uLrEqwbKZZmX8/Kq7NdlFlBox5wPOBT0fE84BHgL+qXSki1kbE8ohYvmTJkn6nsTfqVXxKO19k87ZIOuMMWLBg5/kPPZQr4DzySFpBfdwqjn10LRsYYztiA2OczNqqIFEVIDbe0TpA1Gp14a7krOodH2jdesytkMx6psyAcSdwZ0Rcnb6/hCSAzH71mtA2uuDmuTNetQoWLtx5/hNP1A04ETMtmI7VJJv3GGcbc1jPOAD7s2FHjuJzHDcTIP73+6sDxNKlbXzpVLMLdyXnUO/4XHhhss9WrcfcCsmsd4qoCOl0Av4DeGb6+nTgb5qtP1SV3u3qtOK6VSVxWjlcb9FKJmIL1RXE23eqtS5YoxZQo6PFVTC74tqsCrPkwb23ApOSfgIcApxZcnrKs2JFe/Oh+rmJBjbEsobPQpzJGnanuj5BMJPj6UWDiHq5h4kJuOee4h5S9HjVZj3hAZQGRSete1oM7vMIIzvVQVzHwRzM9c3T4gF1zGaV2dBKyrLabN0jwfbpxvUbW5nLyazl9/lyVUX1wduunck9VLrdqOUKYjOrwwFjULRo3VOvP6aNNL6wz2Mbn+M43vjD91TXFmSHpHMFsZm1wQFjUDR4BuLY6TMa1kG8jzNoWKBYqYd4wQsa73NQOjw0s6Ewr+wEWOKV56/i1x5NKqKXsbHhU9V7cT/3U/NAXu0zHO3kElatcoAws1ycwyjJe99bXcT07W/DRayqegaiEiyydRD3s2jnFrIXXuhcgpn1nANGn3zhC9UB4mMfa7xu1dPUaOcAUcvNSHuniG7jzWYJB4weuf766gDxpjc1XnenALF0afJsQq+ehbB8ascH6eVYHGZDwAGjW+kdaMyZwwaNc6wmkeDggxt/ZKcA8YY3VI80dMcdvjANgnodJXYw4qDZbOGA0aFf/Srph+mR45I7UEUwzjTnspqVVF/odwoQ78k0dZ2YgC99qfOOB6133POtWRUHjJy2bq0uYhoZqd+1xu48Wt1hH4LnPre6DiJbgbFmTXcdD1qiUV1DN3UQ7vnWrIqb1TZQ+4xbPctockEvortvX5jyqR2UqVLX8L3vwbp1O8+HfA0Dzjhj58Ge/GCj7cJmXw6jizvKRYtmchDNgsU25hCIOY0emxsdzZ/eZoMp+cKUT6O6hrVru6uD8IONZlWGK2Dcd1/zYNBmq5bly6uLme6/v/5uH2d+VRHTHCKpxGgnMDTSaLCgU07xhSmvRrm0bdvaW78eN1k222G4Asb0dPNg0KJVyxvfWB0grrmm/m62sHtVgJjP1plxpivTk56UBLB6Gs2vp9FgQWefnX8bu7pGubS5c9tb38yaGq6AUdvldm3xQoM7x5ieZrvm8LdfHN+pBRPAZhZXBYjdeRS+8Y3Wo8sVVSnqu9juNOpEcfVqd65oVqDhChj1VILE5CTbVf/rCJjDTLPXTYxWBYjF3Asf+lB1gDjiiNb7bqe3Vz8x3DuN6hrOPtt1EGZFKmIe8AbIAAAKPElEQVTYvn5Nh9YZZ/TehWN1hxptOb3kJfnGNmw13Gee4UBPPTVZnt3/yIiHDjWzvqCgIVqHa8S9uXNjKlMsVRlR7kzWME7jked2kndEudrmmpDkINq5S52chOOPr9/MttloemZmBdk1R9wbG+PePcbYjtjAGCezlqXcyVg7wQLy1zEU0TWEH8wzs1liuB7cW7SIRScezYMfOItxpvkcxzVff3Q0af7a6YNXRXQN4QfzzGyWKD2HIWmupB9LujTX+i8+jL3mPwrveAds2jTTH1O9yudPfrK7Ss8iWkH5wTwzmyVKDxjA24Cbc699xBHw+ONw1lmwZEkyr9kTud00WS1izGs/mNeaW5CZDYcias47nYD9gG8BrwAubbX+oYceWlSjgfzytILqxzZmq4mJpMWYW5CZ9QyzoZWUpEuAjwILgb+MiKPqrLMaWA2wbNmyQ6en26zgtsE2Pp48tV/LLcjMCjP0raQkHQVsiogGHXQkImJtRCyPiOVLKkVQNnt4zAmzoVFmHcZLgd+TtAH4PPAKSRMlpsfK4DEnzIZGaQEjIt4bEftFxDhwDPDtiGjRTtZmnSIaFphZXwxCKynblVsJecwJs6ExXF2DLF8eU1NTZSejWEV0P2Jm1sTQV3pbqojuR8zM+sABo2xuJWRmQ8IBo2xuJWRmQ8IBo2xuJWRmQ8IBo2xuJWRmQ2K4ujefrSqdJJqZDTDnMMzMLBcHjDx25QfrzMxSLpJqpfbBuunp5D24GMnMdinOYbTiB+vMzAAHjNb8YJ2ZGeCA0ZofrDMzAxwwWvODdWZmgANGa36wzswMcCupfPxgnZmZcxhmZpaPA4aZmeXigGFmZrk4YJiZWS4OGGZmlktpAUPSUklXSLpJ0o2S3lZWWszMrLUym9VuBd4ZET+StBC4RtLlEXFTiWkyM7MGSsthRMRdEfGj9PXDwM3AvmWlx8zMmhuIOgxJ48DzgKvrLFstaUrS1ObNm/udNDMzS5UeMCTtAXwReHtEPFS7PCLWRsTyiFi+ZMmS/ifQzMyAkgOGpPkkwWIyIr5UZlrMzKy5MltJCfgMcHNEfLysdJiZWT5l5jBeChwPvELStem0osT0mJlZE6U1q42I7wIqa/9mZtae0iu9zcxsODhgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpaLA4aZmeXigGFmZrk4YJiZWS4OGGZmlosDhpmZ5eKAYWZmuThgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpaLA4aZmeVSasCQdKSkn0r6maS/KjMtZmbWXGkBQ9Jc4B+A1wLPBlZKenZZ6TEzs+bKzGG8EPhZRNweEY8DnweOLjE9ZmbWxLwS970vcEfm/Z3Ai2pXkrQaWJ2+fUzSDX1IW7cWA/eUnYgcnM7iDEMaweks2rCk85lFbKTMgJFLRKwF1gJImoqI5SUnqSWns1jDkM5hSCM4nUUbpnQWsZ0yi6R+DizNvN8vnWdmZgOozIDxn8CBkvaXtAA4BvhKiekxM7MmSiuSioitkt4C/BswFzg/Im5s8bG1vU9ZIZzOYg1DOochjeB0Fm2XSqcioojtmJnZLOcnvc3MLBcHDDMzy2VgAkarbkIk7Sbpn9PlV0sazyx7bzr/p5JeU2Ia3yHpJkk/kfQtSWOZZdskXZtOPa3cz5HOEyVtzqTnzZllJ0i6NZ1OKDmdn8ik8b8kPZBZ1pfjKel8SZsaPf+jxP9Nv8NPJD0/s6yfx7JVOlel6bte0vclPTezbEM6/9qiml92kc7DJT2Y+W3fn1nWt66EcqTzXZk03pCej4vSZX05npKWSroivebcKOltddYp9vyMiNInkkrv24ADgAXAdcCza9Y5DTgnfX0M8M/p62en6+8G7J9uZ25JafwdYCR9fWoljen7LQN0LE8EPlXns4uA29P/905f711WOmvWfytJw4h+H8/fBp4P3NBg+QrgMkDAYcDV/T6WOdP5ksr+SbrjuTqzbAOweECO5+HApd2eL71OZ826rwO+3e/jCewDPD99vRD4rzp/64Wen4OSw8jTTcjRwLr09SXAKyUpnf/5iHgsItYDP0u31/c0RsQVEfFo+vYqkmdL+q2bLldeA1weEfdFxP3A5cCRA5LOlcBFPUpLQxFxJXBfk1WOBj4biauAvSTtQ3+PZct0RsT303RAeedmnuPZSF+7EmoznWWdm3dFxI/S1w8DN5P0oJFV6Pk5KAGjXjchtV98xzoRsRV4EBjN+dl+pTHrJJLIXvEkSVOSrpL0+h6kryJvOt+QZlEvkVR5gLJfx7KtfaVFe/sD387M7tfxbKXR9+jnsWxX7bkZwDckXaOkK56yvVjSdZIuk3RQOm8gj6ekEZIL7Rczs/t+PJUU0T8PuLpmUaHn58B3DTKMJB0HLAdenpk9FhE/l3QA8G1J10fEbeWkkK8CF0XEY5L+lCTn9oqS0pLHMcAlEbEtM2+QjufQkPQ7JAHjZZnZL0uP5VOByyXdkt5hl+FHJL/tFkkrgH8BDiwpLXm8DvheRGRzI309npL2IAlYb4+Ih3q1HxicHEaebkJ2rCNpHrAncG/Oz/YrjUh6FbAG+L2IeKwyPyJ+nv5/O/AdkruBXmiZzoi4N5O284BD8362n+nMOIaaLH8fj2crjb7HwHV9I+lgkt/76Ii4tzI/cyw3AV+mN0W6uUTEQxGxJX39NWC+pMUM4PFMNTs3e348Jc0nCRaTEfGlOqsUe372umImZ+XNPJJKl/2ZqdA6qGadP6O60vvi9PVBVFd6305vKr3zpPF5JBVzB9bM3xvYLX29GLiVHlXY5UznPpnX/xO4KmYqwtan6d07fb2orHSm6z2LpBJRZRzPdB/jNK6k/V2qKxV/2O9jmTOdy0jq915SM393YGHm9feBI0tM59MqvzXJhXZjemxznS/9Sme6fE+Seo7dyzie6XH5LPB3TdYp9Pzs2cHu4MuvIKnlvw1Yk877EMmdOsCTgC+kJ/0PgQMyn12Tfu6nwGtLTOM3gbuBa9PpK+n8lwDXpyf59cBJJR/LjwI3pum5AnhW5rN/kh7jnwF/XGY60/enAx+r+VzfjifJ3eNdwBMk5bwnAacAp6TLRTIQ2G1pWpaXdCxbpfM84P7MuTmVzj8gPY7XpefEmpLT+ZbMuXkVmQBX73wpK53pOieSNLjJfq5vx5OkWDGAn2R+1xW9PD/dNYiZmeUyKHUYZmY24BwwzMwsFwcMMzPLxQHDzMxyccAwM7NcHDDMzCwXBwwzM8vFAcOsC+l4BEekrz8i6e/LTpNZr7jzQbPufAD4UNrR3POA3ys5PWY94ye9zbok6d+BPYDDIxmXwGxWcpGUWRck/RbJyGePO1jYbOeAYdahdOSySZJRzbZI6tmIemaDwAHDrAPpSGtfAt4ZETcDHyapzzCbtVyHYWZmuTiHYWZmuThgmJlZLg4YZmaWiwOGmZnl4oBhZma5OGCYmVkuDhhmZpbL/wfPdkIFtJMFQAAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 3, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "from random import random, seed\n", "import numpy as np\n", @@ -1102,8 +1048,10 @@ }, { "cell_type": "code", - "execution_count": 5, - "metadata": {}, + "execution_count": 4, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import numpy as np \n", @@ -1164,17 +1112,11 @@ }, { "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "gamma_j after 500 epochs: 9.97108e-05\n" - ] - } - ], + "execution_count": 5, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import numpy as np \n", "\n", @@ -1211,37 +1153,11 @@ }, { "cell_type": "code", - "execution_count": 27, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Own inversion\n", - "[[4.13137049]\n", - " [2.85144795]]\n", - "sgdreg from scikit\n", - "[4.16307568] [2.89603874]\n", - "theta frm own gd\n", - "[[4.13137049]\n", - " [2.85144795]]\n", - "theta from own sdg\n", - "[[4.12146611]\n", - " [2.87377643]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XmcHHWd//HXZ5LJMSQkZBKSEJhJAiEcbiAwCwrIBFHAyOmyu2LkdhFcWXU9fiKKKKKuroKiPjACIiSCLt7uegAmE64EhiPhSgiBHIQjk5uQe+bz+6NqQqfTR3VPdVd3z/v5ePRjuqurqz9V3VOf/h71/Zq7IyIikk9d0gGIiEh1UMIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMKRmmdkUM3sl6TjiZmZuZgclHYf0PkoYUlZmttTMtpjZJjN73cxuN7NBScclIvkpYUgSznD3QcCRwGTgqoTj6ZXMrG/SMUh1UcKQxLj768BfCRIHAGb2ATN70sw2mtkKM7s25bmxYXXMhWa23MxWm9nVKc8PDEss68zsOeAfU9/PzA41s9lmtt7MnjWzM1Oeu93Mfmxmfw5LPw+Z2SgzuzHc3kIzm5xtX8K4LjezxeH2f2RmFj53rZnNyLAffcPHs83s62b2cPjefzSzRjObGR6Hx8xsbNpbTjWzl8Jj8B0zq0vZ/iVm9nwY91/NrDktzn83s8XAYgvcYGarwvd62szeke+zk95JCUMSY2b7A+8HXkxZ/BZwATAU+ABwhZmdnfbSE4CJwMnANWZ2aLj8K8CB4e1U4MKU96oH/gj8DdgXuBKYaWYTU7b7L8CXgOHANuAR4Inw8T3A9/Ls0ukESWpSuK1T86yf6kPA+cCYMP5HgJ8Bw4Dnw31LdQ7QAhwFnAVcEu7nWcAXgQ8CI4AHgLvSXns2cCxwGHAKcCJwMDAkjHtNAXFLL6KEIUn4nZm9CawAVpFyMnT32e7+tLt3ufsCgpNda9rrv+ruW9x9PjAfOCJc/i/A9e6+1t1XAD9Iec07gUHAt9x9u7v/HfgTcF7KOr9198fdfSvwW2Cru9/h7p3ALwmqz3L5lruvd/flwCxSSk4R/Mzdl7j7BuDPwBJ3v8/ddwL/k+G9/yvcz+XAjSn7cTnwTXd/PnztN4AjU0sZ4fNr3X0LsAMYDBwCWPi61wqIW3oRJQxJwtnuPhiYQnCiGt79hJkda2azzKzDzDYQnACHp73+9ZT7mwkSAcB+BEmo27KU+/sBK9y9K+35MSmP30i5vyXD43yN89niiqLQ907fz/3C+83A98NqsfXAWsDYfT93vTZMnD8EfgSsMrPpZrZ3AXFLL6KEIYlx9zbgduC/Uxb/AvgDcIC7DwFuJjjhRfEacEDK46aU+68CB6TW9YfPryww7GK8BTSkPB4VwzbT9/PV8P4K4GPuPjTlNtDdH05Zf7chqt39B+5+NEEV1cHA52KIT2qQEoYk7UbgfWbWXa00GFjr7lvN7BjgwwVs61fAVWa2T9g+cmXKc/MIfvV/3szqzWwKcAZwd4/3IL+ngBPNrMnMhhBPr7DPhft5APBJgiozCBLsVWZ2OICZDTGzf862ETP7x7BUV0+Q2LYCXdnWl95NCUMS5e4dwB3ANeGijwNfC9s4riFIAlF9laB65mWCxu07U95nO0GCeD+wGvgxcIG7L+zpPuTj7vcSnNAXAI8TtJ301O/DbT0F/C9wa/hevwX+C7jbzDYCzxDsczZ7Az8F1hEcuzXAd2KIT2qQaQIlERGJQiUMERGJRAlDREQiUcIQEZFIlDBERCSSqhp8bPjw4T527NikwxARqSqPP/74ancf0dPtVFXCGDt2LO3t7UmHISJSVcxsWf618lOVlIiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpGUPGGY2W1mtsrMnsnw3GfMzM1seKnjEBGRnilHCeN24LT0heFcxKcAy8sQg4iI9FDJE4a7zwHWZnjqBuDzgOaIFRGpAom0YZjZWcBKd58fYd3LzKzdzNo7OjrKEJ2IiGRS9oRhZg3AF4Froqzv7tPdvcXdW0aM6PFw7iIiUqQkShgHAuOA+Wa2FNgfeMLMRiUQi4iIRFT2CZTc/Wlg3+7HYdJocffV5Y5FRESiK0e32ruAR4CJZvaKmV1a6vcUEZH4lbyE4e7n5Xl+bKljEBGRntOV3iIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkJU8YZnabma0ys2dSln3HzBaa2QIz+62ZDS11HCIi0jPlKGHcDpyWtuxe4B3uPgl4AbiqDHGIiEgPlDxhuPscYG3asr+5+87w4Vxg/1LHISIiPVMJbRiXAH/O9qSZXWZm7WbW3tHRUcawREQkVaIJw8yuBnYCM7Ot4+7T3b3F3VtGjBhRvuBERGQ3fZN6YzO7CDgdONndPak4REQkmkQShpmdBnweaHX3zUnEICIihSlHt9q7gEeAiWb2ipldCvwQGAzca2ZPmdnNpY5DRER6puQlDHc/L8PiW0v9viIiEq9K6CUlIiJVQAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhCR6jZzJowdC3V1wd+ZWUcaSmZ7NSSxoUFERHps5ky47DLYHA4YsWxZ8Bhg2rTkt1djrJqGcWppafH29vakwxCRSjF2bHBST9fcDEuXJr+9CmFmj7t7S0+3oyopEaley5cXtrzc26sxShgiUr2amgpbXu7t1RglDBGpXtdfDw0Nuy9raAiWV8L2aowShohUr2nTYPr0oI3BLPg7fXrxDdRxb6/GqNFbRKTGqdFbRETKSglDREQiUcIQEZFIlDBERCQSJQwRkXKr0vGqNJaUiEg5VfF4VSUvYZjZbWa2ysyeSVk2zMzuNbPF4d99Sh2HiJRIlf5ajlUhx+Dqq99OFt02bw6WV7hyVEndDpyWtuwLwP3uPgG4P3wsItWm+9fysmXg/vav5WpOGoUmwEKPQRWPV1XyhOHuc4C1aYvPAn4e3v85cHap4xCREqjiX8sZFZMAsx2DCy/MnHSyjUtVV1fxiTapRu+R7v5aeP91YGS2Fc3sMjNrN7P2jo6O8kQnUi2Srg6qpF/LcRyLYhJgtn3t7MycdDKNV9W9fqWXzty95DdgLPBMyuP1ac+vi7Kdo48+2kUkNGOGe0ODe3BaCm4NDcHycmlu3v39u2/NzeWLwT2+Y2GWeX/Msr8m2zHIdUxmzHDv06ekx277W9v94Z8s8G+eOsuBdo/hXJ5UCeMNMxsNEP5dlVAcItWrEqqDKmV017iORTHDm0+dGgxUmE9qSWTaNOjqyr9eAaWmbRu38eCPF/D1987mfY2PM3Sv7Rz3sX/gqr9OyR9bVHFknXw39ixhfAf4Qnj/C8C3o2xHJQyRFMX8Gi6FGTOCX8Vmwd9ylnC6xXUsCi2pZFo/SgnDPXvJpLExUixb1m3xWTc86ddOmeUnDX3CB7B512qTBiz0KyfN9ns++7Cveq4jthJGOZLFXcBrwA7gFeBSoJGgd9Ri4D5gWJRtKWGIpKiU6qBKEOexKCQBZnvf9ASWKenMmOHer9+er62vfzuGDNte13+knzjkSe/PluCt6PTJA5/zT02e7b+7aq6veXHtHmHGlTA0vLlItUq/AAyC6qDeOH9DUseiri44jWfS3BxULzU1BVV0meIYPhzWrMn4Wl++HMuw7S6MYxueYcphq2h9fwMnXDqRoc1DcoYZ1/DmutJbpFp1n4Cuvjr/ianWJXUsmpqCXlDpmpth6dL8r1+bfsVBoGvZclZwAM1k6IE1Zn8ee+Uw4LCCQo2DxpISqWbTpgUnpq6u4G8tJ4t8DcBRjkXc3ZB70Oi/ftkGNu+d+YqCVTaKeRPOp7N+wO5PmFG3ckVyV9THUa9VrpvaMKTiVUIDcC2Ko9tspm3U1weNzD35vCJ+5mteXOu/u2quf2rybJ888Dk3Ov08Zvgmdo+pa+DAt7eR2pYRpV0kC9SGIVJh1KZQOmPH9qzqJ9c2UsX4eXU8v5o5ty2m7W/baXthFE9vnYBTxwC28K6hC2k9cgOtZw3luIYn6feNr+auSuvh/sfVhqGEIRKXOE5qSZs5szLbRLI1Lptlv54h6jbSFfl5vfFMB223Lqbtvh20Ld6PZ7dNAKCBtzhu2EJaJ79J69n7cMwFh9B/7/6FbbyH+69Gb5FKU0nDZBSjkofdzta4nOuCuqjbSBfx83r1iddp+9lLtN2/k7YlY1i4/UBgBIN4k+MbFzGtdTZT/qmRoz88kX6Djo4eZyGxF7L/cYijXqtcN7VhSEWr9usiosafRDtNqdowCvi8ls9d6Xde/qB/dGKbT6h/adfqe7Pep4541L89dZbPu+0Z37FlRzz7nC/29P3P8blQLRfuxXlTwpCKVgljO/VElKulk9zHOBJV6jYaG/e8cC5lX15+YIXf/tEH/OIJc3xc32W7Vhlq6/zMUXP9u2fO8vY7n/Od23bGuZfRYk/f/zyfixKGSCWq5l5SUUoY1V6KShd+Xl1mvn3fMT7r3V/288c/4E19VuzatWG2xs8Z/YjfeM5sf/LuheVLEDni3eP7ledzUcIQkXhFKT1kK4VAZSfHtBNt150zfNFfXvLp57f5h5sf9DF1r+7ajRG2ys8d87DfdO5sX3DPIu/c0Zl09IFcn0+uz8XMJ8E2j+EcrF5SIvK2fL2kcnVNrdQuxDNn4pddhqV0d97MQD7KT7mLaYyqe4PWMUtoPX4nreftx6GnH4jVRRh9ttxy9cKDnA36LUC7e893Ko6sU66bShgiCcvXcNyTqqkYq/M6d3T6gnsW+U3nzvZVfUZmjHVjw0hf9JeXvKuzq/iY45Rv/3O1MeX5XI4G9xjOwYkngUJuShjSa1VS28gVV2RPGMUOrd7DxvTOHZ3+5N0L/cZzZvs5ox/xRlu9azOdVMgw8LlE2f987Uep3xElDCUM6aUqrfdVrhnmii1hFNiYvnPbTm+/8zn/7pmz/MxRc32ordv1knF9l/lFB83x2z/6gL/8wIrqaKiPEmMh34O07SlhiPQWlXbCK0XDd54uvTu27PB5tz3j3546y6eOeNT3Zv2uVQ6qf9k/OrHN77z8QV8+d2WwvQK6z1aEqBNARS1ppiWXuBKGrvQWqXSVdgV5tquOGxuLb/DOss0N/fflX4e389CaiWzicAAO6beE8w6dT+vJfWm9eDz7HTWWYFLPUPoV62vWQH19EN/atZU15Em3qFdyT5sWPe6BA3cdg07Y2cMIAQ0NIlL5KmVYiG7XX595kMXvf7/oTe740lep+/jl9Nmxddeyt2jgiq3f5RUfwgXveILW99Zz4sUHMWrSgcCB2TeWaX7vHTtg0CBYvbroGEsq2zEtZm70DINg1sU1lUUcxZRy3VQlJTWlyOqFiqhS6WEjfKb5qM9jhr9Ms3divqZ+pM+d+hVf9VxH4bFVylznhYqrY0OGKkxVSYlUs0IG+quBmfU2r97MIz9bSNsfN9K2YCjzNhzCNo7E6OLIgYu4fPKjtJ56EIMveYK6CcMYBhxb7JtVWoksqkKqm3IpZVVlHFmnXDeVMKRmVFpDdiEilHjefO1N/+s32v2Lx83y4wfP93q2ObjXsdNbGp71zxw9y//45Xm+bun6ROKLtI1K6cZcqBKWMBJNAMCngWeBZ4C7gAG51lfCkKr+R06V7yKsJEaDjfqeWZLd5iGj/P8dO8uP3etp78t2B/dp3OGv2mjvwnzzkFH+1g9+Wvp9KXR/Mr220qoAC5Eh/qOg06s5YQBjgJeBgeHjXwEX5XqNEkYvV+3/yKmylTAaG8u/j4Ue1yzJrgv8I9zuxw2e7188bpbP/+frgulGM2231EmxJ9uv5tJft7T9Hw4veQ0kjBXAMILeWn8CTsn1GiWMXq4W/pG7ZTtJNzaWfx8jHNfU+ahX2ujM65M2H3VSSbGnPyyqtdE8B2phtFrgk8AmoAOYmWWdy4B2oL2pqSnWgyhVptb+kTP9Ck5iH7OWGMyvnDTbJw1Y6Eang/sANvt1A7/u2+r6Z00auxJNrgv8SpkUe/rDopZ+mITiShjx9M0tgpntA5wFjAP2A/Yys4+kr+fu0929xd1bRowYUe4wpZJk6+VS6b1fspk2LZg7uqsr+DttWjL7mGXby2ji1gUt7Nuwia+dPIc5N81n/YY6vrT5avrdcWv27XX30ik05rh69+S70HHmzGDk17q64O/Mmbuvd/31wTUQqYq9JqLWxJF1irkB/wzcmvL4AuDHuV6jKqlerpbaMLLJNupoY2Os+7ny8df8F594yD92aJt/ps/3fBO7v+e2uv6+6JJv+rY3t2XfSJTB8JKodssVV9TvUK10rghR7VVSBN2snwUaAAN+DlyZ6zVKGFJr/8juvuc+XXFF5pNq1OSY4Rilzkd9UP3LuzbZPR/1HyZd7VsbR3tXagz5jnOx80yXOvHn2n4NVjdFUfUJI9gHvgosJOhWeyfQP9f6ShiSmFIlqrh/hWfY3lsM9POY4RDMR33GyDzzURdyQi/2uCTVS6oW2sGKOHY1kTAKvSlhSGwK+acr5S/iXEOFZ7ulxdzV2eUv3r/Ub71ojnf0zTxZ0IaBI6PPR13Lv8Krfd+K/C6WLWEA9wJHxPFmPb0pYUgsCv2nK+VJptCeRCm37X36+w8br9ltPupYJguqhV/h2VR7O1iR38W4EkaUXlL/D7jRzH5mZqNjaWkXiVu+ni+pMo1munlzsDyTUg4vnq0nUWPjnj110tR3buPMNbdxwpiX+fGH5vDs71/Emg4o7H0KWTfXNgo5/kmaNi2Yd7y5GcyCv5U4D3k2SQ91HzWzAP8EzAe+Qnh1drlvKmHUiLjrr2O6UjnrL+hSljAyxN7V0ODL/uO//S/HXO2r+oz0Tsy7spQyujJNsBPHOEqFbKPaf7VXk4RLGFGThQHvAC4HVgOvAOfHEUAhNyWMGlCKk0uh/0SFrl/iE2LnHXf6tuH7eRfmb9SN8n/j5l1v09RnhZ8//gHfuFfmtomMMceRkGMYW6pq2gWqSRW0YTwEvErQlnEdcDpwEHATMD2OIKLelDBqQClOLoWWGIr5p4uxVBRlPuqfXRrOR92TmMulXG0etdiluhiV3EsKOBywLM89H0cQUW9KGDWgFCeXQpJQal/8Pn12/1uik1C++agvPXjO7vNRZ1OpJ8xylDAqOWFWgbJWSWV9MYyPI4ioNyWMGlCKk0shV+9muoo65pPQ9re2+8M/WeDfOGWWn9r4mA9i467NT+y3xD92aJv/4hMP+SuPvZp9fyoxMWRTjpO5qr16pCISRrlvShg1oFQnlygn2ajXPBR4Etq6Yas/8KP5ft3Js/y9w9q9gU27NnV4/xf84++Y7b/81EP+2vw3ou1HNf6SLnWSq+WuvmUQV8KwYFvVoaWlxdvb25MOQ3pq5sziphst9nXd6uqC00w+ZsGAgFlsXb+VubcvpO3362l7agiPrD+ErQwEYNKARbQe/Dqtp/TjxEsmMOLQ4dHjg6BLaqbpRZubgwEKeysdlx4xs8fdvaXH21HCkKqQPgc2BNcpFNKHPttJJ13aSSjzfNQDds1H3XrIKlpPHcC7L5lA44RhBe3WHrIltTxJrObF8fn3YnEljMSrmQq5qUqqQiRRxx5HHfaMGe719bmroxoafMuPbs07H/UfvjTP1760rjL3s1ZVW9tOBSGmKqm+Pc9d0quk/9Jbtix4DKX9pRfXFa5muz/u25euQXtj69exvt++3LDz03zz389nJ/X0YSctey3k08c8zJQP7MXxl0xk7/0PAw4rahciuf76zL+kNRdD8P1SaSJRqpKSwiRVlxzH+2bZxjKaGMsy6tnOPw5eyJR/WEvr6YM57uKJDBo1qCdRF6enbTUiaeKqkkpsxj2pUkmNZZNpFjQzmDo170vXLlnH7784D1+WOcYmVnDft59gfcdOHto4iesfmsIpVx2dTLKAzDPxlVLc40BVy7hSUrg46rXKdVMbRgUoZx17ep31ySfv2b0yQ5fTVc91+D2ffXiP+aiXcoDaB9LF3Y23WrsF1zjUrVYSUa7eKpnexyxjD6LO0fvz63/9JW337aBt8X48u21CEBZvcdywhbROfpPWs/fhnQOeov6TH1dPm1RxVzGq+2tFUi8pSU45eqsUMLFQJ+bgPoiNfmrjY/6NU2b5QzcvyDwfdSX2tEkyprgviNMFdhUJlTCkpkW9yA7YMGAki358P0edN5G+A6qs41/S1xeohNErqNFbatbSB19h0177ZnxujxTS0MCQW77LMRcfXn3JAgqfzClumToTFNqNN7WRe9MmqK/v2fakYilhlJp6jOTkXc6Svy/jtosf4IIDH6S57yuMe/f+XLbpu2wOh9vYtW5DA3bFFcnNllaKzzLpGdR6OgNddwlp2bKgRLhmTbCdxsbqnNFOcoujXqvYGzAUuAdYCDwPvCvX+nu0YVRifXQq9RjZQ1dnly/6y0v+k2lt/uHmB3ebj3qErfJzxzzsN5072xfcs8g777izcj7fUn2W1X5ld7XH30tQC20YZvZz4AF3v8XM+gEN7r4+2/q7tWEkXfcbhepz8S7n+T8toe2uV2l7qC9tKw/k9a6RAIysW8WUMS/SevxOWs/bj0NPPxCrszxbTEipPstq+B7norGvqkLV95IChgAvk2Vypky33UoY1fDLphf2GOnc0ekL7lnkN507288d87Dva6t27faYulf9w80P+k+mtfnC/1viXZ1d5Q+w2FJpKT/LnpaUkyxpV8P/ocRWwkgyYRwJPArcDjwJ3ALslWG9y4B2oL2pqentI5DEybjQf8xa/2eaMcO7mpq8y8w3DBzp3xtyrTfa6l272T0f9S0XzvHF9y0NEkSSJ7eeVCtV6meZdLVn0u8vkdRCwmgBdgLHho+/D1yX6zWJljCKnQe6xv6Zuuej/tNRV/sWBuy2b28x0G/e98t7zkfdLenj0ZPvTNKxZ1MJiazS2xKlJhLGKGBpyuN3A/+b6zW7JYxy/wMX+49Z5f9M2eajfpnm6McjdR7tJE9uPS2VVuJn2QurPaVwVZ8wgn3gAWBieP9a4Du51i95L6lc26uGKrAYZJqP+jxm+Ms0eyfmq/uO9Ife9xXvynY8YM99yDePdpzHMNcxq4Rf43GrxX2S2NVKwjgybJ9YAPwO2CfX+iUdGiRfiaUaqsCKsHXDVp9z01MZ56M+rP9iv3XMl3xH3/57xtHYmP3kH+UkXYpjmO+YVWq1Uk/U4j5J7GoiYRR6K2nCyJcQqqUKLI8t67b4rBue9GunzPKThj7hA9i8a9OTBiz0KyfN9ns++7Cveq4jdxyNjdlLXakx5iqJxH0MoxyzSqxW6qla3CeJlRJG3KJUOZXzHzOmeN7qeMvv+/bj/uV3z/IThzzp/dgabIZOnzzwOf/U5Nn+2y/M9dUvrCk8jihVTPnaLuI8hqrPF8koroRRhYPvlEhTU+YLs5qa3r5fziki88WTZarUrRu2MWfDEbT96U3anh7Go28ewg6Ooo5OjmpYxJVHP0Lr+xs44ZKD2WfcocChPYsj3zHLNuVoKS5Mi/IZikjx4sg65bol2oZRbkW2qSylycG9Dzv82L2e9s8fM8v/99pHff2y9fHHEfWYlatkVmmfoUiFQFVSJVBpdcEZ4lm3dL3/8cvzvIvM1S9dmP/l64/5xpUbSxpHpOeSUGnxiFSAuBJGbc+HMXNmMEz08uVBtcT111fH+Dwp1i5Zx5xbXqDtr1tpW7gvT22ZiFPHUpppJsOIpkmOU1UDx1ukFmk+jHzSh10O6/gTH148zxDZHc+v5tefe4T/OKKNIwYuYvhBQzjnW8dy85PH8EF+w9r++9FFHQfsswn69dt920nOO1Cpx1tEYlO7JYxKHCk2w8ikXQMGMm/K55nxykm7zUc9kM0ct89Cphy1Mft81PX1sPfesHZtMr/oU0sUdXXQ2bnnOr1oZF6RShVXCaN2E0YlDrucJYktpZl38AwnNC6k9ehNtH6wkZZpE+k3qF/e1yZ2Qs40LHcmGuZaJHFxJYza7VZbIV0sV8x7lbbbX6ZtVic/WbY8Yx1gM8tZ91Z/6htyfJ5Jz8yWLtPUopmoS6tIzajdNow45iouwtIHX+Hn//YgF094gPH1y2l6536cf/Px3PPCJFbXjcz4GutTR/2g/rmn/cx24k3qhBwlUWkuZ5GaUrsJo6dzFUfQPR/1rRftPh/1RbecwB+WHM4RI17lhrPbeOIXC1m9dTD73vHfeyYxCOr+8zUUJ5QAs8qWqPr00VzOIjWqdtswSsC7nMX3LmX2nStoe7APbSvGs7JrNAAjrIMT93uR1nduZ8p5ozn8rIOo65shH3c3FGeqLuuWrV2ikrqtVvvUoiK9iBq9yyDffNStY15kSjHzUedrMK6WhuJKSmAikpUSRgl07ezi2d+/SNvdr9H2SD/mvHoQq3wEAGPqXqP1gJdoPaGT1mn7c/Cp46IniHTZejx1U1dUEYmRekllUuAv3q6dXSz49WLafvk6bXP7M+f1Cazxg4GDOaDPSk4dt4jWdy+k9fwmDjypCasbHU+cuRqM1VAsIhWqdhJGltFbgV1Jo3N7J0/96gXa/ucNZs8byAOrJrLeJwITGdd3OWcc+BytrUbrBc2MPWF/rG5MaWLN1uW3Tx+1AYhIxaqdKqks1TzbGkfz/WN+QVv7XjzYcTAbGQLAQfVLaR23gtaTjNYLx9H0rhIlh0zUYCwiZaQ2jHRZruzuwuhDFxP7vUTr+FdofU8fWi8ez5iWmKqXiqUGYxEpE7VhANs2buPROxbS9rt1/JuNZKS/vsc6Wwbvy2sPrmLUpPHA+PIHmU05J2MSEYlBVV24553O7Buf4qsnzeY9+zzJ0CFdnHjlEXz5/in83U9ij/KFGXt95IOMmrRv9o3mGT224lRbvCJSMxKvkjKzPkA7sNLdT8+1bp0d7c7jGF0cMfBdEDEsAAAKRklEQVQFphzyBq2nDuDk4fMZfM2nM1/XkKttoNraEqotXhGpCDXThmFm/wm0AHvnSxij9proP/3PO8P5qIe+/USx1zVU2giw+VRbvCJSEWpiAiUz2x/4AHBLlPX3P3QwZ1x3zO7JAvIPhJfp+ZkzsyeZXMknSZU2Yq2I9CpJt2HcCHweyDoOhpldZmbtZtbe0dGReaV8I7amP99dtZP9TSuzbaDSRqwVkV4lsYRhZqcDq9z98Vzruft0d29x95YRI0ZkXinTSK7dMl05nW8uB/dgnUpTaSPWikivkmQJ43jgTDNbCtwNvMfMZuR8xdq1mXsIpQ5lDsEV05B9iO0oVTilquYptpdT93Ubmzfn3z8RkRJIvNEbwMymAJ/N1+jd0qePt6eO4lpsD6F8jeRQmobkYns5qXeUiPRATTR6Fyx9yO/Nm4urOspVhQWlq+bJVBUWZR+KfZ2ISIwqooQRVYuZ7zEwSLFzR6QOzTFsWLBs7drSDtORZfiSvPtQ7OtERNDQIG8rtodQEkNzZBulNkovr2JeJyISo+qqkqpLCzfJHkLFNF4X28tJvaNEpAJUV8Jobg5uZsn2EOpuhF62LKgq6p57I1/SSO3NVcg+FPs6EZEYVVcbRkuLt3/608kPC64hOkSkitTMWFKFaBk/3tvfeCP57qVqhBaRKtI7u9WuXFkZ3Us1RIeI9ELVlTC2b8+8vNyD76kRWkR6oepKGP36ZV5e7l/2aoQWkV6ouq7DGDMGMrVhJPHLXlOsikgvU10ljGHD9MteRCQh1VXCAP2yFxFJSHWVMEREJDFKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkiSUMMzvAzGaZ2XNm9qyZfTKpWEREJL8khwbZCXzG3Z8ws8HA42Z2r7s/l2BMIiKSRWIlDHd/zd2fCO+/CTwPjEkqHhERya0i2jDMbCwwGZiX4bnLzKzdzNo7OjrKHZqIiIQSTxhmNgj4NfApd9+Y/ry7T3f3FndvGTFiRPkDFBERIOGEYWb1BMliprv/JslYREQktyR7SRlwK/C8u38vqThERCSaJEsYxwPnA+8xs6fC29QE4xERkRwS61br7g8CltT7i4hIYRJv9BYRkeqghCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJEoYIiISSaIJw8xOM7NFZvaimX0hyVhERCS3xBKGmfUBfgS8HzgMOM/MDksqHhERyS3JEsYxwIvu/pK7bwfuBs5KMB4REcmhb4LvPQZYkfL4FeDY9JXM7DLgsvDhNjN7pgyx9dRwYHXSQUSgOONTDTGC4oxbtcQ5MY6NJJkwInH36cB0ADNrd/eWhEPKS3HGqxrirIYYQXHGrZrijGM7SVZJrQQOSHm8f7hMREQqUJIJ4zFggpmNM7N+wIeAPyQYj4iI5JBYlZS77zSzTwB/BfoAt7n7s3leNr30kcVCccarGuKshhhBccatV8Vp7h7HdkREpMbpSm8REYlECUNERCKpmISRb5gQM+tvZr8Mn59nZmNTnrsqXL7IzE5NMMb/NLPnzGyBmd1vZs0pz3Wa2VPhraSN+xHivMjMOlLi+WjKcxea2eLwdmHCcd6QEuMLZrY+5bmyHE8zu83MVmW7/scCPwj3YYGZHZXyXDmPZb44p4XxPW1mD5vZESnPLQ2XPxVX98sexDnFzDakfLbXpDxXtqGEIsT5uZQYnwm/j8PC58pyPM3sADObFZ5znjWzT2ZYJ97vp7snfiNo9F4CjAf6AfOBw9LW+Thwc3j/Q8Avw/uHhev3B8aF2+mTUIwnAQ3h/Su6Ywwfb6qgY3kR8MMMrx0GvBT+3Se8v09ScaatfyVBx4hyH88TgaOAZ7I8PxX4M2DAO4F55T6WEeM8rvv9CYbjmZfy3FJgeIUczynAn3r6fSl1nGnrngH8vdzHExgNHBXeHwy8kOF/PdbvZ6WUMKIME3IW8PPw/j3AyWZm4fK73X2bu78MvBhur+wxuvssd98cPpxLcG1JufVkyJVTgXvdfa27rwPuBU6rkDjPA+4qUSxZufscYG2OVc4C7vDAXGComY2mvMcyb5zu/nAYByT33YxyPLMp61BCBcaZ1HfzNXd/Irz/JvA8wQgaqWL9flZKwsg0TEj6ju9ax913AhuAxoivLVeMqS4lyOzdBphZu5nNNbOzSxBft6hx/lNYRL3HzLovoCzXsSzovcKqvXHA31MWl+t45pNtP8p5LAuV/t104G9m9rgFQ/Ek7V1mNt/M/mxmh4fLKvJ4mlkDwYn21ymLy348LaiinwzMS3sq1u9nxQ8NUo3M7CNAC9CasrjZ3Vea2Xjg72b2tLsvSSZC/gjc5e7bzOxjBCW39yQUSxQfAu5x986UZZV0PKuGmZ1EkDBOSFl8Qngs9wXuNbOF4S/sJDxB8NluMrOpwO+ACQnFEsUZwEPunloaKevxNLNBBAnrU+6+sVTvA5VTwogyTMiudcysLzAEWBPxteWKETN7L3A1cKa7b+te7u4rw78vAbMJfg2UQt443X1NSmy3AEdHfW0540zxIdKK/GU8nvlk24+KG/rGzCYRfN5nufua7uUpx3IV8FtKU6UbibtvdPdN4f3/A+rNbDgVeDxDub6bJT+eZlZPkCxmuvtvMqwS7/ez1A0zERtv+hI0uozj7Qatw9PW+Xd2b/T+VXj/cHZv9H6J0jR6R4lxMkHD3IS05fsA/cP7w4HFlKjBLmKco1PunwPM9bcbwl4O490nvD8sqTjD9Q4haES0JI5n+B5jyd5I+wF2b1R8tNzHMmKcTQTte8elLd8LGJxy/2HgtATjHNX9WROcaJeHxzbS96VccYbPDyFo59grieMZHpc7gBtzrBPr97NkB7uInZ9K0Mq/BLg6XPY1gl/qAAOA/wm/9I8C41Nee3X4ukXA+xOM8T7gDeCp8PaHcPlxwNPhl/xp4NKEj+U3gWfDeGYBh6S89pLwGL8IXJxknOHja4Fvpb2ubMeT4Nfja8AOgnreS4HLgcvD541gIrAlYSwtCR3LfHHeAqxL+W62h8vHh8dxfviduDrhOD+R8t2cS0qCy/R9SSrOcJ2LCDrcpL6ubMeToFrRgQUpn+vUUn4/NTSIiIhEUiltGCIiUuGUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDpAfC+QjeF97/upndlHRMIqWiwQdFeuYrwNfCgeYmA2cmHI9IyehKb5EeMrM2YBAwxYN5CURqkqqkRHrAzP6BYOaz7UoWUuuUMESKFM5cNpNgVrNNZlayGfVEKoEShkgRwpnWfgN8xt2fB64jaM8QqVlqwxARkUhUwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJ5P8DY8IBC9SCZV8AAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 6, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -1318,8 +1234,10 @@ "\n", "## Momentum based GD\n", "\n", - "The stochastic gradient descent (SGD) is almost always used with a *momentum* or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically \n", - "implemented as follows" + "The stochastic gradient descent (SGD) is almost always used with a\n", + "*momentum* or inertia term that serves as a memory of the direction we\n", + "are moving in parameter space. This is typically implemented as\n", + "follows" ] }, { @@ -1350,7 +1268,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where we have introduced a momentum parameter $\\gamma$, with $0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a running average of recently encountered gradients and $(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when $\\gamma=0$, this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is" + "where we have introduced a momentum parameter $\\gamma$, with\n", + "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", + "indicate the gradient is to be taken over a different mini-batch at\n", + "each step. We call this algorithm gradient descent with momentum\n", + "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n", + "running average of recently encountered gradients and\n", + "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n", + "used in the averaging procedure. Consistent with this, when\n", + "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n", + "earlier. An equivalent way of writing the updates is" ] }, { @@ -1370,8 +1297,11 @@ "\n", "## More on momentum based approaches\n", "\n", - "Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass $m$ moving in a viscous medium with drag coefficient $\\mu$ and potential \n", - "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$, then its motion is described by" + "Let us try to get more intuition from these equations. It is helpful\n", + "to consider a simple physical analogy with a particle of mass $m$\n", + "moving in a viscous medium with drag coefficient $\\mu$ and potential\n", + "$E(\\mathbf{w})$. If we denote the particle's position by $\\mathbf{w}$,\n", + "then its motion is described by" ] }, { @@ -1420,8 +1350,12 @@ "metadata": {}, "source": [ "## Momentum parameter\n", - "Notice that this equation is identical to previous one if we identify the position of the particle, $\\mathbf{w}$, with the parameters $\\boldsymbol{\\theta}$. This allows\n", - "us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as:" + "\n", + "Notice that this equation is identical to previous one if we identify\n", + "the position of the particle, $\\mathbf{w}$, with the parameters\n", + "$\\boldsymbol{\\theta}$. This allows us to identify the momentum\n", + "parameter and learning rate with the mass of the particle and the\n", + "viscous drag as:" ] }, { @@ -1437,13 +1371,30 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n", + "Thus, as the name suggests, the momentum parameter is proportional to\n", + "the mass of the particle and effectively provides inertia.\n", + "Furthermore, in the large viscosity/small learning rate limit, our\n", + "memory time scales as $(1-\\gamma)^{-1} \\approx m/(\\mu \\Delta t)$.\n", "\n", - "Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. \n", + "Why is momentum useful? SGD momentum helps the gradient descent\n", + "algorithm gain speed in directions with persistent but small gradients\n", + "even in the presence of stochasticity, while suppressing oscillations\n", + "in high-curvature directions. This becomes especially important in\n", + "situations where the landscape is shallow and flat in some directions\n", + "and narrow and steep in others. It has been argued that first-order\n", + "methods (with appropriate initial conditions) can perform comparable\n", + "to more expensive second order methods, especially in the context of\n", + "complex deep learning models.\n", "\n", - "These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). \n", + "These beneficial properties of momentum can sometimes become even more\n", + "pronounced by using a slight modification of the classical momentum\n", + "algorithm called Nesterov Accelerated Gradient (NAG).\n", "\n", - "In the NAG algorithm, rather than calculating the gradient at the current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one calculates the gradient at the expected value of the parameters given our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1})$. This yields the NAG update rule" + "In the NAG algorithm, rather than calculating the gradient at the\n", + "current parameters, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t)$, one\n", + "calculates the gradient at the expected value of the parameters given\n", + "our current momentum, $\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma\n", + "\\mathbf{v}_{t-1})$. This yields the NAG update rule" ] }, { @@ -1496,11 +1447,17 @@ "the steep computational price of calculating or approximating\n", "Hessians.\n", "\n", - "Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM.\n", + "Recently, a number of methods have been introduced that accomplish\n", + "this by tracking not only the gradient, but also the second moment of\n", + "the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and\n", + "ADAM.\n", "\n", "## RMS prop\n", "\n", - "In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule for RMS prop is given by" + "In RMS prop, in addition to keeping a running average of the first\n", + "moment of the gradient, we also keep track of the second moment\n", + "denoted by $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}_t^2]$. The update rule\n", + "for RMS prop is given by" ] }, { @@ -1540,12 +1497,31 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\beta$ controls the averaging time of the second moment and is typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions.\n", + "where $\\beta$ controls the averaging time of the second moment and is\n", + "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", + "typically chosen to be $10^{-3}$, and $\\epsilon\\sim 10^{-8} $ is a\n", + "small regularization constant to prevent divergences. Multiplication\n", + "and division by vectors is understood as an element-wise operation. It\n", + "is clear from this formula that the learning rate is reduced in\n", + "directions where the norm of the gradient is consistently large. This\n", + "greatly speeds up the convergence by allowing us to use a larger\n", + "learning rate for flat directions.\n", "\n", "\n", "## ADAM optimizer\n", "\n", - "A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and $\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below)" + "A related algorithm is the ADAM optimizer. In ADAM, we keep a running\n", + "average of both the first and second moment of the gradient and use\n", + "this information to adaptively change the learning rate for different\n", + "parameters. In addition to keeping a running average of the first and\n", + "second moments of the gradient\n", + "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n", + "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n", + "performs an additional bias correction to account for the fact that we\n", + "are estimating the first two moments of the gradient using a running\n", + "average (denoted by the hats in the update rule below). The update\n", + "rule for ADAM is given by (where multiplication and division are once\n", + "again understood to be element-wise operations below)" ] }, { @@ -1626,9 +1602,16 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n", + "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", + "second moment and are typically taken to be $0.9$ and $0.99$\n", + "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n", "\n", - "Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance $\\boldsymbol{\\sigma}_t^2 = \\hat{\\mathbf{s}}_t - (\\hat{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The update rule for this parameter is given by" + "Like in RMSprop, the effective step size of a parameter depends on the\n", + "magnitude of its gradient squared. To understand this better, let us\n", + "rewrite this expression in terms of the variance\n", + "$\\boldsymbol{\\sigma}_t^2 = \\hat{\\mathbf{s}}_t -\n", + "(\\hat{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n", + "update rule for this parameter is given by" ] }, { @@ -1725,27 +1708,11 @@ }, { "cell_type": "code", - "execution_count": 28, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd0AAAEWCAYAAAAjEk0ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xd8FHX+x/HXZ1MIgUBC7wnSayAECBbkrKgo6qmoWLAhVu7On56np2Kvd/besKCIFBty2MFCgACBUKVLqKGHElL2+/tjBlxjKkn2u+XzfDz28djdmZ15T/1M2xkxxqCUUkqpmuexHUAppZQKF1p0lVJKKT/RoquUUkr5iRZdpZRSyk+06CqllFJ+okVXKaWU8hMrRVdEThCRFTXU7btE5I2a6HaoE5GxIvJQGc0fEpHtIrLFn7lClYgYEWlvO4ctIjJGRN6vYjeGi8hXZTQfJCLZVexHwE+nqozL6lofi8g6ETmlgu2OEJGfqtCvV0TknqP9fTndXiIig2qi21CBouuOyIMikisiu0XkFxEZJSJHXbCNMT8aYzod7e99sv1pgTLGPGKMubaq3Q5GNblyEJE2wG1AV2NMs5roRwVztBURr4i8XMnfVWkhV4HJGDPOGHPa4c9VXQZE5AcRCen1R/FxVF3rY38yxowyxjxY1e6UtKNhjOlmjPmhqt0uTUUL59nGmDggEXgM+Cfw5tH0UEQij+Z3yro2wA5jzLaSGvpxul4B7AKGiUgtP/WzTOII61M1ulwrfxGRCNsZqsQYU+YLWAecUuy7foAX6O5+rgU8BfwGbAVeAWq7zQYB2TiFegvw3uHv3Ob/BCYW6/6zwHPu+6uAZUAusAa43v2+DnDQzbHPfbUAxgDvu+1MA24u1u2FwPnu+87A18BOYAVwURnjoQHwNrAJZ6X/iU+z64BVbnc+A1r4NDPAjcBKdxgeBNoBvwB7gQlAdLFxdRew3R33w3269QNwrc/nEcBP7vuZbr/2u+NimPv9ECAT2O32s6fP73sD891cHwHjgYdKGPZTio3rsUCS279r3Ok+0233HGCJ278fgC7F5qXbgUVuzjeBpu50ygW+ARLKmAYCrAZuwJnPLvBpdjhPZPHxBXQB8oAiN/9ut3l94F0gB1gP/BvwuM0igP+402EtcLNv991uPwz87I6b9pQyr/rkuR3YjDMPXe12r30Az29nAQvc320AxpQwvotP/zS3X7txlrVBPr9pC8xwc30NvIC7rJYw/DOAv7rvj3P7dZb7+WQgsyLLgM8w3gZsc8f/VaX082F3Hslzf/+Czzgd5Y7T3cCLgPj87mp3uu8CpgOJZczDH+OsB/e4ebv5NBvrdnuqO45mA+2KrRc3uNNjHnCCT7Mx/L7emwrcUqy/i4DzyhpHPu22BibjLBc7fMZDO+A797vtwDggvqxa4dOsIc68uheYgzNf/uTTvNR1sTteXga+dHOf4n73kNt8GTDEp/1IN3tKWeMcGAkUAPnuuPjcdzhw6slBoEGxdeZ2IKqy0/5IN8ptoZQRibOg3eC+f9odoQ2AOOBz4FGfBbsQeBynONfmj0U3ETgAxPms7DYDaT4LfjucFe6JbrspPt3OLpZrDL/PfFcAP/s064qz0NTCKdobcFaUkT4js2sp42EqTmFKAKKAE93vT3J/l+J293ncFZDPAvspUA/oBhwCvgWOwVnpLwWuLDau/ut260ScmayTbxHx6fYI/jjj/mEl7g7TNqC/O16vdKdnLSAap9D83R2eC3BmwD8V3ZLGNb+vdN91x2VtoKOb91S3m3fgFIfDK/l1QDpOoW3pZpvv5ozBWaDvK2NePMEdfwnueP68hDx/KroljSv3u3fdaRPn/v5X4Bq32Sh32rRy+/cNfy66v7nTNNId3rLm1cE4Gwrd3fH1QfHpFYDz2yCgB84RsZ5u/nPLmP4tcVbIZ7q/OdX93Nj9zSyffg3EKSylFd0HgOfd93fhbGw97tPs2QouA4eH8QF3PJ7pTpcSN+4otoz5dPMLIB7niE8OMNhtNhRnHu/izgf/Bn4pYx6+Gmd+qwU8g7vxYH4vLjtwdmoicYraeJ/ml+EUr0icjYgtQEwJ672LgNk+v0t2uxtdxjg6vD6OwNlYetqdrjHA8W6z9u40rQU0xilgz5RXK9xm43E2+OrgLAMb+X1jqcx1sTte9uBsfHncTGP5vejeC4zz6ddZwLJKjPOHimU9Mhw466TrfJo9CbxyNNP+SDfKbaH0opsO3I2zgtnPH7fIBgBrfSZo/uGZo5QV+E/AFe77U4HVZeT5BBhdUndKmPni3GyJ7ueHgbfc98OAH4v99lVKWOkDzXH28v60oOLsrT3h87kuTvFK8pnBj/NpPg/4p8/n/+DOuPy+gqjj03wCcE9JKwTKX+G8DDxYLO8KnJXrQJy9KN8t9l+Kz4BlTLMkt3/H+Hx3DzDB57MHZ+Ea5DMv+e5JTQJe9vl8Cz57dCVkeONwc3ceKwCaFMtToaKLs3LJx2cjC7ge+MFnYbvep9kp/LnoPlDOsuM7r74FPObTrGPx6RVo81sJ/X4GeLqM6f9P4L1iv5mOs7HXpoR+fUDpRfdkYJH7/n84RyzS3c8z+P1oVfHpWlJBOVhsvtiGu1FfQn+PzDPFunl8sXF0p/t+Gu6Gms88f4CK7PE4RdwA9d3PY4E3fJqfCSwv4/e7gGT3/Rh+X+/FuM06uJ+fAl4qZxwdLroDcDYqIiuQ/1xggc/ndZRcKyLcebSzz3eP8HvRLXNd7I6Xd4s1H8vvRbc9zgZcrPt5HHBvJcZ5WUX3WuA7973gbBwMrMq0r8p5qJY4hwIaA7HAPPdCq904C0ljn3ZzjDF5ZXTrA+AS9/2l7mcAROQMEUkXkZ1ut88EGlUkoDEmF2eP4WL3q0twJgg4e9j9D2d2uz0cKOkiodbATmPMrhKatcDZYzzcz304W5UtfdrZ6vP+YAmf6/p83mWM2e/zeb3bj6ORCNxWbBhbu91rAWw07tzi06/K2uDzvvi48LrNj3ZcHCEitYELcaefMWYWzp7mpUeRGZx5KIo/DvN6n6wt+OOw+b4v8bty5tXi3StrXAfE/CYi/UXkexHJEZE9OHv/xZc932FKBC4sNr8dj7MR0aKUfpVmFtBRRJoCvXD2qFuLSCOcPcGZZfy2uB3GmEKfzwcoZT4rg+8V+76/TwSe9RnenTgr55bFfo+IRIjIYyKyWkT24qzc4Y/jtLT+ICL/JyLLRGSP26/6lLAudNe1HwGXudcaXIJzWq8iWgPri42vw/1vKiLjRWSjm//9kvpfgsY4e4Klzf8VWReXtPwBYIxZhXOI92wRicU5xfWBm7ki47wsk4ABItIcZ0fFC/zok7tC097XURVdEenrdvgnnMMAB3GOk8e7r/rGGN+Z2pTUHR8fA4NEpBXOeYfDI6wWzkA/BTQ1xsTjHNeXCnYX4EPgEhEZgLMF+L37/QZghk/meGNMXWPMDSV0YwPQQETiS2i2CWfk42aug3MIaGMFspUkwe3GYW3cfoCz1x7r06y8q4g3AA8XG8ZYY8yHOIfwW4qI+LTf5ijy+k6D4uNCcBbiox0Xvs7DOWT6kohscf+21BJnLwqccQOlj5/i88p2nK3vRJ/v2vhk3YxzaPmw1iVkOtLNCsyrm4t1o6xxHSjz2wc4p41aG2Pq41yrIcV+7zteN+Ds6frOb3WMMY/hDH9J/SqRMeYAzl76aGCxMSYf50jMP3COhG2v9JBWTEXWKb424BwR8R3m2saYX0po91KcQ5Kn4BTMJPf74uP0T0TkBJzTNRfhHAGJxznkWtpv38EpXCcDB9yN1IoOT5tSLox7BGf89DDG1MM53F1udpw950JKn/8rsi4ub7p8iLNxMRRY6hZiKH+cl9ldd8P3K5y98UtxDvcf/k1lpv0RlSq6IlJPRIbgHJ9/3xiT5e7NvA48LSJN3PZaisjpFe2uMSYH57DO2ziHpZe5jaJxjsPnAIUicgZwms9PtwINRaR+GZ3/Emcl9QDwkZsXnHM0HUXkchGJcl99RaRLCfk24xxKeElEEtx2B7qNPwSuEpFe7or3EZzzKesqOvwluF9Eot0FbQjORgk4F0SdLyKx7iX/1xT73Vacc3eHvQ6McvdYRETqiMhZIhKHsydRCNzqDs/5OHsQVTEBOEtEThaRKJzzTodwVpZVdSXOIdoeOHs+vXDO8SSLSA93HtqIs3UfISJX45xfPWwr0EpEogGMMUVu3odFJE5EEnFW6If/6zgBGO3Oy/E4h07LUt68OgEYISJd3a3x+0rrUADNb3E4e9x5ItKP8o8qvI+zt3G6Ow1ixPlbXytjzHogw6dfxwNnl9O9GTgXsM1wP/9Q7HNJii8DlVXZ378C/EtEugGISH0RubCUduNwlocdOBuHj1SiP3E4y2sOECki9+JshJbILbJenNMJxfdyyxrGOTgbSI+564sYETnOJ8M+YI+ItMS5MLBc7rI2GRjjrru68vvGMlRiXVyG8TjL2w34HCml/HFeken9Ac71QRcU63Zlpv0RFS26n4tILk5lvxvnYoirfJr/E+eEcrq7C/8NUNn/fX2AszVyZKDcw8O34qywduEs9J/5NF+OsxJa4+7i/+kwrDHmEM4EL6nbp+Ecet6Ec1jn8MVeJbkcZ89oOc45ob+53fkG51zmJJyZtR2/H84+GltwhnUTzqHUUe5wgnNxQz7OjPIOvx8qP2wM8I47Li4yxmTgXOn6gtvNVTjnwHD3HM53P+/E2ZKbXIXcGGNW4Gz9Po+zJ3k2zt/N8qvSXXcBPxnnXOQWn9c8nFMZhxfg63BWBDtwLiLyLfbf4VxVvUVEDu8l3YKzh7wG56jNBziFHZwNlq9wrvpcgLPxVohzdWtJw17evDoN55zodzjT4btyBjsQ5rcbgQfcZf9ed9hKZYzZgLNXcRdOcdiAMz0Or2cuxbmobyfORse75WSbgbPSnFnK55KMwWcZKKf7JXkWuEBEdonIc+W1bIyZgrPeGO+u+xYDZ5TS+rs4h1U34lzQll6JXNNx5vVf3W7kUcYhV5/+9eD3DcnDxlDKOHIL5Nk450l/w7nye5jb+H6cC/j24Jy2q8z64macQ+VbcM6jvu3Tz8qui//E3VCdBRyLc2j9sPLG+ZtAV3dcfFJK5z8DOgBbjDELffpZmWl/hPzxlJ6ySZy7oLxvjGlVXrvKv9w911eMMYnlthwkdH4LbSJyBTDSGHO87Szqd2H9h36lSiMitUXkTBGJdPe07wOm2M6lVEW4pzBuBF6znUX9kRZdpUomOIfTduEcXl6Gc4hVqYDmXk+Tg3Ma6oNyWld+poeXlVJKKT/RPV2llFLKT/Qm5WVo1KiRSUpKsh1DKaWCyrx587YbYxqX32b40aJbhqSkJDIyMmzHUEqpoCIiR3N3u7Cgh5eVUkopP9Giq5RSSvmJFl2llFLKT7ToKqWUUn6iRVcppZTyEy26SimllJ+EXdEVkb+LyBIRWSwiH4pIjO1MSimlwkNY/U/XvXH9rUBXY8xBEZmA8zipsVaDKaXCivF62bRuBVtXpJO/ayMmby8SFYOnbiPiE5Np0yWVmNp1bMdUNSCsiq4rEqgtIgU4DzXeZDmPUipMrFk8m20/vkXbrV/Tkh20LKmlhXDw02jm1+2P9LyQnicPJyIyHFfVoSmspqQxZqOIPIXzcOaDwFfGmK982xGRkcBIgDZt2vg/pFIq5KzM/JED0x8g+eAcWplIltTpx7qkv9Cg4wAat+pA3foNOJR3gF1bs9m2KoOCVTNot/07Gs36kez0R9nc61ZSzx6FeMLujGDICaunDIlIAjAJGAbsBj4GJhpj3i+p/dTUVKO3gVRKHa09u7azfNzt9M2Zwh6JY3nScLqe/XfqN2xa7m+LCgtZ+PV71Mt4nvZFq1kW1ZXYv75IYucUPySvGhGZZ4xJtZ0jEIXbZtMpwFpjTI4xpgCYDBxrOZNSKgQtz/iWg8/2JzVnCnOaXEDE3zIZMOKxChVcgIjISFLOuIpj7prL3J4P0KxgA40/HMycSc9gvN4aTq9qSrgV3d+ANBGJFREBTsZ5OLlSSlWb9A8fpt3nF+KVCFYP/ZS0m96gXnzDo+qWJyKCvuePpmjkz6yJ6UK/rPuY89I1FBbkV3Nq5Q9hVXSNMbOBicB8IAtn+F+zGkopFTK8RUWkvzSStBVPsLhOf+rc+gsdU06slm43apFIl9u/Jb3ZcPpvn8zi/57NgX17qqXbyn/CqugCGGPuM8Z0NsZ0N8Zcbow5ZDuTUir4FRbks+CZC0jb9hHpTS4i+bYvqJ/QqFr7EREZSdqol5jd9W56HJjN2ufOJu/Avmrth6pZYVd0lVKquhUVFpL5/CX0yf2OWcfcSv9Rr+KJiKix/vW/6A4W9HmMLocW8etzQzmUd6DG+qWqlxZdpZSqAm9REfNevJLUvd+QnnQTA6540C9/7Uk9ZxTzku+nZ14GS164GG9RUY33U1WdFl2llKqC2W/9g367vmBWq6tJG/GIX/vd9/zRpLf/Gyn7ZjD77f/za7/V0dGiq5RSR2nupy8xYONY5jQ4m7Sr/2MlQ/9L72NOwhAGZL9FxmcvW8mgKk6LrlJKHYXls78ief49LK7Vi96j3rR2tyjxeOh9w1ssie5Jt3n3snbpXCs5VMVo0VVKqUravuU3Gk27jq2eJrQeOYGo6FpW80RF16Lp1ePYL7F4Jo5gf+5uq3lU6bToKqVUJRQVFrLl7cupYw5QeOG7Fb7DVE1r1KwNW055ntZFG1n6xkjbcVQptOgqpVQlzHnvbrofyiSr59207drXdpw/6H78OcxufRV990xnwVcl3lJeWaZFVymlKmjprGn0W/cqGfVOoe95t9qOU6I+lz/K6oi2tPnlLnblbLYdRxWjRVcppSpgf+5u4r+6lc2epnS+9o2AfcxedK0YOPcV6pl9rHn3RttxVDGBOdcopVSAWfzO32nmzSF38HPUrZdgO06Z2vVIIyPpOvrkfkfm1x/YjqN8aNFVSqlyLP7pM/pvn8ycphfRpf/ptuNUSOrwB1jnaUPTn+/l4P5c23GUS4uuUkqVYX/ubhp8exsbpAXJV9q5AcbRiIquxf5THqc5OWR+cI/tOMqlRVcppcqQ9f6dtDDb2Df4GWrXibMdp1K6HXsmGfVOpU/2e2xYudB2HIUWXaWUKtXapXNJ3fIRcxKGBM1h5eKSLv0vh4hi56TbbEdRQKTtAKFoeca3FH79AFFFeUR58zDiId8TQ35kHHl1WmLi21CnTS8Sew6kXnxD23GVUiUwXi8HP/k7+ySWDpc+ZTvOUWvUrA3pHUaRtuppFv/4Kd1PGGo7UljTolsTvF4iiw6RH1GbA9ENEOMlquggdfNzaHtwMfW274dV4P1WWBuRyJbmg2jQeygdUwYF7N8QlAo3GZ+/Qt/8LOb0GEO/xs1tx6mSXn+9nc2Pv0/MD/fjPXZIjT7rV5VNjDG2M/iViMQDbwDdAQNcbYyZVVK7qampJiMjo9oz7Nmxld+WpLNv9S/U2/wLnQ4tJlK8ZEtzNrS9kI6nX0/Dpq2qvb9KqYrZs2s7hc+msD2yOR3+9UtIFKmMz18ldd4dzO39KH2H1uz/d0VknjEmtUZ7EqTCsei+A/xojHlDRKKBWGNMiXcHr6miW9yeHVtZMXMCdZeOp2vBYvJMFAubDCXxnH/RrHX7Gu+/UuqP0l+5kX6bP2DN+VNpn3yc7TjVwltUxOpH+hFXtJv4OxYSE1u3xvqlRbd0YXUsU0TqAwOBNwGMMfmlFVx/qt+wKf3Ou4Wud//M+ktmsKjBaaRsm0KDN/qR/tJ17Nm13XZEpcLGprXLSdn8ERkJZ4RMwQXwRESQf9IDNGM7mZOftB0nbIVV0QXaAjnA2yKyQETeEJE6vi2IyEgRyRCRjJycHL8HTOzUi36jP2D71bPIbHgm/bZ+jPfZ3syZ9DTeoiK/51Eq3Gye/C8KiSDpwkdsR6l23Y47i6xaKXRc9RYH9u2xHScshVvRjQRSgJeNMb2B/cCdvi0YY14zxqQaY1IbN25sIyMAzRM70e/W91lz/lS2RLWmX9YYlj0+iE3rVljLpFSoW5HxHX1yv2NR68to0rKt7Tg1IuqUu2nAXhZNDt4rsoNZuBXdbCDbGDPb/TwRpwgHrPbJx9H5Xz8xt+cDJB5aSb23T2Tu5GcxXq/taEqFFOP14p1+N9uJp/tFoXsHp859T2FRTCqd1ozVh91bEFZF1xizBdggIp3cr04GllqMVCHi8dD3/NHsvWoG62t1oO+ie8l49hLyDuyzHU2pkJH59Xt0KVjK6m63BvwDDaoq+pS7SGAvi/Tcrt+FVdF13QKME5FFQC8gaE7ctEjqRJd//sCsNiPps3s6G/9zAhvXLLMdS6mgV1RYSIPZT7De05o+595iO06N65x6Moti+tJl7Vj27d1lO05YCbuia4zJdM/Z9jTGnGuMCao5zhMRwYCrnyTrxNdoXLSV2HdPZfncb2zHUiqozZ/6GonebHb0+z8io6Jtx/GL6FP+RTz7WPz5c7ajhJWwK7qhIvmki9h72Vfsl7okfXEx86e/ZzuSUkGpIP8QzTOfZXXEMfQ69XLbcfymc+rJLInuwTErx5J/KM92nLChRTeItWrfndgbvmV9VDt6/XILsyc8YTuSUkFnwecv0cpsIXfAHSFx56nKKBwwmibsZOG0N2xHCRtadINcgyYtSfzHtyyqk0b/pQ+T/sGDtiMpFTQO5R2gTdaL/BrZkeSThtmO43c9T/wraz1JNF70it4HwE+06IaAmNi6dB39CfPrnEDar0+R/t69tiMpFRQyP32eZuSQd/ydYfmwEfF42NFrFEneDSz6foLtOGEh/OayEBVdK4YeoycxL+4vpK1+lvR3Q/d/hkpVh7yD+2m77BWWRXWlx8DzbMexJnnw1WyhMbVm6wVV/qBFN4RERdci+dYJzIs7ibQ1zzFn4n9tR1IqYC384mWasJPCE8JzL/ewqOharOt0FV0Kluo/IfwgfOe0EBUZFU3PW8azMKYvfbIeYP60t21HUirgFBbk03rpa/wa2ZHux59tO451PYbcRK6pzf6ZL9qOEvK06IagqOhadLxlCr9Gd6F7+m1kzfzUdiSlAkrmtDdpYbZyoP/fw3ov97A6cfEsaTaUnntnsG3jWttxQprObSGqdp04Wtz4ORsjWpP07fWsXzbPdiSlAoK3qIhGmS+y1pNIzzC8Yrk0rU+/lQi8rJ6m53ZrkhbdEFY/oRG1r5rEIalF1IRL2Llto+1ISlm38NsPSfJuYEfvm8Luf7llaXlMNxbVSaNT9kTyDu63HSdkadENcc1at2fnOe/QwLuTba9foAuTCmvG66XOnGfJlmb0GnyV7TgBJ3LADTRgL1nT9VqQmqJFNwx0TBnEkv5P0rlgKYtfvlIfC6jC1uKfPqVj4a9s6nZ92NxjuTK6HXc26zytSch6S9cTNUSLbpjoc+ZVzEocRerer5n90aO24yhlheen/7KNBiQPGWU7SkASj4etXUbQvmg1K/TvQzVCi24YSbvyURbEHkuf5f9h+eyvbMdRyq9+nT+DbvmLWNNhBLViYm3HCVg9zriOvdQh95c3bUcJSVp0w4h4PBxz3Xts9TSmwbTr2b5lg+1ISvlN7vdPk2tq021I6D8vtypi69Znyzkf0H2kPgShJmjRDTP1ExqR/9d3iTP72PrWpRQW5NuOpFSN27x+Bcl7Z7Ck+fnE1W9gO07A65gyiNp14mzHCElhV3RFJEJEFojIF7az2HJM9/4s7j2GbvmLmPvOP23HUarGrf/yvxiEpLP+YTuKCnNhV3SB0cAy2yFs63vuTcyNP4N+G95m6axptuMoVWP27t5B9y2fsrD+X2jWur3tOCrMhVXRFZFWwFmAnqwAulz9Mps9zWgw/Wb27MyxHUepGrH0i+eoKweJP/nvtqMoFV5FF3gGuAPQP6ABdeslcODsV2lodrH6rav1f3kq5BTkH6LtqvdYEt2T9snH246jVPgUXREZAmwzxpR5E2IRGSkiGSKSkZMT+nt/HVNOJKPdTaTsm0nGJ8/bjqNUtVo4fSxN2UFB/5tsR1EKCKOiCxwHnCMi64DxwEki8n7xlowxrxljUo0xqY0bN/Z3Riv6Dx/D4lq96LbwYTauCfvT3SpEGK+X+MxX+c3Tkp6DLrQdRykgjIquMeZfxphWxpgk4GLgO2PMZZZjBQRPRASNLnuTIjzsHn8d3qIi25GUqrJls6fTvmg1m7tcow82UAEjbIquKluz1u1Z1vNOuuVnMffjJ2zHUarK8n56iT3UoeeZI21HUeqIsCy6xpgfjDFDbOcINH3Pu5VFMX3psexpNq5ZYjuOUkdty4ZV9Nz3E8uanas3eVABJSyLriqZeDw0vexVivCwZ/z1ephZBa21/3sewdBm8GjbUZT6Ay266g+atmrH8uR/0TU/izkTHrcdR6lKyzu4n84bJ7OozgBaJHWyHUepP9Ciq/4k9dxbWBjTl57Ln2Hz+hW24yhVKVnT3yaBvUQN0Mf3qcCjRVf9iXg8NL30ZQC2jb9Fb5qhgobxeonPept1ntZ0O+5s23GU+hMtuqpEzdp0YFHHm0k+OJsF09+xHUepClkx7zs6FK1ia+crEI+u3lTg0blSlSr1ojtZFdGO1rPvZ+/uHbbjKFWufTNfYi+xdD9D/yakApMWXVWqyKhozJBnaGB2s+z9/7MdR6kybd+0nuS9P7C0ydnUiYu3HUepEmnRVWXq0Hsgc5tcQN+cKazI+M52HKVKtfJ/LxCBl1an3Wo7ilKl0qKrytX98ifZLglEffl3CgvybcdR6k/yD+XR4bcJZMX2pVX77rbjKFUqLbqqXHXrJZCddh/HeNcxb9J/bMdR6k8Wff0ujdgN/fRcrgpsWnRVhfQ+7QoW1+pFl+XPs3PbRttxlPqDOgvfJlua02Pg+bajKFUmLbqqQsTjIe7c/1Lb5LFq/D9tx1HqiLVLZtOlYCnZ7S/RpwmpgKdFV1VYYpc+zGt2Iak7vmDlgpm24ygFwLbvX+GQiaLz6dfbjqJUubToqkrpeskj7JT6eKferg9EUNYd2LeHrjnTyKo/iPhGzWzHUar4KFM+AAAgAElEQVRcWnRVpdSLb8ia5P+jU+Fy5n3+su04Kswtnv42cXKQusfpBVQqOGjRVZWWes6NrIjsTNvMJ/VOVcqqhGXvs9aTSKe+p9iOolSFaNFVleaJiMBz1pM0MHtYOv4e23FUmFqZ+SMdCleyreMlep9lFTTCak4VkdYi8r2ILBWRJSKiT7g+Sh16D2RewmBSNn/EprXLbcdRYWjXzFc5YGrRZbAeWlbBI6yKLlAI3GaM6QqkATeJSFfLmYJW4oWPUkgEmyfpX4iUf+Xu2Un3HV+xOOFk6sU3tB1HqQoLq6JrjNlsjJnvvs8FlgEt7aYKXk1atmVhmyvos+8Hls/52nYcFUaWTn+DWDlE/EB9UL0KLmFVdH2JSBLQG5htN0lwSx52DzkkIF/drQ+7V35hvF6aLB/Hqoh2dOh1gu04SlVKWBZdEakLTAL+ZozZW6zZSBHJEJGMnJwcOwGDSGzd+qxN/gedClcwb9qbtuOoMLBi3ne09a5jR+fhegGVCjphN8eKSBROwR1njJlcvLkx5jVjTKoxJrVx48b+DxiE+px9I6sj2tJy7uPkHdxvO44Kcbk/vc4+U5tup19tO4pSlRZWRVdEBHgTWGaM+a/tPKEiIjKSg4Pupzk5ZH78mO04KoTt2bGVHru/ZUmj06lbL8F2HKUqLayKLnAccDlwkohkuq8zbYcKBd1PGMrC2v3ptvp1dmzNth1Hhahl018jRgpoNOgG21GUOiphVXSNMT8ZY8QY09MY08t9fWk7V6iIH/oYtTnEqon32Y6iQpDxemm+ajwrIjvTrkea7ThKHZWwKrqqZiV2TmF+w7PovW0KG9cssx1HhZil6f8j0ZvN3u6X246i1FHToquqVdJfH8SLh82f/Nt2FBVi8ma9zl7q0OO0EbajKHXUtOiqatWkZVsWtLiElD3fsjor3XYcFSJ2bM2mx94ZLG1yFjGxdW3HUeqoadFV1a7rhfeyT2LZN1X3dlX1WDn9VaKliOYn6QVUKrhp0VXVrn6Dxiw95hqS8+ay5Be9Tk1VjbeoiNZrP2JpdA8SO6fYjqNUlWjRVTWi1wX/ZBsNiPxujN4eUlXJkp8/o6XZyoGeV9iOolSVadFVNSImti7retxKp8IVLPh6nO04KogVzH6TXdSjxymX2Y6iVJVp0VU1JuWcm1jvaUXD2Y9RWJBvO44KQjmb1tFz38+saHY2tWJibcdRqsq06KoaExkVzY7+d5LozWb+Zy/ZjqOC0KrpLxMpXlqdcqPtKEpVCy26qkb1PnU4KyI7k5T1LHkH9tmOo4JIUWEhbddPJKtWb1q17247jlLVQouuqlHi8VD4l3tpwk4yp+gzJlTFZc2YSDO2U9B7hO0oSlUbLbqqxnU77iyyavWm48rX2Z+723YcFSwy3mY78fQ46RLbSZSqNlp0lV9En3ovDdjLoslP2I6igsDm9SvoeWA2K1ueR1R0LdtxlKo2WnSVX3RKPYnM2AF0WzuWPbu2246jAty6r18BIOk0vQOVCi0BWXRF5BYR0SdUh5i4wfdSj/0sm/SI7SgqgBXkH6J99hSyYvvSPLGT7ThKVauALLpAU2CuiEwQkcEiIrYDqapr1/NY5tc9kR4bxrErZ7PtOCpALf5+PI3Zhelzte0oSlW7gCy6xph/Ax2AN4ERwEoReURE2lkNpqqs4ZAxxHCIFZMetB1FBajI+WPZQiN6DLrQdhSlql1AFl0AY4wBtrivQiABmCgiVboSx91zXiEiq0TkzmqIqiohsXMK8+NPo9fmCWzftN52HBVgNq5ZQo9D81mbeAERkZG24yhV7QKy6IrIaBGZBzwB/Az0MMbcAPQB/lqF7kYALwJnAF2BS0SkazVEVpXQYuj9ROBl9eT7bUdRAea3r1+i0Hhof7peQKVCU0AWXaABcL4x5nRjzMfGmAIAY4wXGFKF7vYDVhlj1hhj8oHxwNCqx1WV0fKYLsxvNITeOZ+wef0K23FUgDiUd4DOmz8jq+6xNG6RZDuOUjUiIIuuMeY+Y0yJxx6NMcuq0OmWwAafz9nud0eIyEgRyRCRjJycnCr0SpUl6bz7MHjY8Inu7SpH1jfvk8BeIvtdYzuKUjUmIIuuTcaY14wxqcaY1MaNG9uOE7KatmrHgqbnkbJzGhtWZdmOowJA7KJ32ShN6Xa8HnxSoSvciu5GoLXP51bud8qC9uffSz5RbP1sjO0oyrL1y+fTNT+LDW2H4YmIsB1HqRoTbkV3LtBBRNqKSDRwMfCZ5Uxhq1Gz1ixsOYyUPd+ydulc23GURZu/e5l8E0GH06+3HUWpGhVWRdcYUwjcDEwHlgETjDFL7KYKb13/+m/2E8PuqWNsR1GWHNyfS9dtU8mqdyINm7ayHUepGhVWRRfAGPOlMaajMaadMeZh23nCXf2GTVmceDm99//EyswfbcdRFmR9NZZ67CdmwHW2oyhV48Ku6KrA0+38O9lNXQ5Mf8B2FGVB/SXvs97Tiq5pg21HUarGadFV1tWLb8iytiNIPjiH5XO/sR1H+dHqRb/QqXA5m9tfjHh0daRCn87lKiAk//UOdlCfwm/0nszhZPuMV8kzUXQ5faTtKEr5hRZdFRBi69ZnZcfr6H4okyU/T7UdR/nBvr276L79f2TFn0z9hk1tx1HKL7ToqoDR67x/sI0GeH54GOP12o6jatiS6W9RR/KIO14voFLhQ4uuChgxteuwtusNdClYQtbMKbbjqBpkvF4aLh/HGk8SnfqcZDuOUn6jRVcFlN5Db2Uzjan906O6txvCVmbOpH3RanI6D9cLqFRY0bldBZToWjFsSL6VDoUryfzmA9txVA3ZM/MVDphadD39WttRlPIrLboq4KQMGcUGaUH99CfxFhXZjqOq2a6czfTc9Q1Zjc4grn4D23GU8istuirgREZFszXlbxzjXceC6WNtx1HVbMW0l6glBTQ5+WbbUZTyOy26KiD1PuMa1nna0CjjvxQVFtqOo6pJUWEhiWs+ZEl0T9p27Ws7jlJ+p0VXBaSIyEh29ruNRG8286e+ZjuOqiZZP3xMc3I41Ptq21GUskKLrgpYvU69nFUR7WiR+QwF+Ydsx1HVwJPxOttoQI+TL7UdRSkrtOiqgOWJiGD/cXfQ0mxlwWcv2o6jqmjDyoX0zJvH6sSLiIquZTuOUlZo0VUBreegi1gR2Zk2i1/kUN4B23FUFWz8+gXnQfVn3GQ7ilLWaNFVAU08HvIH3kUztpP5ybO246ijtD93N922fs6ieoNo1KyN7ThKWaNFVwW87sefzZLoHrRb/goH9+fajqOOwuL/vUGcHKTuCTfYjqKUVWFTdEXkSRFZLiKLRGSKiMTbzqQqRjwe5KR/04jdLJzylO04qpKM10uTZe+yOuIYOqWebDuOUlaFTdEFvga6G2N6Ar8C/7KcR1VC17TBLIrpQ6dVb7Jv7y7bcVQlLJs9nbbe9ezoeqXeZ1mFvbBZAowxXxljDt9lIR1oZTOPqryY0+4lgVyyJj9uO4qqhLyfX2YPdegx+BrbUZSyLmyKbjFXA9NKaiAiI0UkQ0QycnJy/BxLlaVjyiAWxB5Lt3XvsGenTptgsDV7NT1zf2RZ03OoXSfOdhylrAupoisi34jI4hJeQ33auRsoBMaV1A1jzGvGmFRjTGrjxo39FV1VUL0z7qMeB1g6+RHbUVQFrJn6DIKhzRl/tx1FqYAQaTtAdTLGnFJWcxEZAQwBTjbGGL+EUtWqXY805k8/kZ4bPmBXzh0kNG5uO5IqxYF9e+i6eRIL655ASlIn23GUCgghtadbFhEZDNwBnGOM0bssBLGGQ8YQwyFWTHrQdhRVhqwvX6U++4kdeIvtKEoFjLApusALQBzwtYhkisgrtgOpo5PYOYX58aeSvPljtm9abzuOKoG3qIgWy95mZWQHOvUt8wCUUmElbIquMaa9Maa1MaaX+xplO5M6es3Pvo8oClk95QHbUVQJsmZ8TGuziT3J1+nfhJTyoUuDCkqt2ndnfoMz6b3tE7ZsWGU7jiomYvbLbKMByaePsB1FqYCiRVcFrTbnjQFg/ZT77QZRf7B2yWy6H8pkddtL9WlCShWjRVcFrWZtOrCg8Tmk7JjKxjXLbMdRrpyvn+GgiabrkFttR1Eq4GjRVUHtmPPvowgPmz+5x3YUBezYmk3yrq9Z1OhM6jdsajuOUgFHi64Kao1bJLGgxTBS9nzD6qx023HC3q9Tn6WWFNDsNL0ZhlIl0aKrgl7XC8eQK7Hsn3q37Shh7eD+XDr/Np6FtfuT2KmX7ThKBSQtuiro1W/QmGXtR9IzL4PFP35qO07YWvT5CySwl6gT/2E7ilIBS4uuCgm9/no7W2hMzA/34y0qsh0n7BTkH6LN8jdZHtmFLv1Osx1HqYClRVeFhJjaddjQ+x+0L1rN/Glv2o4Tdhb+722ak0Ne/1v1ZhhKlUGXDhUyUs4ayRpPEi3mPcWhPL29tr8Yr5cGmS+zztOanicNsx1HqYCmRVeFjIjISPadcA8tzFYWTHnadpywsWjGJI7xrmNbj+vxRETYjqNUQNOiq0JKjxPPZ3GtXnRa8Qq5e3bajhMWImc9y1Ya0uvM62xHUSrgadFVIUU8Hmqd8RAJ7GXxBH0YQk1bkfEd3fKzWNvhSqJrxdiOo1TA06KrQk6HXicwL+4kemWPI2fTOttxQtqB755kL3Xofrbe8lGpitCiq0JSs/MeIYIi1k74l+0oIWvVwp/pfeAXlrS5jLr1EmzHUSooaNFVIanlMV2Y3/xiUndNY2Xmj7bjhKS9Xz3KXmLpet4dtqMoFTS06KqQ1fXiB9ktcRRO/SfG67UdJ6SsXTKblP0/sqT1pdRPaGQ7jlJBI+yKrojcJiJGRHRNEeLqxTdkZbe/0aVgCfP/947tOCFl57RH2Gdq0/Vc3ctVqjLCquiKSGvgNOA321mUf6SeN9q5Ycach8k7uN92nJCwftk8eufOIKvVMH18n1KVFFZFF3gauAMwtoMo/4iIjOTAyQ/RnBwyP3rYdpyQkPPlw+QRTZfz9CI1pSorbIquiAwFNhpjFpbT3kgRyRCRjJycHD+lUzWp+3Fns6DO8fRc+wbbN623HSeorV8+n5S937GwxTDiGzWzHUepoBNSRVdEvhGRxSW8hgJ3AfeW1w1jzGvGmFRjTGrjxo1rPrTyiyZ/fYJIilg74Z+2owS1HZ/fx0Fq0em8O21HUSoohVTRNcacYozpXvwFrAHaAgtFZB3QCpgvIrqpHiZaHtONeS0upu/uaSzP+NZ2nKC0csFMUvbPZFGby2nQpKXtOEoFpZAquqUxxmQZY5oYY5KMMUlANpBijNliOZryo56XPsw2GhA17f8oLMi3HSfo5P3vPnYRR/cL7rIdRamgFRZFVymAOnHxZPe/l3ZFa8iY+JTtOEFl8c+f0+PQfFZ0uI64+g1sx1EqaIVl0XX3eLfbzqH8r/fpV7IoJpVuy5/Ti6oqyHi9RH3/oPMkofP/z3YcpYJaWBZdFb7E4yHhgmeoRQHrxv/DdpygkPnNB3QqXMH6HrcQU7uO7ThKBTUtuirstG7fg/mtryR17zcs/vlz23ECWkH+IRrNepjfPC1JOecm23GUCnpadFVY6nXpA2ySpsR9eyf5h/JsxwlY8yY+SWuziZ3H3UtkVLTtOEoFPS26KizFxNYl54SHSPRmM29cuX/fDkt7dmyly68vkVUrheS/XGQ7jlIhQYuuClvJJ13EvLiT6LP+DdYunWs7TsBZNv5u6poD1B36BOLRVYVS1UGXJBXW2l7+AvulDgWTb6SosNB2nIDx26+Z9Nk2mYxG59C2a1/bcZQKGVp0VVhr0KQlq1PvoWPhr8zVByIcsWvKHeQRTfuLHrEdRamQokVXhb0+Z15LZuwAkn99gexVi23HsW7BV++TfHA2SzpcT8OmrWzHUSqkRNoOEGwKCgrIzs4mL0+veD1aMTExtGrViqioKNtRAOe/uy2Gv0zhawPYM2EULf45A09EhO1YVuzP3U3zX+5jrSeRPhfp7R6Vqm5adCspOzubuLg4kpKSEBHbcYKOMYYdO3aQnZ1N27Ztbcc5oknLtsxN/hd9F91L+viHSBt+n+1IVmSNu4s0trP89JeJiq5lO45SIUcPL1dSXl4eDRs21IJ7lESEhg0bBuSRgtRzb2FB7HGk/PocaxbPth3H79YumU3q5g+ZkzCEzv1Psx1HqZCkRfcoaMGtmkAdf+LxkDjidfZKXWTySPIO7rcdyW+8RUUc+uRv5EpdOg7/j+04SoUsLbpK+WjQpCUbBz5JW+86MsfeZjuO38z56BE6FyxlVa87iW+kj5lWqqZo0Q0Dn3zyCUuXLvV7f8eMGcNTTwXfI/SST7qI2Q3PJW3rh2TNmGw7To3bsHIhySueI7N2Gqnn3GA7jlIhTYtuGKjOolsYJjeQ6Hn1C6z1JNLq+9FszV5tO06NKSosZP+EURRIFC0vf1XvPKVUDdOrl6vg/s+XsHTT3mrtZtcW9bjv7G7ltnfuueeyYcMG8vLyGD16NCNHjqRu3brs27cPgIkTJ/LFF18wcuRIPvvsM2bMmMFDDz3EpEmTyM3NZdSoURw4cIB27drx1ltvkZCQwNy5c7nmmmvweDyceuqpTJs2jcWLFzN27FgmT57Mvn37KCoqYurUqQwdOpRdu3ZRUFDAQw89xNChQwF4+OGHeeedd2jSpAmtW7emT58+1Tp+/KV2nTgiLn6X6HGns/Wdy2hw+w8heTXv3PEPkVawlIyUx0htkWQ7jlIhTzdrg9Rbb73FvHnzyMjI4LnnnmPHjh0ltnfsscdyzjnn8OSTT5KZmUm7du244oorePzxx1m0aBE9evTg/vvvB+Cqq67i1VdfJTMzk4hi/1OdP38+EydOZMaMGcTExDBlyhTmz5/P999/z2233YYxhnnz5jF+/HgyMzP58ssvmTs3uO9n3KZjL5b3e5jOBUuZ9+Zo23Gq3crMH0lZ+RwLYo+jz5DrbcdRKiyE1Z6uiNwC3AQUAVONMXdUpXsV2SOtKc899xxTpkwBYMOGDaxcubJCv9uzZw+7d+/mxBNPBODKK6/kwgsvZPfu3eTm5jJgwAAALr30Ur744osjvzv11FNp0KAB4PzX9q677mLmzJl4PB42btzI1q1b+fHHHznvvPOIjY0F4Jxzzqm24bWlz1nXMnvtz6Rt/ZAF0wfQ+/QrbUeqFvv27qL2p9exS+Jpe/VbelhZKT8Jm6IrIn8BhgLJxphDItLEdqaj9cMPP/DNN98wa9YsYmNjGTRoEHl5eX/4K051/w+2Tp06R96PGzeOnJwc5s2bR1RUFElJSQH5v9vq0uvaF/n1qcV0+uV2VjfvQLuex9qOVCXG62X5G9fS27uFFWeMp6terayU34TT5u0NwGPGmEMAxphtlvMctT179pCQkEBsbCzLly8nPT0dgKZNm7Js2TK8Xu+RvWCAuLg4cnNzAahfvz4JCQn8+OOPALz33nuceOKJxMfHExcXx+zZzk0hxo8fX2b/mzRpQlRUFN9//z3r168HYODAgXzyySccPHiQ3NxcPv/88xoZfn+rFRNLg6snkit1qTP5crZv+c12pCqZO+U5Uvd+w5zEkXRNG2w7jlJhJZyKbkfgBBGZLSIzRKTE55WJyEgRyRCRjJycHD9HrJjBgwdTWFhIly5duPPOO0lLSwPgscceY8iQIRx77LE0b978SPsXX3wxTz75JL1792b16tW888473H777fTs2ZPMzEzuvdd5iPubb77JddddR69evdi/fz/169cvsf/Dhw8nIyODHj168O6779K5c2cAUlJSGDZsGMnJyZxxxhn07Rs6j4Rr1CKR3PPeo57JZccbF5B3YJ/tSEdl+Zyv6bXoAbJqpdDvCn2CkFL+JsYY2xmqjYh8A5R0rOxu4GHge+BWoC/wEXCMKWMEpKammoyMjD98t2zZMrp06VJtmQPJvn37qFu3LuAU8M2bN/Pss8/WSL+CdTzOn/4eKbNuZn7dgST/bQoRkcFzhmZr9moi3jiJPIkh7uaZ1G/Y1HYkFaJEZJ4xJtV2jkAUPGuMCjDGnFJaMxG5AZjsFtk5IuIFGgGBuTtrwdSpU3n00UcpLCwkMTGRsWPH2o4UcFJOv5z07WtIW/lf5rw0gr43vxsUFyEd3J/L3rHDaG4OsX/YZC24SlkS+GuL6vMJ8BcAEekIRAPbrSYKMMOGDSMzM5PFixczdepUGjdubDtSQEobfh+zWo6g387PSX8j8P9KVJB/iF9fOJ9jClax6oSnSewSnP+dVioUhFPRfQs4RkQWA+OBK8s6tKxUWdKueZrZDYcyYNO7zHrrDozXaztSibxFRWS+eBnJB+eQ0f0eep1yie1ISoW1sCm6xph8Y8xlxpjuxpgUY8x3tjOp4CUeD6k3vMXc+oMZ8NurpL8xOuAKr/F6mfvKSPru+YpZiaPof2H4PMBBqUAVNkVXqeoWERlJn1s/YHbDcxmw6V1mvzwSb1GR7ViAs4c758UR9M+ZSHrTS0i78lHbkZRSaNFVqko8ERH0u+lt0psMIy3nYzKfPpeD+3OtZirIP8S854fTf8enzGpxBf2vfykoLvZSKhzokhjGxo4dy80331xuO5s2bTry+dprr7XymMBAJh4P/Ue9QnqHf9Ar90c2PH0S2zett5Jlz67tLP/P6fTdPY1ZbUaSdu2zWnCVCiC6NKoyFS+6b7zxBl27drWYKDCJx0Pa8PtYeNwLtCpYD68NJGvmlPJ/WI02rFzI7udPpFPeIuYkP8iAq5/UgqtUgAmp/+n63bQ7YUtW9XazWQ8447FyWyvt0X6jR4/miy++oHbt2nz66ac0bdqUzz//nIceeoj8/HwaNmzIuHHjaNr09/9p5ubm0rNnT3799VeioqLYu3cvycnJPPHEE2RkZDB8+HBq167NrFmzOOOMM3jqqadITU3lf//7H3fddRdFRUU0atSIb7/9tnrHRRDqfdplrGvdGT4eQbdvr2LW8u/offljxNSuU/6Pj5Lxepn76Qt0z3yIQxLNqsHj6DfgjBrrn1Lq6OlmcJAq6dF++/fvJy0tjYULFzJw4EBef/11AI4//njS09NZsGABF198MU888cQfuhUXF8egQYOYOnUq4Nx3+fzzz+fCCy8kNTWVcePGkZmZSe3atY/8Jicnh+uuu45JkyaxcOFCPv74Y/8NfIBL6pJK09tmkdHgLAZsepecJ1JZ/NNnNdKvzetXkPnUWfRbeA9ra3Ui/9oZdNWCq1TA0j3dqqjAHmlNKenRftHR0QwZMgSAPn368PXXXwOQnZ3NsGHD2Lx5M/n5+bRt2/ZP3bv22mt54oknOPfcc3n77bePFOzSpKenM3DgwCPdOvzYP+WoXSeOfqPHkTVzCgnf30n3by5nwS/HEX/WfbTt1r/K3d+7ewdLJj9O8vqxxGNIb3crfS+9L6huS6lUONI93SDk+2i/hQsX0rt3b/Ly8oiKijryeL+IiAgKCwsBuOWWW7j55pvJysri1VdfLfExfMcddxzr1q3jhx9+oKioiO7du/t1mEJVj4Hn0ej2ecxqcz3t988nccLpZD4xmIXfTaDInT6VsXHNMma9Phqe6c6A315lRd2+7L76J9KueFALrlJBQJfSIFTao/3Kar9ly5YAvPPOO6W2d8UVV3DppZdyzz33HPnO97GAvtLS0rjxxhtZu3Ytbdu2ZefOnbq3W4qY2LoMuPoJ9uy4jTlTHqd99iQazbyOHTP/j9Xxx+Fp/xcad+hHq3bd/1Q4d2/fwqaVC9iz7HviN/9Il4KlNDfCwjrHEnf6XfROPt7SUCmljoYW3SA0ePBgXnnlFbp06UKnTp2OPNqvNGPGjOHCCy8kISGBk046ibVr15bY3vDhw/n3v//NJZf8fqvAESNGMGrUqCMXUh3WuHFjXnvtNc4//3y8Xi9NmjQ5cjhblax+w6akXftf8g89wvzvPsQs/ZTOu3+gXsaXkAGFxsMOieOgxBJpCojlAPEcIB7wGmFN5DGkJ91E21OuoXerdrYHRyl1FELq0X7VLdwe7Tdx4kQ+/fRT3nvvvRrvVyiPx8ooyD/EbysWsGPVXIq2r8JzcCeRBfvwRtTCGxmLSUiidvOOJPYcRHyjkp5aqVTg0Uf7lU73dBXgnPedNm0aX375pe0oYSUquhbteqTRrkfZRyuUUqFBi64C4Pnnn7cdQSmlQp5evXwU9JB81ej4U0qFKy26lRQTE8OOHTu0cBwlYww7duwgJibGdhSllPI7PbxcSa1atSI7O5ucnBzbUYJWTEwMrVq1sh1DKaX8TotuJUVFRZV4RyellFKqPGFzeFlEeolIuohkikiGiPSznUkppVR4CZuiCzwB3G+M6QXc635WSiml/Caciq4B6rnv6wObymhXKaWUqnZhc0cqEekCTAcEZ2PjWGPM+hLaGwmMdD92AlYcZS8bAduP8rfBSoc5POgwh4eqDHOiMaZxdYYJFSFVdEXkG6Cke+XdDZwMzDDGTBKRi4CRxphTajBLRrjdBk2HOTzoMIeHcBxmfwipq5fLKqIi8i4w2v34MfCGX0IppZRSrnA6p7sJONF9fxKw0mIWpZRSYSik9nTLcR3wrIhEAnn8ft62prxWw90PRDrM4UGHOTyE4zDXuJA6p6uUUkoFsnA6vKyUUkpZpUVXKaWU8hMtulUkIoNFZIWIrBKRO0toXktEPnKbzxaRJP+nrF4VGOZ/iMhSEVkkIt+KSKKNnNWpvGH2ae+vImJEJOj/alGRYRaRi9xpvUREPvB3xupWgXm7jYh8LyIL3Pn7TBs5q4uIvCUi20RkcSnNRUSec8fHIhFJ8XfGkGOM0ddRvoAIYDVwDBANLAS6FmvnRuAV9/3FwEe2c/thmP8CxLrvbwiHYXbbiwNmAulAqu3cfpjOHYAFQIL7uYnt3H4Y5teAG9z3XYF1tnNXcZgHAoccVloAAAOHSURBVCnA4lKanwlMw7mpUBow23bmYH/pnm7V9ANWGWPWGGPygfHA0GLtDAXecd9PBE4WEfFjxupW7jAbY743xhxwP6YDwf4cv4pMZ4AHgcdxro4PdhUZ5uuAF40xuwCMMdv8nLG6VWSYQ+p2ssaYmcDOMloZCrxrHOlAvIg090+60KRFt2paAht8Pme735XYjjGmENgDNPRLuppRkWH2dQ3OlnIwK3eY3cNurY0xU/0ZrAZVZDp3BDqKyM/uE7wG+y1dzajIMI8BLhORbOBL4Bb/RLOmssu7Kkc4/U9X+ZmIXAak8vtNSUKSiHiA/wIjLEfxt0icQ8yDcI5mzBSRHsaY3VZT1axLgLHGmP+IyADgPRHpbozx2g6mgoPu6VbNRqC1z+dW7ncltuPemKM+sMMv6WpGRYYZETkF557X5xhjDvkpW00pb5jjgO7ADyKyDufc12dBfjFVRaZzNvCZMabAGLMW+BWnCAerigzzNcAEAGPMLCAG58EAoapCy7uqOC26VTMX6CAibUUkGudCqc+KtfMZcKX7/gLgO+NeoRCkyh1mEekNvIpTcIP9PB+UM8zGmD3GmEbGmCRjTBLOeexzjDEZduJWi4rM25/g7OUiIo1wDjev8WfIalaRYf4N5+Eph59cFgPk+DWlf30GXOFexZwG7DHGbLYdKpjp4eUqMMYUisjNOI8MjADeMsYsEZEHgAxjzGfAmziHoFbhXLBwsb3EVVfBYX4SqAt87F4z9psx5hxroauogsMcUio4zNOB00RkKVAE3G6MCdqjOBUc5tuA10Xk7zgXVY0I5o1oEfkQZ8OpkXue+j4gCsAY8wrOeeszgVXAAeAqO0lDh94GUimllPITPbyslFJK+YkWXaWUUspPtOgqpZRSfqJFVymllPITLbpKKaWUn2jRVUoppf6/vTu0qSgIogB6JxgIQaGQeCwGSSFU8MsgoROCpgAkhjbogpBBvOd5ajebnFPBdTezm8wMonQBYBClCwupqvv9rul5VV3ud2zvZucCjrEcAxZTVc/Z1g9eJPnu7pfJkYCDlC4sZt8L/JXtbu9Dd/9OjgQc5HkZ1nOdbbf1VbaJF1iESRcWU1XvSd6S3Ca56e7T5EjAQa4MwUKq6inJT3e/VtVZks+qeuzuj9nZgP+ZdAFgEH+6ADCI0gWAQZQuAAyidAFgEKULAIMoXQAYROkCwCB/RA0LwsxVL1sAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The max absolute difference is: 1.77636e-15\n" - ] - } - ], + "execution_count": 7, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "\n", @@ -1799,18 +1766,11 @@ }, { "cell_type": "code", - "execution_count": 29, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The gradient of f1 evaluated at a = 1 using autograd is: 3\n", - "The gradient of f1 evaluated at a = 1 by finding the analytic expression is: 3\n" - ] - } - ], + "execution_count": 8, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -1844,23 +1804,11 @@ }, { "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluating at x1 = 1, x2 = 3\n", - "------------------------------\n", - "The derivative of f2 w.r.t x1: 12\n", - "The analytical derivative of f2 w.r.t x1: 12\n", - "\n", - "The derivative of f2 w.r.t x2: -4\n", - "The analytical derivative of f2 w.r.t x2: -4\n" - ] - } - ], + "execution_count": 9, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -1909,18 +1857,11 @@ }, { "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The computed gradient of f3 is: [ 2. 3. 5. 7. 88.]\n", - "The analytical gradient of f3 is: [ 2. 3. 5. 7. 88.]\n" - ] - } - ], + "execution_count": 10, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -1958,18 +1899,11 @@ }, { "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The computed derivative of f4 at x = 2.7 is: 13.8759\n", - "The analytical gradient of f4 at x = 2.7 is: 13.8759\n" - ] - } - ], + "execution_count": 11, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -1999,17 +1933,11 @@ }, { "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The computed derivative of f5 at x = 2.7 is: 5.4\n" - ] - } - ], + "execution_count": 12, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -2067,17 +1995,11 @@ }, { "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The analytical derivative of f6 at x = 2.7 is: 37732.5\n" - ] - } - ], + "execution_count": 13, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -2099,18 +2021,11 @@ }, { "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The computed derivative of f7 at n = 2 is: 1\n", - "The analytical derivative of f7 at n = 2 is: 1\n" - ] - } - ], + "execution_count": 14, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -2155,27 +2070,11 @@ }, { "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [ - { - "ename": "TypeError", - "evalue": "'ArrayBox' object does not support item assignment", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m8.4\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"The derivative of f8 is:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf8_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36mnary_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0margnum\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0munary_operator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munary_f\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnary_op_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnary_op_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 21\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_operator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/differential_operators.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0marguments\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbut\u001b[0m \u001b[0mreturns\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgradient\u001b[0m \u001b[0minstead\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m should be scalar-valued. The gradient has the same type as the argument.\"\"\"\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0mvjp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mans\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 25\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mans\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m raise TypeError(\"Grad only applies to real scalar-output functions. \"\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/core.py\u001b[0m in \u001b[0;36mmake_vjp\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVJPNode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_root\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_node\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/tracer.py\u001b[0m in \u001b[0;36mtrace\u001b[0;34m(start_node, fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtrace_stack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_node\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_box\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_box\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mstart_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_node\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36munary_f\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0msubargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubvals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msubargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m\u001b[0m in \u001b[0;36mf8\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mautograd\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mf8\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# Assume x is an array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mTypeError\u001b[0m: 'ArrayBox' object does not support item assignment" - ] - } - ], + "execution_count": 15, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -2201,27 +2100,11 @@ }, { "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [ - { - "ename": "AttributeError", - "evalue": "'ArrayBox' object has no attribute 'dot'", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0.0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"The derivative of f9 is:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf9_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36mnary_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0margnum\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0munary_operator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munary_f\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnary_op_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnary_op_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 21\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_operator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/differential_operators.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0marguments\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbut\u001b[0m \u001b[0mreturns\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgradient\u001b[0m \u001b[0minstead\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 23\u001b[0m should be scalar-valued. The gradient has the same type as the argument.\"\"\"\n\u001b[0;32m---> 24\u001b[0;31m \u001b[0mvjp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mans\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 25\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mans\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 26\u001b[0m raise TypeError(\"Grad only applies to real scalar-output functions. \"\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/core.py\u001b[0m in \u001b[0;36mmake_vjp\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVJPNode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_root\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_node\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/tracer.py\u001b[0m in \u001b[0;36mtrace\u001b[0;34m(start_node, fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtrace_stack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_node\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_box\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_box\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mstart_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_node\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/usr/local/lib/python3.7/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36munary_f\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0msubargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubvals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msubargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m\u001b[0m in \u001b[0;36mf9\u001b[0;34m(a)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mf9\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# Assume a is an array with 2 elements\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2.0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mf9_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf9\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;31mAttributeError\u001b[0m: 'ArrayBox' object has no attribute 'dot'" - ] - } - ], + "execution_count": 16, + "metadata": { + "collapsed": false + }, + "outputs": [], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -2248,7 +2131,9 @@ { "cell_type": "code", "execution_count": 17, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -2278,7 +2163,9 @@ { "cell_type": "code", "execution_count": 18, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "a += b\n", @@ -2599,7 +2486,9 @@ { "cell_type": "code", "execution_count": 19, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "import numpy as np\n", @@ -2634,7 +2523,9 @@ { "cell_type": "code", "execution_count": 20, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -2652,7 +2543,9 @@ { "cell_type": "code", "execution_count": 21, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -2669,7 +2562,9 @@ { "cell_type": "code", "execution_count": 22, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -2691,7 +2586,9 @@ { "cell_type": "code", "execution_count": 23, - "metadata": {}, + "metadata": { + "collapsed": false + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -3130,25 +3027,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.4" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 2 } diff --git a/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz b/doc/pub/Splines/ipynb/ipynb-Splines-src.tar.gz index 13451208479be4278111c82de5197f673bf36820..72d439fbf7de02428236a9bd1a64fc3a96b5c817 100644 GIT binary patch literal 211 zcmb2|=3oe$+aJro{Pz5LFJ?o5;~(e!7WvHLed*1z^n-FX97}&OY0lntan>47u6MU@ z+b>YxD=N5u{^|b2w#2l}yLRr4H~qTz+&VYS)Z1ZxJGb1my!t3;erB%g>~*Hzxw~6M zm-cQ1oy$8klGozecK9)U@#>SkZ+ zw(0pTP@8$k;^^8hC#MJdZ>qO$d2{rrXUzR}si2bIKWAJHE?d8J=KpWz49MU^eGNm} LYwbP;4F(1Pmq%ym literal 211 zcmb2|=3ppX)f3CW{Pw(e7L%jEvBK-~7Wo;JSo2?R+;Kulz(hP_rrBk?UFvsrQ}X2H z1;Td;PyACqyMIyFqAa`W%G&v&Kh^u+xoKVvnJQC!>&o(w$;GvcN|kr5Ssmn`bvGk$ zmrk~+Q0e~3dfzl>S6D2Kj-J0}zg4+J^xC{Pmiv|Jf2!D7Pk-_@`9$o8N6y<$3z%(s zvHjfbkR5d&yfd49kI!@Yy|?^g*T?1=+e&Y;=5Kp!*S0>RY^8+#FKGs3@WZ~2@%7yQ Jml!k{7yzXwUwr@o diff --git a/doc/pub/Splines/pdf/Splines-minted.pdf b/doc/pub/Splines/pdf/Splines-minted.pdf index 7c47294d4e6a45a5cba3cd315d0bf859c7c93a54..61d97dce62e5e3da394f3d86e848a4623885117e 100644 GIT binary patch delta 40707 zcmZs?LzE>++K8J#2PEsjBm!v zHAdSFM!g6au!Mp;IhF!T3x?>h$}oe)nTT+hlrJ$jf`nlBEEpkL{=g5sAcpgf_E1?? zomG3}5d0^?R#CCMl6)-{DruAQ-NQ2yU=aSwGQeOiNb@0oWtMw7ffETb zf)pual_EZlJ4{1lUG1y4BrnEVnW|aK50r;PV_^BJ?Q_T|W@` znC*L9$6}FlqgzZ;LxJL7yx|+;Cr7OM3|e*_B2^_^-Hv@kCZm}DCqb(gJ&2|dm1mBQ z6+K}T6FSqLTkBNI8Ox<>P?S5yu8*}uv(0HH?Os14ubCfvJ8kayGk_pCL?x>k*(|6@ z_GqOJOq5OShYG9yXLTW!%baD}wA{WE9R29E7fY&1hkf}i!5EU-IVF{&*=7m`Sj9#7JUt$j+%RC z;D$UaI*m8vg1dO-<>d+iI2lrF8-(bHx1hq+<~sTm-bDutDpEzlhi@?t^}uwPXm5ZQ#1wKoFzV zKh-X(#WM6hG&ie8`hWaqD)z7(=+Fc0z(9@z1dU|$7R3G%(K`xWGrn(gJ>sP7V@mSj z)9y9l73(VtzIV91hbp+dfxaXBTs@VonASG?b(LdbilTik%xlha$K_(UwNrh1-JbwReJ#^xl}CR1 zlc|UIXZ*PIau-hUy`W9_WA*9-ID`S(;#Kd3-Hf_6I9JMj*Xu;t8i4Stg|TzVel=Yn z_#~Xaobob-;B5d*`FggY_KR4xbrSy6qwATicFY+*OGtz!!9#CPoAsUQLHj(b0aI?ivY)o|Y2PpA7w;S8p z9YhGcd>hNIs}A1769aS-s7#k?kb;5QlAeSuAdlu+QeNVka~Rx~<3xjD%^~Fl@mI;D z8amS=}-u>=Qti9PD={bZaF+eSq7p7Jkaa_IWDLpOuro7GV$4#L~&8&psl0F&AZSs@?(r zK5{e!WO{B3cKK%rm!(Bs7OVa=G}C);850MAe(LG(W6wd<3ZFNypV#p3z4D0m3Dwp>fgq+z>9J&_#n1lE6hm3|59`3@h}nIkVKTTd0w zU+<9A4(Pq}6C1H;002q%Kl8f)-HDC;-Q#)DiFw~ zp^l5z3VxC;t6ms@LnPc%c|_)LNP^tjqb$-}L@tdiRT!gN6PrJm32lnOpGie*KljO| zu{?IB9c|Ep#$@;trt>K@;h!?o$~*Vj;aoMGXNcin;V=>Yf9qeM3-25b+pi}VA^ZLhc3v$- zyy-_CQ_u1$MNI}NwKdSNf$M)CP6>;r47pc=z1u6D#ciotK|SAyO4!Iwn-E|>Ji(a( zfHY8fFdCq?Y|=&>O3zH)Y=X)~|2_gMW7+sxn2pL>r76D}clC>qY<*~FB0yG|khfO= zfn5j{UnMF8!TX*jE=kG?91Mtvx2)iPAziI=CjuL{UL)Xd- z2Fu#5Q|n8+Yl~x})22Vt=;iHQ0$a^C(?bU z2Y(uj^<{W-&@|`{!+Pt?D4K5IDA6o^0+h^v(-UG<#`)dB7|EvQZ0jyBo9w2q|4)h1nu#Q7QAxEswP z1*K(aIDJkp`+Aq}+}L9GtZifi7Mk@*qkytHj8W3c5#KOX{h?oG;65t0cWy4>%RmoeuA9}I^id=xT!rv> z!nMLOkaaB!)8#(8oOS5o{@61Ph5x3!dbuCal|fUlY)0bcihx>}pUD=Zb4(c`{{(K5 zhoTxiK2WkGCc74JIJqZAp0Wx?7&aj$N?n`Z<$lL1yZa6{JNb-GS{~Cccm0uL)eZ^Xp8Rd$kpAEs)!`2>t~81Ajg71*y_2`HFflzzKO~& zLDGp36ZJbgV1-Z-Q%ZIJ*JiBU=skh*NkDQ7_$wbSrRriH77% zq7*jH%jnZ`)tWNT&sb`!ngjB{gF}1^J7~!uap9 zKvBZ)>w@{yg8E0yCNj@nU6y#L9&rlC&qrsDi-SZ@o>4Lc-6t0)47oofuiAf~%rZ_> zz|Vq++Dt++^p)wdrVoLj06J6AeLq@szZ|`vPL|MLAS{W%N&>fnOMrDR8ol2V8hD=`zndv$y*PSf_bm5E#fqhx^|=(PcXk zd<)%aXi!XY39iRQRy1n>i}_h(1bthSi3DtMCR2D}p&??zEb!-WKF0O48P72Izp$d$ z`VNT+7mK!)29V{hKLoQEf?)NJBafghmO~-xxm9%-)C6kBDyNECpCR)611%;W{Jd+c z;B{z2ZKE!j5CG)@$=5}5q!4NxBorBWOJI|$z;HfbdY!+u`^7zG1#g3E1H5|p09#-m zn-jjIx=nKB75rwvbP;d9U^|B_coBI@mG=HQH`VRLN?{zpkYSVw>(~4#T)U8hj+BX`|d+?0@i!X2PHBd#z=ocH89lz39fC z_zfDFXF-+b_R<#!?Gw`OmOL7K27o8;!~n*$M6$;3_U8ILM^=k7OqOn75qB8wobzw0xLH-OnU40WLnnrNb%mFGjK^W+m4}YBwT$r@xz0a&1z1v-asc zF&1%{Uq^mzE(DLG`yhC^+p2n4n6*pBkPibP$7hxWkQy^g|PXZN!HJU6QQV@ zZKhEo%VMx3j(8vIoSLxaa$N7P?BxobZFd6#7f6roS9}! zmXqo%T()Ezz$Xiriu?s}Q+kY2=*M?}V1h3dfQMW~Xu#PbV1~4F$cg6}u0(1w7-1=k z=a(xr<{nfciYeEOGR1xe1AG!EY$%VUzuYrp+@)Q;l9^%%9FiZ;xwuV4RiCa326A=X zBNx*SJn3E8ld`)qAfz7f()T;WBD1eB{%N=1_LYC~MsP~?GltOB|kq%9Q6v3s` zLcUykZIBpPrt72@oIv(m6`9~M2ibF|jEYhqT}^p+8cAuW(eJD3`S3Nv$cSuv zC&@|R%!HKl?zn6J!}nfuZV=4$3v5M5X)x+(T0-f1#`~NkOif}KLaU%sS(phz%&mi% zMI+!4PU*w=l-e{Oa_i0!Xz7nE`E*ty4WmFBR~01PhAY~Q@QcmQ6vUHN69SA|X~AfV z5Ip)~_T3^2rt;>$vhG5r5{w36r8Omuj@!2|PUO{Mt<-)jc!XH}9cMCc<OWJN4pwt#M%DL%g_*-uGh3sMFTuZg$Z%o}kp$A{7EJl;Y{na%` z>vsYGPB;%Ye-~c%YTUX%Jfn?${|${PidKyDr(>;Bb^cVcRo;R-^E8C_xptEMH+(d$ z_dMT?Hds8QgF;6S5XS9QC{gb2qk+gqHtS$L^f>0k3@Oa3oJ$CgYpP&4Gk|6r>}R-A z^K$X=!i4CwCNzEX3U`B>!$z-bdVxpX3PSxG@H%6HHqO{oO%{a{K{&%a;tXkDkE~*L z96sg<4qHQ1WG*q~PyH+X34xhM<5hMh3+*;?b8l=_&6ffItjSe`#TpSZ=Ye_$(qBQ) zYs9S{6uCr|A@@$GD9q~eN4!=<{bS;G=M(>|q(T;a8#*SPW^fz_jZ?5++Bf8c{bhK_ zoXXo7011Pl$ia7ao9oK&>+h_=)T_FoX)x8YMlM6+dB4)q$V8paN11h8l^~x=l{J<; zVLkj{{)vl0kv?Pr6E(hGkXhHOJX)wE|w%CbZX#~_Ey|MD++L@cfcXV$wuO{R2f1%YMX%3 z61-W97)hGYwRIVjSSObJ#{6T&BNTtEP~FDCC5l}!k=hx6KmBh> z!DZl!?9qPW42>_wv*Opr=smSO5w% z`%G^bqv<~!G*mord~ai1-Jf(i^uG69k(y+nKv79+9g{o8GroF>oTaPWdLyBI7oGYi z%Ot75D`rsWJ)q-u@bLZhjq}|_cux*1aqLrCOUadg_1anVCI!hxFKBo@N;XMLWzdkR zIz;tIe5~OI? zKAq6PF6{@B>80k7H}GPzb|tdI$BzF1^~d#yXO>yom+1TEA(fnNQ>roBGeM@For8tx z^u+?b)UZ^^3Cah6!#VF0c!dNBuxhe3sP%2%j^g3|wL554d z2Z4Nr*e1Q|l_tep57UU3iC!6IO1ai}K(J&$SF0>OQd+6SQrB;FmeGsyJocE$sE)Bd z-l}xV)b7U0-$#17!ZcHHZP^MNYRmkYtwk==mUPhO!Eq{@WrlKp<=I}IXn?JS}@rzNmfV?(edL36_}r<~NsnyUgtm$>EQQr=8~G?ClNC||O4 zB%=gC$-3gvHnc|3=mz6pc7mgvzZ#6Tx*B>?YPll+#$CQg*&M?i;o29F0%+2NozNX0 zbQOLph)aYYtcI(>!$r{PXQd;EU5*|!9Q#ETkHHqDCf8qDC(->G=m6Z_+4^*5JI%+O zlxdhPX4Zb_=yvEuDn}zqR!Q9#TjJgGOQ5N(Ug7p(e8s|2%({f4?=EfzAk3$3c8P&5Z z5Q7XK0-l-o=y(0y0{ToYBE!X&-Jp`fv%*JL*eE+t@zY=`8I*fj zdQOmJslZD|O>>g0?ECuJ#WlXmP8r&{mbc@eT^NyqO2X1qyn%4`>QMFLX#5nt&0sHyIQ*=a5!I#U-+(3{F=Lg}M{}XtG)@wP;PS);K{;670tBd`z~i|& zy9`@Y|E3qMYGnpP^rao4J95~9MzxF?t$s@OEg5E!!p(7r<-e-qOc`Wi2mQbzv$zmj zkf_V@ne2FkNnjx<&WQg28|1q+rnYmch^G&z66;`_Dw+^Zl%&+GFSI`?py|^#g>1Pn zUX{>ziGfIwpYI|7XUsnvDitHPn z9I-kUY(@%%%Ab%9%lv;skZuV=L86CfRTI^f{y{QB6hQfacyTb|-u5u%x;v%w3I61s z++8}_9Ob{ocMFl7`^U2sVk1pFO_5w-{4lovy+~GH|48>_uFIooSJRN*wk`j{<|Dr>fH# z;TEBSxQ{FKI+cz%1|jekADDxNW_zPEfuNquN`Gvmzi+izFwsCmC}SW#xcsg(JNhwu z8$K9Glk!oPi}H@4p~)BW-orG^uY*Nip{!yTCbe2%>Ed00K9uv6iT&HNi&P#aG$eO2 z5}3<+Ws?iwZP+V6KMUz(s{htvKO`jq zH&j}qzmoap`4jy?*s!ot+^na1rk_=!dNDh5(OXn_Ax)7hDK@eA`qdyzEn8U}IC~)E z6eg3&7mQ*}U0f;fpjN|CCq9xbk2Yfl#g`6>Nc$EUk!xnp$3Fqe zjeKrm{CGu#U_>q4wj)dV_gqNL8fMUQ9LHgrO2d|BRC29N1T0rQ-+MgNuRJ?50%-!s zl^@;9@l6B3_Ms=~?dk_R36|Rw7)Yz0yoO{ldnl2ZpoLOzmvdV;3a{RCMUe*RWWVQ6 zVd8e1$TKCyPBV2Cf(S>jHXK`dsF%Z62-(Drqb4l!$YL(%#*wu%dV--5M%Yw^UHoZ1 z`@u=IsypvNxfv>{&JplAI5_!1fTUcAj;L z@}6s0eT!&0+Ns{>>msvCv!F+|Jpgm8>TK@%^KdV*Yips2Mw=%V>= zeq)$h)w4?+QJ>O8u3PYD+Gxm{{!vZ7gOj28?M>1<=8e!G^aU4&6FIb*%C3|;5nE}; zyD5Bqe}zdkI(kTfmAnd@C_wOt)8qXroa7Vi}cW{H(n}@O9Ra4W$ z+oKgB8#_^zyIonKu-g!LzAlL90n_^K1H(-VTKPH5FgM`Cz5w3_kTE;4F5#Wg0hAcU z!!a)VmZimI?4Y*VQrDY@lmzJ2PZrM+PgXrq3CA57XWr0ITt(37+^JKeI5ntzL4Kn! zHQ^Zk1YcnVK7IfALJKi8i^`dVsJfDmxTyTt*~m!JN7+pG5+pb#n#iVmc^d>`cy=uD z>IVVB6na5WPy@yU1l5#YG(c|)j2A3AU2>@i_+$#Sn zeNd-ZYIeimJU=0<%jy#{a^!m_G}J@f310;Q$v+CDH~4-bzR|A`#{=}H8&+=6F+hV( z6=>Qc(gF{?MS91*_>3Og0C`lU_v#N>QF^8B)^w*5s_ysz>|94%^MrLlvi1$^AMDOG zQAzv}38DS^g86BvMZ(Mb(^@dFzi~ep|0KtzL_!?o*bLo0wzV>q8(I29C>l58TR2a& z_VnVbs{Hxq7o9-!(V&vOnzRlvT>%$kJZnX~^mX8qt#;)Clz6j=c=QbFNg`2#x!gn_ z_vBGc&udE&kcQ@D08(|R?rR9FcjK0G`oNJRAg9oJ?ZKN3VwA5*PG@VxbB{D>v0ICP znNZng9p+z52bP5frt24}qZuE$H#M#?->WpF^;HV4Ni1ikA%QA^#KA1G>l{cg9zUPl zT?X&eFK&CobEQ0xDNcSXVAX&Mf6L$%X~*-TI*7;~@Eu7$=!{(xx7Ib{&l1sRrjZ#0 zXniIe>G%TyowJBmuP&Q2hD+82gBWb6EnK4MF*l@+x_;gjzHB+T3nwE< z2bIcj{&&*UHwPGG_qSxf=Cm!ZBQw67vk?{sCcs|R`<6naCimrZ{vz%~go|BUQWmw> z$|(=~D_Hm**m*QB@+V8a1;PX)#dg9aA0m$gDKJg{3(P@5eD06}GAaJcuz@A>$>TRO z3qGjzVcNSb18J&01g#Ex#w#u zaZ9v0Oq0H7UcP5s>|V*2v1k7*LP`+w_b0k8E>wJ?eBq-r>j#WK`cwwa%-+<+)!EF* z?*FX*4+F8VasF=_N@9S82IuBt|NkQu+=;}4);j@%0v*5+iX0zIpu*d3@>Ppoi-PgY z)N=rxGy9}+p@=fCKG3&g4k2<@Df5nuDj-IpXwHE>xwf^+_34=MXa~5tZYIJ>TC7Uw zc(`9sz3?FF+vrG*f3~0?!&q-C<`iJYT~sWd+VST81xBN!T}i53U#7h~k*)G!y%*e0 zZ8-POXV{pP(s&}kdch)Vs(lsp07;-EaPKZ5BC|9xbGd8xa7cN0ppBpP8fqE zsV$-Dsx|@{HvG&&GYX% z6hy9(}80jA`0m81>9j>;RqMPaVQ#29pjvw37pm4Vpp}>HM zh9`JnUVy5$#z-qA%o!2XK>`ERh<@?(6as9YrPQGEyoEkHl#q?FzW#I?rSp$0&w@H^ z&PekF3D4G`Q!frM%U-d|`=f7=y;FO$wnljjYY-iEdWec)QOXupjmrX{_msB66FGM+z0M=iJWccN2VfE^Vb9SW`YUl$-IUrXJ)ckGrcC(-kx)nb(S!`TI z;M(yx#{mBO`ot;@EZyc}s}}#?{O@R8vx0J~lSS*;JU6YLjk0T-; zI1*8j=!s#Pn=f%}Xo8U1!V10Pr_UeT(XLY$71nJ3Bfp8{N+Ama5=pFI>*zAr{_ifX z_^kjbVN0IpuQ3czA5Uw8|3lf}nmQ!-299@yeWZEI$M7hTV#&l6IXfa%cFEee<40KRm@0_`G7a^0Q>O0Z#d!Y(E>Dra8_7A`wrV_0xDW? z5w8rFU>OhHNEBRSycj%&Zm2wH1TJ17kjrWY4*EJ6N>WmWvJhHr|m*-%s)uYH&nWd}T&7~EKsm+4~uLbGr$mkW2Mo?fmZ_`KI~ zXBK)&SnQPDJ3`1`b>r)%Oy^I0e8bPjSA+#>_SkrF%YzrwBAJMWQ4LS>x$516rNCvm zYz=9Tc2zWKaP+d+9uk9_iEg{}I-5&Kxwg#`sAWjJcPPMdXQfH#u z7}okCeRZ`x^d=mcRKr8=2Og27*W^5)NR1k>A1}2s(qB%Jz84^3pq;A}2(%P}Vfg_+ ze}(LLkcZ_`@_F16w0Gw9la4vnQ{peXbeGM9ig?aU#Nx@h0Qyt7uLYuD*%yB80j%69 zg|4C ze%xHTDq7WF)YY;1H{Bdd(8{LFiq!sb-iZI&9!))b&mPWA%L)QRK@XJ{kZge#)c{&* z)qbqXlKR3D6!i}3>VI7EofY=!h2qS3Yy|!nm=5XOONc&BJWJxpHO#9}597v+9NW%) zvE2)OBfLaGn2ewO1I)X07(|AeT_C27njIkjzG9Lrqvu32=}RZK5W8IoSTaIoB?y^b z&qL&G;UfNquY9^*wPE6q=jH%CI%g-wf*0INKqGi@WckzGXiee#48=1$SEMNaLEoz*e@pxHNVY~DAd)|~N4Um{^I5rP^J2KP$ zwN&LqbCh0O&08doy)r(beX;v1xZE4NuG6E!=@$Lt>cI(=_h|JL_gDtfv)}CZ*swU< zhCrE_g31K4!gKG)=P8l3*QG!{bKoz)Ww;3W^+)|AmIe7NOR~yBktmFG^~L>>R{=15 z{|AaTCCOM-y7lysh~-NIwJSUa=1wM7M=olT#*uQz&g`rrGG8L><1Dyl?J@tnmR}VT zvPog-svs(c;ZoHu}3>$Ar)8+fb|wGS&V-BUL2+sEPvl^R_b4A+kMOfK4ocdp9K zvXNiS0eA}H=Q~X3rT3SJ#mbrxLs5#wig9iR24_X%_~VAR{+4qzxVqbuNBK!jQ$9`e zdtYTwZ(Im&27lE1Iaj&XS!J3`vfJ_@`&>~#cfI&0TKvW<2v1-C`B`$Vyqh#KufCQk zr{Sq@CKd9!_;mm~K3>q?2fXjhaQ1Bim$bw_MTBx1AV76Zm&K|CMz&lN^*7{BzGVOg z&uj3!I8q*7T`CRvJC@SN1~tdB*R@V50WqSL;o`Sc9d|DJ`Il-|a=ecTGS?QmmOApB zPElA}v%mNMmKs&UJy?wQ(kd49bLmw(WH6HdR5(Igl_CSrP51JZnuPh71W_(75?OHm zVI}2QML>XGI}Kd~@d7B|eDB|VNbmxH6I!WrXKs5^^}*;O)?)9KuH>6Rx={1|7DJcf zdve^%dRVcOB+K9@C}fXN1*ec=A+aooAAh`UCwks$e%1YgJ6K~*E%*vR{!^#&T6!N4 z1Pw(GroIWByZHO$r=lXtBELk(1^QPA^FwrJZzD*$46`1H@TjOr>;?vOjGGLu1_kd01<a4!+b!V zGBmFtw^12>ip3jaiUllM95yHtzA{aJ?9U59BI_$pZ(u*vV)cUDxge~|u|<~+7mbwF zHpwX^4i7w~&L$}7v8ERMRkfv3m%H=?Y^1SWZcb0vwanz;52PfA%F}(47+O~ndNIKT z@OdV}xPyh0^MB%wBW>xVgI0{5m%7>Cv^vZU;L>lqEr1#>sk>xHoj)JXzsC`jKq%$={}at1h`*YEsf7cItI|lQSxuA^zuD zaLY512ciZk%ZHj}=e-zZblOXFGmn)UwI}LToRhD}=JWsV= zp7Tc_onJlQY1PWYyz0eoAB)=R%&n-jleLQOlsz_9jE9~QbB6*$L(%cJnSFh)f?LC zx^Ql){3z=(RC$1TR(Q}bww%W`uIznmHJ7$y8TYjI`;1SC3ma;V{q^!q5rT1|%m=bE z!Af$lDM+vPPb5DHqVp+)BcRvb9Z3Km`#!vV?Wb8z5JgF?N?Pm5aIB`FJni;lXT`(r(Dxz6b7R|$TVF<+^wm& zx_wgbD5Yg-pb-2l!VF=Y$ij7IKJ3trnT~v%Cc_>9Q84?9QmgH??jN<cT52ch7;j#R$1uCunK84r?Sda93 z#XC!OjjU@MAtKR@mC6eyT4lXqTeyB%^e8shK~q#^*flE8tPw;@rRP(IZ$0)Z7XomQ zwB$gU4ckqrR25A!IP>gmBM29D30=SmI%?_p!>Lhv{U6Tr#UN^;I#}#52J{9=wuW>Q z)Noe)%Mp>se9mDLV&A>t3wfh@GyOI{Ze_R2qQ!C~r@A=b(4pZ6%j36lMWz{iIGRNw z586eXF($~}6j4OkNOCo2RgHnT9pZO6{dV9fkiIR`l?rNX+_RhJeGf01sLmtSl}o$> zy+|MRu0;@44IND^fZymtz$X#dcrYiE*Y_>Hx#2Sa2Z^7Ij#d?WB<^tkHZz-9(Ps>G{fcjNb3kzR9`1V$1eb?STCA+ACB3unmu`D;s1rQcHSojDD2 z(mEl-#9Tidt-Ghc<6p4b+vGL(Jq<8|+Tak3qrHPe7ncHYI%kJewj_hHYp#0ty{i#E zbduvd^GCltnIXKJdT>+yJquHHC4~XB_Wj^g8{i-J=5U_bE3;xus;zAe{_i((npx>B zQ(@_*ft4UZce)Gp*I2Tz(^zpT+W0gzubXfk84VgAw{}9%Il};!&wR%x0tpb`ao#%U zRKxPYyTHtT7WH?aWO)rSm)l%^NuQ4Z*QuQDpep2Cp6$Muv9zN!4ce2+x}NqzfT*_- zlC&Yh>X7Pehk6=|q$%phl^Gc;yudS4-sbMQ-6-K6BB*ex(aeY-FngRhIM^E_nTUDt zizQ*UTV>|n<|zHhm=Zw^*$gNrWh=Sq zIrOIR)T}TdoEiRxszXx$Pc={W{D&@lBC`59AycV#^(0YGYzUpLXYv zgfSoz-Sfm|i+?%}%yPt%BahvpX_dW~!PDZW2w^Gwj$lU*WxPg<@TDV|vEcuc{jdn| zRcKY2C>Mdm7SQ@n4+Dmnp`Y9uJV%SKTr^XRmWI7wsQ#9xb>6os^Lv?knVI^|=mtQY z2O!-gEn79zT{MfQ%3v1IOV6-Fs%5wVv%`;Rvx^b_gvZ!PLTByOd~>vfkcsh4ykaYC z?4NYVq?~Vc$bSuXd@6G%XXk@5-HlfGY%XmX5kREeLe20Hh5~bz-AHo0KG;&6NY*)0 zBwI3fn6eAbNxt}ko8)L|O9PAJs516q@&{(B7_+gt@`h2e{6iDZhwj!?Z?Zb7B!Xt{ zKvh_1F}AA=a*g&IZl#ZQ3+Q!2-~^ukO=LcGLaHirx&i$4)KpNm>XFX;2ajbF;i$49 z0v%vp-UI&2iGe;>LZ+88e59!VzHngkEI8@+#@l4|n7vP(DG*Tqssa|@gfRL`Mp?Ii z=l=;si})-?zE(|dVE zKYn--ob+h?eezWc>CCZ40+&-cTL7-*8VA*!!x*A#H3)=ZO20Gqw`Zwv-8U3bzyWur z_AOHg*mqSHIe`ufm%VU}@JGDm%4hX0eqi7>48jM;^Bp$l5a5a!fkV%b=^YdryDqa$(Cbe< zbBnzg(bO|#7q&6GU#Sv^3~NUqYh3!`n9VAJTxZTtgrd`BcrZqlDBA}Fwx{eX8A9lVXi(`Y0)ZO8^nT`&s|0k5K9rtZZhsG^{9WKj zpJ_fYh^x~fk5$`~vRB_G|I^A6y~8yjzKi)A84MmI)^hM#LqHTYu_V9G#Hd44&9&;o{_&ZnI*vjD5bxo72Q+F_` z)*dBDBYJJD^WkH8?h8X-0ln3?`m1cwQW*G8is~u=Bid)-@~@ZjY?3ttvUZGBNzjcs ze_NFvQxVTNNzM?}9>Et5WW_(F!&1`YRf;Bk2|PV#qW$WXtf#L5ym@z78E}NmPpU~* zpyj?sx3_?+hMc-42-7DC0kh4Cq;O;gP#z1q2&VmN=1b$lP8{E*Jj@-=)mR z*!eAE(vggIsgCuZV`LSI$!oW;lW1G0y_2gOPQC$cD!0XdMI&))R!u}-*2_LedV;%? zb~>2v2MqnJunoNRn|e81ttK3e762C0AR+1XO}4QeHT<;FCi%u%3o}t;Nac$sUd7rc z+ml+z9?f%*9zh(*@f^!lEuyljjuep$X}BsEEDTbD1WH!F*Oe*pRlaxrHL{IHeHq=D zMT0HC9ChWo5@5SAvFlq!rUnHROWzK=En_+L0YS1yVgmjbqa^iJ@Po4d4~YzEZP?>+ zA^Yys?V7-&>rRLK9e|0K&4IEEAXW%yttG)MQJN#8LoM9L&$NA566bH_j8ppkV7t)4v&e42u&eO>SO$fB586uw`2Ai0oV7$p43Qfy zGH~Da++`cSBh(*g3oB!+xN}Sf;N;y!Z+f=PpvzG&_zx0pdt)P=VzF?^C97bfp*IDx zfx0^n{9}e3@yxsZm*x+P`Wt*DnA!5*A_P+r_HXRC^hl3L z#A-}Enh}m1zSOPqe;Z9U>sBQrP zVWOyE3VchT#Nhyg$TbkIJz7G7X+Tk0h)Cjr#t0&qsDRqRkZOdL65^1WJsvDswN3B$ zZ?Pb5D6!&l@M6b=u)=^g8vXxpX?=Xp$-$DMKoRe4Xm7EhQQ2_&F2Tp1`^NE%D&<|XhdR zD6<%u!PGdOO>qkc3{OfTbmSEgEp`NMX@2u<-vB>hPgW(U*?AmO{|tWoDRy5*~ZyAC~=POFbbhw2S8@J=f%xC2}y*kd=mi|wuEl1mgW zJLJTv!1Umt*z|A*?>~q4o=fXxM%oPS~sE&mT{)A zvJFsGB&#sc*7cWTm8A~iK5<=$M*O6_q^GEB7&6=z%QtiJI-o`_+YyFqq?(kd^iL2I z{4j(Al_pt^ri+H$fUF@wn0}0^Cdutx1?B{)r!28bIWpyz%uxZ??u%4LH_cdL3zM+z zgltBVD&V3~c>J{Wc>-r6M;xjrU2sIP*BbfM0{c9lM$&TFcK3T|edcPaX$2IWuwAs9 zyH=4}LYQWSo`^T;S^PHI;Oe*pXGZJvYlJaO7ftByHCgEMnNN>@&Y&_Wy5N8tPXo(2 z+*0!l=1E!8LyEA+;`e5cb>0E|zMCo^AmHS@5r_Z-I@~l(@;7#jwh|0oVbvG@diQ<{ zm2>021=)?Z1MbAz=qiC{|Rf^O|_~z)tn^E0on5JF}@7k6iZYss=eQ#^#W#c zzy`R;j_LORIZGJ2(cs4A_C8O6;evI&+??c&G!D0>YI~@qwZDQ)8xgV`_6$p!e^rxg zFE<+Wx$X!%aQ^+d+)%b!F{*JPScbQznxm~gHWejMP1J1r9cE@xbj(vt#|^D!gd<5H zXnww`;>ZPmM#$G$8cSna$$iGoLG|Br9Z=xWo&(8Sw9h$_K zo}WNg+IvHni~lOT;O~^%!loYlP<0mR7@s`3R;BoS*fv9=FuivbXVNLVXr?(?7ry;wo48QVq zA=XHNrSRl@X3|TXMpFMCEkng`e#hJj9$ouhH<4qTfC|7Z1y6n?)(sRPeP zXHKG3V9{uZDyw1|Rzez&i}r|sgXK?8+#;IEkDx2Z+@7DPB?0KQPnbxD8vO70d*7!! z!M4gtpS0TmQ1m3rZgg-iX2AbkP9<8hahv~rF8t7VWWdeWp^Nx)%n5IigHSJQIKwa< zkP6s}rf5hjuW0z-^a~u@& zn$>B1U+g(_lG~sEk+s$TrzPLq>MXJ@Mvx)-{`^OEUU{RkNNu5b3Wohe9P{hkv5pqpZ5BDVas@$GbAZk{o?5&G*Y{3>*h*Fy zfk4iA{w5GjS=uEo|B)$-7l`TAg={v592Td*Q(LI3z3=j5b9ZI@nH%A2E!6Z{)F%Zq zCP~Bk^wL`%ilUqHdCSVKRSwvhg$ZHq+Wyz|Mr*ZCjB;_6VGrpf-vyJ%r6M&pEJj`n zmtWcC>?99Lq=0zYi?g8lzZa6)UcmEz3?DPq71~xS$sfsU^`k)6XC@p#TnJ3v!Qcck z2)KYbgRX@AE9v)KkTInRk^+li38gdGGwy_1E!}u8PAg3#F|*26WV8cNT@_5kfNFgO z65pGE#w_G3er@T>8`T*iu9bu4J%<}|T?X9BK<*w(7rkKS9 za8Ow>ZwUBz^m4ffq9YIw{c3jvlm1L$RH2QI@O3{G3_c1zcpLVo3Le`STqp(Uf3od` zMHc^Qtg2tmOo_I>Sb+L~&L};Nw-&c;$5&qLdIzLtRg!1|!Lvw_9CjNUhd+&R2aF#@ z1aK6L#6g-A_hkCU!=qh>{2RqQSW~6&Is$A{IaZk(jA<-uwn~o?wEC!wk9NOko6mY- zROuiRX@OL?6Ehf8DP&X3jQ2-p6UOy+i^LMa>@IT0%?`tJ5XJ>Tasg2NWW8v$w29YF(dodb$>tfY7Hf@ z6wDnk!J-(?@uT%OFLF>p6T{j0f!s~!kb~P>XJBk>tI#wiRD-37jR^UaCykQc+_Xd! zRVSC)B*DhH115i6j_(MfSxuPAx==O)hjaGcWP@}&bGtqL^1|!-vH4L4O2w4c{Wl0@ zBm{gsBHjR9VNYWivqj7cyWKc+B@$L00+$XXK~ldmAQpsGAyrw1IHZoGZQ!&yB-6Xg zvu1Hil#r%x9|i|R^3=#2-<2ZXNGrhe?kd<@V|guHi&gi-MO2v)JhOqNa&OJ>>rh&N zV*9o6JSPg|AY_MBr(yKk5kX~h?OD4ypj?EATp(z@ywxR&G7#te~VB{%jf_;ms=ln0F`NKtOK#-4b@ckP$ zTcD$c=}Fx^M#C$XsKGCwzG29hx?%z3JTbhvxUfW>R@wz5!NJ!uGZ_(is+40aRFgA) z`W&_p&6K~oKH{7@0UEdO?|*I=n}QOd6q^DwC7(R4sd0z|YB{a=$H=KsV5yf9qmI$Q zUcS~5G^_DWur%ByIZVE%0qGNOn-?jCA8ZA9hjIRWi7-7t4p_aUNWO{~b*{#!O0%|v zcwmI$r=VQuwz4Yvjxa^PqKn*d#_tAogod>>FPiafP z|1{!b`gS314_Hm>h6paOEr3;eEdR?jNfcrGY`O8pqKS+4E-@YmzZqH2exe-1Elc9- zqspDz6HNC&iy3hOVCB6MOo1MH-55Lc=~%N%wAo9<;~$1w0MEdwbsHcKT~~l92zp-} z5JQO?&xa1+J!q$%TFd2tT`+#u))T>iPJqD08kqNSFbQJqWibUceqxe&d5>| zs48qz@*6%m_FR$h;VCkC5WdLF1y5PO2H6l8QuCN=kmO3V5JBICDaB;4c|CGviGWnP zJ%+lUuPUU=HJto`w$B}>|L*vW2ekdg%)y0yo4@DU3T%bYaJfpfm{3?PLa3QTH$_fs3yEw``}Soj7?0d2P%DaN!4=3&n=tn3K~D=uJF z+Cte9VRFLBXQPegkP%{d?TDI*T6y~ei*F`W`|gzWgprC#jX}wSb1&cA$0y$3(je}1 zl-)(CI#qTN4H^ps{~Xi48$cZM)|wReV}JAns17|Ok(Y6B0rf~JjMhcvmr>HmfhHt- zJ=-0qbCcM3rjhqy7|B@1Xwnbj!2>xw0t{WwpMk(+Ij0?{h7Nxg%wUCKSZ0w=eQ^#p zWGF?$4rzXv;HQZiEo69HB0wPSlCWZQ)~Fr1T+P8cThaRsykr&}mXw8}PuX~I&%swv z#*&kL+o4-%?d*(Hl!T37b+X4WgflDL)w{VRfMF-(#PK?i@ z258DS9dN+)+|=GG<&5aw5r^|U#$7pYoXPU?x0Tu=M$m7ZPo&65XBPGL00E!xQLXVm znnoZZ#Uc_4jaDAe=zl#j9Pc1*xZMa&SSLsFshwGQX-Xs0YDQ={l1HPbt5&j&2Q7?S zdFg*)e|>LeXH0q)R!7N;%LLYF*{Nc%1Db!-=kpz!<~TC-lva*>zgC`8UkH;kwue@z zG9#AO-`K@>*iQ;7T`!`UwVG-LhP6{`x~iHZI$+d=pcx<7ucDJkn9Uf}t2m`|Bn8=0 zTh>#nOx0U&_JBqsOXoT;}ouOWU3~QnxBXYHbK=aDq z8{`w9^4`ExG5nEb@?}F>BXrlXnVK@zp!$SR;jgqJqEN7{*JWM1f%6alAT892SgbV3 zyef%MJOhSeKV3h`3`c^OwHAhG)e`{zmgMatESq&ZFjOYeWIemAVuJc6$SP zYI0}HUmk%?59j=PK7X;(P9dzqdsX#Ysl6B*P>JYft zR0J}0zZy@KJvob`P}1JKE$vrGr{51lv}oSa8Q+-Xk~h_|0A=o*rFfTcF!RnXd25i4 zu*H~Dj9gAXAnI7~=^rTj%ZM?5`{y~hv;*Jw!P)Hrw#KLStgiib@F8+qOIXn=9F$uQ z5uX8t^Dv*DVht5UKIa+ox*0L*H8pphmmhi_q*YE|V%F?MxHE=A(vSk|o0O#WVh+17 z*_0wOwo9$sfGwIm9ecK3aC1Iu)pMi51WVJUI9{rNdP{zeyR1S3d73z&yBU$L4f~!N zOEH15e#pG9nyf8paGMqOYg_8Bbr300X9eAm-803guVP?Np^|6ncyYYzh&K~!ZpnkX z4c&lL%x9`BLu~Lb+4?S7Q4z|N@-VLjgK^gK?)=Opz)To|fEUAj9B3BFZtS{FU{hbG zcM(-2&1m`w2N;+;h2>88Yi2m^fKpzo&rx5m(5zk0GlgtmCtoU;Kv>(RMnWG5n+p(A=BuW6JKo(RlnA(ejTN_PnH)Zrk;*_ndL`% zpZNn^4WCYv0>s&-xy8sAEGzi;L5bO`PkoI5KpRKgIBmQlhC916T~&F%GYb)>-JP|A zWb|g;%>0u{3}KY*J|yeOxjPPz`~fUfYe^rJxQ-F$K^Zq$HM~>W$CQDPz4zOrf5%d1 z8C-d?SE^EfyqT-Zq9vYD-8M}#j#nn})$9)(m`Kd<-}N8civ z1V-w;^krsl^Q|uiz+dzl=*b#c-(8pv@bVUyC9)EQ4r(U-E{&J;?ll)lqtAvlp836j zA(S-%;1006<{#T|FCjmXjD)ZR+j&sa{Q~v z|7j#$XxviBCqqPH60TO`T~=3haa)sj{yH+`RhvUvVA_jgAxud!$c5G%T(ogU2;^m$37LK&20aGDMI$EHE$g!L(pxBJc?Qg@B z&meQ|PV%Opiu#5(qUi#Vg&42Qr~)10#ko67gX5$b*jQ;gtgy~MT5NB+Z7G^xU%SQ| zmN*gzSLzE-wx>UErfRQjL}^>|tm8Y;VtQi2nK*k8J^-wJni}UQ$euro|;!z+~LQ8`aN0|g303<|B+|79jQW$5HS# z;G2Do=h~r52isnwxC#EV&q9kMqBmPYT_ZiNk1M+?>HG@{H%EZc;cINlC?*mhee{7d zx|;yvV52S}Uj1<#+Zon`NN2llTiueUJ>VgZYavAkXP|xvNY|}BBb@44>dmuj`#|}y zCPyE57))wBk@hwLS>B(PShy?C{ofw_L3x@{AG>yACE|1wNU+6=ZIWq<)a2B7!K#Lu z74k9!3P3DXbTR-sS59HeGP^;iPX7qbbxQBns&jAs2c2KGnt?*CQo86cCy3~~4lR_p;|`M^mi(a=yzEJ>`dAB?Dm za9n%`l~+FNA1f>}J{4vp-FMERAHc(83mQ&U*S{dFb&nfbjthe|bmS&(DusrG{+>mJ%&hew^yL zQNk)bSLFtgh<|lH1G{e&iUL`;Sn9g)g>1*mu#SPjQ<$dnup5Qh-fM3CGSO5rL%QGH zIF)|P7TY<~&AaJ2Y+c1FrqWa|6T=>FB3cTRVdG2^6lWR|y58F85N~kc5E?~tjv7iq zy!imktqYorXT2_h1G zyO*2L+~DvcIoBJWJ&x9VjXK*&@;Ja>HR!VuB@=A6l@<}ksO-hA^q2=dJME!^A0>PX zgdf(fU{aDNLI@|P(nWS>E~tIbt+)F-OLPOwvW5HqcuVjb(h-+)Jy6~ZKI8wTz^W@q z7=1qa?L5s!S(8(3*Xg>kRNd%9=dfjSV4g6@)ViZ_Or3lLq=f@*ceQJQak6}L2hb~IE~*|Ivme5;ir_{`gH zT^jvl(gq3zTg_tv!$Kydp=g>Nnk_Kut?|5qf8~dhF!K|kc38X_{r>JnL z51?VgDopE*@$JvlPMz2Kf*~zCt<;`ak?J|3-xl=3QP&Dq-k~DLeD{;kZve=qs?}MA!iio!IBS5V0j01bP zu&;$!v8vgxyu?>&?0PEu&D2d;m?9y;IPoxl!ZUJFD*nxP8HL7kyXhz=hc}zZId^i#=&~1aTB-;X`%KjH)gwj@#7D=e75Sq zSMoJ2>dRXnJaZOk8Pz_|SL%%p|MW^Mt?vt<=^?9mXe{6MLbR=)P<<36**C-bh`6qB zuJ1!CwVBlbwu}ZOB({1gZLjD(S3^%fY4mXB)hzR3`IK56<6TCa-=rNd533GxAPEkQ zmu6W;-baduWR&xEFg;uvxq3x5Riu0lL=LULgU_%HIFts@>?x%Udo|(mlyZJ$8cw8m zyV_@b1b<}94_8C@6CpN{Cq1*9dyA?806??9^>^DKOSLsM1GfGJPoGWaXDOKtoxDUm{V9S{LZ?a zSWqY&T>YR$@BxL(bS*h0#h-J((|kY=S5=JdIwMIWJ_{JLgMIq*mkQdclBbbk2}dzX zAQ9oo*bjJ;2J;&DN$Pz=anb*l|Ktq_y|&ivayYO2tI%&X<*ovN(uu|Fd>4uDqp8(Y z;x-4c+8BxDSe`UPwdm|JSw1)mjxM=hMD%^K?B&fEk-rh`gSs!2YC-bRLb??5aD1 zkmW9o-(ZYf-$9dW)C=y}DJb8Lb?wy(J*hHJG$W~0;(dO+ScdHj8UkGt; zzACU#u|@HnSsE=c_*=L|O7bEGW8+>~#d&sQ`YZYR_^O-z#Q;!qQIe6oHx9{m;Ni3(6qo3od=F_JX0I6{jCYCW1}UC^?b&Wq>C%lzfF2dDxpttW zO>5d$8oyL=CxJd`hQC(+V58@=4w0#$Xsf=Anc4I_KcL6!KJ+QyPxR7E<^3w;K>Sh%l5!6{3Hh zxYe8eE)Gv9z9WKcsOM&;b5IQ?G&dTl_|gF$=p9ApyZ9HFy$EAE(x~3sD}BAGp+KLX zXJt8Rh)_15&w>i=(PQ2o>=yKbGcNYoD4E$hN=z*zRV%_y!O00U7nU*@xbJ);9cRd>qqR zy4%7u2o2V&^&Yh)^lzJVE0Z7~2+IReP>hHlthb02{=8)6#_tYE#bqLBop*Z*-UT=z zFW45~Dav zv*Y;i;E**J_sRDh`ta+1<`@aebgt&_5y4b^WcVo8CBp;(} z-v=>xIf+TqaL+*_#7i#zpK0*Lqs{>T9T8ObT$$|SZQ_ML^KLg%xfZ(OVIZU;4jeK2 zK}tj}BRMooy!X+~mvh*(p3aOzh^>(2dXeZTBp*7CIp+s>63*(#DTmH@E=uIdHE9!*S%eLR9%X?G|$wd>ww#WT^;%TUO?yuQ+Y;Z@w9Dk|2i$PI% zPV5g-gWr~nDF zJZVy;mxJA|qe6t&JzMrgE_q~=DDMt{^?nhZS$}L$LIzabRfRv^>dBRkL7lgSueVB+ zfd-+7f_1Ti6IK1t#Kgu$xnCU8(4|% z*MQ6sUqtLTRb>UkG5puISXw_~p1#$MiHQu^_N2LU7tb(s^U*AKuf?H)cGYZ)7w{2; z)kHs{^6x*qjO#@1ZUM`j0h&>;+&wnl0{-pdQlfsnIR@4*a<98dN#Zq&NWuKtLWzzb zypcz6z7#++MtWSe5w6B6KEB5`l~#amK0!p%-tCsiCtJ2{a!5Ga)f4ZR#US7jMd0A* zwL+~c`z1ofj8pV7gdd=4v&6ptrSNk9FMJm>H+!P+9W|g?Yb$nx0|hW^u;-N+pLX;j ziO^3>&oKN{-zo&dMxVF3*$kp{uXRtc(fe&CHIafhvCVNk`Ku)4VK{wf%Kq%qNWxsO zC-3a-Yb)|*+kwL*imUkb^4fPKh)7~1T|LA;-^{IYt#){8eHL?|%ybQ673(S+Qh+;F1>by4`oY#cepw4(~ z1VG)|87+yi(`{mQ9JZ)i2oC!zsJ;=rQA`uFy{B5EP7X@7ImYgHq=FV*rYQeO%V_}n ztx>CG$Ln)V#D?je{jHOFQ?){-?+AYAQ2F^cvjISgI`ZYBA#!Ks2q9{>V`F>_fm#0`C2#EL|=hX$TW?Obir*hzo; zqcEpYdJvFG=sQ$;_7)&r`MSrNks7A{4T|0%-@9&BPsfRy3zZ6*9N)73@mqu41s9O& zQUJJ!GumAji;oo^K*`#sxC!&8rb{P|Cd$gF$L=kKf4-c`4(9KVw_%_OvJd(N;ed=l ztXvR)eHTM@OI$x7(EuGUYnYZj))A#EhsR5!e(eL7-cqG(>LRE6hV_{%5W5Kj?3$~R zAVWefh6IbkaQ4>xr%iMij>cCoQ4IHWFbyEJmJ@IG&w49bOMf1GMNt?Px=V$-y3 zH1KE8iD=}pfbWH8L7e6-9wi2ys!f44N~`<1m*0YouBA`A0m)6XLX#SPG2cvb02H7i zL-gG~0%b^vV-6o!7e^*Z_B<9_0nLELs3Z)+I{-BK z7D`o(!+KYHlgf$q*H`&FU2@3ZBq$1L-Qu?LNROcOWB1 zI9I&P?}+8wI=97-;sb_rG`V{xDQ%%P9yOB+MVd69-3Td=H2Pbc7$1iQ4rgN2+G}|B z68J=q>Jfzo);nHtDG3$xi8lqDQ_m&zmxwt;Ir+%1o?pL=c-_l?;;_$#fwc;&xpTX!+Ym(?D9R@?Es-qtB@9&;9_%4CwVh+(@x8kQDj#+ z^*IGX;hrR>q$^H|3MSwm5jhh~VYH+MVu?-YL#!{cu-eaQvrIk|Pz~EvrP(iomtDzF zpDmH!v&|$2M9}W<7f_f{!vrB*J`3!lo(ObfVWnjn9U@ab^&CCn&^ zGJ|03x-9g$Wly?)(m0P2vjPVH~(~SSxulV(cN1#r|#N8zv8RN!4u>km}p)523y)aAw`RY4qVJt*X=RM=Tpe_g z1j{bS;4_i}(nVyXQsf8~^lK~0%bqn|b{|5fx}ZTkOLE*v1aPG3zEB$NpWeXQRDo}+ zYsNmDR>i8zbn_Y_6gNfP!}PZed+bg8iD?MkH?${gGXR@~V}~DfVD12HamPzruPO># z6}JyHF?pxX%LeDiRO2_I!Jq_?L0t z)Sd5v+9$UMyBi!Hs7j$TYY4h_dnC2}unNYzm+$r-f(ar~NMYHMB3uDtj5iT){{Z*P&6||5@Bwm^lA4J^*3nNZhle0ch*k9kikQ{)@OM z1c_?j&))3kvyQZ#2ahPQ@zXP8^rJ|@cd@Y?NuW&pditVLjyTQ7A020i=n+G48FJcd z_dekgZ<&d>tkn>2`#w89ue8ISrXpAWm9kaEGQmARVfxEzpl!-FWi9qCcjH*_!(~ll z1-9mQ&vx=;hgA*tG{CF3q58cOqm|7-|Hu1bOJhelTgw<-#a6*>v=N*=2=~IRtKG@V zA!lcw)$IPfG@w$am73svrkB@CRlMrhEH5r-Jc@;c>_s)Ybl3*OX)NQIR}0!}|xWM0VV*in?=Vn$}toZ#JgTm%>HV{R}Lbe zX+4_-R;1ST>aWa3EO33fy3gg4o(QaZoJe{9ym-Gs1>7W9ei9%*YKcaf0;L~GhsxUm zp1(gA8Qir^if6>ckx_e$5<#4Qz7I5D$~h6wM^YUvgc#rHMD26vA}8#f6ki9Rq8H?CU8NxO8hC zqHqq!24Jf*SgM2b-gKZTg7}bBiz%{$lAgVT7oe@x0JOKiyB{f=#Lq`zW=W|cV5L}C zdvtcc-<^=?>>s(=g8pgv@ORE+#w&FlZ6E*kd71^nI1#xMLE2jg+8cuTy7wM50(<5g zOzWQ{kB{mVLHC>vLiKWsYS?Szf{-IdL83Lm1oT}R^+U4Rp7*RL`)qOOk)%-tCG35c z1NEG+q2i4JqhFfAjsSFwFoT}H{5U_qs%4O4a2aecG5pI;#h4N?VhJiYpRX8#H6&h4 z{vrjEVbue|C&@tG%dYV#`-;NTcM2vouQ)S9{|w%9%41$R8n9{&QU20xDfV)|Wz1-K z0aVJV(SIqouf1lv9%ZL$@}Bdzs0CP(L_Jsan+X97SKt49^e|qG?4({B-#07te&4Wk zz26$G@OS}gS_jKVJA;<0A8NKPLsv!-gJ`4kFzXG?N$bXzoP63{nPf&b5-z~+HNL-^ zkv-z{L9pDL$biHTHHzg5-vCR%PUsb3m}w3 z&tfnPaqWeU^?AD}!X|Wn*5~oQka}so{?cMi$*T>9!$j_{Q|v76>cPY zPU^>EN3pA5GMIW^M$Tc>1rUS^Q2=gVsf!WGBx2qicH3pg7YAKOOvl#C_izb5JmJEY zQN_VJh%9)lz6LzVAC(thXVgsugfyl^)ghvII`3|8TLt$aJ{Fc);A0bA!ahLBv>2|D zt`HM%G-JY8CQg#`!3cSVCV?U}9;@G3NoDFX0}S6DD(I(MW&#9XMSXXGrBoLLin$oAG# zE_fX{rIVCl^OYH?{u0KAx#+u@DiexfoA?$>U8ISfH(Rk~gIb{})AVPdoJbs5#5mdZ z$IPdht_JfLnN1)}=}|dzy4%udb=j;7al|xIUL!}U@0u`6C>HpPG=SuI?~9U0N)Uo) zTS#69I?o5kH~_vN4?^)JH1Z^*_=|)N(VTf41Ero;VP?vARu|<+kn56kj5p;+H$i=N zRx5O5PXQb9aA{hMvVhgSTMn`6>0tARWl)S%Kjqb-AAYcO;;uGmqHk7qM$Hx_Aa6+Y zv3gF@oTKyye?n(;>hI%o&isM9JDwuXNU-H{;$7dR7DqXE?s{B8qtL!O9Fu?+A^?hQ+!^vId6;7|)lC z=Bmnx`(@w9>;f@o;FfO(xsPm~OcYPbCnVTUMV0db3Ju{0@Mxx=jLn{drYtRC8Kd ztiMc=u|=%`etv6<$6W5gqjuLM)1H(WkdM}v$|4|#=hHXCdW*Xg*pW^-ZC84xm}fn4 z^(Z$2s zquuhg!Dwdz1EosMZ_idnj0;E3JvKn~R-%noEsgGiGxM9z^9$!qnY?2HZOLWUa|ES| z6%=QC=3QxnYf#h+lyJWz=wF-+fG5miVVap(Sd8%v^}SfUu9FEh7;^iN@hfu7c`THS zW>|fZq%&_iOX51bW6ob%Ks_Vk#c=&-5J9;G%R*_^WEG6rp@F@G$*(OPs zi!!b&CF=X3(@?reV~W18DeRQQrKn46Sci|^!gmH1Ot92)*1tt<7Ssf!1&Lz)BE|(5 zv!qfv^*yIO;TE|lbEdC_3i5yyBS=}wM*5LYUH0!{tjo$UPf6Z&#err7$=+2KiLByT z4+g1}aQmyL{G$y!Lk$wms-G;T>~~5AQ(23SM#Miab=neXwr=C)M54&_qv;2-NI6G) zX-Z0t2$iY#BuX?hp(z8od!kec@R~0(OUgtY$02&1*Q^+x=`igB>&Ae(lNFAb#YaVW zwzUw?A&J^1agVfA{YY1tFl95tzbvz1OHl&^1Hq##yBSZOI@Y5t(`by_5Io)!W)>15%@Zt02b;%DBj8fLHLqo8 zoyYEoSC9LzFBDrl>CXXsJsWa_BO+j-4~p?pHn+UC1`^K9_X1W1*hi_W8mLy8!@(o? zu$YX`a=(htx(yL;jYrkqJha%G6|IKmNd476Tu1hiy7ZAHYm^%2|MOTSUN9k7nl<8wJJUS_Sl! zyHBfYjWd)P@>#>WHO#Moe|Mnd44EI%3`IWN=B&i8{~ z2E*Ob30~6P$3m(VTGV}B-9Gd1Ynrhq?d)O72v6t@#WFmmU_6qY$PhxqQm7%d66gq( zUjt9?+J!V3=rp){<+%7yeOc&eej`+#lV-AfIz54t7y$s5x#(5-_n9C?NUQj6*Xmn? z-GOo(eB;K4iJpL(e>{nUUaA+QFS0$SfB@#a=`<1nn+tl*J4nuxR)D1j=)H`U(0BTu zzRv<2Cc;|aqcxmco**9*JqMC4qi0~8%jt*IFg1Ct6cZU9x=%v&i}q%!9abvAf$aj0 zVPf=TStr0i`a4_&OKz=xFV#5IbPh(+nrVfYvj`ca&+omN!thPh?g)0*46bERt#uaD{ER9Wt&8i=2&e|6oTd?xyx(ng%IQz zwy~Z8`33Rz7ub;a{J$1Z^g7k>U~%F;P8MyKj)MSn6JL`04vWj{ivVWxJaB$OTl!r# zKDKM&w(f`SD)*seF`_V8?9eknwbth!nsBnA$M&xKAM0PDNg4gwTtUNZMgzEmA5YJ| zpk-sGDIBRfu}ZKj<#n@_#UD0I#oedEI$pRUIv+)=n`$2}-RmvEVR0A;?6f^w8Y%aS z1)Bh$u{7^i>Wg-tI+~Z~?@+5oO^gp6uP1*IvQRt8{a>!#jfY2?qFDB8Km=fu45aGw zC&{G5*c@0(6h)jjj_5WpIu{3fX`@cC4*w9ER{{pHS*S!&JBf4qO;9$q7>6?Oe5fC0 zgn#ZBw~IE-o>W;ly2~uJmE9VPwqnF0n%v>`0|ul5CE)?AtKY$)MV+iwFwiC(phi8< zS=Sh(e9iP0CCDpCwz>Qqb(85i!sccBfxdT4^QWd~ z#=87gj*VFKP=tt~$^y}T%&L?;*oBCx#v=#5`YMP6xZw?$RNovhXqqAvm&eXW^w}`d z=CPa;p~u5aOY_#pdbdSNKmR5{!zQ3D^az-FF!{l~**aIm6z}E~9kBqVsN~63D2I`z|f@XrF0;#s14G zetS)Y6>zKDrtv*iRI|}yuVCM8a$ZurZImXex9MT?9BZv@L(*o2VUbw*s0hILXm6fM z?lgcy&^0C)bC-{U9K`;q)L(KMrIuweB*dAy|qW zaPE1|>*n=m_;FlsCHu2tx}HBqQ7!>dAG6a~Web1u+GWYzZ;p+~v(bIU>Al^%Y^!UI z74Gn$-d&zCi{v(=j?4=;dI6w(j)elFdr@TIM#xB$)Rk9eA;@*gnF-OxGE|D8dTo*! z1Ugy6-yM~*-44eCTdLC-T!&Y=Yn?R5SiUQNZp*FiCK$ya;fY8+bc4I=T&AEv_O*JWp_%BK6)P&+S^#I?qFRuCLum=>3K4 zf)v^9|MCu6^D{(ZC{CmL!IHLSGH`=ev{B_o@9Ql6H#5)Gfah_3M_h{&QkjEB^Y(r5 z^IP2=f_M$|YVq)K% zFyp2$5Q4NoJ8t8}PAcSK1B)`{>v)DyT=6Al(kZYi`T%thz)6S+0sR6_E#1XOp3P>i zueVJP_v){=q; zK89EF7h9=H5s7qqR4*s5e{+knwG8W061dNVk5u-f&}0Op;KrQu2UskYil}k=)i4}^ z^Z&&!asLBW{wsb-$Hoqq6Xml4MOVvq z;lsZ7>o`4MJ&C^aPw5Ion53>F7nwQ13|u?zfoLi3fyoK~R}ll|`*^m4dy(Tn<5gAh z$u3VW$X5mf+Bv#N>MOVFT2eT4%OtY*l7WtK*rPd|gBU@GVJqq~fnVkzzrPl?E%JdYQej%;~>Nk1`Onp}0PtUfz5yPCK&fr8@i>x>Sp zI0Q{xOf@hQQM&4ZXpn@=Nnyyhlp)+wcFG}XNFHGMJ&2gjFF|WGdv->FZhFDIEC_86 zHiSs?6$I7|___H}RNL*xx<$ZWIjaBYV>BW<`%de5+(OQ5&Rj=>Ag!vON5LMZw#G|v0PC&1ULY+d9 zIzRL3NwciisNDockEIHP)t~e%sP;TN?oWh<9~%&9V>jvr$Dx-|&L zv?;|qWu@ooo??ha^PE^d5{nS4y8ZXggu9d|{;^x}SD>r<`kze{@#h=EXS`3l-#!FF z{AtyglPXWjU$^_%-vVBp`nlOZa^-+G%OBr6TZa&r$I0dRfkaRHdi@;PmB_VTM2yf- zr>ft~!9lZ@f8k38=v#zW0|haSZ^n{;{9petw+3bcLktDxm4&7N*SkSTwZp_Y)A}`e zPPEQh_Ean)DHBdhnf-$G%-#H#ydtBZ?~Ya*9YS{@$>Oz@`p=V|ICmkzp-UU^0ww)J z>D`=W^|cVA@JCPSViCvf=?+Tv7zV_oy)rW5?Laq40Zo-EKRh21>NA}L1^joaMdPTu z7va8@W#bn8K~@9;K@f#{F@y6zpnS7Ax_cn%p}ro^-|N2$|} zc*EjnF9q%dVlog&1)nv?_{;^Q@A{MC$Cln$V@%z5X+igrN>xtcZx}dl zXTKBsp|1~{t+aZ@6S3%%>QW+ZRbfwMgEs~Qbx%=MPuPS-;$pZ#7Z2={}2+0!*HP+|AU=oX6=5m_$Q4+N&qluPee%!PdvSUOF~# zEk9V(gO&2m?DlMSu|DBc=2-DG+f$)89g*f2uL}!U-*8ix({1mk9-PM=C(H$EsZ`X( zepe=TlBfdonqb|%M1VWd%Lk;BdkACZq1`N<_ScA4!{o6^kG!nKKg#(m=_6r~g;J(8gbmqP96DLLfUSmi-H`$hQ>D|5AxU5<_Myh>I@ zYl>!ACkC(7fVkX_`|Uq{V6F{@;%@v^tg&u~ns#cGl)=5Y-t}uj>*xST+gjrd!}Q(3 zbBg=vT1sYRn_GlX{uHqQKa-FAb6h5u)l^+F6X0|*4#-~%0 zFJ|dTa;GA}2%*Y_F=HvvnX~L^vYbkhh#HpqGqs0fykCHQ_hh0`FiwR?8l@Ja7yMtf zNgwL*?P1W=h=Cb~NjO2GlUC}$&^cy9ov}?aR;tHN;yyYyyhQM~j##3}ay68;-nC1W z``ejII;wN)ar6{zRPOqDre|!~7!YA@APYnYsGFS>#8v%IN->_AdnH_RPH{J>B?ae- zLO^@-=QXZDe4yuduzp9gb%!Fq9?0)9!(i#d50YtZY@UKk)nVq`6h;K8cGLl~O{L>z zSu8t`FL`XeRDa+ju0t%O)ZQtkIgEl)L#x-SW->x%qgMF^(S@HBM%jQ_GX_u=Nd(2F z&+rjMy?s2xqda!YDFce3swLo^wiwJ2Q}CI2yPaHbyPtg%_*!2*3PbVd4H04>b4ROk z5wu4pIk-~##66)6Ds8&ynwAsnpvQ-|ZI{e2E|bwY%~vaw28zdF9M3fcZkc=-Z=Q=b z@jTI7GH^B>%IXEY$bZT?(vsJR-rE%oHC7?_G7H722k#&Apmx9%1R0 z8nVUnr8#@EU7p#`%^|%`Jllj?^AvJ6lGr1@Am&r`mduGPs~~q>-}&3c)FF#es0B!4 z5=zVsvZI}?T6NVAq2@YrnWlp>!r7@v3@Udv}}{6P8WIb*t;`W@GdpAb11Ig^}G z(<*kbmd;mNZ8xXK7dwkHJGqwBc^aA$0Qy#T*N33jq$78z&XmB5cr4D7tD4>zVG?mF zCA5;}ASj?)5k;E0E8&|CBm9-L#gT}AZ~Q15%&RaJD@0ujx05A7hkIrR@~jyvkJ%8hdJ z_0Uj<)42T4e)+gsh9i@bA*^4>8y^63H)72TG&HHwK$vmJF;~BXec}qVoOTZ;TPE}Y zoVT@pNtE)?ur?I$e-;j@^E=jf=hvtR$X`u2{ElR9R}*K6;z+ZQlBQXbF(07iv#{`; z)t-+M_#q?jHPKt&m@#wSuxGe+)nJ?b3tuZBDUE8?R)rbpHBh|}eG9b+&>VFPY@zc& zAYZ(WM?FVKGv4iEteqWB7TG@T(3C#_Gh1Y-9xz{2e{%Uem>zIz>{&$e9A}qOiFsPc zw#w%oyS%P^`Y-;3EceWBcyLO6-;F?e+K~^baNSP3^F4>i7-nK%?&*v$rIe*X5i{N%9}d?s+7+&D*#mo-TQJJ*#F$}{4*iLP!$Ma zOu8Fn8c^b}Um$Ypm=ujoehiq@co;j1A+=v}zP5GhLb9EgWheDGfH~cV2jQ^+^;sJN z*WFEsqB;u3)yV~0$=0L8J6x`e^rB?PeCs#bkOy2%@P*YVELR|K>Zz8&GmOHn$TvZe=kJ@kIYo20g z@vU~5e1H4^#_Zj`|Hwvl2sb#L4-InV&d4P&Ozt)6kEA7cOLx38RAcSZiNDWWbJTw~ z)i&b%pkn{{xSaZ{Xe$q#;l)2fv)<^bWwX4nxgQiOrWhc&6_dF8%*jd2wMm40LA7M! z%(yW1M(vD>R3Zw@BOh~!uXXY+3+P)5aHYj?7c)`VW)fRE(?=ok)k$30J|8y<>*JW4-`EEEo+ z@|$dS4RJ!8rralUu9m&M+kd{l*afmw(A3&>Cly{HguF^33-(EOiAhbfci zdX)x$K?4PmR4*V{OVL5+Eo^9__{$>OX`6$;=UuOcf$z)T{%ccj>)Vd7(Vd`wCpY{v zuqdzeWUo@`PL*1eDcs&6-fhC*vuCmrspA^&yiRq6dEVm|?_B*-4eL^WWiEhDxj6xS z*x(JuvFP}Gu0R18#jRkyw5cnzBQv`+}b-OM^PWvBsUgpJLbVx6@J{d({^a3fEh*Fw_K?cY}R5xF*zuj~3I2 zeD4LQ>>Qk_-X`!Q|2xSvf$xFi<>yH?GKK#F)^b^C#P?sU>NfEeWDO6|G#ALc!XA{i zrEkcr$OKODu-t%jXud?eoLulz&;$iA=tS&KaO1;3y2CEY`4qkEr9sWx5}F9v353*} zDp&&z!HD8N>oXiPv5N3e=ch2W5nzUR@yULm3D?E^ zJGO-p27~O*^f3X>GG_x?Xs6OKR_z;Oe*-PTP9cXc2HlkouvUl$Q_@8A<8u z;(oh5KD_KeUwp7*#9{2~0R}dMAuvICBGk2?Te?JE#a@?_bB5bU3J_f>3%`QTr#qJ2 z_)EnG4Nt{5_*u(dGaa@AAJ=RjhTz{u>d&5B83F^rUA;zP4wllR z({dbb`AxU@uVt3m>(gy7!+QLAc>mb`eMNNrCgmzi@2V_2I-S#AkzUNWos;OS9pi9l z%AP%4b$imDF^@O)<|bS@YTlPLacCQ0Gv_MbHGOf5*irm;&h1L7RxOEsoE?9&o(&e3 zMx{RplL~VXMJ<-}d-y8_){8Z-rWn_ra|zRj!6Uo`!3TlxYHuJEq4tSGY2n!>%NmmDhAdd$Zt^^%MW(%m_>9F| zNho_hbzV8r>NwEE-8NhfD{hM0)eK*ELhz^t*2XvFGYnpTj@+7;6GPmyMR4cK2x1iA zUsU60{@ra#aIXOXyQ|+BC4@B0d+N$9hYrtK0eH`uBQ|vs1{}LOCanML;#WF^(yia-#7!xLN_($tL3%I)Oe4bLxV1 zpg*I*WW44>eg=E+=Q`OkV*b`!xSceBK#nSpSlK_508ab083E^Z8VkB}cJ?8E*Y2*O z;E^v&q3!E_2~jd8@8IPFz45{Ld^ef__b=Y%Mg z=DBT$!^-%UY~9I>-ZyGTljBqX`ON-A=2`#<5E;BJD^bFGy_-F;4t{)es!_Td0#P=s zyV1k&v2TKI4AvP0`eXf`NgH{a;JP(03j9Y^7cUz?oj4>@fvuvHp)oP)DtV>dd5464n@jbvDH(GT$mkwuyo6uQb?Se{v>!gb%=UIo1b)ne*(j74IKY6G z7%1maB#*Ssmn?g~S*@%O6=!9%FZtZ{9K;bk^p;&2-XA*6CFpq&nN>#Oo{~C`WXn*S zH7Sz>aN;HoyZ+60;^}u4cwKK!C^J4z$ahWy(>v`-?mnqQd;()hti2#*<$m;--u#!SA-gt z%yQgL?rU&q5)$t)cEWHy*U=U5^&<1xq1faTI{Dvk9bV_Lln-+HhX?X_z<*`Fy|Nnu zmkKK5a*bV?8*KjdZ88-xo|~ShGop6rdN{g%yh!9h`tj&Hc5jyz1clwuUa^atvMVQK zK-|0YbQZBo#TWzmKkxLvmen0}W$K@1*OpTc!fbsnr$TX@O+h^1|7}`~bNmIR!B#QC zQD*YU%ZA=P#Kup!m25*bUzqvhpg=V$xVrK4Q`m1nF)TE#YFiBV#taBd=N`nP3Kr?d zH0p)xD{Y4@m@2@RE4k#D51SmY=Wt|onWlSl#9At3JO8j9OC@K%V|=+Cygm6vB4SSq z@%pL26-i3q8v+?DAqNn+gf$9vA+5=CtEX0fu_PF1ai_eo*}VVPb9j^37gtdN&Sx1h zileGx=x#W%X0o=IhOqyB7DL4HRTFa1Dx&skEQ))avelr-&ogeLIFgC(Gd)u#rD{(8QY|^F+>~^#o0i2yEJcs_QKK zG;ub51UB+E6HZtxJsSLaS{F%yCge5dxl*3eQZJq?QcRCFWI$Z~+G(v4@$A_EL@Wz` zwKh8UueYmg=CuCehv`jK=oqnJK|UeBmWyqDCQEEoj$_{Wk9!j*K=Gc~=e)j=rTwU-ylueL|Hft;+E+jio z2$3j?s=mEs!5VEKA&~(FnwtaCxGkLa1BXy3or$HohZv5I(HYWvZ#yhd^3>u&%M_U= zs*-uSxbNwA&Vf|NA%V;YRirQxn9KOlR>JJ>6^*pJ%Yh>E%$Ebp zJeNY+lM&+*6&gL=XTx)6Y1!gwXw~Sy{t%9*8*ov2Jj?k1(Y!}Mz_Kh$V zFZ=AEl#j4h+V)hqCF=?mH6Giv+O(0oOFkZYSHyE5($dpu(o zQH3lOuEruo;#?iND(yF!j~*$;Ukb1gtaN#;_#cP`I!u!EW~Bna1)pZd;NXHQGFPmGjTB;nd%d8UwLz z3y9R+>X*RAs<1YafyejEd@P(#_+)uurR{N!$*ogb4wM+@f{~Iil7t7t8m#57eCAi`9oB>Lv`-C7+gHVog3 z{S9LShHYlx2HNLUOK8iX;Y81#bkdAtIg-c^qwG?T+|%e-k5C1Po}(2F5J~<3gPj~9 z2$2Tk<+~*Oc=?c^AEbF!kLW=SQkxx4s4j}=aoU`vp@t}=_h)t&%Td75l?hN%$1ZFF z;vfpdH^c86q9E~cV^2$d>1-Dj4s(ch#*zJ$cd<|_alI&cYX1lLXMVbE{aamTb!W(W zRR0L=R8Y-NG>Crmjr?YK5x^VG(G?7YfWya}Ki|RHW4SQIFV_>K;Q`OxIL6$2c?8d< z*QyJnd}@9cqWExx&Gd(8HWWso4QmpRC!A48m!>;N%o24V@Nt$Icf#a`<55YOt1$R&F?Y5vqWoX zB-J$5d2w1e4xpBw_zob-I=c1fujX>?pF?QU>^pVrb%xza;+Og)nP_;N29YBgknrD~ zeB=nZzYMYzTD-^V5(Wn{^`oxF;x{7TQO9Y`S8=Z2 zaPfWDSkwpY+#9ffV{hw!(7vsK+~o-HU@x#qQix+ z_0z~I6t8!)QzU)ScT%X!TzF!shC1It4_pNOpsqX<9U|4s6_+|7_yxP{hLD6 zv!BJ$_W016i->;h-`XTTHJWR0gZaEx*)?kh=SUda&49alK6b*KTp+Gn_K!Feiw~jL zXzc79gGCsf-jGDzIqm)xHzNqC32{f;7hwWQb5hTiYB+K>jK-1{`k4=dV3TUO=pBM^ zlUR--)+SaH4=I1?_?!u52;pQ%`S)C`pWm9(S`zBq)Py1+mZ=0|ivK@KA{>RRAUASZ zuda*CHAoOPh-&n#BkJ`N7)X(WV zM6^b>j->Xrc=Z?K=78olo)acECN3DGGSD;75Ac_fOFmf+4%1t*&;>XLIvZmjeklsj z5BzU!=dD!9`(4SL1DF8gts>hZXIeM6*DN5I8O32()WSEe+@j61`9f#9NYdfs#Z z9(ht1F(BEq=pd~~d2$TRQhPUzmyLTba`_yXGKn^eKe!^0F z`{)o)E;%dfsr)yxb}rfG6m(JE+W<0*Fj0k2;lnT(KGbJ; s#M}>3Vm|_n6f9OEbv#~@|G^_(;%B9&%6q_*BXhBHAk)%HDu9sx3()v`+yDRo delta 40554 zcmZsiLv$rv8)jqMso1t{R%}*m+qtoA+qP}n$&GEhQvI#JSI>HS=I>zd{hW?F%(q+2 z22n78nW!^0!EeSU6u}A2K~|GvLH-yCPjXmvG2WPESUjBUvAC6QyZ2rY&)k z&_C##d3j3eGEEGKl%4X=clQ`heJBqmQ7$k+TI(d;PnS>^s;}8w)BKC+Y#5+%L>Pff zw9zRXF{*+qT3=mN*>ORE!wrKy$o`(&oSNzY<>i5~lQqRPuls~*fp;au)^Wer!oZ7G zCgY4tjS8x2QUvdcJ+CAmX?%?j=-P`Yi3B1ob=&e zi1{u8`bT(Ll5$B|NtxG>F?#0s(fkcA_^VW-fH=IQTTq}ek_Lq39()?8sReT!| z)-6>+f9*A+Yy4Ov(~y&G*-Q|GNJ8)9JdB;3eiUo+E9Ab2l}@$)A{)1xtiH*X$RLP9 zPYyfxsc|eZf9fJDA0-5i3cJde7f%a-UKeT1>qEH?G44r*vF%J#5<^PH4+hN2tSq*ep6~j!10*#e*dM zCQ}b;!$_1s>o8|B6|!S#o~<+JbAV}{vvX>IE|=v4*9E+kMSj$zr|#WV_v8f7mTgL{ z@(NdW6XkwUxLaXOf`||h3}%cTq0&AH96H~~bp}qNG|QDPr=DH9z$JRI>JMZ6&MkW= zO;&$u)C6vgl>##lBsDS6kn@82Y}Y|UBMLZ9dkI(L;8ZstTCYN&^_LV+x*WVhV-^PV?}0mv!OUIRRVBXM$xVM;+*Q&}&TCKcif`=*z>(nKTv9>3}q* z?}8UHq)eK4$G~@LW(-6m+`+?(!qpNZR@m1)&bn(5peM|h8-;LOzB2#^dOq7&m%B$Y z*s;KLWET(d4hiwDt6{Ycepz2Ny!{*NJTgKsm4o$kuGX*9ZThl^;evchjCj2-yJ+Jz zPAG*)OYx!5o^!ZUWAFCUaRE%q3;PflntC>TN=)xjo6`!)_oK9v8n7%Vv+$3joAWtk zR(^93)Bjn_DfqQ<`wb9H2WI_b_{3~U(G*@R?{Ww@lCuTJ!?D(PuLd@-_`%Gh`Y6e7 z(D6NnLYC|ln&^LtRM=)>FWh?*i363Z{n5utsBR+^5~)+|7UC5!D`fShFst4DOPy9Zei$$oz(3^`c6;$= ziPiRZ!yBkq;^~pP5f+}w8G(FybUFK*lT&FdH>7*rKEfk{Q!hU;Mf^T2$JZbMP)fZ_ zTb>5kxA(MQ8Gup^`hHqEa%&bU7T+w&b6Mrz7rc|OPJBlyv@6yLk1+OyV_L}U^Kf^t z)esqaQe;K-6@yJ;E7UBU+yxEMcvA09lGLz&mjd^fW0b}?%w0s~-Bjj6@J*7(Vdd0OXJm~i1mZWQOJ_v9m4AMD$|ufZ~z#v%&w!<`Q=}!oJzk zCRea%xvBNi9<&G10+LY|4b0ycv%NX_Sens+Am%3S7Cn}6;cLa%vlBJ0*w69P(f|Hj zJHehs3}SFLODho6UPk8wP8NR=!BQ`rocEH|l0`$bV?3)r9~|x-l(`SFUQ@6d@+2c# zGnS?_WCMJ1TnL=m&_6HQRur9be}H=B`G>rJDPSuUDE3Xn*o>7#q>hE zpM8XfjO!d7DNp{D4mx$DwT}+YmWHRD%!cLtB?AaHi>136xMyE*s6SLXJgEn`uG^{l zh;}XeqdM+lU{)!kbHu`vWj0-;6F#B~sHQ0(>0g>#fH;h)(e^)Q}yt))Der|+_E zvdbhb`f2rbktNrU)lZ@}6l}roAB7y*$WGHyzrfI&iDAz@!C5)D zlZ`N_0joN{9gaEC`o1)ul_kgE_EI<}z&81zw8;nT#|8*lc_x5aZAlu_bd}?oe_wP> zF)5VI+XXfu2LIS{j7l|X#oy6I*}dYmdHIc=|J$>#7yK)Ek9g))*}AdQx9v#dViOg+ z$ce4uVDFJl)XWub4gfHCaS{sZ?MYWR>p!K*0JPs;7P(AJUhWq>R$q0Zu75tuxBCH~ zi0#>|OtH-VmpLrpS51g-(-Y1nris{ayKN0q*yHv+^*$N!lrS!lV5g4^v->~Ow%7M> zg@g~FPzS@zYsd#cC!r*SBCRRmtKWkzCPy!OtaVl}(rM`$SoRTfos67Z{FKtO*Y31pv|NltE=k}UEPx+Qch5XhA=KrQFF5+oq}kf z{tR_I{WpZ3DVn*Q)_Hn#j2Sexq>9Nfcqjmr5D3Uy?bo;&F7!WPH|$-^ZKoxf%4sIe zYbXPCW-%lQNuXTU8#NL1kfLXEw_=!eskY<4ZMIljqC5+Y8^8Y@jl<0Flo*WzK&)%h z5!cfhY0}eqa*EbU#kMWw@N!CI<&yA5ox}!Ls|(Q`_4(8tdw)UXa`$F^e`W`v52Y*A zuD`a{3{G9MRNXzvO|(<&g)iKvBjcpk>v3iL!+R#|vU9L8ja6X^8)(_;iereSEPvWn zQXwfKXl7f0plXkmD=21HVovW2pmb$7nQGG0Xq?9V=lxNTiiskoi8ZnGnNN@%6w26% zJ%!x$lvT@uAHyjA&b({{Uc&Zb(>qM0O3pY`;XhIDkg~MNwZ5NTPM8EA64rO)jg7mL zF?~j=eK4#+zL>H5*@8If#)uma?dQ)@ZFLWaid=y%iBu^vJu#tbarZ0_@Y^~Q4^^uA zjaUCYNukVvphQc!!0%FrCK|0({|J*&*w>;`>0G8s*E(NQf+H^5xI|8hg30Q>q;L;u zdad@jgc6cL0h?vwq8Xb3Vl%`!H;pa`Zm*@lQYnKUIw|3KqY}=rR4K-E#)JKRJso7g z&xQk!PZ=E~o~<>Y?W)WO&@hYyc9c)@sy1@6%HL(scK6oDFFs6M11~xV@CLLMA)!EB ziittsxHF4j4*8)$+#o>63f1RC+M^X5G8G7dX5jg(XfZ*VS;xa$kfvP1c(J!{ej9iH z%@^zLNud`8$FF6-~IN#eXHICORYsIAYn1ns5L4`;NYS zy3^DijGKF^+y36i}w-^pion^f(Enn#kC6Xz^M$x7{@z zYD^$S4xJ+tXd3SeA|@^KTru&Z`bN;E2HuunfN?gYAtGKZhmN>sVi4alS9tl>`Xs>6 zD;z`q{{6q_ufr)uEiS^Ms4u#cF? z+lu9@<$Z65Dx!U8ivJXz4N=uT$z1s+Ss?k*CWEp8xMS0e5+HnwkK_aJDqUH`di;yOZGS&FIAgW-Pjtv$tcA7C@GuyD z+Mk$UW%b+@;J;JvU4&NURW}Y*jYhf6<)S=bjKVriiq%7A_36z`FjiU9v@3c*3RY=k z8Mm#pEQ~vV0<+Ni++FA`gUoxSW83}Y9y{OdZTTv-5gX289K_;_D(m$cQ1<};Q#6{v zRbt}SF#>(Cfw11VbEfF`>l6%XLzta@#p1nW)@auk0C!7!uW%k$j0_a-ybO$DH53@r zMn2$>6M_f9wlxiN=DFdh(q;{C!rFi^y8=7|`|5RSWcseWeQ|t^A(&htr@yRmL`r`k zpMe~Oagoi@WK)R$r8;%{%*UR(M1soY#1IV0e-3HFme-+CFr*4l z-mNAY!tfl2wzbyVY$;_!Oii_hG50G7U@ovGX{A@A%Tb2C=eJZvyLt8lyUWa>!4o-W znT@O6rRSv}L7V2_Vq-&mXGN3q!5{Ds7FH2AjBdn$mLJW>`50n6l*+(I|EA2DMzZt9dx&Hn!^=U!V$&Atuq@Aq+41M0Xfo(PQj&OzhYWY zXV@QM{yk^&8>?e#Zw^~jMpPfP{O)amAw;qwh|^avM%YEm)F7>17|RoGUj0f%RXFT8C(i_0{UVnc$kbcPl;U^8#_Y46bNIzL#sb$ z(wI|5_m@r$H5LuiswU;NP_H*LT4O|WBhzFBW78H#2EaF$eyy5$rG``#Xo5XioJZ2T zkWH6_C6o@x@F2LwB}agD7Qqwd9MLV@mVrZxBI|% zeD;jft)|T+*`U8Q&C%bC>0)SNrr_nzY1LB19i@LG(S#x+)c1exhKAwme%H#!e@_>e z2j1>n0rJ1K5$n&%zz^{vY{c6HRzNep=<==Q22&$BM2xAYwHD%|A{7N`G6pN5j{CBB zVPsYcgb3mh6AlXmo7(`e48*=raEhDKpH`;4OehsnP zO8b@8bA%Glc!)iImJ>za#ff0qlor(s5R)#70HU{>Ly5#xdH}#43eq1DlZx(zb!>bT z0d86W(&=My`W-Ptg4@h3op2MSzE|MLqdJgCR(m33F2PYTap5xhQv z&lHNU&7Aw;7p9Ejpm=PmtU`aPaBM7_CHPN!DCFn{3%k}VGyb6!%k6SAg<~aj4U`ra z)>jZKnogPv<%a<(CyoOkIqvv{eE1o``yf9%Y~nDnkZoU~BDmAWhYHL$@}Leop}<@K zmY2Jgn;F;9?7istAR-mx!pAlhKTQ(?w7Lx!7mO}dy8EENdD1NvGSEB{*1~99DOYCb zxf^0|fxekIg<9mO*Yl&ROow6B@FaW&Fn>{XeP{#$-obzeG$_k|5Kxu|Lm<=G%)&3- zECZyoAo+R+m#c_&7`XNjo;hoct{S9QZLA(+sAH>E+> z&#??|2C;yW=w@fk_s-)^kmc9Ff8E{)>MF8If|J}GyUK2y&z8!Z=gTN3b`cu`(;gw^ z)3AzSMN^cl~Acwzqs1P8u56h1O&*xG1ZMk ze4if>!VlDe$z(2})?F^#2!%`&nDkf6Mz5dJ$^P7khON9; z?@<2(&!RJrn=BHCj>E=G%uMWHYz@cH562{DW^dtYNzBH^#r!|ynbrNBeAI^8H($Tc zpmqa8xm%#BTUlc3P+NVic3Am1_!>y(L|#VjG&4S9@9)9*zS?QDzvGg9$r@=#rwrA_j4rPLOMgMT2qKQ_ufWb@ zD@LghL8-+t!7QbXd+PSrVeq4m zEbb)8a-CUoYW{<+j0!mBp^^)`#MWlH^*?6T= zC8;jXhAzIlv)Gv|ihk;ZSqH7)Q8-1ig2N#|RPDN|^At^dB3jn5>AErj(=ajbITBWT zke)Qb35J>-TihyD8gWI=A4$nsf*PAmI5R5zA%>a^t$A(e> z-7}9dS0Z?kv#y7>cGd?NZH^sj*p7kEI&uS9l z*^S_zHkQ09KIf{D%M<~S7vLHbPc>IVlb=iD>_g=eE-<79Vi?G{1Ngj zOmN^^H?6wDpu>^^tpwp`3@q+~h&u(qIbzHQJ$iK@W=P6|!J-Wu@kF17;grouZs%dt ztZ>Sc#(Lok1a8d_9B$5;;>L3i{2oBP&LQSKQ{AyF`V3R61A*{DB+OVJ2an*>R*OSP zXDz1WZI2J=njm4AGY|upRME5o=`8{MPG#S(2`q++sD#Q9DP@Ww6)Q-~%&Y=LLfas? z(M%9MYDcf>purw2CYfliKZScETM3n0t>1xKwT={tH!^iMMQRGq)R2N&1=QYdB#p$o zY1V>tlb}Le|52KtP80rxmjUhaeUXu7__d%@+nAG5E)3R;_Pf|!pD^pOvC3z!Q82~- zxYx#W$mk?z>Th`<>OWPjKaCiGOP_4%@wL(lRW7}Bs>1Q8*_%vW42~^YW8s5~Aa)WA zb7xMvu1UEMR^qwaP3DO);vGL&hX&nUAyG@h>VZP0OecuX`8l@H02ueE`#q!`Ph8Tmf7;HFXQ`oHZ1LG zi%w(2hA<_8&xZde4LWN)gcW@%LT&YtoDD!K&_7BOOm|p(vo~fp?25jzc+y806EUrwcJw^jE*22BQ#444Oqu(9A`r z7m_!4qV`q&P0is)T>LFd%(t)|e0RIT}aRlx3kgT!!cSKUbV z-M1t6`YqA`esQ^ObTV~A^{=gQq{g?cxi}hh42U6!U5ny$)aWUKIb^4@o9Pm2{7e^- z@Ndr%gx4p0wpoIr3hWYRkc8kS3))rZ60N@?OIL(`=vPEYFV|H8=T68toku08vdf{Y}X-pKk#HQh1>+7z=Is}r+xXKZ$2|X&Yp9jZ~@o+QX@1~v%r;xt$=|RZj zrR0YIo}YX3ovs#kvj+H2$Bv{xnhsp?|2kaEwP;(Tm&xBL)MP3u@V$Z3p~-JET$$UQ zxz_HlwYO3JD({}203KVp3wmMqN zobP(U)Gm+ou8KTpG@@TMqNPglY8*CQyPb~D&0j9p>Il;RaOw~@e8bxe^9;E zVU3cVCEiSfNc|PWI{@k7GlaJWDn{J+$VU>cg>bT|UfOSU(4m5r!hjPiLhyZVv$#|R z;26WVV;-EM;fon7WSn>ezjj9eVUe;7^Wf^q2&fbTg6GvjFTFyAmTtBGAw*;IwYBm9 z5xXsm&xcC3XyDMh0K>H5K2|@P;)fZqc1IEKN}6gg4yUs3v_7OS>V{WDz}~qH(m+TT z^bdjQbbb9kZ+owQ@6$=`Ml<2vy8(^>a67Qri;3*^JS_2wuS&i97EwArl+OH6!iJQ{ z3zj_uKM6GUgEcbzZ7>O_fLQ+Y%>9?B=e4exJY!46!;g8h+?L#P(7|72nLBMw-&36x zO*xJv7H(0muW}hOBwJi4{7_*QyFw|L!bhG4 zsbn4!q&k>ZV$11^%iSGw`P_eg8P#R{>DH}_2#PUfUF3l5+PL622BXp{TSPCvisNSb zqIa2H?U`V)03jJYbX2}}IrdVKI;S95tS=?I%!z{`4Vu`G01re-#{k8_f@3N` zID{|t`ZPw1U61cLMy&=Q=#R|M3dNV6Q;S zq2-{ZeLTMI*@YDQLKTD|fc$w44c!8)#(pubFP$aep`N?qEcionn@s;_PuoHD;YtBK) zaZS(H)pD%;JHXf@#I@6{Fcdvmz$N)u54#I9@-Ek7Y&fo8s zw2!FjDmf&#uH}9bW9tz+Hlxmu12YoaARQAHQlnl|RGz&!Rrk2}DHl)^X-wPA3L*W9OYw?TMly)P_IH3p3?l;ePmh@?b@S zV;b#6tcgA{x_}F0BWQJPA>W|LPDQxS_4JAB$}Pg|C~9Oosn~;#`nUKUB-+`hlFBH1 zQjbxkmlEyNd5-C(0gV9jw5IPo+DVshO7z_e{4<;bk%x8_9EXXu`VBb?Sib^tIA^DNLHi!$~4N%KhuqLMbYTDli$!F$NysdLdZQ&lEGy|rh^03ruRKV`5zNwL0mj>cXBvv<6F|4f=yz=+A4x(4s^I#= z>ZliKiO$*HIlrMfe0$mk?6}=A*>=IvPgyXCj2{E)%gzZqVaNEN>A&-26dS^^);*E7 zol!csu!gn2OckW9Ohy{0h*gtEkD>KEAkCVs!@=P)sQ_!;+FZxJkve0q@#=f)gD+0}m!NoGZQ$}N`4*~G%6~j)8x}=NYqqrE zpy-s3i`;INV-#)As3?Pk0?3npLl7w@1BilWT&H|T{4@x*xx3iSz>X!x<;QcBb;k>Z zm!Ytc#j1r&K;AeS`KrAcBw`L6mnwD!o(rqpF`ZRAW9=ga_}D2*xL3^4V~|8X(50{u z5UtoDqLDA5mRKFZ`gwT;nh?Tylz1}H8ao_3dELOxbn(NoUKN5ey>@xA!I$7e0cY5^ z@v(-%#V&HI`I(%KVW9PeT|twICbn~`=xz$3;J)~x4EK=)Mx4n{&e8@hCW-%DQMrQU zjm*B7W_n=csc%twTzn*>8uUWr;k&fC#n~_nJA~61U6vkqCFhuKp^L^MlRAC+LLS^r zr8@WLy}^0mj*p`Ec?Fp?^j@?f09inbq;>og6#OE_u+yn}A+CAcu3 zY**n%;r)7^{xWd*x7iB)Q{nu2I(ra zkiEQ3IpOSYPgXd)NQ@dG4iKgGw9mLZ_5_T!n&L8iUw$F#H^6}T}L-m z$@^(VpI-<{RdM0{%EpNgP)Dc5h_zUXN=*0{t5=eG@^WNe^J~l&6*v}yw_ilTT~<65 z=pe7E46zB9Z0E|+p?dW%TKseBn@zX&W4bDOq}~+z$X25 zSag5|q!?7BJ`X@EN>_m`PZ`f@=J*p-ta?(dkg5&JB@XE&+Qdt!*-a5tgTFK(pnT4|AUbT=_Tuya_OkAao(F3Xi(yl)8`hp6SXe-p~Jh8Z) zs#d3}TnG$*_p1#~4zVUP$@eNP51Gwbkb$pr{gvf(&WNp<#=6vdKdY5X%q4M{{PTXn zs#fb!U6{py$bNfY&2b`jxMX;qKG=gIHxzl$q6?eL71_c}beb_5giX)g{HVQe<2HH% zn+(gW%$ULnFv$6=>+A0RFZ9Apog_%QK%C`B*KVhDSMU8ubLpOs&}E15ZM)!fZ4VsS zA&yBlf4p28KrAsnGe5AB&TqUR)7f})`xc(Bk43SwrLU<$DUh`5xm?U9rn{56+{$mB z4~YcuCcCKmzl2G<9(cAGBI-HPtNeTAokAF|NJFXvAn4f(390M5fr_RoDRbASC70Nr{5aP)koAU_rbX9spFi;no{VM={RYcXc00i5PgY2-ZtN@{lgOPyUsY z&4q;o(CBI1uaCAkI11#Jq^E)J%`DsU+|}3DnC(xvP}zK7vN4?wX!$hF%~A}~d-CGr zPW(c)Yo`)PBHi59T{URP8~enq6h;|73>iR?Ce3g4OYoRbrY zF(x%XNtUOtk`3aiNY}xyVXu-MNT%o_v*je9zIy&%Ofk?vKBcRWPET6U4Jk@T!7Mbf zbzePdr0M!}K{dd`I4$z0HbxSK>)odD)%}>Tf?zrl-(l@o7tr`>BDH0a-zCZ}6l6~T z6oDPV80Y?(P-hHnwUroRM=7fjS`!$C7Tk>$%TA*WZ-ZYD<6akb=Vs)orwwNOoK9Ib2a z&Yx3*r?A7H>9ch~@Me}H1~tW>9h7JIclYn=b+`Kj{tbE=`suRT`v*z{NZuQ69kBf! z%#SyBL(kKT_{NpGa~jhgtJVDmAc7=SdV_t!KpaA^$}Rvk%@zazjMi1Ss;5bi?PEdK z5a$UvdtVI%Il*N!%-5yafiud50#MI|}mVm;@OlJ4P*|>UJ{Ue(5b`Wt!cm}a)LbNQENI0Slv`(NNDeo0v ztD`~7_N8S{j)g7;LCvIc5OaNozOd%%j(}nKapNI6=IW*U#9V2>e@_M6^@^Bp1}1?n z$|xG{REPaw1+>|sn5BSpm$58+KX4@LzjBP_-v*9Osnw21eL%zA2n4tsCYB6I;*G3! zJh^hHsZm>?;Z=v{D;7CztmQGC&LUcm$}|%z-Je_oTeLWo;)Wl5fHDWOKhRh7IrgjE z1r|Ib+&x3fhZOV6B{gG3DRLBhGh;0(Z1?%rh8E`G6d2pz3Ja+B`xe(qIe_6&;87oq z;qyWBrJf>5g0ml$dLAe}l3Yb@egm=$RzHC7wZm7SHF^5|dLM@pTxr{#N8zkdw(XuF~a`>0#^Yb`l0up$#2cl$%Ys-fKhFIMSLy{zkGv( zag8ee*WPsl%&3(3BuQ%u43$cTlfZ#CiAeWivxy1qHs~* z*ILy$oV7+2%+{lHfIz?O7_=cR`Y3@s5?a*I`9{zY-uOFIg=7EoI|}MRF+uH<1t#2b z{Cn6VZfGW>ok3M}2_9~18rt-Ggca@%#WlM8a?o&s9uMZ*MJ7Z04C>L5DR^^83;yep zh)5&2uzSJ7NB}>jWv$)*FI|LR%p7CW15uD;;AMv~#1NiC0Doa&0#Yo)@nOcY2|{=9 zYlsbrQtq_@VbOp=v|ka-IjJeEeF!ZMlLn3&q^ z8O2iKO}xTzW?nGANcuqq=IS7V!{%kd5_gZe8qir_;aNV(SG2-%F&-juMNW1IYBCb` z0+6QILem2^aZoZ>jS5SM_ZLLV2bKN7kqB0doBMh>0JFP!*2wB#-VgSLTbh>>CNDke zH$Pq;HF*NVXJvkGAIoyB&{Y3svR+S&cV|z5p;5T&Ee;mg(bFdv*I>V`Uk=wdH$Nt$ zM^kxOy$DF-Nd8-2-2Q$J*3oQeJz+Zf`!k+@AL)AUrK3Ow_cR?(6hEHOBkE1m5(-6I zkdSWP08j=kO=17)1BYY#*P%t|&&!ib(|7CQxa8J+qPO_cFQo5QkKCPXUv_;dv3yIW zMGNgIbVkx0M}MN%Xl2pHX*Z#@W|#gnR+fvt(S| z&mpn?%2N7j*vT!B<8}%2+^0o9 zZ$6zW;-%cI68yW-h7Fwjc8IC6ut>l88a-=%eeSP*iyjSs z4exgZAvHPn@#LM|Og&Ye^D$|eEN4I?K*d?<=AGht#JyexpIKHG9JRH1{83w zTX8ADA{ybQDkbq^!_1d&s7;?}r>yi={kJ!pf~%U?Hb}b}DgEW1W?i9MmDv91Ua>CU z5pECFh~$+`ck`>pt!kKeY%{VHj*iCK0QzL&m;H*-orYB zjp$VDSz)Qk^R{&ytNiSudUYpqFF=1*Oe<3Q8vGGwsA-;0vtlSI0qc#rBtEs=gu;qx zHuc1cBJ>J{* zKb6PM@&8eI|9vL-pUOMY`G0eX1!dJggPy_CQ_xh6$sO0u#)|4lf6AlioXj&yr9(^H z-Tz)aae&B{ge95(l}6&3L4m*%_9*_kvKRJ!XxNVy+Pi=L9tO^q8^E>S&*)lVd%VHV z`GzU_l3rC5rj{QUsI{~(D>ejum22oX=qemAUTtUFP zmZjTtNnLo4oWa{m znPqn$EI@P-(7q$?d`T8nlJu^kHi`vSP;4NOs()!$q!7i)yd^&tt0sfy`&rsvT#J^( zmR&h2>q!}-yJJT@Ax56!Z{Bbm4jJ|GJE%N* zIelN3(-r!2hkvz+ex+}!ZP9#$6dWt=>~h70UK9am|?09obh!eIrvw;MqyyAZ=sd` z&tr@NV0VA8tf#WPI%aL)L>bUUHyzV_CJ#j!v2i}3!GO-5Lj8{iK6JoG>C5GD=+#%cr{+Rr2*(Q5KTLClLA-_`usG<1I(si z_0O5xsAPOO*tfk`%%-tjEKO+%$A$IbH}44j&MpI;e*btL%?rLzJVlVBE{n<9yn(eS zXE{I8%A(FAt<5RA{GGxyTI2PBee4J#YeBzcV0G;PnXVeGA&t)OY$x{Wy;+nx65?;Y zi0(&l79>GMw1>j&w2EN(&+2s%Fn0)6}o z)$cKA<5u(qCNNMW&MMRQ>dl2BL@lB|2@b+MR7eBOJmOL-XIhyg%5OO{2!|J|s4*#b zaz2bjqY2@`jP=|btcVJSLzPxUoZza&5;)E%nQ`#m^cnZT?|>e7W0)1`4MUt z=wdvO^5e0N#S&yTCJ+7b0U+Q-YMP)%l`pH}&Y0P3#V1CZ_;L8TL?g1!C#*mJedp`S8XL)fxG)_4=kFg)x3c!qrifH4}o*dD*kND73pwTh@WY z<=QA7c8iU-C*Rik?U5W_zP!V%ZL|DtXABMT6V@ zZ?3ePSa`ey9Nu0YfNX7Efs**IUwcD^#zT|{KJ?3a5Saej&jOoH#%@pM7qdP-x z_ru{BPs)BTVa-9n9sxB^o_@>rKu)JLvr-%F}9EXg{J*F5pY=KaX{ zHYsFrwj+goinko;L^0_sNw9Y1R|&dA67=(r(D0LwKm}85usqW+pp>Q$XZwk$+_&?+zdz z>?wqW1q@(9m%D=^TVyKceAmvdi`AVXEVb!JD23bXsbWW^Ytez8&3&iDF5=+K<3jp} z+m>#ImB5RztLyUXJ)bs)Hl-;=_?XH${H6^iT(Z$lujcaV?~_pZ&~nuT*%OnEJ(n#1 zxr}(5*!bB(x4iT;xlV}FR1ER(>Y%w@YPXX+1Z2B&m`dy-b33pOVA&>!mbp3Eml%=U z5$s)p#RYdXwk=QJvg~jV*XfI%`-41^3+c%G!0aV_ko4h~j$WL$&6uvbvXy0x(Lw}Sl z0$6UfRUoCWoQn)ezg>%9#q>C2-?UKTc%CdeNcAlBdUW%QRC_C_;9@4$9VWlVfyO$k zp;61KCOBRv8Pyl1CK*|X`9CR#`FUJ1*m+2GLlGiCD&bf+I~?h1-@X3DEKWt5hz_7; z8$mLY(vH&i*FUx+P^aE;yy>@ZgOs6f009DRhekAQY1(*E>svM0C|aMSC&JB zZh}jyVN(i>{*LpXtIszn0ln!URi%=!7x# z*DfOv=+(;J^u=y}a`u&dOF$e!TP8eP)ad;>4p}0kw|Jiw4^OiBc8MfG{)Nw;0#NkA zOL<#fp&2)S37a_=%X0o#hFpQGuS#&yE|GWb;)?McqiXkGIpwgWs4p_4%Z*fPGf$$W zM5ilryM%qAkq#mlrHpVT_Ev=yzMoM4_ka@@7o~OE=43mAhNos%AkJe+ez{X+GxsQ+ zStH$lw22WcmYm@ZV#k-kj>V~=1aP7ibFvoryAzsor<7B4b-{9)azJrX%r1fL1Gc_; z>u;dkddV+rDm@iW0j=w2UZcXC?(U!KL3QS(u08Bz+|59!a`7uhDNGtA>Yfn|!z&#L zy4cS&;QD%fs&(ZX+%{qI8$$|`T_rBzXz-BQt2*%h=v^fHR*cp9!^6rk_F1SauE zVu9cA#USD=qKk8i<40#j;_IHIt(}Me+|M>0*5_aA-DffhSwLP4p9$hn@tiDwnb<^L z%@s|?21+eb;k6;=mZuu6c+1_;WIDme;>3$TFGMGMxQt7Qh~mXDCAUU4byU_pff1F_ z5JD75zIc%dl-$G@x};(QGvJlu;Jm~u`*6af*^aQ_5N9XIH8QbEHt{RKRy9bb6w=D$ zqs!1j+bzl~b!zEu`W>Bh;H3>VJA94ukWR;KQ7ZZWko8XOb#QICXl&cI)y8&X+qRvV z8QZpPn~j~uwr!_beZRG@b+XUKKN!z#3W1G&8gCO#s8eE~ROq&c8LD%ZVVdVG`qKdY z+;v#FTfemHHYes+@f35|UJzuL$_%6Olq~XcLMwJ0UnWp}Yq6?bdo8Z76rpGg*-2N~ zHIf3fQ-8R3OsLQ=PkwLN$`cH&c>?F`<>*gk5OUE&L&eXAlxj|fIRq2{U3>vK5->4{ zz(@=-{>?5a!XdNs65QW1!*gyd==rO%!@$i8E}aL2*EzZnaYKU|yel3p+W_K-YrBCvT_@fq}VDL z0*@|?ypiK?BuXRGRkKGjJeseOml8spJ*c_jejukWwo-iZ8{=^&4fFtc?XjQaTxjett@En;Q=)E+G*TCW{YG+J&yFSYhAYQlcNxBI8a(MaP{MdWM(o* zmVamF(5R=R;dotXS3c{Ovjdm;rRXZEa8*gmtMTaA$h)au7;jDqybcVT_K?(gz|;R2 z-{{Y&3}p<8(ysF%+ZU@z(fCC(&IC$s%FCwm!m7TWCw|oR(UUUgR*`Ake`RDa502MD z&$A8NI|iNcBCP2PwPfhyAGXNuJJq6O#ZasyW4F0I)KysZhgKpWVEqDfu!cxxduDsqBg`*Cv`1POziNu*U6D=kmgDYBs znc{DTHz)!)4MM1YPpQ<~9(Br;tDvEZco>AA(iXYx7pr{Z5+1FEjit0~(#~~Cuwf_C z61HuD#K@)gYp-e_bhYs8Frq~?bo!>sY=hlj4#lQ&Vkf5n;5Yxn?HX`8B-fH(W=~+H zy|x?~gU~Tn?2w9b$rISH*XTuX)B)lkK5{IwkzM^=MGtxp!@=R%ctIK%cBffDY>HLn zoMcm8CisElXM36?1*rpxfATX$4wO(5YbA~{HqQ(4r56yE;{fc1-y_%XWYwaIp0B>2 zCH5NtvF@S=hSrRGh`Uf+_749wx$C8GH*qu?bHVtOhn^T)k1UYz_D=N|Xmy8NZhuq1 zns|0*wmK;l$OM18>Gw+-w_3GCYQ%t7+ zz*^?RC{Dr8acPu2>}d|n6D`St7;`9sGliu=L5hw5y-cIgDolg?qUCg9q4^5uqbbYW zSMp;<;H`&D!!G4b1jPGI73)EmyK2k)UQ~HlkoU)cP4$;l3=h9qnyvgP*5XSQsk{ow zbRP5dz$$UUVT@iKNX4khsMkEyt zjIz@KzX;6#g+ti=d5gHhGF)biSsA|syxW(al8KhrCv{C>yMlBHW}9k%k$`s4KXKY$ zj*YJpDLx)-o4mLHC6jzemezG}jyIi$oi4)-NY6C#L^DfiP2He{co9s%1vdBXl1$0i z-61bzP#J@hy32G5z9RzFCwz24SJG6RD5NRQddbCW&8MSP zlf|d0ZZw9FX4~R2{W(T-tc=tL$wvIJXE115VG*bZP?r+03dC4MOjblnidSma#hgOm z+~?C*R*Y*l6tWTC40(-<5s8HtwAatks5#psk}~dmCI$w#AEHy%1|Csmt#M7_FC8%w?z>W8sDM zkIaGY`G65UV>bM@J-6s8v)NJyhvuV3SZ$*zL*oTH#Y5bS*n`J-8>?2tK~n&s>m;!- zPA%g-`ikmGOSnNgtwz>XJmT7hf1JjmUd4JWp5^MoqWNSR%qiUTYiWK!+AXxSI+3w4 zc9i?&zq+-+do`1U48#!?#Qp~9rEWD*zRSS@8NSo>l4Zq?&hrnM)QVFGGvQZ7r@ZTL zcvqIomtW4LEAL>-3ZctT7oG!l6krG(`ob!>A-LDgf&f67`G--g;fc(--F)`3HqG1B zTEDEZducz%&=NHA-hLTjzxovjEE=dc6l93fOW1QRBizn3X#HvMY%pDYpN}G04b#-* z*=0IxcRPHnnmZF9b?&DooV4#EFF@EYHH;Byx!-5Pk_R8om|n$=KI#XW2QcxYYqprc zW0`RqcMptzR?O+khQ_Hz`Q;*bUFmAc3A7GWfn@n*9)QF9G26}77-u*JRVtPpB_i$^ zM8*rtu&nocF#IMCbsMWnYdtu3g!OHK^sT0hGE;*EomX>g%BhqF6yelS7VFo%AsLH! z529};#wf&Zj7!{sL&yTH2u@Er`+TpCv0P%?>h#yKw49u*K5qG{3^v}YGMz~0{4x7p zh%){X(|yqwXFA|WzoF#u1s+ko2GyxtmV~h=ot5HkY8#)md;3wYOiaPS4#*#EK2-Bi zt5|`rD0G9{nY$7^vHmk~&Z0}6-&-y1$_)=RHV^q9rF4f@nM(rm1o#voNLI_t$zj7* zPi7N;BUeA;LTGTsCGK4VSC^8jn0)Uup9I0pRZ!pQGtQHjN=WV*vLzL^yl74~|s5<2Y* zV^Wox{a4{wpZ5r9RGafux=uP1$V4uGb?%!;5FW1rWaQ}TP8`^U8Ld`Afe3cPwh@L4Rxtwv|gf42a0(C9hRf|2d?R1i6~h&ELM zJU9e0a%@03@q0>FpmWR3(Gd}#)rD(U<4Z{XoUxo(sYqmAWW+=08F>Iph#5-gTaz;d z=_Qu!?DHqBfl?I0NHZJa;33>rakrR1Pyp!zEMuwOg+A@?$DoLaJp?0(+% zx5VwOhh3!o;K<)i;nATI3x!Iycwh(PxDC%dw(XFJM4%!xZ|VgjdtgMV&=7`jJH-kt zmWmdKFx!F6zr1*r?j;sH)_G0X2_ybe`!wCS)hagww~}F!4`W*8DZpGVjzaInFlU99 zr{dbh4=Vs>e zEL=@?vWgQYa1KN`j#DF-U(R#u&C?R8y1&--URFFp*^?>j+Qx>d!VcrJM>&a8#6D;u z{D5h<-=kq3!~(Rq(zKNfDt`u<61Ahqn2Y%6gXgs#>tlL{4SY7PHEi@uo%oTqwwE@R z;=%+4d=tf0Bj_}^flP8@YF2YuV)um-X@#;zuQ1jE_!dG@GY2OrbSCy$DbWVLgiY7fwv+3nDkr6Qn#&GxH<)Lwq^~u+bL}MEVBM?F3_nl>-Lzd_KG;-&U#lB z{ApfG9o0w8W7p=X(@wly-rXBNA+9CeIp^4a1Z%J8z?j>852S{cZbPqf%ba-K1Mzui z)?48qB)ZDDge~lw87#1A$YEdt6>T{%1IB#`vB#F?P9@D<-R}HEly!PCw9Mqh(ie&p zJ8r^1rJFFLqnjoI<;8TmJ=GSqP&qI%jQR(PV@93wM=Ds|9LjsxOZQUQ86L{+`~ZUm zfA_Zw=Fa;)fw67< zTxRP`3B^q&6M+arAEi`G@sF=xlNm}Rr*DgM_oMeUYs(bvm!CX8`jN-^ekkLbX)(c} zy!-G~{~|1UGd!sYAz6&4Gv$=gU$XuMh0D8;G(KgX4{Qj)Mi%J2ZDe@OHI+Uoj|mIw zEv9VQh#TPMm&<4HA1lIDu??izyk;6o(GFEhEiIIq%rZ*BcCzWtaE?Wb|Fq@Jev(zr z+av)9tRhAtQ}}6{fpE__7w$XfIkT`6gb%q(&P%evmtc=M;JAFyRnOsTM-&=6K!{!# zp0eR+0|)Ph=`E#Xl%z1t=uk_{(S6V6>?()Q4~|r|{;Vm9xeqKKVXgW|)HlWp$cfUg z+vg`FU3_x49*pSH;%{=P*p#`7!tx}~^jDRBnKIZ56J(Nsjxk|EmvXjEcw*Z!dI(1# z9wNz-^{$TlSW@iiUv z_7e#wa2t0`6`3rQdcJZF*L8d|Cot!X-T9NwJ z(%VHEwSj`;@P4VS_so%;GsluZRUHc+gsr9VZVSb zDQ)cy3F`7H{8mJL^#jz_415^e9V7N#&6tc|x)wsS!VKS<{bhxPzBP<2!Ba4Gj9{*n z;gH>!RquK}|HSJi^NSnQ;uEO+wzGKPSmvxWwkgh`3Y@nSb{w{c*iosId!0kmH!$Dr zNHCt@>Mx7a%wie}RT&7Zn?u4ZS7Dgzd9u>f*CCM*frGqnknts!&mYP_`lhr=P-dDRyUvEagRC@2Z2vCz>voV_ZsR({0Y!GC%PjcfFJ^9I&DZA2 zdCyQj5QKvxH^=I|{abLZCEX^nb6_HI{*xJBE{gsPsBgZvH3KjEYL~Zz?rf)n!Ce!S z9F*AcA~V*Q#tlw?Ix%RvZ3-j9TBNZKb9X)_WSUdBGyiQ1feW z_PW5ktESqSLm_#e~I9Pt+U~+V@q!?dK5-vQGth>aAGMM zWbltkD~@w4vgzxP_p*w<6@lqGFbyi+A*U@@%eU4zVKd#yp_vgN{GY$70d^m#$U#VM zeM3!ke9+yhw2aA}d>#{TE|Me70Se{@C&__0&o}MtPwu5e5<)b{_Trx%CTQS4sb_>8 z(h=lD86@ro5nDVYu?7kfnt{Qxigdw9Q;iJ5jUEeXh6Vk4$0w99E4ll5Us|F4*FW-c zG$br%>ljNtjNHWy{vA??%8eM^|2&&!@S82AeISIPosYZlYSAMLxO&jzfpO#>UDIiX z6t_d?OBYlbKktcAOh1?L#Zp5&56q(6Ky!hgk7es>vS(Q?WSDm!KEP^f-A=)G?1@RA z%tlHmd(E)2<5j~nj8imaV`vwIBamqv9Lf>JTyOa=iGn0?s`+3*zjgc&!y?PHQD;+0 zCJ3XHiYyNW;&_SSAP9;D3-|?I_K)txEH`C^t2Br6QQOB~=-laGAf4LXPZ?sz{atTF z4u`VJdM74i2SLpFuofuD#@7 zY_zGjh)180r>o{9Tp_;{$a$kOX1F9~stJ@4E-F`oa4f$$l3GE4OBBbsLX~9>J?Rh9 z90n{GKOfH*V#s&OkN?5U| zCH-9ABCVPdbfsz}7CWRDnf!L^d4I7W#7UuC%Ew-z`b_o%rv6V{oq=qjUb0vlC+run ztHbM|XBYKks<3D;!#FyBZq}Z>7UZ@?U5`%hxR02%*UuXr5E3-LVIhqWtfLF zFmrPNdDH^=qfz)=k4#NPbs8y6bQkP~#OYF-#8WqV!|2KHp^kIsfsbvy}Bb)MB#9nguvYKTT1O?Y)Z-i(i92|2a34SxgugW^MWeaWg zIUMGkkJ-C}cvc+>>#J~trhG*kSd`y~&`N77N%tbEMFgNo;SW()dcM{t672_rsTI@k z-s@F`VSxg@-?;C+f+0jE(vtcpcN?Ld+ZJ|8;b=?BE|Fkjl*gAk*NPBN9oBbZjnwOw?K)d*FleFy?WpJ6=?*Nr znAhayasp2nruXD?Styme(PazKMDLP(#1V$JXVeiB-K_Xs5rju!KzM6kmV53|RgSF; zP0mG&R${_ucJm-K&HW`SBNdj*@W&IJB6`f z-;({C_2a&S86?;J)+TYFxNwjH(oua`dYOja@ay$*3c@i(kCSsCWRX2K-0KVA38)GAW_D9}>pnSy=7EfE)@d-$6qECRk0v&!&s*i`+CyRM0}h&+&Gk`%oyM1PMcm;Q%n(WnNc^c0G#f}PYVI$IO%p&}4vJ#v4F{{~30?$tExSS#&%+IB zsdF>Yb^QyY77dibL$9%JF_rh05opvxnA&P;-Bn13hCKDA*7~s z5-U{g2k|s zBn<=^=>K7UoR^h_%vrDuOP-Dr--3W}r$uCk%;~U0tpu7?TPSn;8YJi)Zm7F(-NN&J zbW9C=Ct*RhG%g{1#}x0oP3*s%*osqSW;BG-4@xc!YD62_&0e*|LeG$913`u`eFs){ zHpwEjO5K;Sd4bKFGuK_GjVLMLLL1WRdWW7T_LhqQzK+hQ9yk3-PG2BeWmQYuPH&ps zdn^TA?Yo_>ELNMTCqh`~x!f4mB5RZ56eNLmH6~Be|$LGM?9=As6jvb0f}D=}ld^go!H{ zC@lT3&=s^R3)I=-+?%g(w3oBvpgL(41L8deoAG?4HK6$6QC(lyAgt`XezD+SrLA{D z$a<@72mVo=i0Y$WOmQA&;Dl<3LczeZcPFRoeZ{C`8IxDDIi#oqOh4q|RC<8bv2Usu zlXa^If;@=c?tp-_(MY;t;%udBLs;a_ECa~?$; z^Ji>~UWyY142qSz`G_5C#L_ark?-c`1~tVOU(#uxFBp($zSI|E>dZW{S)3XqVhA9` z;R2$js4t^a|0YR)whtHT|3lR(VL_?ZA}pWw6TC)M8u;JkfX&7F|F@B`{yQfOG3bE* zM4SHr{7s#r1KUsYS3eGz$mUb5u@XCDk}wL$=GJ9(6PZ{_>a4%-NW_VhVSc6u5Hxd> zwPH!^*$(EtxXN4^)Ld^@aADtq3bjN#iis$J(I^}lo0P$<#l`Z)is5VtvgYF#k7h|H z7k|%@2_FRTRcl9~xvGl8K4YTFESp2cUjkM8%McZf<6!V=0>;wGjX<-@Qv zhxs81Vr(%SVFnucM=xj)$|mzx>hEFRtYlU{3VFUxp)@uE4F;*?cUbA?2XZt6&X%3oy)J&<^;185N#FX-Wfz3*<$Pk$?25>>;UZqZP~ zX)|RtH0pNtO`!w@pf_%jd<1*$c z!ZZ50Xa`$46J*ua&kQsiY^*_YM`)iDoN~7D(pv4@?HC(SMoeADzN(K*sP@k!)zQv# zXw*tbQ%?a;(wd0%T%A`Fd;8!G&^FLiGB@a3I8Xg z$+aq}N;}M}7<`x%faUm8>0#1|{PfayMg>DOXpJ_UHWPG10zot*4&RU*$CYYpvM*wl zQ`3M^LSa8m9tA>(8u^PGGR*Ugb<+M??Yjtl_;{kKv)Rq(`2`83%img~)b+e1#Xc)D z_0oS`pk31@hl4mZqk@7Uh1pc20tGIfv=RG<{o_6ymXUv$gAmFrtC;-&ThO`WxUNk}58O_GjYz>ZG9SSCmj)Lb3S2 zlxVZnI>A?>UN%a~cBc_tY>35pbE%r~=Sm8?tClKM@in zYYTCX>;0*tMeS0s2jbcq?!{k0`K{qpNoe|{$a1EN!Y~ti7GjOUSEI;$sK-Pl8>jYI zuIqR-xfWb3qYDWCsZaHJ>L0}u8gBv1A$dzkd2WoacI$i0d_#FHH9Z%?B0~X#m%t`! z?l55DORQ@vBvmE& zNbt6nS?X1Ltoir{0jB3$xz}w>T7020QaGTG9%#%2?DU%A%=Qa4NSNDR_e7?g)A!g- z4b9j0*^?X2+kr|``TC!NGy>Lkqr8W4O<`EjTB5G3;Qr(!@$}w`Ryn0(9|61dJe|k0 zlUV-8yUD;y)idUgP^{nd5@8@QHcmboP_xyOgVRE5gztoi@>Ml`4UcVw40_B}jyvMc zDz9dIISwu)Go3Soh1l-HKc(n0iAnsq6bw3M?Vn%7TlF_KayXbD))Z=8TZz%pUkUN> zq=;%gV9?pqA>5t~&QG($|A&vcIse1Q;4J?u(540c>(Abh_>Y;sl_LFPz7vPR%_WT9 zEW{iVfMtp2uWlJtA=XK8H1zhA15n(bkdcnS%!l-5tvGVj=Y5!-n&;{ocYJ<^MOx8F zQ^@Trpdg39CAD<-%0V58Pvl7wA)OF&eDb4l=ol5EqxVQqEJz~3ff8)v9Py94{VV*h96=vKj|IMAxOIbdwHeHjOO*)0Q3(^Nt=5y`20z?RfY@&N7mT8# zEq7!$*}>vF%{iPQjR~|A=>wRe7*(s$FTg!Wes7~#RCc^=r`SuhZO{rRBl*UT9Vi7W zOxbWwDSzYy^YtXTaEe=B0kR8pXG%8HD$PGUk`3MTbSJpEFEIuk&VP~P$lDsbJLPGm zx%c_9R04}xp`UzD@vN(jC+PfgoG36qNrSyICxba+5U`dtjfOzAFr*C;v4sN_n1ERL?4nx1lsD=$UWo`@Rx#jE0!R~Uf^(lTvlE@Cl3&iK5(1{EVvk<$dG1-)lJESdD zo@H*7Any5eiE=sAel_`#I9EWfaJ3i~oP(L6x4!`#>ayIgG&&nCu~T0f5t`MmN;ap1 z{c$QYN^a2mrJ1_ID|sXQ?9&;zJQ zlsmSRk+3DzDA}Rr`SDSIkaAvEy^qinqC_8s02ej4Zd&t)AH;P6RHTr4EA!q;IyKX( zK7y@O$qM9UijyukW+^;P?8V2%4#nJ^OgX$v7oL5t|oSbrc)t~ca zl?L9Gt(Ug$Jlp}I2UhQ9m_zDrdaVK$fu*x*D0~5QSx-YW1HswfTkuSuIhkDR)KR zSt7^nfM2tqRlBgkKOw+)%XetaJgO;9el}@VA2#JLqVZIVBEqyec`?--M}w2@IiV}N zjaTM9rm0GGh`n`s6I<<_13Y8F!peDbME6*b&@kr%vrBkeJPs+e3Q)g|Ta)NLDIH;~?P1yy-2rv%k|q(@Rv8xvJD zKJ{ha_VJtpA-{+xX@L8Q0OGzg8psGGlfJKb5%@?_HP$GThlR%bEu(B^gNdKAXj$HB z?s_8Sp;)o@?0uQ55KR}Q)UD2w`v(y1^@FD({_w51P7y|Q4-nh99V%$@d4 zt=$b17b6Nq*OGk<)gYH6DhKb9wmI-}B8#eER(s5%r23x9JHUQUy(Q?OK#h~84z;91 zKlj@Aes{c(P;YQbxZMQWXpmo}lhYIkvfLqq&E+_8Is1z9+NuCrl`Q^ldK}pu20ukl@`ezEO^WMG}d&J0)TzZRuj;hSP!K!G1lk5h3(XHzUr17ipuHLd{imr?J#t zTfN!jIoY)$93X&(KcD$nhrkE!L*7bWc{&V=Vdr@_Pr^moWeK}AsOn{6LH}$A$>y$F z{ZhdimJnBRKYCEsSQ8wImoAcHjci0E5zg+|ojDgB=dtQ0Zhvy60`n%>&?G@2DZ!FA z(XJUSH6F^TWL3$^|CkMpbI++KDtDvUDt)fFo|V{<8mMGq#2njCdI-b7CX&#AYUFa* zmVw1Y5iu2ow9o2KoB zB9(X09a;hyxI8tN*_BsXjfx(*rrUFGfA z6op32!tT?3H}##(m%afN8wll(-!7tb0L{J|Uc}-0;@a0o7)C^b04W`BqkBv9YB(yH zXv==2YZ7Yo`~ty=ped}!SUQxgDQb|Y3Kt+S@;qlfciv}E_Z$XBPB=4->(6(zb1}8& z3|w(~6E(A(J1V>%g#mE!!{k|&B9Ug3D7rNZLi~blK#vhNl}e2=EkfqMXOburQGA6l z{holqD3iz!hmb(pfD|X`Y?tc-0|x}IZ7)XtP#_Q)(!p>`)qq7WQjlg&eBf%}m;X-^CZ|arc_*ac?wnH65B*rkIzesEjZ<<3`)pPLA|@CAn!i z`34vBGC1GxkRii+%7S+nPi`bpPGN+4>=SAQ*7PzBJ9rv&g)&pv!@T2*Iqi!b8>PBm z8jHRl2V0nD_z%6HzQ7xF^#R+=|E4y;Sh)XpEBs((`(Ko*M_VUuqaD*P&!9jOKL+`}I*85-peBVDSOs>2g4_1a_jnYT-;nJEU3730>a zUxMuuK2u|SSije-R6!Z1Fd0G=vB5xjVvr$= z#`p)P{Xb8|gQ;*t{cnZO>DkCt&S2KuM=CQD4D3a;&=;&JH0xJ4SK*WP!cd4-g_oGD zWPhX*0-IBo&mC5KW#eG$VW8meAc|eaJ+VLaq%YIJbE`&0F-Jfqqcp=cC2C@BX&xFA zDJeOeSeDk!W_4HJ<_@5}C_j;kV~>_vxkBtNS!y8Z==u7@V8JvbOt2lZS~@qgVk6jy zw`F5S?mn*z=!GBEzF$ES*eE8JXYw%Q1mY=U}}vYn>%V z%9=juFlxe<$GS2$ShEuI33#&2>d2b?E8J1_Wtnk8FPLoi^RMyCZcLiy#>7^rNd?zl z#eK1``Gqk9%=PE@wO$TpN#D$7UPcYz^{KDw_VnxHYv$}jrNPdfU!OAAp1;aHOY{Nw z`WWh1viC@4emN9MGqZvC#!-0Q+pWtEL66HUa|{nrzoy_qZXPf=vd@;MVodt{# zS6-LDWpHNM$K5pbVQEHMuOM#njz8BJCL+45(@}#gI@rc^50h)JY1-i#_atv>n?Kg$ ze5!YyB_4b*&O&GAl-9zILilrQG|quacDbi8gWOP=pPxNAoq1unXkSkg)d?K?>E{X< ztD1-t6l&e_V|#Wij~h9xWq-;=VJc8g3w*CKS7D7;ky8dR5_>Mj1;F)gC~%(P+Gmhi zmxHf1%gYtNL1FMfTBy0Lh7Od0Q8|@=1x?qg`NDV@1oOak8y8bI%*b#pmzL+q`Pxq- zL2(sQRfG$ZXubM!XT&gT!~w$>+53GFV90;_N_m%N;T!3WnYzh5JcT!6FOrnthrr7* zSol#Pu=}4i(y6}*IP>j+gI{{^mLi+l(vgJnlNGp-jLHWPSokcENNR7)yaH5>LOsyI zgydn>_|A-GEl3vA96+UFoBUs+`PEU+R&$l4thDBno~Tl|pym5wN`Ox*pt`YDQA*&( zR={{UY{8AYm>bN2`B`v_5o3mpT`7|TIqpYHjh(||pV+NBR6XhoI6 zfeX%-Wp}@3epQ)^;wM(` zLE!p#5;NsqBtOHKVbE-e*Oe16lI(TN7u03ZkJ*q(V|*FbN&>?@=-x{Z+s?=idmZoHfA>j>L3x`uGCOnshC$=gyqWy8 zCQvS+lepDo<&yVTwF3xQEE)C8RPk(r{=h!s(fznou&BLit;5@#Hox&FwDP~M)=*Ob z`?aCzJv4A6-U7^Is64|xBLz`nE4`v8iM;H!lnKslA8d?CXg zo91Y=uv8}A*FKp5!?^@Y!(8r-<@#d>zHAtB0VII#pbI$b%!V|0h2oTp>}{ZcZABO; zw8|k>zU?`=6#S$d;SVW9&~u-n#?gEitG8O02)F{fjWtUroFYOHNgh6w2B` zpS){P7_B@C)KWYudzE#ILyKvZ^{0AiHd+wu+M^yBl5Uxlv}xdGRFf#k=Er25oW`G* zUp?h!iyc4}CU}fJtqnuvFUQJ#{WND%4~*c_e$GITFNvG_FPP8Y8u_&c(iW4ooEYY* zYpuLc=YI~+dw%eL5UE9WImfsQExQjBKm2*MhiN+Ld>3(LedC*Mn5{IGR+XHEQOD|;LatQ zzP57@Ac|$qBVz?{YqiarKPg_TpH{HLpP$oW49#dUwzTDL+{?_^#X zJTVTFK3DrZ#SRR-y1~1BB2zsy8o49KIag?U#4fQe%M!9~s&Y0cCx(q#X<4XrC9VM~ z4&Gk!;>=<1%o9xP!HPlOq~^t)U_5Q-i1S+Ep(p97fS)pF$EydqM2BseAK(uZOa|=V zF8Gk-o(HRd!qSLlA$uxU_D!c@(=BVRi}wl6v8H;e)IBYG9ef7IH6GzPsp_R5LCXn7 zn$2Km3$e*I$n=W->k+qmxV(v9ud_3Hy&~RdZ{_2MpR5bO3qsD#XHQ1&#Rgrof8T{2 z*&`8{;4e8{Sq1%*d*j=5lXLmApeToEozP|fEnzz~IO^1Nxp#amGj5bwP|I(T z?=D;n>oZ{a2sd^1ZM~GI^S`SPlr?E*80&xTiD@ml|74Gzf4)T;a#nJ`NU=X)(tnyy z3&Pw3h%&L}*0ptFG}gzQo6hQPilnT>vr;+NK&H(Yl|*M8kDZPYIc0uIr+&UNJoMGo z<@WdvT8hxJDmNaUQx#+<>7I#=XR|ORRk__uz5`Ug0jn5#9h&#LrCF_-8fHDa99Nsb zF}pQ8e7WyN<)XbI$lslu06Ap7UYTbL&#+f#Cx&qvS zUlh?a<}(`^Mr?!bM8&l^7-i_R;zZB)KPMZ@Ey6hv&M9(~EHIruJd(XJwPesUD};A& zZ|InyB0j=BD3u&XO}gpjyLp!&n13;rZGTaLO;S!Gv?hnOzt-2NV=3!Em*4XPL9ih) z!!%SIIf6F=Iu!3lgMR@#G=&#R43ESPFNF)Huw*C;;hhUzyl5@?@R}7#ORVt{PEhrj zl%my@h?L4;W|^y!D6qXNuMtREa7cZyw{#@d#LWWJqvcWt3Lc~BC3ka$i>VN#H0QH% zW^zmyB>vRO7V{T_<$VZim?#c>kcBoa`A;Ue+5^yQJD@qEN;hI^PQ0{7*lQ`kq)o@U@)i1) z*?JG7C8rzJA}VLu=s^yp>2i2eAEO8-zj0+q<-|*eyubwY8nZ30Y+f9IOIKS+rU~N= zSHuqdGLQtV1REp?Sv6`iK-&-zCP(AQ6;R96n}PD!ql1mP)S~3+ zoOYkXHuURb$-rq0&Y!me-&Y(G;iFrof4JWiSN7mv;=TCkEU9-#3olk38BXCrkShA1 z2MjN&$0io9{V6V>=F;ODm zd<^ek)sx#`Wr+zHTUbg4G9JOh59&Jd;HZ4_kB7`1nC+++PtApl^+@l7@j?Bk4Tn$$MhLi{d-U#VN5PyJNyb-C8HOMrl5Ik( zX-dlsE~JqLszAMT)+ahD&U?PM|MjHcbsww_rtxxWUMH~YO2#R0^24+|+xNd5&-m?h zD7}gaZ5dr>!N^+RL9+o6pBn z^OljUwCTP;=>pI@Ne_%ECof@c^>CWS>!)+ z&6sc^d*<&qIk1T90jiJjL+lz30tM7 zBcHfhkp)ve06$~{O?9@4X5j1pNTZwgA;KYf0Hn3ZH)_J>U7bzYq|0#`c;#vrEfDdF zrOus!WAMzZbI~XpHKPE0(k zV?d2WiM?zQV}eXfGjP3`TukF9GhC$%kP#8d);dtr2qaP0RVsR;9IcQVM26esCX5hz ziE$lAsSjvo($l_?5monh0AXLxnaI#@Z zeK|f|3Wm6kKK;MgsNH|BG~y31$R@rwxdSoBv`FkwZR zsx?%O*>9k97BrE%{n)4WSl0MVok{iIS?3kr=ErKJKpckHeNy6fk zf&|9?2t-%{D)3-49r?Y>1RUaXm5DJq@kjpWVbd7m(<2)B7o2*u zmV9~YeO((2V}CfP(}4XneAt0 zfd)In_{yj3Seb#44!Qp}3C_YkVaGLlr+$~sj{>%lw7L3pd%8387y_XcP9nxa7`o)NBbf3`!^BhKLbaa5Syn(L6mpCP>2 zF9p3Da^NxiwsX4dwzI$k1}%IBn#di80hf-(Cx<(ILPGce%NfMj#LHXiKSr;J2Lrx} zJ{Uf-B(bs6P@aB){Wx*9k+RCm7)`5M82)97SU25v%=-X=tT|;9>-Vqr&CGWg3rU(G zYHrTL(>1)lnR;h3D{m#<>Th@V>oA}9tD*!;oc!|SB>cnt=i@%_->>T{Z-CpE18{_a z>HTHWAXP(G3fRHJ?#mWuznpe+d~q}m%wgK;)^cQ>)g&}tWtdR2jw<_@ zc1d@S@c|gU@S0`|>c$Wk(oIfHfYs<3fKx3;S(DaRajfj2=-|NBv+tB#zURptjh&U* zM&8>dzD>^1l6Ny!hC_j-NZQ^w26q$)P0V0CZ=CN3lhu!xn~$W6zlU3LeOx`P-w$zF zZ_H898V^;&=j*!vP|8Wv#RW8V8aA+xJE)O3yXLq|rn;t&2jD`VZ^jD%!e2f38N(Br z5ry^=xipwe(Uj%4u*IxN?jYOmb0~&Cc}ey4JZ!^t7lH61b^8Q(>(bhS-vBs;8^W^k zad2l7Rc8NLOxTgw6eY5OW7*aQE?O z?jG9>hnRKt#KqaQ^(8mssOK}y(E>Z){keO0BR+t}L<><<&h?d@RoV60)~dzLO21wa zn4>?*Z*5P}3{!EcRM|or@j^zeceaG$VrG-Cnh{Mmm zsX{l5e$IbuayB*ZxW+8|GtQBtWHKa1T~gQ+`S-Wu+*0_>#?%RkUM8;poiU*p7@ui< zp77W^IvJfkI@v#)_58PYxRo=U+#BOe4sI_h(T1@;yz#wbq}6aT3{oged4})qPIF0J znP<(|-nCcxmu=dG=bxB^jKP ztjm(;qk40MMigEKv=>7vboOf0hBuXoCR1=GqzJA|QS#lm8zDsC2 z`-rgBb0+Or1~hvp(2N#bM?=&-Cc4tfDMEEcBQ2GWKZ0}3Xd-I}XJ9~u;ppX4UZX!b zF<-1ycNWmD{O;0bYG;%0;;+=uGUY{I6BcaCC1z&aNhHMymwRm2t^J#!m(1+@G7EmV$SI!$$J~dT9eC z7t~oT(?Z1oCi}1$h}CYPT4q1HMh0Aw%`Z8{x;R#2&ZVMAznCe8nsI&8-HPaxITc+vyN0Q!{^m^wCGVW4k?S^~R-a3n>fzUG znEKver3qa-gw%@Rn}xn*iq91gwtw=JfxQ&)>s{~bazq6Y@p4$~C4EvrGwMeYv3!C% zOLa0VyAa)Ow1*ntKout{c?am4qtY{xI20mzFVQ&JHU~vr1!`s7aqm#9&zKQFUe zkh67u$NT_^hpBMnZZ*e+9*>iaP&x0_V5}<%X=D*lHZzw#>1U8ORWaj>Jo-{)b9JI8EEgRw}>uCluAlMKE!VRjSyM^d|_TFX?f0%z;Z?|em_wuS4B zu15JZFK*Mxh+~w&P12gY_dl~kf8h?w=~Y-jsrou!z{z+gN#@Q7>@Y|qt;ObK3NVvc zNv5{5R7fBASp~8SxsE{ar5((})U;j{$tpp#$t}PgI(1Dal%2?sYwDy~%152~p7V+_ za_%sdY4yZfHP>amx@cetwoQH3Ldo%^mU`oK^xP8}KG|3H(2@>`A1)&`KXMquL>=OD7evNo?~K@F1eeG917B9ru4dhZn3 zt9Q>C(BDApXmbYLf1tqd`yA>E$QXXJPZ3buH}*+cV#)Q0?VcwnBuPos^&_0C7IvYd zuatknDRINwW8@R1CfcD~q0i{GoRbaW4>C#sw-+ioqYdW*u^#;!Y6vTCtY%zl{;o>L z-I2x<6)wrN(;7a)Z-Ia^^Vkeg7u6&rm3?^GLO?N>i6KRHL~G~nlKRpgbsY-I0j$fm zr=WTvbmb(e4ZivS^7;c0;-^cs-i)ZM>BbqgY;}M==p9l7-c~P_8-821Hx73_13;mLpr?X) zA=LhRpfPMe=XtZNyw;Rl`Ki#E1Il@LM{$d`rLc$e~#KC=?n@ajx z64xT68-v?Le0v6^WV+fJrbLUQwC(Ggb0f8{B$7mc%{UDR-PZSPRcS{%0)c(C*XfrN zZ<=OzRX@pmdZ$8&0v6ZSnBJUth9+woI(WO3rBU2C_C+|m4_yS=zSW?*Dt^ls(dCGb zwymAxu6stG)6vCzaPwgM)!BTCp+6|bOwJoskr7pte8w5H+|psQl}dL|s0Hv`g)cgC zvi({fFJ$Pv9$V7-Z`WD(V;Z4lWlw4x>P4>p)^{$gu|_0annAkD1m3Up>!#i<^6i|z9+T-$<4ZI6Jv)A# z2xSjTz8V<*q>wMX^xHFKJ45`mNJ`_QzoK?aB+l=WdGn1(^Z7kUdIQ)W#Q_I_mv2#^ zXzgO~Ye|LTwp7!FeA-}*<73TSRK*|*64~(Y4i_p?y5n0*9A4869=vra*&e(W*}mMV znwCQr24rJ=!-8g24O2#+&&Q3^;R4DfP+lcm<-f)|+N>orT4gY5{RFYIG3saKRId#! zOXT@ZHf*;FYxe3G3~OM@^BON>ql?tn>E7ExJD>VCgF~j5_L23x-M=FKZ#RPwu=mx= zqgVWCP}=q5q6cBhZnb~#Wk=sx@XKF{K-tjl?&Ay}_kgZ*cGcX`k>tl&NX+iyGLL;# z#wN(Y&t&mo)iuYTw zRA(Ig_u9Kb(wXqS?eSrz>)+L@ipam4Vv(1}$CzQSYqZaT-u^y;zWxDuG|UNaxCu~9 z_3HwBH@ReO(8(O(nZY{sb&nv=_z2&g{&lO5lTrGQJsB=OkxbmLH3nHLEq-6y9!!)+ zyIMG$FPz^i6ZB=T@82!F=uMU$4gu(iOg1Rz66LJZ_EZRpZNWYsbu9UCq2~ z@42m-L3&!AP9`k}kL0)BpVmmnM z@6u0C$68*0dfj$qr5pDoL5S_ZYOM}uB_ci?a%R&h*DfXL^hx#^Q3;FahxzDklY-5h zE{Zf4YsL8GSmZck-E?h!A{!tWd#=OePm~2Pi zXTf8YO_^q|t&;qTBHQ#y&lZs#e=?2Vn{r96V>EIQP=~=06yH&hNKAcK7#3 zAuBRh)E(R;5Va|l)Gv!6WEa|{~}W|f6+SjG|MiSzz7wbcLEcFYJC@=2MdEK z!ZW}UH~LF!fImo@3dGuFCj`QS4}3ygCQeeoAv`ooADH1)5T((^F67eJD8M%dyRW|~Z)^n1 zFk=>P>wal$+m1uVuRgW#fA+oH+c|+GPhVfMQ*g9FM95bX2epeH9AzDSCMROT{H#O&45xb#E?3D9%4LkT9&=j)d5^GAlWq^B zG||@!!hLVduM<;1{f5G=^o&EYQkPzRtW7pxDS<1e~DofYrxn zLtp;o!MT97SPVl8R>kMRWZ5|)PzQHICbQ)HPLy-k)V%%LjpAC9?IxS;C=XKS_`$-O zT4l?K&*$lFH=nV*R*{~tJq?7@S2o+AjAbj%ZMzMbBk~K8v0E|v3!bk4Oj0CX z)7s~&%~>U7q+SQaXBSXJ#lfvX>f=UH7z=a(Yzb~19}-rqS#X>sQ~rl>w;a#8=(fQM zQSCnNCM4`<)3Z~xuTS6+`1p~^vjFZUiA>;ODoxRmda<{E+iQHij)v>_G=?XiquZ0_ z=Vn%Zo~t46-uCzXhU(Za!{`J4eO_z7LlfC2SXbFI$#h8qU@Z8%)j zds(U{ZPla{e)M)al;}^w>vy0@OsI6!Vzea;*@cH z$y*#s1`GJTZEp8}%k>S-P0%}kO#T2MQdPYj4#`(PNvKV%Vp-7|dpW;<9yb=$M}~kq z;}t+0r<-z#45#(Kn`qRHb?}Qt7gVo>kKkKOMJJB+_ zd3inDG`1Fl{Stc4NwvI#)pcadnNOLjwrZsjvj+aPp#M0NoJ=@w6ky{v5c9k_>&?FG z4e+oS&ymjfONCG)q5SyO_V)>IG#q=lKW9c9cxH??N4bC9UfV5ICBRLCdHOe)w!87C z1S;&Bttz^y?Ty+l#9uYGGNzZKF#8W+HLakg`Mx-%*%iT|iu+@(A;3kC{yl>TSxJ6am# zGN*A#8O>uUD403Zk547}HTqN6H>5~S9|W5`WtJn=$(|Wn|G8%ZxP4B-ifb&2;DDzs zEXsfMYSz5Mh-}LAqq^U0ss)clg>zC^H)W9qJ0w0(r$r!r-qP`|q*ktr^X$o>UHax) zX*lpg()hY(a`X7bZa`}WJV~(2E}N*0x3T0=kIw`7Et;ty0RvwI@-cJ#XrF64v4J6scY^7@2PGq?L?;DBS?$`?0dtY)3~S)mJ!Ts$oXoQ~ zi;Tj9ErlSSQO%^TjYQ6L5E8a^pyng3r})jvxHUrCr<>7b?Fgj)cCugDZ7cV#fAJ|% zv_1MHQPbZUGY+a@cD26hRoTNeLk&xpw5x?6r@YU+|2GPVyaxcANEG(f> zzR53goxywQgd`h89js5yXO!Jr!}`3 za~H$?Z2I1t!wZTnoi#Dv(BvXgMU+3%5Yc!j&)N6GWk4V+Y6c}j1jeP?(lAz~7}`*{ z!@NKeatgZ<{ux4Ka-`)Q)4_bvx5#?oSiax0EcutOv*UDd=5#SqL0Lt4T=Z&5B`5wW9)mb7{B{^9PnAyQg5Ibq6ONjKN#3`qqDf%+@a8@4Us=_$Wg6 z$?&sUS(fN=n|Ev;a7f*-Z|syc*JB;fH;^Ps={T$uhROpCnhTgGlu!VqyOaHvgP>Wo zC#g4pz6PIL$BW(Zo)gxR3lWW9{>(!of;1D> zxq1c~m78H0|KRRx@ljeO0atOLO0n+2`9hp9q%E2tmF^CB?$85+g^>hPvOCJazHaic zG2MmG5`>t|`guc4rHffMRmsVaz0QWJpmN31DkPA#0Gc234H5Po=%5rIH($T98Bh-< zA(9=Ihz6pY0?aLdQkER`9Nijm9!NYKpM4%ng*avC@1w`%Kv)}?6VfQl;%o5r7>E1{~RV* zgS|jnz>DLDJA>*6E%ru;EN%~#ASDqKq&m2e^1M?XSb5_NNh$(^VY{3O!Xf}nE z_f(>t&Fv>v_v+<*>FLt3;6-60$WWTXiW#|r2PO4DgO|154=?c+DXZiQvqatO4>DFh z=l}sr2)L@EgRC)Pk6AZ|1jTtuuML#WYm4N}iE*~lT)S4r3^5Yr-eVFPE^+=kb8UG-1Uel%G*pVbVD5SMfe4&m#S_4gBKV>p27URoJp z3}L4M))b>Bx!M7VF{N)v<*&(=|3{qvDF2FJBOxg%EiT0=#m^zhF3Aq!=HTa%;FT1U zkN|OWit~#}atTuk{(mk%wA~T-!Ez2trbH2!_d@sCW1156Jas%XF8b;UhFLOYDL?hg zq%fjV5peW9!*yg^&<^|LnQ3@+5@l)3ENwrT0?i^)kvE0kM%m5}AIF;R{qGMsxSoHc z_tKvPw<4&0^?R912Sl%w%SWJC;9NV9&4{Cwhgs=hD(L*Mv=Xo|EX;29>C;JePE~2P95WrLHbDZA zTdz80V}Ax_IH3`eB3~V>?@o_sdM&70-SRfDEOSVQc@VKN$Ag$kn{a}OWaluYD&huF zrn*ioVFQKc+*QYy__G=^@KqI@KrU{0?y{q=__1#;;ZttT^Wh?*NrHsXoD-2J^g;Q@ zX_z59>^A$6=WXJcnq2b_`+Bf>71muhJirC|Pdx^!>k|lFrcu!kj2G7_m+lUI*473@ z;Ij9cIgaV3duBq5fLt!9%H722g{yd|r@{e!@4M$N42aIXO6P-ddsPv?)fgW^(96)u z&|st7AP|U0h(H0(0Q3KVo`XP)N7XPvW2Uin{20+gRJ(Hx$*|YW!LT@A+ zkq5!`9fk;oy?PB{$f1qmxNrD*Z-OMKOOY5(IoSt+JGc&J2h86P-qEW?IcbB$7}c^o zk7Ec{FcA9>!X`1?@!j!La9^E7+?qow%|6m|ga`i&`Dqr$R=NuOxzL~2;3SgVkUY2b zK~-5*8L(IGsr_ivKS(z~hs;VF>=xo?hI+tQ6l@goj;+|`zvEBMwP&+zJXs#?^Cl~6 zfJ8s^-?8^NrxahG8K`#&EreOi` zw2JJ1BHElw`k!cu{*8c{$TUR`H>Ep)eGP${;@t%qA(&^61u{9T?lRpWP;g=88U2X& SLZCq9QU43}i(0Y( diff --git a/doc/src/LogisticRegression/LogReg.do.txt b/doc/src/LogisticRegression/LogReg.do.txt index f2fe7576e..4aa899374 100644 --- a/doc/src/LogisticRegression/LogReg.do.txt +++ b/doc/src/LogisticRegression/LogReg.do.txt @@ -450,7 +450,8 @@ if __name__ == "__main__": !split ===== The Credit Card example ===== -Here we use the the "credit card data":"https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients". More text to come. +Here we use the the "credit card data":"https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients". +The data are from an extensive database from Taiwan and include more than ten predictors. !bc pycod import pandas as pd diff --git a/doc/src/Splines/Splines.do.txt b/doc/src/Splines/Splines.do.txt index 35bbeb89a..0c30f2a66 100644 --- a/doc/src/Splines/Splines.do.txt +++ b/doc/src/Splines/Splines.do.txt @@ -881,15 +881,29 @@ _Challenge_: try to write a similar code for a Logistic Regression case. !split ===== Momentum based GD ===== -The stochastic gradient descent (SGD) is almost always used with a *momentum* or inertia term that serves as a memory of the direction we are moving in parameter space. This is typically -implemented as follows +The stochastic gradient descent (SGD) is almost always used with a +*momentum* or inertia term that serves as a memory of the direction we +are moving in parameter space. This is typically implemented as +follows + !bt \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t) \nonumber \\ \boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}, \end{align} !et -where we have introduced a momentum parameter $\gamma$, with $0\le\gamma\le 1$, and for brevity we dropped the explicit notation to indicate the gradient is to be taken over a different mini-batch at each step. We call this algorithm gradient descent with momentum (GDM). From these equations, it is clear that $\mathbf{v}_t$ is a running average of recently encountered gradients and $(1-\gamma)^{-1}$ sets the characteristic time scale for the memory used in the averaging procedure. Consistent with this, when $\gamma=0$, this just reduces down to ordinary SGD as discussed earlier. An equivalent way of writing the updates is + +where we have introduced a momentum parameter $\gamma$, with +$0\le\gamma\le 1$, and for brevity we dropped the explicit notation to +indicate the gradient is to be taken over a different mini-batch at +each step. We call this algorithm gradient descent with momentum +(GDM). From these equations, it is clear that $\mathbf{v}_t$ is a +running average of recently encountered gradients and +$(1-\gamma)^{-1}$ sets the characteristic time scale for the memory +used in the averaging procedure. Consistent with this, when +$\gamma=0$, this just reduces down to ordinary SGD as discussed +earlier. An equivalent way of writing the updates is + !bt \[ \Delta \boldsymbol{\theta}_{t+1} = \gamma \Delta \boldsymbol{\theta}_t -\ \eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t), @@ -900,20 +914,28 @@ where we have defined $\Delta \boldsymbol{\theta}_{t}= \boldsymbol{\theta}_t-\bo !split ===== More on momentum based approaches ===== -Let us try to get more intuition from these equations. It is helpful to consider a simple physical analogy with a particle of mass $m$ moving in a viscous medium with drag coefficient $\mu$ and potential -$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$, then its motion is described by +Let us try to get more intuition from these equations. It is helpful +to consider a simple physical analogy with a particle of mass $m$ +moving in a viscous medium with drag coefficient $\mu$ and potential +$E(\mathbf{w})$. If we denote the particle's position by $\mathbf{w}$, +then its motion is described by + !bt \[ m {d^2 \mathbf{w} \over dt^2} + \mu {d \mathbf{w} \over dt }= -\nabla_w E(\mathbf{w}). \] !et + We can discretize this equation in the usual way to get + !bt \[ m { \mathbf{w}_{t+\Delta t}-2 \mathbf{w}_{t} +\mathbf{w}_{t-\Delta t} \over (\Delta t)^2}+\mu {\mathbf{w}_{t+\Delta t}- \mathbf{w}_{t} \over \Delta t} = -\nabla_w E(\mathbf{w}). \] !et + Rearranging this equation, we can rewrite this as + !bt \[ \Delta \mathbf{w}_{t +\Delta t}= - { (\Delta t)^2 \over m +\mu \Delta t} \nabla_w E(\mathbf{w})+ {m \over m +\mu \Delta t} \Delta \mathbf{w}_t. @@ -922,26 +944,51 @@ Rearranging this equation, we can rewrite this as !split ===== Momentum parameter ===== -Notice that this equation is identical to previous one if we identify the position of the particle, $\mathbf{w}$, with the parameters $\boldsymbol{\theta}$. This allows -us to identify the momentum parameter and learning rate with the mass of the particle and the viscous drag as: + +Notice that this equation is identical to previous one if we identify +the position of the particle, $\mathbf{w}$, with the parameters +$\boldsymbol{\theta}$. This allows us to identify the momentum +parameter and learning rate with the mass of the particle and the +viscous drag as: + !bt \[ \gamma= {m \over m +\mu \Delta t }, \qquad \eta = {(\Delta t)^2 \over m +\mu \Delta t}. \] !et -Thus, as the name suggests, the momentum parameter is proportional to the mass of the particle and effectively provides inertia. Furthermore, in the large viscosity/small learning rate limit, our memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$. -Why is momentum useful? SGD momentum helps the gradient descent algorithm gain speed in directions with persistent but small gradients even in the presence of stochasticity, while suppressing oscillations in high-curvature directions. This becomes especially important in situations where the landscape is shallow and flat in some directions and narrow and steep in others. It has been argued that first-order methods (with appropriate initial conditions) can perform comparable to more expensive second order methods, especially in the context of complex deep learning models. +Thus, as the name suggests, the momentum parameter is proportional to +the mass of the particle and effectively provides inertia. +Furthermore, in the large viscosity/small learning rate limit, our +memory time scales as $(1-\gamma)^{-1} \approx m/(\mu \Delta t)$. -These beneficial properties of momentum can sometimes become even more pronounced by using a slight modification of the classical momentum algorithm called Nesterov Accelerated Gradient (NAG). +Why is momentum useful? SGD momentum helps the gradient descent +algorithm gain speed in directions with persistent but small gradients +even in the presence of stochasticity, while suppressing oscillations +in high-curvature directions. This becomes especially important in +situations where the landscape is shallow and flat in some directions +and narrow and steep in others. It has been argued that first-order +methods (with appropriate initial conditions) can perform comparable +to more expensive second order methods, especially in the context of +complex deep learning models. + +These beneficial properties of momentum can sometimes become even more +pronounced by using a slight modification of the classical momentum +algorithm called Nesterov Accelerated Gradient (NAG). + +In the NAG algorithm, rather than calculating the gradient at the +current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one +calculates the gradient at the expected value of the parameters given +our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma +\mathbf{v}_{t-1})$. This yields the NAG update rule -In the NAG algorithm, rather than calculating the gradient at the current parameters, $\nabla_\theta E(\boldsymbol{\theta}_t)$, one calculates the gradient at the expected value of the parameters given our current momentum, $\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1})$. This yields the NAG update rule !bt \begin{align} \mathbf{v}_{t}&=\gamma \mathbf{v}_{t-1}+\eta_{t}\nabla_\theta E(\boldsymbol{\theta}_t +\gamma \mathbf{v}_{t-1}) \nonumber \\ \boldsymbol{\theta}_{t+1}&= \boldsymbol{\theta}_t -\mathbf{v}_{t}. \end{align} !et + One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\gamma$. @@ -965,12 +1012,19 @@ adaptively change the step size to match the landscape without paying the steep computational price of calculating or approximating Hessians. -Recently, a number of methods have been introduced that accomplish this by tracking not only the gradient, but also the second moment of the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and ADAM. +Recently, a number of methods have been introduced that accomplish +this by tracking not only the gradient, but also the second moment of +the gradient. These methods include AdaGrad, AdaDelta, RMS-Prop, and +ADAM. !split ===== RMS prop ===== -In RMS prop, in addition to keeping a running average of the first moment of the gradient, we also keep track of the second moment denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule for RMS prop is given by +In RMS prop, in addition to keeping a running average of the first +moment of the gradient, we also keep track of the second moment +denoted by $\mathbf{s}_t=\mathbb{E}[\mathbf{g}_t^2]$. The update rule +for RMS prop is given by + !bt \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ @@ -978,13 +1032,34 @@ In RMS prop, in addition to keeping a running average of the first moment of the \boldsymbol{\theta}_{t+1}&=&\boldsymbol{\theta}_t - \eta_t { \mathbf{g}_t \over \sqrt{\mathbf{s}_t +\epsilon}}, \nonumber \end{align} !et -where $\beta$ controls the averaging time of the second moment and is typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a small regularization constant to prevent divergences. Multiplication and division by vectors is understood as an element-wise operation. It is clear from this formula that the learning rate is reduced in directions where the norm of the gradient is consistently large. This greatly speeds up the convergence by allowing us to use a larger learning rate for flat directions. + +where $\beta$ controls the averaging time of the second moment and is +typically taken to be about $\beta=0.9$, $\eta_t$ is a learning rate +typically chosen to be $10^{-3}$, and $\epsilon\sim 10^{-8} $ is a +small regularization constant to prevent divergences. Multiplication +and division by vectors is understood as an element-wise operation. It +is clear from this formula that the learning rate is reduced in +directions where the norm of the gradient is consistently large. This +greatly speeds up the convergence by allowing us to use a larger +learning rate for flat directions. !split ===== ADAM optimizer ===== -A related algorithm is the ADAM optimizer. In ADAM, we keep a running average of both the first and second moment of the gradient and use this information to adaptively change the learning rate for different parameters. In addition to keeping a running average of the first and second moments of the gradient (i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and $\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM performs an additional bias correction to account for the fact that we are estimating the first two moments of the gradient using a running average (denoted by the hats in the update rule below). The update rule for ADAM is given by (where multiplication and division are once again understood to be element-wise operations below) +A related algorithm is the ADAM optimizer. In ADAM, we keep a running +average of both the first and second moment of the gradient and use +this information to adaptively change the learning rate for different +parameters. In addition to keeping a running average of the first and +second moments of the gradient +(i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and +$\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM +performs an additional bias correction to account for the fact that we +are estimating the first two moments of the gradient using a running +average (denoted by the hats in the update rule below). The update +rule for ADAM is given by (where multiplication and division are once +again understood to be element-wise operations below) + !bt \begin{align} \mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\ @@ -995,9 +1070,18 @@ A related algorithm is the ADAM optimizer. In ADAM, we keep a running average o \boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\ \end{align} !et -where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and second moment and are typically taken to be $0.9$ and $0.99$ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop. -Like in RMSprop, the effective step size of a parameter depends on the magnitude of its gradient squared. To understand this better, let us rewrite this expression in terms of the variance $\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - (\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The update rule for this parameter is given by +where $\beta_1$ and $\beta_2$ set the memory lifetime of the first and +second moment and are typically taken to be $0.9$ and $0.99$ +respectively, and $\eta$ and $\epsilon$ are identical to RMSprop. + +Like in RMSprop, the effective step size of a parameter depends on the +magnitude of its gradient squared. To understand this better, let us +rewrite this expression in terms of the variance +$\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t - +(\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The +update rule for this parameter is given by + !bt \[ \Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.