lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

replacing hats with boldface

Browse Source
This commit is contained in:
mhjensen
2020-09-25 05:19:28 +02:00
parent 873469a414
commit f7578d8e14
29 changed files with 528 additions and 528 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week39/week39.do.txt
+87 -87
View File
@@ -41,33 +41,33 @@ defined probabilities
!bt
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
p(y_i=1|x_i,\bm{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\bm{\beta}) &= 1 - p(y_i=1|x_i,\bm{\beta}),
\end{align*}
!et
where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.
where $\bm{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.
!split
===== The equations to solve =====
Our compact equations used a definition of a vector $\hat{y}$ with $n$
elements $y_i$, an $n\times p$ matrix $\hat{X}$ which contains the
$x_i$ values and a vector $\hat{p}$ of fitted probabilities
$p(y_i\vert x_i,\hat{\beta})$. We rewrote in a more compact form
Our compact equations used a definition of a vector $\bm{y}$ with $n$
elements $y_i$, an $n\times p$ matrix $\bm{X}$ which contains the
$x_i$ values and a vector $\bm{p}$ of fitted probabilities
$p(y_i\vert x_i,\bm{\beta})$. We rewrote in a more compact form
the first derivative of the cost function as
!bt
\[
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}} = -\bm{X}^T\left(\bm{y}-\bm{p}\right).
\]
!et
If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as
If we in addition define a diagonal matrix $\bm{W}$ with elements
$p(y_i\vert x_i,\bm{\beta})(1-p(y_i\vert x_i,\bm{\beta})$, we can obtain a compact expression of the second derivative as
!bt
\[
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} = \bm{X}^T\bm{W}\bm{X}.
\]
!et
This defines what is called the Hessian matrix.
@@ -81,14 +81,14 @@ Our iterative scheme is then given by
!bt
\[
\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T}\right)^{-1}_{\hat{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}}\right)_{\hat{\beta}^{\mathrm{old}}},
\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\frac{\partial^2 \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T}\right)^{-1}_{\bm{\beta}^{\mathrm{old}}}\times \left(\frac{\partial \mathcal{C}(\bm{\beta})}{\partial \bm{\beta}}\right)_{\bm{\beta}^{\mathrm{old}}},
\]
!et
or in matrix form as
!bt
\[
\hat{\beta}^{\mathrm{new}} = \hat{\beta}^{\mathrm{old}}-\left(\hat{X}^T\hat{W}\hat{X} \right)^{-1}\times \left(-\hat{X}^T(\hat{y}-\hat{p}) \right)_{\hat{\beta}^{\mathrm{old}}}.
\bm{\beta}^{\mathrm{new}} = \bm{\beta}^{\mathrm{old}}-\left(\bm{X}^T\bm{W}\bm{X} \right)^{-1}\times \left(-\bm{X}^T(\bm{y}-\bm{p}) \right)_{\bm{\beta}^{\mathrm{old}}}.
\]
!et
The right-hand side is computed with the old values of $\beta$.
@@ -189,10 +189,10 @@ which we Taylor expand to obtain
\end{array}.
\]
!et
Defining the Jacobian matrix ${\bf \hat{J}}$ we have
Defining the Jacobian matrix ${\bf \bm{J}}$ we have
!bt
\[
{\bf \hat{J}}=\left( \begin{array}{cc}
{\bf \bm{J}}=\left( \begin{array}{cc}
\partial f_1/\partial x_1 & \partial f_1/\partial x_2 \\
\partial f_2/\partial x_1 &\partial f_2/\partial x_2
\end{array} \right),
@@ -210,13 +210,13 @@ where we have defined
!bt
\[
\left(\begin{array}{c} h_1^{n} \\ h_2^{n} \end{array} \right)=
-{\bf \hat{J}}^{-1}
-{\bf \bm{J}}^{-1}
\left(\begin{array}{c} f_1(x_1^{n},x_2^{n}) \\ f_2(x_1^{n},x_2^{n}) \end{array} \right).
\]
!et
We need thus to compute the inverse of the Jacobian matrix and it
is to understand that difficulties may
arise in case ${\bf \hat{J}}$ is nearly singular.
arise in case ${\bf \bm{J}}$ is nearly singular.
It is rather straightforward to extend the above scheme to systems of
more than two non-linear equations. In our case, the Jacobian matrix is given by the Hessian that represents the second derivative of cost function.
@@ -391,7 +391,7 @@ Using the definition of convexity, try to show that a function satisfying the pr
Before we proceed, we would like to discuss the approach called the
_standard Steepest descent_, which again leads to us having to be able
_standard Steepest descent_ (different from the above steepest descent discussion), which again leads to us having to be able
to compute a matrix. It belongs to the class of Conjugate Gradient methods (CG).
"The success of the CG method":"https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf"
@@ -401,7 +401,7 @@ the class of iterative methods for solving problems from linear
algebra of the type
!bt
\begin{equation*}
\hat{A}\hat{x} = \hat{b}.
\bm{A}\bm{x} = \bm{b}.
\end{equation*}
!et
@@ -409,12 +409,12 @@ In the iterative process we end up with a problem like
!bt
\begin{equation*}
\hat{r}= \hat{b}-\hat{A}\hat{x},
\bm{r}= \bm{b}-\bm{A}\bm{x},
\end{equation*}
!et
where $\hat{r}$ is the so-called residual or error in the iterative process.
where $\bm{r}$ is the so-called residual or error in the iterative process.
When we have found the exact solution, $\hat{r}=0$.
When we have found the exact solution, $\bm{r}=0$.
!split
===== Gradient method =====
@@ -422,28 +422,28 @@ When we have found the exact solution, $\hat{r}=0$.
The residual is zero when we reach the minimum of the quadratic equation
!bt
\begin{equation*}
P(\hat{x})=\frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T\hat{b},
P(\bm{x})=\frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T\bm{b},
\end{equation*}
!et
with the constraint that the matrix $\hat{A}$ is positive definite and
with the constraint that the matrix $\bm{A}$ is positive definite and
symmetric. This defines also the Hessian and we want it to be positive definite.
!split
===== Steepest descent method =====
We denote the initial guess for $\hat{x}$ as $\hat{x}_0$.
We denote the initial guess for $\bm{x}$ as $\bm{x}_0$.
We can assume without loss of generality that
!bt
\begin{equation*}
\hat{x}_0=0,
\bm{x}_0=0,
\end{equation*}
!et
or consider the system
!bt
\begin{equation*}
\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0,
\bm{A}\bm{z} = \bm{b}-\bm{A}\bm{x}_0,
\end{equation*}
!et
instead.
@@ -452,22 +452,22 @@ instead.
!split
===== Steepest descent method =====
!bblock
One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form
One can show that the solution $\bm{x}$ is also the unique minimizer of the quadratic form
!bt
\begin{equation*}
f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n.
f(\bm{x}) = \frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T \bm{x} , \quad \bm{x}\in\mathbf{R}^n.
\end{equation*}
!et
This suggests taking the first basis vector $\hat{r}_1$ (see below for definition)
to be the gradient of $f$ at $\hat{x}=\hat{x}_0$,
This suggests taking the first basis vector $\bm{r}_1$ (see below for definition)
to be the gradient of $f$ at $\bm{x}=\bm{x}_0$,
which equals
!bt
\begin{equation*}
\hat{A}\hat{x}_0-\hat{b},
\bm{A}\bm{x}_0-\bm{b},
\end{equation*}
!et
and
$\hat{x}_0=0$ it is equal $-\hat{b}$.
$\bm{x}_0=0$ it is equal $-\bm{b}$.
!eblock
@@ -477,32 +477,32 @@ $\hat{x}_0=0$ it is equal $-\hat{b}$.
We can compute the residual iteratively as
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1},
\bm{r}_{k+1}=\bm{b}-\bm{A}\bm{x}_{k+1},
\end{equation*}
!et
which equals
!bt
\begin{equation*}
\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{r}_k),
\bm{b}-\bm{A}(\bm{x}_k+\alpha_k\bm{r}_k),
\end{equation*}
!et
or
!bt
\begin{equation*}
(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{r}_k,
(\bm{b}-\bm{A}\bm{x}_k)-\alpha_k\bm{A}\bm{r}_k,
\end{equation*}
!et
which gives
!bt
\[
\alpha_k = \frac{\hat{r}_k^T\hat{r}_k}{\hat{r}_k^T\hat{A}\hat{r}_k}
\alpha_k = \frac{\bm{r}_k^T\bm{r}_k}{\bm{r}_k^T\bm{A}\bm{r}_k}
\]
!et
leading to the iterative scheme
!bt
\begin{equation*}
\hat{x}_{k+1}=\hat{x}_k-\alpha_k\hat{r}_{k},
\bm{x}_{k+1}=\bm{x}_k-\alpha_k\bm{r}_{k},
\end{equation*}
!et
!eblock
@@ -567,23 +567,23 @@ pt.plot(it_array.T[0], it_array.T[1], "x-")
===== Conjugate gradient method =====
!bblock
In the CG method we define so-called conjugate directions and two vectors
$\hat{s}$ and $\hat{t}$
$\bm{s}$ and $\bm{t}$
are said to be
conjugate if
!bt
\begin{equation*}
\hat{s}^T\hat{A}\hat{t}= 0.
\bm{s}^T\bm{A}\bm{t}= 0.
\end{equation*}
!et
The philosophy of the CG method is to perform searches in various conjugate directions
of our vectors $\hat{x}_i$ obeying the above criterion, namely
of our vectors $\bm{x}_i$ obeying the above criterion, namely
!bt
\begin{equation*}
\hat{x}_i^T\hat{A}\hat{x}_j= 0.
\bm{x}_i^T\bm{A}\bm{x}_j= 0.
\end{equation*}
!et
Two vectors are conjugate if they are orthogonal with respect to
this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is conjugate to $\hat{t}$, then $\hat{t}$ is conjugate to $\hat{s}$.
this inner product. Being conjugate is a symmetric relation: if $\bm{s}$ is conjugate to $\bm{t}$, then $\bm{t}$ is conjugate to $\bm{s}$.
!eblock
!split
@@ -592,7 +592,7 @@ this inner product. Being conjugate is a symmetric relation: if $\hat{s}$ is con
An example is given by the eigenvectors of the matrix
!bt
\begin{equation*}
\hat{v}_i^T\hat{A}\hat{v}_j= \lambda\hat{v}_i^T\hat{v}_j,
\bm{v}_i^T\bm{A}\bm{v}_j= \lambda\bm{v}_i^T\bm{v}_j,
\end{equation*}
!et
which is zero unless $i=j$.
@@ -602,20 +602,20 @@ which is zero unless $i=j$.
!split
===== Conjugate gradient method =====
!bblock
Assume now that we have a symmetric positive-definite matrix $\hat{A}$ of size
Assume now that we have a symmetric positive-definite matrix $\bm{A}$ of size
$n\times n$. At each iteration $i+1$ we obtain the conjugate direction of a vector
!bt
\begin{equation*}
\hat{x}_{i+1}=\hat{x}_{i}+\alpha_i\hat{p}_{i}.
\bm{x}_{i+1}=\bm{x}_{i}+\alpha_i\bm{p}_{i}.
\end{equation*}
!et
We assume that $\hat{p}_{i}$ is a sequence of $n$ mutually conjugate directions.
Then the $\hat{p}_{i}$ form a basis of $R^n$ and we can expand the solution
$ \hat{A}\hat{x} = \hat{b}$ in this basis, namely
We assume that $\bm{p}_{i}$ is a sequence of $n$ mutually conjugate directions.
Then the $\bm{p}_{i}$ form a basis of $R^n$ and we can expand the solution
$ \bm{A}\bm{x} = \bm{b}$ in this basis, namely
!bt
\begin{equation*}
\hat{x} = \sum^{n}_{i=1} \alpha_i \hat{p}_i.
\bm{x} = \sum^{n}_{i=1} \alpha_i \bm{p}_i.
\end{equation*}
!et
!eblock
@@ -629,18 +629,18 @@ The coefficients are given by
\mathbf{A}\mathbf{x} = \sum^{n}_{i=1} \alpha_i \mathbf{A} \mathbf{p}_i = \mathbf{b}.
\end{equation*}
!et
Multiplying with $\hat{p}_k^T$ from the left gives
Multiplying with $\bm{p}_k^T$ from the left gives
!bt
\begin{equation*}
\hat{p}_k^T \hat{A}\hat{x} = \sum^{n}_{i=1} \alpha_i\hat{p}_k^T \hat{A}\hat{p}_i= \hat{p}_k^T \hat{b},
\bm{p}_k^T \bm{A}\bm{x} = \sum^{n}_{i=1} \alpha_i\bm{p}_k^T \bm{A}\bm{p}_i= \bm{p}_k^T \bm{b},
\end{equation*}
!et
and we can define the coefficients $\alpha_k$ as
!bt
\begin{equation*}
\alpha_k = \frac{\hat{p}_k^T \hat{b}}{\hat{p}_k^T \hat{A} \hat{p}_k}
\alpha_k = \frac{\bm{p}_k^T \bm{b}}{\bm{p}_k^T \bm{A} \bm{p}_k}
\end{equation*}
!et
!eblock
@@ -649,24 +649,24 @@ and we can define the coefficients $\alpha_k$ as
===== Conjugate gradient method and iterations =====
!bblock
If we choose the conjugate vectors $\hat{p}_k$ carefully,
If we choose the conjugate vectors $\bm{p}_k$ carefully,
then we may not need all of them to obtain a good approximation to the solution
$\hat{x}$.
$\bm{x}$.
We want to regard the conjugate gradient method as an iterative method.
This will us to solve systems where $n$ is so large that the direct
method would take too much time.
We denote the initial guess for $\hat{x}$ as $\hat{x}_0$.
We denote the initial guess for $\bm{x}$ as $\bm{x}_0$.
We can assume without loss of generality that
!bt
\begin{equation*}
\hat{x}_0=0,
\bm{x}_0=0,
\end{equation*}
!et
or consider the system
!bt
\begin{equation*}
\hat{A}\hat{z} = \hat{b}-\hat{A}\hat{x}_0,
\bm{A}\bm{z} = \bm{b}-\bm{A}\bm{x}_0,
\end{equation*}
!et
instead.
@@ -676,22 +676,22 @@ instead.
!split
===== Conjugate gradient method =====
!bblock
One can show that the solution $\hat{x}$ is also the unique minimizer of the quadratic form
One can show that the solution $\bm{x}$ is also the unique minimizer of the quadratic form
!bt
\begin{equation*}
f(\hat{x}) = \frac{1}{2}\hat{x}^T\hat{A}\hat{x} - \hat{x}^T \hat{x} , \quad \hat{x}\in\mathbf{R}^n.
f(\bm{x}) = \frac{1}{2}\bm{x}^T\bm{A}\bm{x} - \bm{x}^T \bm{x} , \quad \bm{x}\in\mathbf{R}^n.
\end{equation*}
!et
This suggests taking the first basis vector $\hat{p}_1$
to be the gradient of $f$ at $\hat{x}=\hat{x}_0$,
This suggests taking the first basis vector $\bm{p}_1$
to be the gradient of $f$ at $\bm{x}=\bm{x}_0$,
which equals
!bt
\begin{equation*}
\hat{A}\hat{x}_0-\hat{b},
\bm{A}\bm{x}_0-\bm{b},
\end{equation*}
!et
and
$\hat{x}_0=0$ it is equal $-\hat{b}$.
$\bm{x}_0=0$ it is equal $-\bm{b}$.
The other vectors in the basis will be conjugate to the gradient,
hence the name conjugate gradient method.
!eblock
@@ -700,22 +700,22 @@ hence the name conjugate gradient method.
!split
===== Conjugate gradient method =====
!bblock
Let $\hat{r}_k$ be the residual at the $k$-th step:
Let $\bm{r}_k$ be the residual at the $k$-th step:
!bt
\begin{equation*}
\hat{r}_k=\hat{b}-\hat{A}\hat{x}_k.
\bm{r}_k=\bm{b}-\bm{A}\bm{x}_k.
\end{equation*}
!et
Note that $\hat{r}_k$ is the negative gradient of $f$ at
$\hat{x}=\hat{x}_k$,
so the gradient descent method would be to move in the direction $\hat{r}_k$.
Here, we insist that the directions $\hat{p}_k$ are conjugate to each other,
so we take the direction closest to the gradient $\hat{r}_k$
Note that $\bm{r}_k$ is the negative gradient of $f$ at
$\bm{x}=\bm{x}_k$,
so the gradient descent method would be to move in the direction $\bm{r}_k$.
Here, we insist that the directions $\bm{p}_k$ are conjugate to each other,
so we take the direction closest to the gradient $\bm{r}_k$
under the conjugacy constraint.
This gives the following expression
!bt
\begin{equation*}
\hat{p}_{k+1}=\hat{r}_k-\frac{\hat{p}_k^T \hat{A}\hat{r}_k}{\hat{p}_k^T\hat{A}\hat{p}_k} \hat{p}_k.
\bm{p}_{k+1}=\bm{r}_k-\frac{\bm{p}_k^T \bm{A}\bm{r}_k}{\bm{p}_k^T\bm{A}\bm{p}_k} \bm{p}_k.
\end{equation*}
!et
!eblock
@@ -726,26 +726,26 @@ This gives the following expression
We can also compute the residual iteratively as
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{b}-\hat{A}\hat{x}_{k+1},
\bm{r}_{k+1}=\bm{b}-\bm{A}\bm{x}_{k+1},
\end{equation*}
!et
which equals
!bt
\begin{equation*}
\hat{b}-\hat{A}(\hat{x}_k+\alpha_k\hat{p}_k),
\bm{b}-\bm{A}(\bm{x}_k+\alpha_k\bm{p}_k),
\end{equation*}
!et
or
!bt
\begin{equation*}
(\hat{b}-\hat{A}\hat{x}_k)-\alpha_k\hat{A}\hat{p}_k,
(\bm{b}-\bm{A}\bm{x}_k)-\alpha_k\bm{A}\bm{p}_k,
\end{equation*}
!et
which gives
!bt
\begin{equation*}
\hat{r}_{k+1}=\hat{r}_k-\hat{A}\hat{p}_{k},
\bm{r}_{k+1}=\bm{r}_k-\bm{A}\bm{p}_{k},
\end{equation*}
!et
!eblock
@@ -776,22 +776,22 @@ with $x_i \in [0,1] $ chosen randomly with a uniform distribution. Additionally
The linear regression model is given by
!bt
\[
h_\beta(x) = \hat{y} = \beta_0 + \beta_1 x,
h_\beta(x) = \bm{y} = \beta_0 + \beta_1 x,
\]
!et
such that
!bt
\[
\hat{y}_i = \beta_0 + \beta_1 x_i.
\bm{y}_i = \beta_0 + \beta_1 x_i.
\]
!et
!split
===== Gradient descent example =====
Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\hat{y}} = (\hat{y}_1,\cdots,\hat{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$
Let $\mathbf{y} = (y_1,\cdots,y_n)^T$, $\mathbf{\bm{y}} = (\bm{y}_1,\cdots,\bm{y}_n)^T$ and $\beta = (\beta_0, \beta_1)^T$
It is convenient to write $\mathbf{\hat{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by
It is convenient to write $\mathbf{\bm{y}} = X\beta$ where $X \in \mathbb{R}^{100 \times 2} $ is the design matrix given by
!bt
\[
X \equiv \begin{bmatrix}
@@ -827,7 +827,7 @@ where $X$ is the design matrix defined above.
The Hessian matrix of $C(\beta)$ is given by
!bt
\[
\hat{H} \equiv \begin{bmatrix}
\bm{H} \equiv \begin{bmatrix}
\frac{\partial^2 C(\beta)}{\partial \beta_0^2} &; \frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} \\
\frac{\partial^2 C(\beta)}{\partial \beta_0 \partial \beta_1} &; \frac{\partial^2 C(\beta)}{\partial \beta_1^2} &; \\
\end{bmatrix} = 2X^T X.
@@ -1447,9 +1447,9 @@ again understood to be element-wise operations below)
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
\mathbf{m}_t &= \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\beta_2 \mathbf{s}_{t-1} +(1-\beta_2)\mathbf{g}_t^2 \nonumber \\
\hat{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\hat{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \hat{\mathbf{m}}_t \over \sqrt{\hat{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\beta_1^t} \nonumber \\
\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\beta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\end{align}
!et
@@ -1460,13 +1460,13 @@ respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
Like in RMSprop, the effective step size of a parameter depends on the
magnitude of its gradient squared. To understand this better, let us
rewrite this expression in terms of the variance
$\boldsymbol{\sigma}_t^2 = \hat{\mathbf{s}}_t -
(\hat{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The
$\boldsymbol{\sigma}_t^2 = \bm{\mathbf{s}}_t -
(\bm{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The
update rule for this parameter is given by
!bt
\[
\Delta \theta_{t+1}= -\eta_t { \hat{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
\Delta \theta_{t+1}= -\eta_t { \bm{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
\]
!et