lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-09-07 22:05:07 +02:00
parent 20c87e7b95
commit 151ee978c0
57 changed files with 3198 additions and 1922 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week37/week37.do.txt
+85 -36
View File
@@ -51,6 +51,7 @@ o The cost function is convex which guarantees that gradient descent converges f
We revisit an example similar to what we had in the first homework set. We have a function of the type
!bc pycod
import numpy as np
x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
!ec
@@ -857,7 +858,7 @@ o Initialize $r_0 = 0$ (an all-zero vector in $\mathbb{R}^d$).
o At each iteration $t$, update the accumulation:
!bt
\[
r_t =; r_{t-1} + g_t \circ g_t,
r_t = r_{t-1} + g_t \circ g_t,
\]
!et
o Here $g_t \circ g_t$ denotes element-wise square of the gradient vector. $g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.
@@ -892,9 +893,9 @@ Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\dis
===== AdaGrad Properties =====
o AdaGrad automatically tunes the step size for each parameter. Parameters with more *volatile or large gradients* get smaller steps, and those with *small or infrequent gradients* get relatively larger steps
o No manual schedule needed: The accumulation $h_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\eta/\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
o No manual schedule needed: The accumulation $r_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\eta/\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
o Sparse data benefit: For very sparse features, $r_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal
o Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. $O(1/\sqrt{T})$ regret bound) comparable to the best fixed learning rate tuned for the problem
o Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem
It effectively reduces the need to tune $\eta$ by hand.
o Limitations: Because $r_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)
@@ -906,11 +907,11 @@ Addresses AdaGrad’s diminishing learning rate issue.
Uses a decaying average of squared gradients (instead of a cumulative sum):
!bt
\[
v_t = \alpha_2 v_{t-1} + (1-\alpha_2)(\nabla C(\theta_t))^2,
v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2,
\]
!et
with $\alpha_2$ typically $0.9$ (or $0.99$).
o Update: $\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t)$.
with $\rho$ typically $0.9$ (or $0.99$).
o Update: $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t)$.
o Recent gradients have more weight, so $v_t$ adapts to the current landscape.
o Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 -- unpublished.)
@@ -923,7 +924,7 @@ FIGURE: [figures/rmsprop.png, width=600 frac=0.8]
!split
===== Adam Optimizer =====
Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.
Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.
o Fast convergence by smoothing gradients (accelerates in long-term gradient direction).
o Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).
@@ -946,43 +947,91 @@ problems involving lots data and/or parameters. It is a combination of the
gradient descent with momentum algorithm and the RMSprop algorithm
discussed above.
In addition to keeping a running average of the first and
second moments of the gradient
(i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and
$\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM
performs an additional bias correction to account for the fact that we
are estimating the first two moments of the gradient using a running
average (denoted by the hats in the update rule below). The update
rule for ADAM is given by (where multiplication and division are once
again understood to be element-wise operations below)
!bt
\begin{align}
\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\
\end{align}
!et
!split
===== Why Combine Momentum and RMSProp? =====
where $\theta_1$ and $\theta_2$ set the memory lifetime of the first and
second moment and are typically taken to be $0.9$ and $0.99$
respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
o Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).
o Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).
o Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)
o Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)
Like in RMSprop, the effective step size of a parameter depends on the
magnitude of its gradient squared. To understand this better, let us
rewrite this expression in terms of the variance
$\boldsymbol{\sigma}_t^2 = \bm{\mathbf{s}}_t -
(\bm{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The
update rule for this parameter is given by
Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice
!split
===== Adam: Exponential Moving Averages (Moments) =====
Adam maintains two moving averages at each time step $t$ for each parameter $w$:
!bblock First moment (mean) $m_t$
The Momentum term
!bt
\[
\Delta \theta_{t+1}= -\eta_t { \bm{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
m_t = \beta_1m_{t-1} + (1-\beta_1)\, \nabla C(\theta_t),
\]
!et
!eblock
!bblock Second moment (uncentered variance) $v_t$
The RMS term
!bt
\[
v_t = \beta_2v_{t-1} + (1-\beta_2)(\nabla C(\theta_t))^2,
\]
!et
with typical $\beta_1 = 0.9$, $\beta_2 = 0.999$. Initialize $m_0 = 0$, $v_0 = 0$.
!eblock
These are _biased_ estimators of the true first and second moment of the gradients, especially at the start (since $m_0,v_0$ are zero)
!split
===== Adam: Bias Correction =====
To counteract initialization bias in $m_t, v_t$, Adam computes bias-corrected estimates
!bt
\[
\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}.
\]
!et
* When $t$ is small, $1-\beta_i^t \approx 0$, so $\hat{m}_t, \hat{v}_t$ significantly larger than raw $m_t, v_t$, compensating for the initial zero bias.
* As $t$ increases, $1-\beta_i^t \to 1$, and $\hat{m}_t, \hat{v}_t$ converge to $m_t, v_t$.
* Bias correction is important for Adam’s stability in early iterations
!split
===== Adam: Update Rule Derivation =====
Finally, Adam updates parameters using the bias-corrected moments:
!bt
\[
\theta_{t+1} =\theta_t -\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}\hat{m}_t,
\]
!et
where $\epsilon$ is a small constant (e.g. $10^{-8}$) to prevent division by zero.
Breaking it down:
o Compute gradient $\nabla C(\theta_t)$.
o Update first moment $m_t$ and second moment $v_t$ (exponential moving averages).
o Bias-correct: $\hat{m}_t = m_t/(1-\beta_1^t)$, $\; \hat{v}_t = v_t/(1-\beta_2^t)$.
o Compute step: $\Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$.
o Update parameters: $\theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t$.
This is the Adam update rule as given in the original paper.
!split
===== Adam vs. AdaGrad and RMSProp =====
o AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)
o RMSProp: Uses moving average of squared gradients (like Adam’s $v_t$) to maintain adaptive learning rates, but does not include momentum or bias-correction.
o Adam: Effectively RMSProp + Momentum + Bias-correction
* Momentum ($m_t$) provides acceleration and smoother convergence.
* Adaptive $v_t$ scaling moderates the step size per dimension.
* Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.
In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone
!split
===== Adaptivity Across Dimensions =====
o Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.
o This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.
o Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction.
===== ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" =====