x = 2*np.random.rand(m,1)
+ import numpy as np
+x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
-
@@ -367,7 +394,7 @@ $$
- 45
- 46
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs037.html b/doc/pub/week37/html/._week37-bs037.html index 1b1a97fa2..53c5d12df 100644 --- a/doc/pub/week37/html/._week37-bs037.html +++ b/doc/pub/week37/html/._week37-bs037.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -365,7 +392,7 @@ $$
- 46
- 47
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs038.html b/doc/pub/week37/html/._week37-bs038.html index 0c701d227..8cbd77ca4 100644 --- a/doc/pub/week37/html/._week37-bs038.html +++ b/doc/pub/week37/html/._week37-bs038.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -323,9 +350,9 @@ MathJax.Hub.Config({
- AdaGrad automatically tunes the step size for each parameter. Parameters with more volatile or large gradients get smaller steps, and those with small or infrequent gradients get relatively larger steps -
- No manual schedule needed: The accumulation \( h_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate. +
- No manual schedule needed: The accumulation \( r_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
- Sparse data benefit: For very sparse features, \( r_{t,j} \) grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal -
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. \( O(1/\sqrt{T}) \) regret bound) comparable to the best fixed learning rate tuned for the problem +
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem
- 47
- 48
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs039.html b/doc/pub/week37/html/._week37-bs039.html index 54be03a14..19306d445 100644 --- a/doc/pub/week37/html/._week37-bs039.html +++ b/doc/pub/week37/html/._week37-bs039.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -325,12 +352,12 @@ MathJax.Hub.Config({ Uses a decaying average of squared gradients (instead of a cumulative sum): $$ -v_t = \alpha_2 v_{t-1} + (1-\alpha_2)(\nabla C(\theta_t))^2, +v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2, $$ -
- Update: \( \theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \). +
- Update: \( \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \).
- Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
- Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
- 48
- 49
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs040.html b/doc/pub/week37/html/._week37-bs040.html index f3953990b..9f854ada3 100644 --- a/doc/pub/week37/html/._week37-bs040.html +++ b/doc/pub/week37/html/._week37-bs040.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -321,7 +348,7 @@ MathJax.Hub.Config({
- Fast convergence by smoothing gradients (accelerates in long-term gradient direction). @@ -356,7 +383,7 @@ MathJax.Hub.Config({
- 49
- 50
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs041.html b/doc/pub/week37/html/._week37-bs041.html index 5c5afb175..efa04d70a 100644 --- a/doc/pub/week37/html/._week37-bs041.html +++ b/doc/pub/week37/html/._week37-bs041.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -330,54 +357,6 @@ gradient descent with momentum algorithm and the RMSprop algorithm discussed above. -
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,11 +346,15 @@ MathJax.Hub.Config({
- Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction). +
- Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients). +
- Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients) +
- Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations) +
- 51
- 52
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs043.html b/doc/pub/week37/html/._week37-bs043.html index 2959b7480..6d0ceebc4 100644 --- a/doc/pub/week37/html/._week37-bs043.html +++ b/doc/pub/week37/html/._week37-bs043.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,14 +346,32 @@ MathJax.Hub.Config({
- Randomize the data when making mini-batches. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented. -
- Transform your inputs. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the cost function is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case. -
- Monitor the out-of-sample performance. Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This early stopping significantly improves performance in many settings. -
- Adaptive optimization methods don't always have good generalization. Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications. -
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,92 +346,17 @@ MathJax.Hub.Config({
- When \( t \) is small, \( 1-\beta_i^t \approx 0 \), so \( \hat{m}_t, \hat{v}_t \) significantly larger than raw \( m_t, v_t \), compensating for the initial zero bias. +
- As \( t \) increases, \( 1-\beta_i^t \to 1 \), and \( \hat{m}_t, \hat{v}_t \) converge to \( m_t, v_t \). +
- Bias correction is important for Adam’s stability in early iterations +
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,82 +346,23 @@ MathJax.Hub.Config({
- Compute gradient \( \nabla C(\theta_t) \). +
- Update first moment \( m_t \) and second moment \( v_t \) (exponential moving averages). +
- Bias-correct: \( \hat{m}_t = m_t/(1-\beta_1^t) \), \( \; \hat{v}_t = v_t/(1-\beta_2^t) \). +
- Compute step: \( \Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \). +
- Update parameters: \( \theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t \). +
- 55
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs046.html b/doc/pub/week37/html/._week37-bs046.html index f8e5b041a..55eb231b0 100644 --- a/doc/pub/week37/html/._week37-bs046.html +++ b/doc/pub/week37/html/._week37-bs046.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,108 +346,19 @@ MathJax.Hub.Config({
- AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting) +
- RMSProp: Uses moving average of squared gradients (like Adam’s \( v_t \)) to maintain adaptive learning rates, but does not include momentum or bias-correction. +
- Adam: Effectively RMSProp + Momentum + Bias-correction +
- Momentum (\( m_t \)) provides acceleration and smoother convergence. +
- Adaptive \( v_t \) scaling moderates the step size per dimension. +
- Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on. +
- 56
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs047.html b/doc/pub/week37/html/._week37-bs047.html index d570040e5..97c36f3e9 100644 --- a/doc/pub/week37/html/._week37-bs047.html +++ b/doc/pub/week37/html/._week37-bs047.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,96 +346,20 @@ MathJax.Hub.Config({
- Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps. +
- This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems. +
- Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction. +
- 56
- 57 +
- ... +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs048.html b/doc/pub/week37/html/._week37-bs048.html index c893b5a1f..dfcc79107 100644 --- a/doc/pub/week37/html/._week37-bs048.html +++ b/doc/pub/week37/html/._week37-bs048.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,64 +346,11 @@ MathJax.Hub.Config({
- 55
- 56
- 57 +
- 58 +
- ... +
- 63
- » diff --git a/doc/pub/week37/html/._week37-bs049.html b/doc/pub/week37/html/._week37-bs049.html index 0de04ba98..a89883cd2 100644 --- a/doc/pub/week37/html/._week37-bs049.html +++ b/doc/pub/week37/html/._week37-bs049.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -319,79 +346,14 @@ MathJax.Hub.Config({
- Randomize the data when making mini-batches. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented. +
- Transform your inputs. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the cost function is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case. +
- Monitor the out-of-sample performance. Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This early stopping significantly improves performance in many settings. +
- Adaptive optimization methods don't always have good generalization. Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications. +
- Using gradient descent methods, limitations
- Momentum based GD
- Improving gradient descent with momentum -
- Same code but now with momentum gradient descent +
- Same code but now with momentum gradient descent
- Overview video on Stochastic Gradient Descent (SGD)
- Batches and mini-batches
- Pros and cons @@ -292,22 +313,28 @@ MathJax.Hub.Config({
- RMSProp algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html"
- Adam Optimizer
- "ADAM optimizer":"https://arxiv.org/abs/1412.6980" -
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" -
- Algorithms and codes for Adagrad, RMSprop and Adam -
- Practical tips -
- Sneaking in automatic differentiation using Autograd -
- Same code but now with momentum gradient descent -
- Including Stochastic Gradient Descent with Autograd -
- Same code but now with momentum gradient descent -
- But none of these can compete with Newton's method -
- Similar (second order function now) problem but now with AdaGrad -
- RMSprop for adaptive learning rate with Stochastic Gradient Descent -
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" -
- Material for the lab sessions -
- Reminder on different scaling methods -
- Functionality in Scikit-Learn -
- More preprocessing -
- Frequently used scaling functions +
- Why Combine Momentum and RMSProp? +
- Adam: Exponential Moving Averages (Moments) +
- Adam: Bias Correction +
- Adam: Update Rule Derivation +
- Adam vs. AdaGrad and RMSProp +
- Adaptivity Across Dimensions +
- ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" +
- Algorithms and codes for Adagrad, RMSprop and Adam +
- Practical tips +
- Sneaking in automatic differentiation using Autograd +
- Same code but now with momentum gradient descent +
- Including Stochastic Gradient Descent with Autograd +
- Same code but now with momentum gradient descent +
- But none of these can compete with Newton's method +
- Similar (second order function now) problem but now with AdaGrad +
- RMSprop for adaptive learning rate with Stochastic Gradient Descent +
- And finally "ADAM":"https://arxiv.org/pdf/1412.6980.pdf" +
- Material for the lab sessions +
- Reminder on different scaling methods +
- Functionality in Scikit-Learn +
- More preprocessing +
- Frequently used scaling functions @@ -361,7 +388,7 @@ MathJax.Hub.Config({
- 9
- 10
- ... -
- 57 +
- 63
- » diff --git a/doc/pub/week37/html/week37-reveal.html b/doc/pub/week37/html/week37-reveal.html index f7c498920..841138031 100644 --- a/doc/pub/week37/html/week37-reveal.html +++ b/doc/pub/week37/html/week37-reveal.html @@ -253,7 +253,8 @@ desirable properties such as:
- AdaGrad automatically tunes the step size for each parameter. Parameters with more volatile or large gradients get smaller steps, and those with small or infrequent gradients get relatively larger steps -
- No manual schedule needed: The accumulation \( h_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate. +
- No manual schedule needed: The accumulation \( r_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
- Sparse data benefit: For very sparse features, \( r_{t,j} \) grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal -
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. \( O(1/\sqrt{T}) \) regret bound) comparable to the best fixed learning rate tuned for the problem +
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem
- Update: \( \theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \). +
- Update: \( \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \).
- Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
- Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
- Fast convergence by smoothing gradients (accelerates in long-term gradient direction). @@ -1371,51 +1372,120 @@ problems involving lots data and/or parameters. It is a combination of the gradient descent with momentum algorithm and the RMSprop algorithm discussed above. +
- Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction). +
- Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients). +
- Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients) +
- Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations) +
- When \( t \) is small, \( 1-\beta_i^t \approx 0 \), so \( \hat{m}_t, \hat{v}_t \) significantly larger than raw \( m_t, v_t \), compensating for the initial zero bias. +
- As \( t \) increases, \( 1-\beta_i^t \to 1 \), and \( \hat{m}_t, \hat{v}_t \) converge to \( m_t, v_t \). +
- Bias correction is important for Adam’s stability in early iterations +
- Compute gradient \( \nabla C(\theta_t) \). +
- Update first moment \( m_t \) and second moment \( v_t \) (exponential moving averages). +
- Bias-correct: \( \hat{m}_t = m_t/(1-\beta_1^t) \), \( \; \hat{v}_t = v_t/(1-\beta_2^t) \). +
- Compute step: \( \Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \). +
- Update parameters: \( \theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t \). +
- AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting) +
- RMSProp: Uses moving average of squared gradients (like Adam’s \( v_t \)) to maintain adaptive learning rates, but does not include momentum or bias-correction. +
- Adam: Effectively RMSProp + Momentum + Bias-correction +
- Momentum (\( m_t \)) provides acceleration and smoother convergence. -
- Adaptive \( v_t \) scaling moderates the step size per dimension. + +
- Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on. +
- Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps. +
- This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems. +
- Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction. +
- At each iteration \( t \), update the accumulation: $$ -r_t =; r_{t-1} + g_t \circ g_t, +r_t = r_{t-1} + g_t \circ g_t, $$
- AdaGrad automatically tunes the step size for each parameter. Parameters with more volatile or large gradients get smaller steps, and those with small or infrequent gradients get relatively larger steps -
- No manual schedule needed: The accumulation \( h_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate. +
- No manual schedule needed: The accumulation \( r_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
- Sparse data benefit: For very sparse features, \( r_{t,j} \) grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal -
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. \( O(1/\sqrt{T}) \) regret bound) comparable to the best fixed learning rate tuned for the problem +
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem
- Update: \( \theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \). +
- Update: \( \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \).
- Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
- Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
- Fast convergence by smoothing gradients (accelerates in long-term gradient direction). @@ -1378,46 +1400,96 @@ gradient descent with momentum algorithm and the RMSprop algorithm discussed above. -
- Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction). +
- Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients). +
- Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients) +
- Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations) +
- When \( t \) is small, \( 1-\beta_i^t \approx 0 \), so \( \hat{m}_t, \hat{v}_t \) significantly larger than raw \( m_t, v_t \), compensating for the initial zero bias. +
- As \( t \) increases, \( 1-\beta_i^t \to 1 \), and \( \hat{m}_t, \hat{v}_t \) converge to \( m_t, v_t \). +
- Bias correction is important for Adam’s stability in early iterations +
- Compute gradient \( \nabla C(\theta_t) \). +
- Update first moment \( m_t \) and second moment \( v_t \) (exponential moving averages). +
- Bias-correct: \( \hat{m}_t = m_t/(1-\beta_1^t) \), \( \; \hat{v}_t = v_t/(1-\beta_2^t) \). +
- Compute step: \( \Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \). +
- Update parameters: \( \theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t \). +
- AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting) +
- RMSProp: Uses moving average of squared gradients (like Adam’s \( v_t \)) to maintain adaptive learning rates, but does not include momentum or bias-correction. +
- Adam: Effectively RMSProp + Momentum + Bias-correction +
- Momentum (\( m_t \)) provides acceleration and smoother convergence. +
- Adaptive \( v_t \) scaling moderates the step size per dimension. +
- Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on. +
- Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps. +
- This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems. +
- Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction. +
- At each iteration \( t \), update the accumulation: $$ -r_t =; r_{t-1} + g_t \circ g_t, +r_t = r_{t-1} + g_t \circ g_t, $$
- AdaGrad automatically tunes the step size for each parameter. Parameters with more volatile or large gradients get smaller steps, and those with small or infrequent gradients get relatively larger steps -
- No manual schedule needed: The accumulation \( h_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate. +
- No manual schedule needed: The accumulation \( r_t \) keeps increasing (or stays the same if gradient is zero), so step sizes \( \eta/\sqrt{r_t} \) are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
- Sparse data benefit: For very sparse features, \( r_{t,j} \) grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal -
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. \( O(1/\sqrt{T}) \) regret bound) comparable to the best fixed learning rate tuned for the problem +
- Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem
- Update: \( \theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \). +
- Update: \( \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t) \).
- Recent gradients have more weight, so \( v_t \) adapts to the current landscape.
- Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
- Fast convergence by smoothing gradients (accelerates in long-term gradient direction). @@ -1455,46 +1477,96 @@ gradient descent with momentum algorithm and the RMSprop algorithm discussed above. -
- Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction). +
- Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients). +
- Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients) +
- Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations) +
- When \( t \) is small, \( 1-\beta_i^t \approx 0 \), so \( \hat{m}_t, \hat{v}_t \) significantly larger than raw \( m_t, v_t \), compensating for the initial zero bias. +
- As \( t \) increases, \( 1-\beta_i^t \to 1 \), and \( \hat{m}_t, \hat{v}_t \) converge to \( m_t, v_t \). +
- Bias correction is important for Adam’s stability in early iterations +
- Compute gradient \( \nabla C(\theta_t) \). +
- Update first moment \( m_t \) and second moment \( v_t \) (exponential moving averages). +
- Bias-correct: \( \hat{m}_t = m_t/(1-\beta_1^t) \), \( \; \hat{v}_t = v_t/(1-\beta_2^t) \). +
- Compute step: \( \Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} \). +
- Update parameters: \( \theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t \). +
- AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting) +
- RMSProp: Uses moving average of squared gradients (like Adam’s \( v_t \)) to maintain adaptive learning rates, but does not include momentum or bias-correction. +
- Adam: Effectively RMSProp + Momentum + Bias-correction +
- Momentum (\( m_t \)) provides acceleration and smoother convergence. +
- Adaptive \( v_t \) scaling moderates the step size per dimension. +
- Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on. +
- Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps. +
- This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems. +
- Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction. +
It effectively reduces the need to tune \( \eta \) by hand.
-
@@ -356,7 +383,7 @@ MathJax.Hub.Config({
with \( \alpha_2 \) typically \( 0.9 \) (or \( 0.99 \)).
+with \( \rho \) typically \( 0.9 \) (or \( 0.99 \)).
-
-
Adam Optimizer
-Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.
+Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.
In addition to keeping a running average of the first and -second moments of the gradient -(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and -\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM -performs an additional bias correction to account for the fact that we -are estimating the first two moments of the gradient using a running -average (denoted by the hats in the update rule below). The update -rule for ADAM is given by (where multiplication and division are once -again understood to be element-wise operations below) -
- -$$ -\begin{align} -\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) -\tag{1}\\ -\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\ -\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\ -\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\ -\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\ -\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\ -\tag{2} -\end{align} -$$ - -where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and -second moment and are typically taken to be \( 0.9 \) and \( 0.99 \) -respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop. -
- -Like in RMSprop, the effective step size of a parameter depends on the -magnitude of its gradient squared. To understand this better, let us -rewrite this expression in terms of the variance -\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t - -(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The -update rule for this parameter is given by -
- -$$ -\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}. -$$ -ADAM algorithm, taken from Goodfellow et al
- --
-
diff --git a/doc/pub/week37/html/._week37-bs042.html b/doc/pub/week37/html/._week37-bs042.html index 394ec7925..e17e7709a 100644 --- a/doc/pub/week37/html/._week37-bs042.html +++ b/doc/pub/week37/html/._week37-bs042.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
-
Algorithms and codes for Adagrad, RMSprop and Adam
+Why Combine Momentum and RMSProp?
-The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8.
- -The codes which implement these algorithms are discussed below here.
+-
+
Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice
@@ -350,7 +381,7 @@ MathJax.Hub.Config({
-
Practical tips
+Adam: Exponential Moving Averages (Moments)
+Adam maintains two moving averages at each time step \( t \) for each parameter \( w \):
+The Momentum term
+$$ +m_t = \beta_1m_{t-1} + (1-\beta_1)\, \nabla C(\theta_t), +$$ +The RMS term
+$$ +v_t = \beta_2v_{t-1} + (1-\beta_2)(\nabla C(\theta_t))^2, +$$ + +with typical \( \beta_1 = 0.9 \), \( \beta_2 = 0.999 \). Initialize \( m_0 = 0 \), \( v_0 = 0 \).
+These are biased estimators of the true first and second moment of the gradients, especially at the start (since \( m_0,v_0 \) are zero)
--
-
diff --git a/doc/pub/week37/html/._week37-bs044.html b/doc/pub/week37/html/._week37-bs044.html index e8a316a03..08721665f 100644 --- a/doc/pub/week37/html/._week37-bs044.html +++ b/doc/pub/week37/html/._week37-bs044.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
-
Sneaking in automatic differentiation using Autograd
- -In the examples here we take the liberty of sneaking in automatic -differentiation (without having discussed the mathematics). In -project 1 you will write the gradients as discussed above, that is -hard-coding the gradients. By introducing automatic differentiation -via the library autograd, which is now replaced by JAX, we have -more flexibility in setting up alternative cost functions. -
- -The -first example shows results with ordinary leats squares. -
- - - -# Using Autograd to calculate gradients for OLS
-from random import random, seed
-import numpy as np
-import autograd.numpy as np
-import matplotlib.pyplot as plt
-from autograd import grad
-
-def CostOLS(theta):
- return (1.0/n)*np.sum((y-X @ theta)**2)
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-XT_X = X.T @ X
-theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
-print("Own inversion")
-print(theta_linreg)
-# Hessian matrix
-H = (2.0/n)* XT_X
-EigValues, EigVectors = np.linalg.eig(H)
-print(f"Eigenvalues of Hessian Matrix:{EigValues}")
-
-theta = np.random.randn(2,1)
-eta = 1.0/np.max(EigValues)
-Niterations = 1000
-# define the gradient
-training_gradient = grad(CostOLS)
-
-for iter in range(Niterations):
- gradients = training_gradient(theta)
- theta -= eta*gradients
-print("theta from own gd")
-print(theta)
-
-xnew = np.array([[0],[2]])
-Xnew = np.c_[np.ones((2,1)), xnew]
-ypredict = Xnew.dot(theta)
-ypredict2 = Xnew.dot(theta_linreg)
-
-plt.plot(xnew, ypredict, "r-")
-plt.plot(xnew, ypredict2, "b-")
-plt.plot(x, y ,'ro')
-plt.axis([0,2.0,0, 15.0])
-plt.xlabel(r'$x$')
-plt.ylabel(r'$y$')
-plt.title(r'Random numbers ')
-plt.show()
-
-Adam: Bias Correction
+To counteract initialization bias in \( m_t, v_t \), Adam computes bias-corrected estimates
+$$ +\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}. +$$ +-
+
diff --git a/doc/pub/week37/html/._week37-bs045.html b/doc/pub/week37/html/._week37-bs045.html index daf64c06b..d3df1ef26 100644 --- a/doc/pub/week37/html/._week37-bs045.html +++ b/doc/pub/week37/html/._week37-bs045.html @@ -154,6 +154,27 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Why Combine Momentum and RMSProp?', + 2, + None, + 'why-combine-momentum-and-rmsprop'), + ('Adam: Exponential Moving Averages (Moments)', + 2, + None, + 'adam-exponential-moving-averages-moments'), + ('Adam: Bias Correction', 2, None, 'adam-bias-correction'), + ('Adam: Update Rule Derivation', + 2, + None, + 'adam-update-rule-derivation'), + ('Adam vs. AdaGrad and RMSProp', + 2, + None, + 'adam-vs-adagrad-and-rmsprop'), + ('Adaptivity Across Dimensions', + 2, + None, + 'adaptivity-across-dimensions'), ('ADAM algorithm, taken from "Goodfellow et ' 'al":"https://www.deeplearningbook.org/contents/optimization.html"', 2, @@ -264,7 +285,7 @@ MathJax.Hub.Config({
-
Same code but now with momentum gradient descent
- - -# Using Autograd to calculate gradients for OLS
-from random import random, seed
-import numpy as np
-import autograd.numpy as np
-import matplotlib.pyplot as plt
-from autograd import grad
-
-def CostOLS(theta):
- return (1.0/n)*np.sum((y-X @ theta)**2)
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x#+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-XT_X = X.T @ X
-theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
-print("Own inversion")
-print(theta_linreg)
-# Hessian matrix
-H = (2.0/n)* XT_X
-EigValues, EigVectors = np.linalg.eig(H)
-print(f"Eigenvalues of Hessian Matrix:{EigValues}")
-
-theta = np.random.randn(2,1)
-eta = 1.0/np.max(EigValues)
-Niterations = 30
-
-# define the gradient
-training_gradient = grad(CostOLS)
-
-for iter in range(Niterations):
- gradients = training_gradient(theta)
- theta -= eta*gradients
- print(iter,gradients[0],gradients[1])
-print("theta from own gd")
-print(theta)
-
-# Now improve with momentum gradient descent
-change = 0.0
-delta_momentum = 0.3
-for iter in range(Niterations):
- # calculate gradient
- gradients = training_gradient(theta)
- # calculate update
- new_change = eta*gradients+delta_momentum*change
- # take a step
- theta -= new_change
- # save the change
- change = new_change
- print(iter,gradients[0],gradients[1])
-print("theta from own gd wth momentum")
-print(theta)
-
-Adam: Update Rule Derivation
+Finally, Adam updates parameters using the bias-corrected moments:
+$$ +\theta_{t+1} =\theta_t -\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}\hat{m}_t, +$$ +where \( \epsilon \) is a small constant (e.g. \( 10^{-8} \)) to prevent division by zero. +Breaking it down: +
+-
+
This is the Adam update rule as given in the original paper.
@@ -421,7 +389,7 @@ delta_momentum = 54
In this code we include the stochastic gradient descent approach
-discussed above. Note here that we specify which argument we are
-taking the derivative with respect to when using autograd.
- In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone
@@ -447,7 +385,7 @@ theta = np.55
@@ -434,6 +385,8 @@ delta_momentum = 55
Note that we here have introduced automatic differentiation The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8. The codes which implement these algorithms are discussed below here.
@@ -401,6 +375,9 @@ training_gradient = grad(CostOLS)
Running this code we note an almost perfect agreement with the results from matrix inversion.
It effectively reduces the need to tune \( \eta \) by hand. with \( \alpha_2 \) typically \( 0.9 \) (or \( 0.99 \)). with \( \rho \) typically \( 0.9 \) (or \( 0.99 \)). Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp. Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp. In addition to keeping a running average of the first and
-second moments of the gradient
-(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
-\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
-performs an additional bias correction to account for the fact that we
-are estimating the first two moments of the gradient using a running
-average (denoted by the hats in the update rule below). The update
-rule for ADAM is given by (where multiplication and division are once
-again understood to be element-wise operations below)
+
+ Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice Adam maintains two moving averages at each time step \( t \) for each parameter \( w \):
+ The Momentum term
+ The RMS term with typical \( \beta_1 = 0.9 \), \( \beta_2 = 0.999 \). Initialize \( m_0 = 0 \), \( v_0 = 0 \). These are biased estimators of the true first and second moment of the gradients, especially at the start (since \( m_0,v_0 \) are zero) To counteract initialization bias in \( m_t, v_t \), Adam computes bias-corrected estimates Finally, Adam updates parameters using the bias-corrected moments: where \( \epsilon \) is a small constant (e.g. \( 10^{-8} \)) to prevent division by zero.
+Breaking it down:
+ This is the Adam update rule as given in the original paper. where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
-second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
-respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
- Like in RMSprop, the effective step size of a parameter depends on the
-magnitude of its gradient squared. To understand this better, let us
-rewrite this expression in terms of the variance
-\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t -
-(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
-update rule for this parameter is given by
-
+
+ In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone
It effectively reduces the need to tune \( \eta \) by hand. with \( \alpha_2 \) typically \( 0.9 \) (or \( 0.99 \)). with \( \rho \) typically \( 0.9 \) (or \( 0.99 \)). Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp. Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp. In addition to keeping a running average of the first and
-second moments of the gradient
-(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
-\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
-performs an additional bias correction to account for the fact that we
-are estimating the first two moments of the gradient using a running
-average (denoted by the hats in the update rule below). The update
-rule for ADAM is given by (where multiplication and division are once
-again understood to be element-wise operations below)
+ Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice Adam maintains two moving averages at each time step \( t \) for each parameter \( w \):
+ The Momentum term
+ The RMS term with typical \( \beta_1 = 0.9 \), \( \beta_2 = 0.999 \). Initialize \( m_0 = 0 \), \( v_0 = 0 \). These are biased estimators of the true first and second moment of the gradients, especially at the start (since \( m_0,v_0 \) are zero) To counteract initialization bias in \( m_t, v_t \), Adam computes bias-corrected estimates Finally, Adam updates parameters using the bias-corrected moments: where \( \epsilon \) is a small constant (e.g. \( 10^{-8} \)) to prevent division by zero.
+Breaking it down:
This is the Adam update rule as given in the original paper. where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
-second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
-respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
- In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone Like in RMSprop, the effective step size of a parameter depends on the
-magnitude of its gradient squared. To understand this better, let us
-rewrite this expression in terms of the variance
-\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t -
-(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
-update rule for this parameter is given by
- It effectively reduces the need to tune \( \eta \) by hand. with \( \alpha_2 \) typically \( 0.9 \) (or \( 0.99 \)). with \( \rho \) typically \( 0.9 \) (or \( 0.99 \)). Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp. Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp. In addition to keeping a running average of the first and
-second moments of the gradient
-(i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and
-\( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM
-performs an additional bias correction to account for the fact that we
-are estimating the first two moments of the gradient using a running
-average (denoted by the hats in the update rule below). The update
-rule for ADAM is given by (where multiplication and division are once
-again understood to be element-wise operations below)
+ Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice Adam maintains two moving averages at each time step \( t \) for each parameter \( w \):
+ The Momentum term
+ The RMS term with typical \( \beta_1 = 0.9 \), \( \beta_2 = 0.999 \). Initialize \( m_0 = 0 \), \( v_0 = 0 \). These are biased estimators of the true first and second moment of the gradients, especially at the start (since \( m_0,v_0 \) are zero) To counteract initialization bias in \( m_t, v_t \), Adam computes bias-corrected estimates Finally, Adam updates parameters using the bias-corrected moments: where \( \epsilon \) is a small constant (e.g. \( 10^{-8} \)) to prevent division by zero.
+Breaking it down:
This is the Adam update rule as given in the original paper. where \( \theta_1 \) and \( \theta_2 \) set the memory lifetime of the first and
-second moment and are typically taken to be \( 0.9 \) and \( 0.99 \)
-respectively, and \( \eta \) and \( \epsilon \) are identical to RMSprop.
- In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone Like in RMSprop, the effective step size of a parameter depends on the
-magnitude of its gradient squared. To understand this better, let us
-rewrite this expression in terms of the variance
-\( \boldsymbol{\sigma}_t^2 = \boldsymbol{\mathbf{s}}_t -
-(\boldsymbol{\mathbf{m}}_t)^2 \). Consider a single parameter \( \theta_t \). The
-update rule for this parameter is given by
-Including Stochastic Gradient Descent with Autograd
-
-# Using Autograd to calculate gradients using SGD
-# OLS example
-from random import random, seed
-import numpy as np
-import autograd.numpy as np
-import matplotlib.pyplot as plt
-from autograd import grad
-
-# Note change from previous example
-def CostOLS(y,X,theta):
- return np.sum((y-X @ theta)**2)
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-XT_X = X.T @ X
-theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
-print("Own inversion")
-print(theta_linreg)
-# Hessian matrix
-H = (2.0/n)* XT_X
-EigValues, EigVectors = np.linalg.eig(H)
-print(f"Eigenvalues of Hessian Matrix:{EigValues}")
-
-theta = np.random.randn(2,1)
-eta = 1.0/np.max(EigValues)
-Niterations = 1000
-
-# Note that we request the derivative wrt third argument (theta, 2 here)
-training_gradient = grad(CostOLS,2)
-
-for iter in range(Niterations):
- gradients = (1.0/n)*training_gradient(y, X, theta)
- theta -= eta*gradients
-print("theta from own gd")
-print(theta)
-
-xnew = np.array([[0],[2]])
-Xnew = np.c_[np.ones((2,1)), xnew]
-ypredict = Xnew.dot(theta)
-ypredict2 = Xnew.dot(theta_linreg)
-
-plt.plot(xnew, ypredict, "r-")
-plt.plot(xnew, ypredict2, "b-")
-plt.plot(x, y ,'ro')
-plt.axis([0,2.0,0, 15.0])
-plt.xlabel(r'$x$')
-plt.ylabel(r'$y$')
-plt.title(r'Random numbers ')
-plt.show()
-
-n_epochs = 50
-M = 5 #size of each minibatch
-m = int(n/M) #number of minibatches
-t0, t1 = 5, 50
-def learning_schedule(t):
- return t0/(t+t1)
-
-theta = np.random.randn(2,1)
-
-for epoch in range(n_epochs):
-# Can you figure out a better way of setting up the contributions to each batch?
- for i in range(m):
- random_index = M*np.random.randint(m)
- xi = X[random_index:random_index+M]
- yi = y[random_index:random_index+M]
- gradients = (1.0/M)*training_gradient(yi, xi, theta)
- eta = learning_schedule(epoch*m+i)
- theta = theta - eta*gradients
-print("theta from own sdg")
-print(theta)
-
-Adam vs. AdaGrad and RMSProp
+
+
+
+
+Same code but now with momentum gradient descent
+Adaptivity Across Dimensions
-
-# Using Autograd to calculate gradients using SGD
-# OLS example
-from random import random, seed
-import numpy as np
-import autograd.numpy as np
-import matplotlib.pyplot as plt
-from autograd import grad
-
-# Note change from previous example
-def CostOLS(y,X,theta):
- return np.sum((y-X @ theta)**2)
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+np.random.randn(n,1)
-
-X = np.c_[np.ones((n,1)), x]
-XT_X = X.T @ X
-theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
-print("Own inversion")
-print(theta_linreg)
-# Hessian matrix
-H = (2.0/n)* XT_X
-EigValues, EigVectors = np.linalg.eig(H)
-print(f"Eigenvalues of Hessian Matrix:{EigValues}")
-
-theta = np.random.randn(2,1)
-eta = 1.0/np.max(EigValues)
-Niterations = 100
-
-# Note that we request the derivative wrt third argument (theta, 2 here)
-training_gradient = grad(CostOLS,2)
-
-for iter in range(Niterations):
- gradients = (1.0/n)*training_gradient(y, X, theta)
- theta -= eta*gradients
-print("theta from own gd")
-print(theta)
-
-
-n_epochs = 50
-M = 5 #size of each minibatch
-m = int(n/M) #number of minibatches
-t0, t1 = 5, 50
-def learning_schedule(t):
- return t0/(t+t1)
-
-theta = np.random.randn(2,1)
-
-change = 0.0
-delta_momentum = 0.3
-
-for epoch in range(n_epochs):
- for i in range(m):
- random_index = M*np.random.randint(m)
- xi = X[random_index:random_index+M]
- yi = y[random_index:random_index+M]
- gradients = (1.0/M)*training_gradient(yi, xi, theta)
- eta = learning_schedule(epoch*m+i)
- # calculate update
- new_change = eta*gradients+delta_momentum*change
- # take a step
- theta -= new_change
- # save the change
- change = new_change
-print("theta from own sdg with momentum")
-print(theta)
-
-
+
+ADAM algorithm, taken from Goodfellow et al
+
+
But none of these can compete with Newton's method
+Algorithms and codes for Adagrad, RMSprop and Adam
-# Using Newton's method
-from random import random, seed
-import numpy as np
-import autograd.numpy as np
-from autograd import grad
-
-def CostOLS(theta):
- return (1.0/n)*np.sum((y-X @ theta)**2)
-
-n = 100
-x = 2*np.random.rand(n,1)
-y = 4+3*x+5*x*x
-
-X = np.c_[np.ones((n,1)), x, x*x]
-XT_X = X.T @ X
-theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
-print("Own inversion")
-print(theta_linreg)
-# Hessian matrix
-H = (2.0/n)* XT_X
-# Note that here the Hessian does not depend on the parameters theta
-invH = np.linalg.pinv(H)
-theta = np.random.randn(3,1)
-Niterations = 5
-# define the gradient
-training_gradient = grad(CostOLS)
-
-for iter in range(Niterations):
- gradients = training_gradient(theta)
- theta -= invH @ gradients
- print(iter,gradients[0],gradients[1])
-print("theta from own Newton code")
-print(theta)
-
-Similar (second order function now) problem but now with AdaGrad
-
-
-# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent
-# OLS example
-from random import random, seed
-import numpy as np
-import autograd.numpy as np
-import matplotlib.pyplot as plt
-from autograd import grad
-
-# Note change from previous example
-def CostOLS(y,X,theta):
- return np.sum((y-X @ theta)**2)
-
-n = 1000
-x = np.random.rand(n,1)
-y = 2.0+3*x +4*x*x
-
-X = np.c_[np.ones((n,1)), x, x*x]
-XT_X = X.T @ X
-theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)
-print("Own inversion")
-print(theta_linreg)
-
-
-# Note that we request the derivative wrt third argument (theta, 2 here)
-training_gradient = grad(CostOLS,2)
-# Define parameters for Stochastic Gradient Descent
-n_epochs = 50
-M = 5 #size of each minibatch
-m = int(n/M) #number of minibatches
-# Guess for unknown parameters theta
-theta = np.random.randn(3,1)
-
-# Value for learning rate
-eta = 0.01
-# Including AdaGrad parameter to avoid possible division by zero
-delta = 1e-8
-for epoch in range(n_epochs):
- Giter = 0.0
- for i in range(m):
- random_index = M*np.random.randint(m)
- xi = X[random_index:random_index+M]
- yi = y[random_index:random_index+M]
- gradients = (1.0/M)*training_gradient(yi, xi, theta)
- Giter += gradients*gradients
- update = gradients*eta/(delta+np.sqrt(Giter))
- theta -= update
-print("theta from own AdaGrad")
-print(theta)
-
-Practical tips
+
+
x = 2*np.random.rand(m,1)
+
import numpy as np
+x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
$$
-r_t =; r_{t-1} + g_t \circ g_t,
+r_t = r_{t-1} + g_t \circ g_t,
$$
@@ -1305,9 +1306,9 @@ $$
$$
-v_t = \alpha_2 v_{t-1} + (1-\alpha_2)(\nabla C(\theta_t))^2,
+v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2,
$$
-
-
@@ -1348,7 +1349,7 @@ $$
Adam Optimizer
-
Why Combine Momentum and RMSProp?
+
+
+
+Adam: Exponential Moving Averages (Moments)
+
+$$
+m_t = \beta_1m_{t-1} + (1-\beta_1)\, \nabla C(\theta_t),
+$$
+
+
+$$
+v_t = \beta_2v_{t-1} + (1-\beta_2)(\nabla C(\theta_t))^2,
+$$
+
+
+Adam: Bias Correction
+
+$$
+\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}.
+$$
+
+
+
+
+Adam: Update Rule Derivation
+
+$$
+\theta_{t+1} =\theta_t -\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}\hat{m}_t,
+$$
+
+
+
+
+
-$$
-\begin{align}
-\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta})
-\tag{1}\\
-\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
-\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
-\boldsymbol{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
-\boldsymbol{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
-\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \boldsymbol{\mathbf{m}}_t \over \sqrt{\boldsymbol{\mathbf{s}}_t} +\epsilon}, \nonumber \\
-\tag{2}
-\end{align}
-$$
-
+Adam vs. AdaGrad and RMSProp
-
+
+
-
+
-$$
-\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
-$$
-
+Adaptivity Across Dimensions
+
+
+
+ADAM algorithm, taken from Goodfellow et al
diff --git a/doc/pub/week37/html/week37-solarized.html b/doc/pub/week37/html/week37-solarized.html
index 5c85a93e4..07225dd2c 100644
--- a/doc/pub/week37/html/week37-solarized.html
+++ b/doc/pub/week37/html/week37-solarized.html
@@ -181,6 +181,27 @@ div.toc p,a {
2,
None,
'adam-optimizer-https-arxiv-org-abs-1412-6980'),
+ ('Why Combine Momentum and RMSProp?',
+ 2,
+ None,
+ 'why-combine-momentum-and-rmsprop'),
+ ('Adam: Exponential Moving Averages (Moments)',
+ 2,
+ None,
+ 'adam-exponential-moving-averages-moments'),
+ ('Adam: Bias Correction', 2, None, 'adam-bias-correction'),
+ ('Adam: Update Rule Derivation',
+ 2,
+ None,
+ 'adam-update-rule-derivation'),
+ ('Adam vs. AdaGrad and RMSProp',
+ 2,
+ None,
+ 'adam-vs-adagrad-and-rmsprop'),
+ ('Adaptivity Across Dimensions',
+ 2,
+ None,
+ 'adaptivity-across-dimensions'),
('ADAM algorithm, taken from "Goodfellow et '
'al":"https://www.deeplearningbook.org/contents/optimization.html"',
2,
@@ -339,7 +360,8 @@ desirable properties such as:
x = 2*np.random.rand(m,1)
+
import numpy as np
+x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
@@ -1320,9 +1342,9 @@ $$
@@ -1335,12 +1357,12 @@ $$
Uses a decaying average of squared gradients (instead of a cumulative sum):
$$
-v_t = \alpha_2 v_{t-1} + (1-\alpha_2)(\nabla C(\theta_t))^2,
+v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2,
$$
-
-
@@ -1356,7 +1378,7 @@ $$
Adam Optimizer
-
+Why Combine Momentum and RMSProp?
+
+
+
+
+Adam: Exponential Moving Averages (Moments)
+
+Adam: Bias Correction
+
+
+
+Adam: Update Rule Derivation
+
+
+
+Adam vs. AdaGrad and RMSProp
-
+
+
+
+
+Adaptivity Across Dimensions
-$$
-\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
-$$
+
+
ADAM algorithm, taken from Goodfellow et al
diff --git a/doc/pub/week37/html/week37.html b/doc/pub/week37/html/week37.html
index c67da1a41..a24a65f19 100644
--- a/doc/pub/week37/html/week37.html
+++ b/doc/pub/week37/html/week37.html
@@ -258,6 +258,27 @@ div.toc p,a {
2,
None,
'adam-optimizer-https-arxiv-org-abs-1412-6980'),
+ ('Why Combine Momentum and RMSProp?',
+ 2,
+ None,
+ 'why-combine-momentum-and-rmsprop'),
+ ('Adam: Exponential Moving Averages (Moments)',
+ 2,
+ None,
+ 'adam-exponential-moving-averages-moments'),
+ ('Adam: Bias Correction', 2, None, 'adam-bias-correction'),
+ ('Adam: Update Rule Derivation',
+ 2,
+ None,
+ 'adam-update-rule-derivation'),
+ ('Adam vs. AdaGrad and RMSProp',
+ 2,
+ None,
+ 'adam-vs-adagrad-and-rmsprop'),
+ ('Adaptivity Across Dimensions',
+ 2,
+ None,
+ 'adaptivity-across-dimensions'),
('ADAM algorithm, taken from "Goodfellow et '
'al":"https://www.deeplearningbook.org/contents/optimization.html"',
2,
@@ -416,7 +437,8 @@ desirable properties such as:
x = 2*np.random.rand(m,1)
+
import numpy as np
+x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
@@ -1397,9 +1419,9 @@ $$
@@ -1412,12 +1434,12 @@ $$
Uses a decaying average of squared gradients (instead of a cumulative sum):
$$
-v_t = \alpha_2 v_{t-1} + (1-\alpha_2)(\nabla C(\theta_t))^2,
+v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2,
$$
-
-
@@ -1433,7 +1455,7 @@ $$
Adam Optimizer
-
+Why Combine Momentum and RMSProp?
+
+
+
+
+Adam: Exponential Moving Averages (Moments)
+
+Adam: Bias Correction
+
+
+
+Adam: Update Rule Derivation
+
+
+
+Adam vs. AdaGrad and RMSProp
-
+
+
+
+
+Adaptivity Across Dimensions
-$$
-\Delta \theta_{t+1}= -\eta_t { \boldsymbol{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
-$$
+
+
ADAM algorithm, taken from Goodfellow et al
diff --git a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz
index 4e9183dc9..56a733323 100644
Binary files a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz and b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz differ
diff --git a/doc/pub/week37/ipynb/week37.ipynb b/doc/pub/week37/ipynb/week37.ipynb
index 9daff5edb..eb66de6d4 100644
--- a/doc/pub/week37/ipynb/week37.ipynb
+++ b/doc/pub/week37/ipynb/week37.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "f9d2c9f2",
+ "id": "50812059",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "e33dc2f5",
+ "id": "7793a9e0",
"metadata": {
"editable": true
},
@@ -29,7 +29,7 @@
},
{
"cell_type": "markdown",
- "id": "386091d7",
+ "id": "a3583f46",
"metadata": {
"editable": true
},
@@ -52,7 +52,7 @@
},
{
"cell_type": "markdown",
- "id": "f0f100b2",
+ "id": "aeab1b07",
"metadata": {
"editable": true
},
@@ -69,7 +69,7 @@
},
{
"cell_type": "markdown",
- "id": "cad77084",
+ "id": "113b8927",
"metadata": {
"editable": true
},
@@ -79,7 +79,7 @@
},
{
"cell_type": "markdown",
- "id": "b8c81325",
+ "id": "d6aea694",
"metadata": {
"editable": true
},
@@ -103,20 +103,21 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "966c81ee",
+ "id": "4ac32eb7",
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
+ "import numpy as np\n",
"x = 2*np.random.rand(m,1)\n",
"y = 4+3*x+np.random.randn(m,1)"
]
},
{
"cell_type": "markdown",
- "id": "6518b96a",
+ "id": "e4d1a33d",
"metadata": {
"editable": true
},
@@ -127,7 +128,7 @@
},
{
"cell_type": "markdown",
- "id": "7a37718c",
+ "id": "c4c6aadb",
"metadata": {
"editable": true
},
@@ -139,7 +140,7 @@
},
{
"cell_type": "markdown",
- "id": "e29be680",
+ "id": "211ddc94",
"metadata": {
"editable": true
},
@@ -149,7 +150,7 @@
},
{
"cell_type": "markdown",
- "id": "86eb083a",
+ "id": "264d3a45",
"metadata": {
"editable": true
},
@@ -161,7 +162,7 @@
},
{
"cell_type": "markdown",
- "id": "bd11b54f",
+ "id": "2f40241b",
"metadata": {
"editable": true
},
@@ -175,7 +176,7 @@
},
{
"cell_type": "markdown",
- "id": "f400a51e",
+ "id": "1bf35a52",
"metadata": {
"editable": true
},
@@ -191,7 +192,7 @@
},
{
"cell_type": "markdown",
- "id": "03ef0dd8",
+ "id": "44613300",
"metadata": {
"editable": true
},
@@ -201,7 +202,7 @@
},
{
"cell_type": "markdown",
- "id": "0d11bbf8",
+ "id": "438523f1",
"metadata": {
"editable": true
},
@@ -213,7 +214,7 @@
},
{
"cell_type": "markdown",
- "id": "e8958c12",
+ "id": "f0e7f06b",
"metadata": {
"editable": true
},
@@ -223,7 +224,7 @@
},
{
"cell_type": "markdown",
- "id": "02645e27",
+ "id": "ff2b1f63",
"metadata": {
"editable": true
},
@@ -235,7 +236,7 @@
},
{
"cell_type": "markdown",
- "id": "4c7e3a2c",
+ "id": "6a100c53",
"metadata": {
"editable": true
},
@@ -249,7 +250,7 @@
},
{
"cell_type": "markdown",
- "id": "4912e3e5",
+ "id": "322be116",
"metadata": {
"editable": true
},
@@ -259,7 +260,7 @@
},
{
"cell_type": "markdown",
- "id": "815019f2",
+ "id": "78435579",
"metadata": {
"editable": true
},
@@ -270,7 +271,7 @@
},
{
"cell_type": "markdown",
- "id": "22dd6eb2",
+ "id": "bc7776c0",
"metadata": {
"editable": true
},
@@ -285,7 +286,7 @@
},
{
"cell_type": "markdown",
- "id": "25eb6eaf",
+ "id": "a10aa000",
"metadata": {
"editable": true
},
@@ -295,7 +296,7 @@
},
{
"cell_type": "markdown",
- "id": "ff53c5d0",
+ "id": "b52074ac",
"metadata": {
"editable": true
},
@@ -307,7 +308,7 @@
},
{
"cell_type": "markdown",
- "id": "0e8a41a5",
+ "id": "91581f81",
"metadata": {
"editable": true
},
@@ -319,7 +320,7 @@
},
{
"cell_type": "markdown",
- "id": "33540fbc",
+ "id": "4416c640",
"metadata": {
"editable": true
},
@@ -334,7 +335,7 @@
},
{
"cell_type": "markdown",
- "id": "62be036e",
+ "id": "ea0cefb8",
"metadata": {
"editable": true
},
@@ -347,7 +348,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "4f008c07",
+ "id": "61693bed",
"metadata": {
"collapsed": false,
"editable": true
@@ -406,7 +407,7 @@
},
{
"cell_type": "markdown",
- "id": "f361836a",
+ "id": "648a50c5",
"metadata": {
"editable": true
},
@@ -418,7 +419,7 @@
},
{
"cell_type": "markdown",
- "id": "6f580bd1",
+ "id": "b2705e40",
"metadata": {
"editable": true
},
@@ -430,7 +431,7 @@
},
{
"cell_type": "markdown",
- "id": "b5beb077",
+ "id": "0b1d169a",
"metadata": {
"editable": true
},
@@ -440,7 +441,7 @@
},
{
"cell_type": "markdown",
- "id": "01108292",
+ "id": "151c7c6a",
"metadata": {
"editable": true
},
@@ -454,7 +455,7 @@
},
{
"cell_type": "markdown",
- "id": "9955f452",
+ "id": "40aa26f3",
"metadata": {
"editable": true
},
@@ -464,7 +465,7 @@
},
{
"cell_type": "markdown",
- "id": "b008799a",
+ "id": "e4016061",
"metadata": {
"editable": true
},
@@ -476,7 +477,7 @@
},
{
"cell_type": "markdown",
- "id": "a2506a06",
+ "id": "7fa28627",
"metadata": {
"editable": true
},
@@ -487,7 +488,7 @@
},
{
"cell_type": "markdown",
- "id": "773a857a",
+ "id": "97e1c1a8",
"metadata": {
"editable": true
},
@@ -502,7 +503,7 @@
},
{
"cell_type": "markdown",
- "id": "cc3907e2",
+ "id": "40e8f9a6",
"metadata": {
"editable": true
},
@@ -516,7 +517,7 @@
},
{
"cell_type": "markdown",
- "id": "8d71ca48",
+ "id": "627e04e5",
"metadata": {
"editable": true
},
@@ -527,7 +528,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "a59e601a",
+ "id": "33ea2bc1",
"metadata": {
"collapsed": false,
"editable": true
@@ -588,7 +589,7 @@
},
{
"cell_type": "markdown",
- "id": "e81eaae6",
+ "id": "d4f9401f",
"metadata": {
"editable": true
},
@@ -610,7 +611,7 @@
},
{
"cell_type": "markdown",
- "id": "cbb62bab",
+ "id": "bb52c3f1",
"metadata": {
"editable": true
},
@@ -625,7 +626,7 @@
},
{
"cell_type": "markdown",
- "id": "93812907",
+ "id": "e0b7af14",
"metadata": {
"editable": true
},
@@ -636,7 +637,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "a45b8555",
+ "id": "a3ec834a",
"metadata": {
"collapsed": false,
"editable": true
@@ -702,7 +703,7 @@
},
{
"cell_type": "markdown",
- "id": "de094662",
+ "id": "1d1552c1",
"metadata": {
"editable": true
},
@@ -713,7 +714,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "8717679c",
+ "id": "4d263b16",
"metadata": {
"collapsed": false,
"editable": true
@@ -787,7 +788,7 @@
},
{
"cell_type": "markdown",
- "id": "9a8e189e",
+ "id": "e04acafb",
"metadata": {
"editable": true
},
@@ -806,7 +807,7 @@
},
{
"cell_type": "markdown",
- "id": "88315a32",
+ "id": "a51bbde0",
"metadata": {
"editable": true
},
@@ -827,7 +828,7 @@
},
{
"cell_type": "markdown",
- "id": "2159f671",
+ "id": "b63bb5be",
"metadata": {
"editable": true
},
@@ -843,7 +844,7 @@
},
{
"cell_type": "markdown",
- "id": "728e571e",
+ "id": "7fc69be3",
"metadata": {
"editable": true
},
@@ -857,7 +858,7 @@
},
{
"cell_type": "markdown",
- "id": "e1718d7d",
+ "id": "ff9e2f5b",
"metadata": {
"editable": true
},
@@ -885,7 +886,7 @@
},
{
"cell_type": "markdown",
- "id": "9f2a002d",
+ "id": "551735b2",
"metadata": {
"editable": true
},
@@ -917,7 +918,7 @@
},
{
"cell_type": "markdown",
- "id": "3c8f1f32",
+ "id": "aaea617e",
"metadata": {
"editable": true
},
@@ -934,7 +935,7 @@
},
{
"cell_type": "markdown",
- "id": "7f6416ce",
+ "id": "7072bd4a",
"metadata": {
"editable": true
},
@@ -947,7 +948,7 @@
},
{
"cell_type": "markdown",
- "id": "d105c6ba",
+ "id": "321840eb",
"metadata": {
"editable": true
},
@@ -960,7 +961,7 @@
},
{
"cell_type": "markdown",
- "id": "f95e6d96",
+ "id": "a4d8ed58",
"metadata": {
"editable": true
},
@@ -973,7 +974,7 @@
},
{
"cell_type": "markdown",
- "id": "99c71758",
+ "id": "66a000dd",
"metadata": {
"editable": true
},
@@ -987,7 +988,7 @@
},
{
"cell_type": "markdown",
- "id": "32870c63",
+ "id": "69eeffa0",
"metadata": {
"editable": true
},
@@ -1009,7 +1010,7 @@
},
{
"cell_type": "markdown",
- "id": "25d1b17d",
+ "id": "2a3d9f38",
"metadata": {
"editable": true
},
@@ -1024,7 +1025,7 @@
},
{
"cell_type": "markdown",
- "id": "32b38308",
+ "id": "04bdfa1b",
"metadata": {
"editable": true
},
@@ -1036,7 +1037,7 @@
},
{
"cell_type": "markdown",
- "id": "feace9cf",
+ "id": "fea9e66c",
"metadata": {
"editable": true
},
@@ -1049,7 +1050,7 @@
},
{
"cell_type": "markdown",
- "id": "2fecd201",
+ "id": "8dc0899c",
"metadata": {
"editable": true
},
@@ -1063,7 +1064,7 @@
},
{
"cell_type": "markdown",
- "id": "10d92a8b",
+ "id": "00f5161d",
"metadata": {
"editable": true
},
@@ -1074,7 +1075,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "a7b7f74f",
+ "id": "8ab6969d",
"metadata": {
"collapsed": false,
"editable": true
@@ -1099,7 +1100,7 @@
},
{
"cell_type": "markdown",
- "id": "bf547895",
+ "id": "f7999a9a",
"metadata": {
"editable": true
},
@@ -1115,7 +1116,7 @@
},
{
"cell_type": "markdown",
- "id": "5d99a32b",
+ "id": "21179d56",
"metadata": {
"editable": true
},
@@ -1136,7 +1137,7 @@
},
{
"cell_type": "markdown",
- "id": "9d414bec",
+ "id": "0c72dd40",
"metadata": {
"editable": true
},
@@ -1156,7 +1157,7 @@
},
{
"cell_type": "markdown",
- "id": "b5448f64",
+ "id": "acfb2fac",
"metadata": {
"editable": true
},
@@ -1175,7 +1176,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "a9d971e2",
+ "id": "022d8b91",
"metadata": {
"collapsed": false,
"editable": true
@@ -1210,7 +1211,7 @@
},
{
"cell_type": "markdown",
- "id": "5fc808be",
+ "id": "143b7dc9",
"metadata": {
"editable": true
},
@@ -1223,7 +1224,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "5fafe385",
+ "id": "0951608f",
"metadata": {
"collapsed": false,
"editable": true
@@ -1300,7 +1301,7 @@
},
{
"cell_type": "markdown",
- "id": "0e4c7a12",
+ "id": "1f3febe4",
"metadata": {
"editable": true
},
@@ -1315,7 +1316,7 @@
},
{
"cell_type": "markdown",
- "id": "e5ba945d",
+ "id": "d9e0ce4c",
"metadata": {
"editable": true
},
@@ -1346,7 +1347,7 @@
},
{
"cell_type": "markdown",
- "id": "85175ac4",
+ "id": "ebcb2e04",
"metadata": {
"editable": true
},
@@ -1368,7 +1369,7 @@
},
{
"cell_type": "markdown",
- "id": "902d59a3",
+ "id": "6afd3d9f",
"metadata": {
"editable": true
},
@@ -1388,7 +1389,7 @@
},
{
"cell_type": "markdown",
- "id": "897f935c",
+ "id": "afe7ceae",
"metadata": {
"editable": true
},
@@ -1404,7 +1405,7 @@
},
{
"cell_type": "markdown",
- "id": "2e7a7078",
+ "id": "ea5412a1",
"metadata": {
"editable": true
},
@@ -1424,19 +1425,19 @@
},
{
"cell_type": "markdown",
- "id": "289c60b8",
+ "id": "1e300383",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "r_t =; r_{t-1} + g_t \\circ g_t,\n",
+ "r_t = r_{t-1} + g_t \\circ g_t,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "31654b40",
+ "id": "810cb488",
"metadata": {
"editable": true
},
@@ -1448,7 +1449,7 @@
},
{
"cell_type": "markdown",
- "id": "3731c41a",
+ "id": "178f8e1e",
"metadata": {
"editable": true
},
@@ -1460,7 +1461,7 @@
},
{
"cell_type": "markdown",
- "id": "9ca5512b",
+ "id": "73454d26",
"metadata": {
"editable": true
},
@@ -1472,7 +1473,7 @@
},
{
"cell_type": "markdown",
- "id": "43538765",
+ "id": "30e79c94",
"metadata": {
"editable": true
},
@@ -1483,7 +1484,7 @@
},
{
"cell_type": "markdown",
- "id": "35712ddb",
+ "id": "a9626453",
"metadata": {
"editable": true
},
@@ -1495,7 +1496,7 @@
},
{
"cell_type": "markdown",
- "id": "85533783",
+ "id": "9961d1a9",
"metadata": {
"editable": true
},
@@ -1505,7 +1506,7 @@
},
{
"cell_type": "markdown",
- "id": "72f24dc5",
+ "id": "984eb258",
"metadata": {
"editable": true
},
@@ -1517,7 +1518,7 @@
},
{
"cell_type": "markdown",
- "id": "d14bc442",
+ "id": "740227ff",
"metadata": {
"editable": true
},
@@ -1527,7 +1528,7 @@
},
{
"cell_type": "markdown",
- "id": "28ba6dce",
+ "id": "1bbb0321",
"metadata": {
"editable": true
},
@@ -1536,11 +1537,11 @@
"\n",
"1. AdaGrad automatically tunes the step size for each parameter. Parameters with more *volatile or large gradients* get smaller steps, and those with *small or infrequent gradients* get relatively larger steps\n",
"\n",
- "2. No manual schedule needed: The accumulation $h_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n",
+ "2. No manual schedule needed: The accumulation $r_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\\eta/\\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.\n",
"\n",
"3. Sparse data benefit: For very sparse features, $r_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal\n",
"\n",
- "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. $O(1/\\sqrt{T})$ regret bound) comparable to the best fixed learning rate tuned for the problem\n",
+ "4. Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem\n",
"\n",
"It effectively reduces the need to tune $\\eta$ by hand.\n",
"1. Limitations: Because $r_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)"
@@ -1548,7 +1549,7 @@
},
{
"cell_type": "markdown",
- "id": "2b07cd34",
+ "id": "7a1c4723",
"metadata": {
"editable": true
},
@@ -1561,25 +1562,25 @@
},
{
"cell_type": "markdown",
- "id": "cdc6fad8",
+ "id": "8e05a132",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "v_t = \\alpha_2 v_{t-1} + (1-\\alpha_2)(\\nabla C(\\theta_t))^2,\n",
+ "v_t = \\rho v_{t-1} + (1-\\rho)(\\nabla C(\\theta_t))^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "3410448c",
+ "id": "27a9e5c9",
"metadata": {
"editable": true
},
"source": [
- "with $\\alpha_2$ typically $0.9$ (or $0.99$).\n",
- "1. Update: $\\theta_{t+1} = \\theta_t - \\frac{\\alpha}{\\sqrt{v_t + \\epsilon}} \\nabla C(\\theta_t)$.\n",
+ "with $\\rho$ typically $0.9$ (or $0.99$).\n",
+ "1. Update: $\\theta_{t+1} = \\theta_t - \\frac{\\eta}{\\sqrt{v_t + \\epsilon}} \\nabla C(\\theta_t)$.\n",
"\n",
"2. Recent gradients have more weight, so $v_t$ adapts to the current landscape.\n",
"\n",
@@ -1590,7 +1591,7 @@
},
{
"cell_type": "markdown",
- "id": "6ecbee1b",
+ "id": "897391da",
"metadata": {
"editable": true
},
@@ -1606,14 +1607,14 @@
},
{
"cell_type": "markdown",
- "id": "bc48c220",
+ "id": "28fccc5b",
"metadata": {
"editable": true
},
"source": [
"## Adam Optimizer\n",
"\n",
- "Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n",
+ "Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.\n",
"\n",
"1. Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n",
"\n",
@@ -1628,7 +1629,7 @@
},
{
"cell_type": "markdown",
- "id": "403a6c11",
+ "id": "6843042b",
"metadata": {
"editable": true
},
@@ -1641,148 +1642,216 @@
"parameters. The method is efficient when working with large\n",
"problems involving lots data and/or parameters. It is a combination of the\n",
"gradient descent with momentum algorithm and the RMSprop algorithm\n",
- "discussed above.\n",
+ "discussed above."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4c5e2e6d",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Why Combine Momentum and RMSProp?\n",
"\n",
- "In addition to keeping a running average of the first and\n",
- "second moments of the gradient\n",
- "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n",
- "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n",
- "performs an additional bias correction to account for the fact that we\n",
- "are estimating the first two moments of the gradient using a running\n",
- "average (denoted by the hats in the update rule below). The update\n",
- "rule for ADAM is given by (where multiplication and division are once\n",
- "again understood to be element-wise operations below)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "75a83893",
- "metadata": {
- "editable": true
- },
- "source": [
- "\n",
- "\n",
+ "1. Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).\n",
"\n",
- "$$\n",
- "\\begin{equation}\n",
- "\\mathbf{g}_t = \\nabla_\\theta E(\\boldsymbol{\\theta}) \n",
- "\\label{_auto1} \\tag{1}\n",
- "\\end{equation}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "3df5c7d4",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbf{m}_t = \\theta_1 \\mathbf{m}_{t-1} + (1-\\theta_1) \\mathbf{g}_t \\nonumber\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "747fef3f",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\mathbf{s}_t =\\theta_2 \\mathbf{s}_{t-1} +(1-\\theta_2)\\mathbf{g}_t^2 \\nonumber\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "daa4e645",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\theta_1^t} \\nonumber\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "b2205a46",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\theta_2^t} \\nonumber\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "a5814442",
- "metadata": {
- "editable": true
- },
- "source": [
- "$$\n",
- "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "86237a0f",
- "metadata": {
- "editable": true
- },
- "source": [
- "\n",
- "\n",
+ "2. Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).\n",
"\n",
- "$$\n",
- "\\begin{equation} \n",
- "\\label{_auto2} \\tag{2}\n",
- "\\end{equation}\n",
- "$$"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "ff104975",
- "metadata": {
- "editable": true
- },
- "source": [
- "where $\\theta_1$ and $\\theta_2$ set the memory lifetime of the first and\n",
- "second moment and are typically taken to be $0.9$ and $0.99$\n",
- "respectively, and $\\eta$ and $\\epsilon$ are identical to RMSprop.\n",
+ "3. Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)\n",
"\n",
- "Like in RMSprop, the effective step size of a parameter depends on the\n",
- "magnitude of its gradient squared. To understand this better, let us\n",
- "rewrite this expression in terms of the variance\n",
- "$\\boldsymbol{\\sigma}_t^2 = \\boldsymbol{\\mathbf{s}}_t -\n",
- "(\\boldsymbol{\\mathbf{m}}_t)^2$. Consider a single parameter $\\theta_t$. The\n",
- "update rule for this parameter is given by"
+ "4. Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)\n",
+ "\n",
+ "Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice"
]
},
{
"cell_type": "markdown",
- "id": "454f5c52",
+ "id": "ce71b8ba",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Adam: Exponential Moving Averages (Moments)\n",
+ "Adam maintains two moving averages at each time step $t$ for each parameter $w$:\n",
+ "**First moment (mean) $m_t$.**\n",
+ "\n",
+ "The Momentum term"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d5666bf4",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n",
+ "m_t = \\beta_1m_{t-1} + (1-\\beta_1)\\, \\nabla C(\\theta_t),\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "d4638f79",
+ "id": "564bb697",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "**Second moment (uncentered variance) $v_t$.**\n",
+ "\n",
+ "The RMS term"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1589c6f3",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "v_t = \\beta_2v_{t-1} + (1-\\beta_2)(\\nabla C(\\theta_t))^2,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "72ee2d0a",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "with typical $\\beta_1 = 0.9$, $\\beta_2 = 0.999$. Initialize $m_0 = 0$, $v_0 = 0$.\n",
+ "\n",
+ " These are **biased** estimators of the true first and second moment of the gradients, especially at the start (since $m_0,v_0$ are zero)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3d03c54f",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Adam: Bias Correction\n",
+ "To counteract initialization bias in $m_t, v_t$, Adam computes bias-corrected estimates"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4ff05e26",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\hat{m}_t = \\frac{m_t}{1 - \\beta_1^t}, \\qquad \\hat{v}_t = \\frac{v_t}{1 - \\beta_2^t}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7629b25b",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "* When $t$ is small, $1-\\beta_i^t \\approx 0$, so $\\hat{m}_t, \\hat{v}_t$ significantly larger than raw $m_t, v_t$, compensating for the initial zero bias.\n",
+ "\n",
+ "* As $t$ increases, $1-\\beta_i^t \\to 1$, and $\\hat{m}_t, \\hat{v}_t$ converge to $m_t, v_t$.\n",
+ "\n",
+ "* Bias correction is important for Adam’s stability in early iterations"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "91889f1c",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Adam: Update Rule Derivation\n",
+ "Finally, Adam updates parameters using the bias-corrected moments:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "de09f733",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "$$\n",
+ "\\theta_{t+1} =\\theta_t -\\frac{\\alpha}{\\sqrt{\\hat{v}_t} + \\epsilon}\\hat{m}_t,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2ec1c97",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "where $\\epsilon$ is a small constant (e.g. $10^{-8}$) to prevent division by zero.\n",
+ "Breaking it down:\n",
+ "1. Compute gradient $\\nabla C(\\theta_t)$.\n",
+ "\n",
+ "2. Update first moment $m_t$ and second moment $v_t$ (exponential moving averages).\n",
+ "\n",
+ "3. Bias-correct: $\\hat{m}_t = m_t/(1-\\beta_1^t)$, $\\; \\hat{v}_t = v_t/(1-\\beta_2^t)$.\n",
+ "\n",
+ "4. Compute step: $\\Delta \\theta_t = \\frac{\\hat{m}_t}{\\sqrt{\\hat{v}_t} + \\epsilon}$.\n",
+ "\n",
+ "5. Update parameters: $\\theta_{t+1} = \\theta_t - \\alpha\\, \\Delta \\theta_t$.\n",
+ "\n",
+ "This is the Adam update rule as given in the original paper."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "37b4758a",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Adam vs. AdaGrad and RMSProp\n",
+ "\n",
+ "1. AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)\n",
+ "\n",
+ "2. RMSProp: Uses moving average of squared gradients (like Adam’s $v_t$) to maintain adaptive learning rates, but does not include momentum or bias-correction.\n",
+ "\n",
+ "3. Adam: Effectively RMSProp + Momentum + Bias-correction\n",
+ "\n",
+ " * Momentum ($m_t$) provides acceleration and smoother convergence.\n",
+ "\n",
+ " * Adaptive $v_t$ scaling moderates the step size per dimension.\n",
+ "\n",
+ " * Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.\n",
+ "\n",
+ "In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d75712f4",
+ "metadata": {
+ "editable": true
+ },
+ "source": [
+ "## Adaptivity Across Dimensions\n",
+ "\n",
+ "1. Adam adapts the step size \\emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.\n",
+ "\n",
+ "2. This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.\n",
+ "\n",
+ "3. Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5041844",
"metadata": {
"editable": true
},
@@ -1798,7 +1867,7 @@
},
{
"cell_type": "markdown",
- "id": "15d51068",
+ "id": "00e66b1d",
"metadata": {
"editable": true
},
@@ -1812,7 +1881,7 @@
},
{
"cell_type": "markdown",
- "id": "3fc7c784",
+ "id": "83d5a029",
"metadata": {
"editable": true
},
@@ -1830,7 +1899,7 @@
},
{
"cell_type": "markdown",
- "id": "8b23e797",
+ "id": "723bf8b1",
"metadata": {
"editable": true
},
@@ -1851,7 +1920,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "ef50a89a",
+ "id": "4c5339d0",
"metadata": {
"collapsed": false,
"editable": true
@@ -1911,7 +1980,7 @@
},
{
"cell_type": "markdown",
- "id": "e11e52e8",
+ "id": "96801735",
"metadata": {
"editable": true
},
@@ -1922,7 +1991,7 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "32bb444b",
+ "id": "ea189034",
"metadata": {
"collapsed": false,
"editable": true
@@ -1986,7 +2055,7 @@
},
{
"cell_type": "markdown",
- "id": "d8765ab0",
+ "id": "68e93f81",
"metadata": {
"editable": true
},
@@ -2001,7 +2070,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "fd19d605",
+ "id": "bbb479a7",
"metadata": {
"collapsed": false,
"editable": true
@@ -2085,7 +2154,7 @@
},
{
"cell_type": "markdown",
- "id": "a1372cf6",
+ "id": "37b1e945",
"metadata": {
"editable": true
},
@@ -2096,7 +2165,7 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "6e738e45",
+ "id": "59d8c89a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2174,7 +2243,7 @@
},
{
"cell_type": "markdown",
- "id": "028b6f6b",
+ "id": "19ff08d8",
"metadata": {
"editable": true
},
@@ -2187,7 +2256,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "9623fde5",
+ "id": "b9373caf",
"metadata": {
"collapsed": false,
"editable": true
@@ -2231,7 +2300,7 @@
},
{
"cell_type": "markdown",
- "id": "28dd8452",
+ "id": "ebb24d8f",
"metadata": {
"editable": true
},
@@ -2242,7 +2311,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "891c0f44",
+ "id": "921bd177",
"metadata": {
"collapsed": false,
"editable": true
@@ -2301,7 +2370,7 @@
},
{
"cell_type": "markdown",
- "id": "18435145",
+ "id": "789528fb",
"metadata": {
"editable": true
},
@@ -2311,7 +2380,7 @@
},
{
"cell_type": "markdown",
- "id": "6aab62c2",
+ "id": "d3eee3bd",
"metadata": {
"editable": true
},
@@ -2322,7 +2391,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "b05dd4e6",
+ "id": "9c483eda",
"metadata": {
"collapsed": false,
"editable": true
@@ -2387,7 +2456,7 @@
},
{
"cell_type": "markdown",
- "id": "7fe3c208",
+ "id": "18713b46",
"metadata": {
"editable": true
},
@@ -2398,7 +2467,7 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "643d540a",
+ "id": "f867cb1a",
"metadata": {
"collapsed": false,
"editable": true
@@ -2468,7 +2537,7 @@
},
{
"cell_type": "markdown",
- "id": "6afbc076",
+ "id": "700401c6",
"metadata": {
"editable": true
},
@@ -2485,7 +2554,7 @@
},
{
"cell_type": "markdown",
- "id": "4bd64d07",
+ "id": "9a76e50d",
"metadata": {
"editable": true
},
@@ -2513,7 +2582,7 @@
{
"cell_type": "code",
"execution_count": 17,
- "id": "233ca6bb",
+ "id": "d00e0f77",
"metadata": {
"collapsed": false,
"editable": true
@@ -2533,7 +2602,7 @@
},
{
"cell_type": "markdown",
- "id": "4a4e7e02",
+ "id": "5236f9a5",
"metadata": {
"editable": true
},
@@ -2549,7 +2618,7 @@
},
{
"cell_type": "markdown",
- "id": "ff9a4ae1",
+ "id": "c05d7dc6",
"metadata": {
"editable": true
},
@@ -2569,7 +2638,7 @@
},
{
"cell_type": "markdown",
- "id": "59e3ab97",
+ "id": "7b7b6cf3",
"metadata": {
"editable": true
},
@@ -2596,7 +2665,7 @@
},
{
"cell_type": "markdown",
- "id": "7023f973",
+ "id": "6c08d35c",
"metadata": {
"editable": true
},
@@ -2609,7 +2678,7 @@
},
{
"cell_type": "markdown",
- "id": "6938220d",
+ "id": "2eb821a9",
"metadata": {
"editable": true
},
@@ -2621,7 +2690,7 @@
},
{
"cell_type": "markdown",
- "id": "d58b1afa",
+ "id": "9a960d29",
"metadata": {
"editable": true
},
@@ -2636,7 +2705,7 @@
{
"cell_type": "code",
"execution_count": 18,
- "id": "1740008f",
+ "id": "57504163",
"metadata": {
"collapsed": false,
"editable": true
@@ -2663,7 +2732,7 @@
},
{
"cell_type": "markdown",
- "id": "4d99f002",
+ "id": "4a431690",
"metadata": {
"editable": true
},
@@ -2677,7 +2746,7 @@
},
{
"cell_type": "markdown",
- "id": "a391f689",
+ "id": "b3010e14",
"metadata": {
"editable": true
},
@@ -2689,7 +2758,7 @@
},
{
"cell_type": "markdown",
- "id": "11817188",
+ "id": "96f0d6df",
"metadata": {
"editable": true
},
@@ -2706,7 +2775,7 @@
},
{
"cell_type": "markdown",
- "id": "7b5b8b7a",
+ "id": "46d0d2a1",
"metadata": {
"editable": true
},
@@ -2718,7 +2787,7 @@
},
{
"cell_type": "markdown",
- "id": "495d0eb9",
+ "id": "3a2eda90",
"metadata": {
"editable": true
},
@@ -2728,7 +2797,7 @@
},
{
"cell_type": "markdown",
- "id": "5eef0f3a",
+ "id": "1191e11c",
"metadata": {
"editable": true
},
@@ -2740,7 +2809,7 @@
},
{
"cell_type": "markdown",
- "id": "c710f430",
+ "id": "fe46dfdd",
"metadata": {
"editable": true
},
@@ -2750,7 +2819,7 @@
},
{
"cell_type": "markdown",
- "id": "833cb381",
+ "id": "86deeaa9",
"metadata": {
"editable": true
},
@@ -2762,7 +2831,7 @@
},
{
"cell_type": "markdown",
- "id": "9e1ed690",
+ "id": "525ee0f5",
"metadata": {
"editable": true
},
@@ -2773,7 +2842,7 @@
},
{
"cell_type": "markdown",
- "id": "d0e4fb31",
+ "id": "0bc83987",
"metadata": {
"editable": true
},
@@ -2785,7 +2854,7 @@
},
{
"cell_type": "markdown",
- "id": "a8eacba6",
+ "id": "574751fd",
"metadata": {
"editable": true
},
@@ -2795,7 +2864,7 @@
},
{
"cell_type": "markdown",
- "id": "9817b0f8",
+ "id": "0326f019",
"metadata": {
"editable": true
},
@@ -2807,7 +2876,7 @@
},
{
"cell_type": "markdown",
- "id": "5d9f9d4e",
+ "id": "4d7402c2",
"metadata": {
"editable": true
},
@@ -2817,7 +2886,7 @@
},
{
"cell_type": "markdown",
- "id": "0b9f155d",
+ "id": "3f3155bf",
"metadata": {
"editable": true
},
@@ -2829,7 +2898,7 @@
},
{
"cell_type": "markdown",
- "id": "f7087fa4",
+ "id": "7159445b",
"metadata": {
"editable": true
},
@@ -2839,7 +2908,7 @@
},
{
"cell_type": "markdown",
- "id": "789f7b34",
+ "id": "9eef8c8c",
"metadata": {
"editable": true
},
@@ -2851,7 +2920,7 @@
},
{
"cell_type": "markdown",
- "id": "f7201d61",
+ "id": "a0e96ef0",
"metadata": {
"editable": true
},
@@ -2861,7 +2930,7 @@
},
{
"cell_type": "markdown",
- "id": "2d40822c",
+ "id": "cc17d03f",
"metadata": {
"editable": true
},
@@ -2873,7 +2942,7 @@
},
{
"cell_type": "markdown",
- "id": "e1d0c6fb",
+ "id": "99988d41",
"metadata": {
"editable": true
},
@@ -2883,7 +2952,7 @@
},
{
"cell_type": "markdown",
- "id": "848ae697",
+ "id": "5f9a2a05",
"metadata": {
"editable": true
},
@@ -2895,7 +2964,7 @@
},
{
"cell_type": "markdown",
- "id": "b3b4946b",
+ "id": "4c0b0b11",
"metadata": {
"editable": true
},
@@ -2905,7 +2974,7 @@
},
{
"cell_type": "markdown",
- "id": "813ce75d",
+ "id": "77243441",
"metadata": {
"editable": true
},
@@ -2917,7 +2986,7 @@
},
{
"cell_type": "markdown",
- "id": "a059f74b",
+ "id": "6647833f",
"metadata": {
"editable": true
},
@@ -2927,7 +2996,7 @@
},
{
"cell_type": "markdown",
- "id": "376c7d7f",
+ "id": "3f9dbf20",
"metadata": {
"editable": true
},
@@ -2939,7 +3008,7 @@
},
{
"cell_type": "markdown",
- "id": "6d02b71f",
+ "id": "3290cd69",
"metadata": {
"editable": true
},
@@ -2951,7 +3020,7 @@
},
{
"cell_type": "markdown",
- "id": "af586c60",
+ "id": "400b3a14",
"metadata": {
"editable": true
},
@@ -2963,7 +3032,7 @@
},
{
"cell_type": "markdown",
- "id": "90f435f9",
+ "id": "0321f9fe",
"metadata": {
"editable": true
},
@@ -2973,7 +3042,7 @@
},
{
"cell_type": "markdown",
- "id": "da6005e4",
+ "id": "955ed99b",
"metadata": {
"editable": true
},
@@ -2985,7 +3054,7 @@
},
{
"cell_type": "markdown",
- "id": "3c83f6c2",
+ "id": "f080e69a",
"metadata": {
"editable": true
},
@@ -2998,7 +3067,7 @@
},
{
"cell_type": "markdown",
- "id": "5b536d6b",
+ "id": "a7ff8bf8",
"metadata": {
"editable": true
},
@@ -3010,7 +3079,7 @@
},
{
"cell_type": "markdown",
- "id": "adf8118d",
+ "id": "89a9d63b",
"metadata": {
"editable": true
},
@@ -3024,7 +3093,7 @@
{
"cell_type": "code",
"execution_count": 19,
- "id": "97cbe17c",
+ "id": "2f17916c",
"metadata": {
"collapsed": false,
"editable": true
@@ -3121,7 +3190,7 @@
},
{
"cell_type": "markdown",
- "id": "85156b7a",
+ "id": "938e24aa",
"metadata": {
"editable": true
},
@@ -3142,7 +3211,7 @@
},
{
"cell_type": "markdown",
- "id": "5431640a",
+ "id": "e3979f55",
"metadata": {
"editable": true
},
@@ -3154,7 +3223,7 @@
},
{
"cell_type": "markdown",
- "id": "94ac146e",
+ "id": "2ffb16fe",
"metadata": {
"editable": true
},
@@ -3164,7 +3233,7 @@
},
{
"cell_type": "markdown",
- "id": "a5827db7",
+ "id": "417d919e",
"metadata": {
"editable": true
},
@@ -3176,7 +3245,7 @@
},
{
"cell_type": "markdown",
- "id": "9be598fe",
+ "id": "dbb044d9",
"metadata": {
"editable": true
},
@@ -3186,7 +3255,7 @@
},
{
"cell_type": "markdown",
- "id": "cb3b7130",
+ "id": "39ac3a58",
"metadata": {
"editable": true
},
@@ -3198,7 +3267,7 @@
},
{
"cell_type": "markdown",
- "id": "8ab51a34",
+ "id": "839d8c1f",
"metadata": {
"editable": true
},
@@ -3216,7 +3285,7 @@
{
"cell_type": "code",
"execution_count": 20,
- "id": "150bee16",
+ "id": "1a39a394",
"metadata": {
"collapsed": false,
"editable": true
@@ -3292,7 +3361,7 @@
},
{
"cell_type": "markdown",
- "id": "7444117c",
+ "id": "20df8406",
"metadata": {
"editable": true
},
@@ -3306,7 +3375,7 @@
{
"cell_type": "code",
"execution_count": 21,
- "id": "d283a960",
+ "id": "688f0a26",
"metadata": {
"collapsed": false,
"editable": true
@@ -3395,7 +3464,7 @@
},
{
"cell_type": "markdown",
- "id": "c2cb7fac",
+ "id": "fea0581c",
"metadata": {
"editable": true
},
diff --git a/doc/src/week37/week37.do.txt b/doc/src/week37/week37.do.txt
index b13809744..80220d91b 100644
--- a/doc/src/week37/week37.do.txt
+++ b/doc/src/week37/week37.do.txt
@@ -51,6 +51,7 @@ o The cost function is convex which guarantees that gradient descent converges f
We revisit an example similar to what we had in the first homework set. We have a function of the type
!bc pycod
+import numpy as np
x = 2*np.random.rand(m,1)
y = 4+3*x+np.random.randn(m,1)
!ec
@@ -857,7 +858,7 @@ o Initialize $r_0 = 0$ (an all-zero vector in $\mathbb{R}^d$).
o At each iteration $t$, update the accumulation:
!bt
\[
-r_t =; r_{t-1} + g_t \circ g_t,
+r_t = r_{t-1} + g_t \circ g_t,
\]
!et
o Here $g_t \circ g_t$ denotes element-wise square of the gradient vector. $g_t^{(j)} = g_{t-1}^{(j)} + (g_{t,j})^2$ for each parameter $j$.
@@ -892,9 +893,9 @@ Equivalently, the effective learning rate for parameter $j$ at time $t$ is $\dis
===== AdaGrad Properties =====
o AdaGrad automatically tunes the step size for each parameter. Parameters with more *volatile or large gradients* get smaller steps, and those with *small or infrequent gradients* get relatively larger steps
-o No manual schedule needed: The accumulation $h_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\eta/\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
+o No manual schedule needed: The accumulation $r_t$ keeps increasing (or stays the same if gradient is zero), so step sizes $\eta/\sqrt{r_t}$ are non-increasing. This has a similar effect to a learning rate schedule, but individualized per coordinate.
o Sparse data benefit: For very sparse features, $r_{t,j}$ grows slowly, so that feature’s parameter retains a higher learning rate for longer, allowing it to make significant updates when it does get a gradient signal
-o Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate (e.g. $O(1/\sqrt{T})$ regret bound) comparable to the best fixed learning rate tuned for the problem
+o Convergence: In convex optimization, AdaGrad can be shown to achieve a sub-linear convergence rate comparable to the best fixed learning rate tuned for the problem
It effectively reduces the need to tune $\eta$ by hand.
o Limitations: Because $r_t$ accumulates without bound, AdaGrad’s learning rates can become extremely small over long training, potentially slowing progress. (Later variants like RMSProp, AdaDelta, Adam address this by modifying the accumulation rule.)
@@ -906,11 +907,11 @@ Addresses AdaGrad’s diminishing learning rate issue.
Uses a decaying average of squared gradients (instead of a cumulative sum):
!bt
\[
-v_t = \alpha_2 v_{t-1} + (1-\alpha_2)(\nabla C(\theta_t))^2,
+v_t = \rho v_{t-1} + (1-\rho)(\nabla C(\theta_t))^2,
\]
!et
-with $\alpha_2$ typically $0.9$ (or $0.99$).
-o Update: $\theta_{t+1} = \theta_t - \frac{\alpha}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t)$.
+with $\rho$ typically $0.9$ (or $0.99$).
+o Update: $\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{v_t + \epsilon}} \nabla C(\theta_t)$.
o Recent gradients have more weight, so $v_t$ adapts to the current landscape.
o Avoids AdaGrad’s “infinite memory” problem – learning rate does not continuously decay to zero.
RMSProp was first proposed in lecture notes by Geoff Hinton, 2012 -- unpublished.)
@@ -923,7 +924,7 @@ FIGURE: [figures/rmsprop.png, width=600 frac=0.8]
!split
===== Adam Optimizer =====
-Why Combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.
+Why combine Momentum and RMSProp? Motivation for Adam: Adaptive Moment Estimation (Adam) was introduced by Kingma an Ba (2014) to combine the benefits of momentum and RMSProp.
o Fast convergence by smoothing gradients (accelerates in long-term gradient direction).
o Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).
@@ -946,43 +947,91 @@ problems involving lots data and/or parameters. It is a combination of the
gradient descent with momentum algorithm and the RMSprop algorithm
discussed above.
-In addition to keeping a running average of the first and
-second moments of the gradient
-(i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and
-$\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM
-performs an additional bias correction to account for the fact that we
-are estimating the first two moments of the gradient using a running
-average (denoted by the hats in the update rule below). The update
-rule for ADAM is given by (where multiplication and division are once
-again understood to be element-wise operations below)
-!bt
-\begin{align}
-\mathbf{g}_t &= \nabla_\theta E(\boldsymbol{\theta}) \\
-\mathbf{m}_t &= \theta_1 \mathbf{m}_{t-1} + (1-\theta_1) \mathbf{g}_t \nonumber \\
-\mathbf{s}_t &=\theta_2 \mathbf{s}_{t-1} +(1-\theta_2)\mathbf{g}_t^2 \nonumber \\
-\bm{\mathbf{m}}_t&={\mathbf{m}_t \over 1-\theta_1^t} \nonumber \\
-\bm{\mathbf{s}}_t &={\mathbf{s}_t \over1-\theta_2^t} \nonumber \\
-\boldsymbol{\theta}_{t+1}&=\boldsymbol{\theta}_t - \eta_t { \bm{\mathbf{m}}_t \over \sqrt{\bm{\mathbf{s}}_t} +\epsilon}, \nonumber \\
-\end{align}
-!et
+!split
+===== Why Combine Momentum and RMSProp? =====
-where $\theta_1$ and $\theta_2$ set the memory lifetime of the first and
-second moment and are typically taken to be $0.9$ and $0.99$
-respectively, and $\eta$ and $\epsilon$ are identical to RMSprop.
+o Momentum: Fast convergence by smoothing gradients (accelerates in long-term gradient direction).
+o Adaptive rates (RMSProp): Per-dimension learning rate scaling for stability (handles different feature scales, sparse gradients).
+o Adam uses both: maintains moving averages of both first moment (gradients) and second moment (squared gradients)
+o Additionally, includes a mechanism to correct the bias in these moving averages (crucial in early iterations)
-Like in RMSprop, the effective step size of a parameter depends on the
-magnitude of its gradient squared. To understand this better, let us
-rewrite this expression in terms of the variance
-$\boldsymbol{\sigma}_t^2 = \bm{\mathbf{s}}_t -
-(\bm{\mathbf{m}}_t)^2$. Consider a single parameter $\theta_t$. The
-update rule for this parameter is given by
+Result: Adam is robust, achieves faster convergence with less tuning, and often outperforms SGD (with momentum) in practice
+!split
+===== Adam: Exponential Moving Averages (Moments) =====
+Adam maintains two moving averages at each time step $t$ for each parameter $w$:
+!bblock First moment (mean) $m_t$
+The Momentum term
!bt
\[
-\Delta \theta_{t+1}= -\eta_t { \bm{m}_t \over \sqrt{\sigma_t^2 + m_t^2 }+\epsilon}.
+m_t = \beta_1m_{t-1} + (1-\beta_1)\, \nabla C(\theta_t),
\]
!et
+!eblock
+!bblock Second moment (uncentered variance) $v_t$
+The RMS term
+!bt
+\[
+v_t = \beta_2v_{t-1} + (1-\beta_2)(\nabla C(\theta_t))^2,
+\]
+!et
+with typical $\beta_1 = 0.9$, $\beta_2 = 0.999$. Initialize $m_0 = 0$, $v_0 = 0$.
+!eblock
+ These are _biased_ estimators of the true first and second moment of the gradients, especially at the start (since $m_0,v_0$ are zero)
+
+!split
+===== Adam: Bias Correction =====
+To counteract initialization bias in $m_t, v_t$, Adam computes bias-corrected estimates
+!bt
+\[
+\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}.
+\]
+!et
+* When $t$ is small, $1-\beta_i^t \approx 0$, so $\hat{m}_t, \hat{v}_t$ significantly larger than raw $m_t, v_t$, compensating for the initial zero bias.
+* As $t$ increases, $1-\beta_i^t \to 1$, and $\hat{m}_t, \hat{v}_t$ converge to $m_t, v_t$.
+* Bias correction is important for Adam’s stability in early iterations
+
+!split
+===== Adam: Update Rule Derivation =====
+Finally, Adam updates parameters using the bias-corrected moments:
+!bt
+\[
+\theta_{t+1} =\theta_t -\frac{\alpha}{\sqrt{\hat{v}_t} + \epsilon}\hat{m}_t,
+\]
+!et
+where $\epsilon$ is a small constant (e.g. $10^{-8}$) to prevent division by zero.
+Breaking it down:
+o Compute gradient $\nabla C(\theta_t)$.
+o Update first moment $m_t$ and second moment $v_t$ (exponential moving averages).
+o Bias-correct: $\hat{m}_t = m_t/(1-\beta_1^t)$, $\; \hat{v}_t = v_t/(1-\beta_2^t)$.
+o Compute step: $\Delta \theta_t = \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}$.
+o Update parameters: $\theta_{t+1} = \theta_t - \alpha\, \Delta \theta_t$.
+This is the Adam update rule as given in the original paper.
+
+!split
+===== Adam vs. AdaGrad and RMSProp =====
+
+o AdaGrad: Uses per-coordinate scaling like Adam, but no momentum. Tends to slow down too much due to cumulative history (no forgetting)
+o RMSProp: Uses moving average of squared gradients (like Adam’s $v_t$) to maintain adaptive learning rates, but does not include momentum or bias-correction.
+o Adam: Effectively RMSProp + Momentum + Bias-correction
+ * Momentum ($m_t$) provides acceleration and smoother convergence.
+ * Adaptive $v_t$ scaling moderates the step size per dimension.
+ * Bias correction (absent in AdaGrad/RMSProp) ensures robust estimates early on.
+
+In practice, Adam often yields faster convergence and better tuning stability than RMSProp or AdaGrad alone
+
+
+!split
+===== Adaptivity Across Dimensions =====
+
+o Adam adapts the step size \emph{per coordinate}: parameters with larger gradient variance get smaller effective steps, those with smaller or sparse gradients get larger steps.
+o This per-dimension adaptivity is inherited from AdaGrad/RMSProp and helps handle ill-conditioned or sparse problems.
+o Meanwhile, momentum (first moment) allows Adam to continue making progress even if gradients become small or noisy, by leveraging accumulated direction.
+
+
+
+
===== ADAM algorithm, taken from "Goodfellow et al":"https://www.deeplearningbook.org/contents/optimization.html" =====