diff --git a/doc/pub/week39/html/week39-bs.html b/doc/pub/week39/html/week39-bs.html index 8da1f374f..63c281832 100644 --- a/doc/pub/week39/html/week39-bs.html +++ b/doc/pub/week39/html/week39-bs.html @@ -203,6 +203,10 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Algorithms and codes for Adagrad, RMSprop and Adam', + 2, + None, + 'algorithms-and-codes-for-adagrad-rmsprop-and-adam'), ('Practical tips', 2, None, 'practical-tips'), ('Automatic differentiation', 2, @@ -342,7 +346,7 @@ MathJax.Hub.Config({
  • Program example for gradient descent with Ridge Regression
  • Using gradient descent methods, limitations
  • Improving gradient descent with momentum
    • -
  • Same code but now with momentum gradient descent
    • +
  • Same code but now with momentum gradient descent
  • Overview video on Stochastic Gradient Descent
  • Batches and mini-batches
  • Stochastic Gradient Descent (SGD)
    • @@ -362,28 +366,29 @@ MathJax.Hub.Config({
  • Second moment of the gradient
  • RMS prop
  • "ADAM optimizer":"https://arxiv.org/abs/1412.6980"
    • -
  • Practical tips
    • -
  • Automatic differentiation
    • -
  • Using autograd
    • -
  • Autograd with more complicated functions
    • -
  • More complicated functions using the elements of their arguments directly
    • -
  • Functions using mathematical functions from Numpy
    • -
  • More autograd
    • -
  • And with loops
    • -
  • Using recursion
    • -
  • Unsupported functions
    • -
  • The syntax a.dot(b) when finding the dot product
    • -
  • Recommended to avoid
    • -
  • Using Autograd with OLS
    • -
  • Same code but now with momentum gradient descent
    • -
  • But noen of these can compete with Newton's method
    • -
  • Including Stochastic Gradient Descent with Autograd
    • -
  • Same code but now with momentum gradient descent
    • -
  • Similar (second order function now) problem but now with AdaGrad
    • -
  • RMSprop for adaptive learning rate with Stochastic Gradient Descent
    • -
  • And Logistic Regression
    • -
  • Introducing "JAX":"https://jax.readthedocs.io/en/latest/"
    • -
  • Weekend challenge
    • +
  • Algorithms and codes for Adagrad, RMSprop and Adam
    • +
  • Practical tips
    • +
  • Automatic differentiation
    • +
  • Using autograd
    • +
  • Autograd with more complicated functions
    • +
  • More complicated functions using the elements of their arguments directly
    • +
  • Functions using mathematical functions from Numpy
    • +
  • More autograd
    • +
  • And with loops
    • +
  • Using recursion
    • +
  • Unsupported functions
    • +
  • The syntax a.dot(b) when finding the dot product
    • +
  • Recommended to avoid
    • +
  • Using Autograd with OLS
    • +
  • Same code but now with momentum gradient descent
    • +
  • But noen of these can compete with Newton's method
    • +
  • Including Stochastic Gradient Descent with Autograd
    • +
  • Same code but now with momentum gradient descent
    • +
  • Similar (second order function now) problem but now with AdaGrad
    • +
  • RMSprop for adaptive learning rate with Stochastic Gradient Descent
    • +
  • And Logistic Regression
    • +
  • Introducing "JAX":"https://jax.readthedocs.io/en/latest/"
    • +
  • Weekend challenge
    • @@ -438,7 +443,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 87
    • +
  • 88
  • »
    • diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index 612a3d1a6..981e59452 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -2370,10 +2370,17 @@ learning rate for flat directions.

    ADAM optimizer

    -

    A related algorithm is the ADAM optimizer. In ADAM, we keep a running -average of both the first and second moment of the gradient and use -this information to adaptively change the learning rate for different -parameters. In addition to keeping a running average of the first and +

    A related algorithm is the ADAM optimizer. In +ADAM, we keep a running average of +both the first and second moment of the gradient and use this +information to adaptively change the learning rate for different +parameters. The method isefficient when working with large +problems involving lots data and/or parameters. It is a combination of the +gradient descent with momentum algorithm and the RMSprop algorithm +discussed above. +

    + +

    In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM @@ -2419,6 +2426,14 @@ $$

     

    +
    +

    Algorithms and codes for Adagrad, RMSprop and Adam

    + +

    The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8.

    + +

    The codes which implement these algorithms are discussed after our presentation of automatic differentiation.

    +
    +

    Practical tips

    diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index 5eceeedf7..413df7668 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -230,6 +230,10 @@ div.toc p,a { 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Algorithms and codes for Adagrad, RMSprop and Adam', + 2, + None, + 'algorithms-and-codes-for-adagrad-rmsprop-and-adam'), ('Practical tips', 2, None, 'practical-tips'), ('Automatic differentiation', 2, @@ -2318,10 +2322,17 @@ learning rate for flat directions.









    ADAM optimizer

    -

    A related algorithm is the ADAM optimizer. In ADAM, we keep a running -average of both the first and second moment of the gradient and use -this information to adaptively change the learning rate for different -parameters. In addition to keeping a running average of the first and +

    A related algorithm is the ADAM optimizer. In +ADAM, we keep a running average of +both the first and second moment of the gradient and use this +information to adaptively change the learning rate for different +parameters. The method isefficient when working with large +problems involving lots data and/or parameters. It is a combination of the +gradient descent with momentum algorithm and the RMSprop algorithm +discussed above. +

    + +

    In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM @@ -2363,6 +2374,13 @@ $$ $$ +









    +

    Algorithms and codes for Adagrad, RMSprop and Adam

    + +

    The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8.

    + +

    The codes which implement these algorithms are discussed after our presentation of automatic differentiation.

    +









    Practical tips

    diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 7f1b69db8..d3a763cce 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -307,6 +307,10 @@ div.toc p,a { 2, None, 'adam-optimizer-https-arxiv-org-abs-1412-6980'), + ('Algorithms and codes for Adagrad, RMSprop and Adam', + 2, + None, + 'algorithms-and-codes-for-adagrad-rmsprop-and-adam'), ('Practical tips', 2, None, 'practical-tips'), ('Automatic differentiation', 2, @@ -2395,10 +2399,17 @@ learning rate for flat directions.









    ADAM optimizer

    -

    A related algorithm is the ADAM optimizer. In ADAM, we keep a running -average of both the first and second moment of the gradient and use -this information to adaptively change the learning rate for different -parameters. In addition to keeping a running average of the first and +

    A related algorithm is the ADAM optimizer. In +ADAM, we keep a running average of +both the first and second moment of the gradient and use this +information to adaptively change the learning rate for different +parameters. The method isefficient when working with large +problems involving lots data and/or parameters. It is a combination of the +gradient descent with momentum algorithm and the RMSprop algorithm +discussed above. +

    + +

    In addition to keeping a running average of the first and second moments of the gradient (i.e. \( \mathbf{m}_t=\mathbb{E}[\mathbf{g}_t] \) and \( \mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t] \), respectively), ADAM @@ -2440,6 +2451,13 @@ $$ $$ +









    +

    Algorithms and codes for Adagrad, RMSprop and Adam

    + +

    The algorithms we have implemented are well described in the text by Goodfellow, Bengio and Courville, chapter 8.

    + +

    The codes which implement these algorithms are discussed after our presentation of automatic differentiation.

    +









    Practical tips

    diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index d0530ffbe..9637528b3 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 10aeb280c..c993ceaf5 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "831ab16c", + "id": "f5eeed58", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "1a89a1bf", + "id": "041e3d30", "metadata": { "editable": true }, @@ -29,7 +29,7 @@ }, { "cell_type": "markdown", - "id": "a48d4553", + "id": "df619ba6", "metadata": { "editable": true }, @@ -55,7 +55,7 @@ }, { "cell_type": "markdown", - "id": "b76fe6e1", + "id": "1a9d86fd", "metadata": { "editable": true }, @@ -76,7 +76,7 @@ }, { "cell_type": "markdown", - "id": "8efd7af4", + "id": "ff350bb2", "metadata": { "editable": true }, @@ -93,7 +93,7 @@ }, { "cell_type": "markdown", - "id": "46208ba4", + "id": "959e1599", "metadata": { "editable": true }, @@ -108,7 +108,7 @@ }, { "cell_type": "markdown", - "id": "a2d78f95", + "id": "dadc0b80", "metadata": { "editable": true }, @@ -118,7 +118,7 @@ }, { "cell_type": "markdown", - "id": "6089b10e", + "id": "1542dbf4", "metadata": { "editable": true }, @@ -134,7 +134,7 @@ }, { "cell_type": "markdown", - "id": "7d1ae41a", + "id": "6690998e", "metadata": { "editable": true }, @@ -146,7 +146,7 @@ }, { "cell_type": "markdown", - "id": "d132fa52", + "id": "585d65ca", "metadata": { "editable": true }, @@ -157,7 +157,7 @@ }, { "cell_type": "markdown", - "id": "b56b1ac3", + "id": "43872b1a", "metadata": { "editable": true }, @@ -169,7 +169,7 @@ }, { "cell_type": "markdown", - "id": "466913e9", + "id": "8e8d2ad6", "metadata": { "editable": true }, @@ -179,7 +179,7 @@ }, { "cell_type": "markdown", - "id": "30c1f3cc", + "id": "b085449f", "metadata": { "editable": true }, @@ -193,7 +193,7 @@ }, { "cell_type": "markdown", - "id": "09c04703", + "id": "4f533137", "metadata": { "editable": true }, @@ -205,7 +205,7 @@ }, { "cell_type": "markdown", - "id": "99eea598", + "id": "37895ab0", "metadata": { "editable": true }, @@ -215,7 +215,7 @@ }, { "cell_type": "markdown", - "id": "7604810b", + "id": "cfb0635e", "metadata": { "editable": true }, @@ -227,7 +227,7 @@ }, { "cell_type": "markdown", - "id": "44b47354", + "id": "bc765b30", "metadata": { "editable": true }, @@ -239,7 +239,7 @@ }, { "cell_type": "markdown", - "id": "1190ad62", + "id": "e99b34ec", "metadata": { "editable": true }, @@ -259,7 +259,7 @@ }, { "cell_type": "markdown", - "id": "abf9f046", + "id": "4f129fed", "metadata": { "editable": true }, @@ -275,7 +275,7 @@ }, { "cell_type": "markdown", - "id": "4b991659", + "id": "decab89c", "metadata": { "editable": true }, @@ -291,7 +291,7 @@ }, { "cell_type": "markdown", - "id": "fa285a83", + "id": "3bb0a4c7", "metadata": { "editable": true }, @@ -302,7 +302,7 @@ }, { "cell_type": "markdown", - "id": "2163d473", + "id": "2f6c4862", "metadata": { "editable": true }, @@ -314,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "d7edfa0e", + "id": "b6bb5588", "metadata": { "editable": true }, @@ -324,7 +324,7 @@ }, { "cell_type": "markdown", - "id": "650aea98", + "id": "39c30f20", "metadata": { "editable": true }, @@ -336,7 +336,7 @@ }, { "cell_type": "markdown", - "id": "136077c3", + "id": "32cb43d3", "metadata": { "editable": true }, @@ -346,7 +346,7 @@ }, { "cell_type": "markdown", - "id": "6100b858", + "id": "c93f9b58", "metadata": { "editable": true }, @@ -358,7 +358,7 @@ }, { "cell_type": "markdown", - "id": "09d246f7", + "id": "1f05a82b", "metadata": { "editable": true }, @@ -380,7 +380,7 @@ }, { "cell_type": "markdown", - "id": "cf082864", + "id": "3bda01dd", "metadata": { "editable": true }, @@ -393,7 +393,7 @@ }, { "cell_type": "markdown", - "id": "a83bd82a", + "id": "09af289b", "metadata": { "editable": true }, @@ -406,7 +406,7 @@ }, { "cell_type": "markdown", - "id": "04bc02b4", + "id": "7587e9b7", "metadata": { "editable": true }, @@ -416,7 +416,7 @@ }, { "cell_type": "markdown", - "id": "c34bc1b5", + "id": "363582f4", "metadata": { "editable": true }, @@ -434,7 +434,7 @@ }, { "cell_type": "markdown", - "id": "c1a8421b", + "id": "a4f36649", "metadata": { "editable": true }, @@ -444,7 +444,7 @@ }, { "cell_type": "markdown", - "id": "3b9ae588", + "id": "9d3b6c37", "metadata": { "editable": true }, @@ -459,7 +459,7 @@ }, { "cell_type": "markdown", - "id": "c83b53c3", + "id": "8a225302", "metadata": { "editable": true }, @@ -469,7 +469,7 @@ }, { "cell_type": "markdown", - "id": "4e116635", + "id": "765ad822", "metadata": { "editable": true }, @@ -483,7 +483,7 @@ }, { "cell_type": "markdown", - "id": "519b8461", + "id": "41e026a9", "metadata": { "editable": true }, @@ -493,7 +493,7 @@ }, { "cell_type": "markdown", - "id": "175a09e1", + "id": "5b1b1fdc", "metadata": { "editable": true }, @@ -507,7 +507,7 @@ }, { "cell_type": "markdown", - "id": "fc897577", + "id": "66134dcd", "metadata": { "editable": true }, @@ -522,7 +522,7 @@ }, { "cell_type": "markdown", - "id": "dd88592c", + "id": "6415873d", "metadata": { "editable": true }, @@ -539,7 +539,7 @@ }, { "cell_type": "markdown", - "id": "7f0c3e7d", + "id": "61e7d8e3", "metadata": { "editable": true }, @@ -551,7 +551,7 @@ }, { "cell_type": "markdown", - "id": "f4f75ab5", + "id": "6ab0f2f3", "metadata": { "editable": true }, @@ -565,7 +565,7 @@ }, { "cell_type": "markdown", - "id": "7872bbf4", + "id": "76f334ba", "metadata": { "editable": true }, @@ -580,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "9dc53b5c", + "id": "ee44185b", "metadata": { "editable": true }, @@ -592,7 +592,7 @@ }, { "cell_type": "markdown", - "id": "d14ffd8a", + "id": "7ba50769", "metadata": { "editable": true }, @@ -603,7 +603,7 @@ }, { "cell_type": "markdown", - "id": "51d0991d", + "id": "4715abbd", "metadata": { "editable": true }, @@ -631,7 +631,7 @@ }, { "cell_type": "markdown", - "id": "f8f5d523", + "id": "a184c9b1", "metadata": { "editable": true }, @@ -653,7 +653,7 @@ }, { "cell_type": "markdown", - "id": "bd56a953", + "id": "1a88b5f0", "metadata": { "editable": true }, @@ -675,7 +675,7 @@ }, { "cell_type": "markdown", - "id": "da048fc7", + "id": "48588784", "metadata": { "editable": true }, @@ -687,7 +687,7 @@ }, { "cell_type": "markdown", - "id": "1fa9515a", + "id": "47212f96", "metadata": { "editable": true }, @@ -724,7 +724,7 @@ }, { "cell_type": "markdown", - "id": "4603676b", + "id": "ba003ec5", "metadata": { "editable": true }, @@ -752,7 +752,7 @@ }, { "cell_type": "markdown", - "id": "a1034044", + "id": "2d3cc1ac", "metadata": { "editable": true }, @@ -782,7 +782,7 @@ }, { "cell_type": "markdown", - "id": "81655e20", + "id": "9092858f", "metadata": { "editable": true }, @@ -802,7 +802,7 @@ }, { "cell_type": "markdown", - "id": "4db9bda7", + "id": "f177a49d", "metadata": { "editable": true }, @@ -814,7 +814,7 @@ }, { "cell_type": "markdown", - "id": "3e840e3a", + "id": "a03b3ade", "metadata": { "editable": true }, @@ -824,7 +824,7 @@ }, { "cell_type": "markdown", - "id": "600e0070", + "id": "f18a3516", "metadata": { "editable": true }, @@ -836,7 +836,7 @@ }, { "cell_type": "markdown", - "id": "d2fe45d9", + "id": "3d664192", "metadata": { "editable": true }, @@ -848,7 +848,7 @@ }, { "cell_type": "markdown", - "id": "faed650b", + "id": "9f860f40", "metadata": { "editable": true }, @@ -860,7 +860,7 @@ }, { "cell_type": "markdown", - "id": "0561cb94", + "id": "674748a4", "metadata": { "editable": true }, @@ -872,7 +872,7 @@ }, { "cell_type": "markdown", - "id": "e5a243bf", + "id": "ce6c46f8", "metadata": { "editable": true }, @@ -883,7 +883,7 @@ }, { "cell_type": "markdown", - "id": "b96eac4e", + "id": "ad7edeae", "metadata": { "editable": true }, @@ -896,7 +896,7 @@ }, { "cell_type": "markdown", - "id": "f5b81f53", + "id": "d73c5e6c", "metadata": { "editable": true }, @@ -908,7 +908,7 @@ }, { "cell_type": "markdown", - "id": "9d6f7709", + "id": "19a9b9e0", "metadata": { "editable": true }, @@ -918,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "d1719061", + "id": "6d3a6453", "metadata": { "editable": true }, @@ -930,7 +930,7 @@ }, { "cell_type": "markdown", - "id": "baa11a7a", + "id": "45c9f786", "metadata": { "editable": true }, @@ -940,7 +940,7 @@ }, { "cell_type": "markdown", - "id": "b46fb88b", + "id": "4695ba9c", "metadata": { "editable": true }, @@ -951,7 +951,7 @@ }, { "cell_type": "markdown", - "id": "ecd6c1c0", + "id": "353aa093", "metadata": { "editable": true }, @@ -963,7 +963,7 @@ }, { "cell_type": "markdown", - "id": "759598fd", + "id": "d92a06b9", "metadata": { "editable": true }, @@ -975,7 +975,7 @@ }, { "cell_type": "markdown", - "id": "0486631f", + "id": "a5568ef0", "metadata": { "editable": true }, @@ -987,7 +987,7 @@ }, { "cell_type": "markdown", - "id": "86721e06", + "id": "fbf7f79c", "metadata": { "editable": true }, @@ -998,7 +998,7 @@ }, { "cell_type": "markdown", - "id": "fe4f0621", + "id": "b3472f49", "metadata": { "editable": true }, @@ -1009,7 +1009,7 @@ }, { "cell_type": "markdown", - "id": "93aa0a1b", + "id": "600c0bda", "metadata": { "editable": true }, @@ -1021,7 +1021,7 @@ }, { "cell_type": "markdown", - "id": "e3db98cd", + "id": "52184918", "metadata": { "editable": true }, @@ -1031,7 +1031,7 @@ }, { "cell_type": "markdown", - "id": "01fe93a3", + "id": "c4ef33bd", "metadata": { "editable": true }, @@ -1043,7 +1043,7 @@ }, { "cell_type": "markdown", - "id": "03c84175", + "id": "fc17dee3", "metadata": { "editable": true }, @@ -1053,7 +1053,7 @@ }, { "cell_type": "markdown", - "id": "8179368d", + "id": "d1254e14", "metadata": { "editable": true }, @@ -1065,7 +1065,7 @@ }, { "cell_type": "markdown", - "id": "dd73dfbe", + "id": "5fba6788", "metadata": { "editable": true }, @@ -1075,7 +1075,7 @@ }, { "cell_type": "markdown", - "id": "572abe34", + "id": "042a4df4", "metadata": { "editable": true }, @@ -1087,7 +1087,7 @@ }, { "cell_type": "markdown", - "id": "c89821cc", + "id": "6a711a48", "metadata": { "editable": true }, @@ -1097,7 +1097,7 @@ }, { "cell_type": "markdown", - "id": "5a690582", + "id": "95c4182e", "metadata": { "editable": true }, @@ -1109,7 +1109,7 @@ }, { "cell_type": "markdown", - "id": "0abad81b", + "id": "178acda1", "metadata": { "editable": true }, @@ -1120,7 +1120,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "f6a032e9", + "id": "1e575c26", "metadata": { "collapsed": false, "editable": true @@ -1153,7 +1153,7 @@ }, { "cell_type": "markdown", - "id": "77b7155c", + "id": "a41c1a9b", "metadata": { "editable": true }, @@ -1164,7 +1164,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "28f56acb", + "id": "6dca64f1", "metadata": { "collapsed": false, "editable": true @@ -1178,7 +1178,7 @@ }, { "cell_type": "markdown", - "id": "609d822d", + "id": "28ecfa4c", "metadata": { "editable": true }, @@ -1189,7 +1189,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "bbfb7f41", + "id": "fe79d750", "metadata": { "collapsed": false, "editable": true @@ -1202,7 +1202,7 @@ }, { "cell_type": "markdown", - "id": "651bc36e", + "id": "1dc0d664", "metadata": { "editable": true }, @@ -1213,7 +1213,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "bca365f4", + "id": "eaf477ea", "metadata": { "collapsed": false, "editable": true @@ -1231,7 +1231,7 @@ }, { "cell_type": "markdown", - "id": "a7e8c4e3", + "id": "98d06c7b", "metadata": { "editable": true }, @@ -1242,7 +1242,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "ae10a006", + "id": "5a1c85b0", "metadata": { "collapsed": false, "editable": true @@ -1257,7 +1257,7 @@ }, { "cell_type": "markdown", - "id": "1f64a7d9", + "id": "d6d958db", "metadata": { "editable": true }, @@ -1267,7 +1267,7 @@ }, { "cell_type": "markdown", - "id": "50bef8bb", + "id": "95d53abb", "metadata": { "editable": true }, @@ -1281,7 +1281,7 @@ }, { "cell_type": "markdown", - "id": "ebf2dcec", + "id": "c2018a9c", "metadata": { "editable": true }, @@ -1293,7 +1293,7 @@ }, { "cell_type": "markdown", - "id": "e4ce5a29", + "id": "e5c9ebb6", "metadata": { "editable": true }, @@ -1304,7 +1304,7 @@ }, { "cell_type": "markdown", - "id": "dddd83fc", + "id": "afdf1eef", "metadata": { "editable": true }, @@ -1316,7 +1316,7 @@ }, { "cell_type": "markdown", - "id": "b1be09ba", + "id": "4ba8c994", "metadata": { "editable": true }, @@ -1327,7 +1327,7 @@ }, { "cell_type": "markdown", - "id": "35b17982", + "id": "507eb4e1", "metadata": { "editable": true }, @@ -1338,7 +1338,7 @@ }, { "cell_type": "markdown", - "id": "56f8f452", + "id": "cb6a068f", "metadata": { "editable": true }, @@ -1350,7 +1350,7 @@ }, { "cell_type": "markdown", - "id": "c840acb5", + "id": "7ffadb53", "metadata": { "editable": true }, @@ -1360,7 +1360,7 @@ }, { "cell_type": "markdown", - "id": "8aabb73e", + "id": "a14e8c65", "metadata": { "editable": true }, @@ -1372,7 +1372,7 @@ }, { "cell_type": "markdown", - "id": "cd71a3b6", + "id": "0410f657", "metadata": { "editable": true }, @@ -1384,7 +1384,7 @@ }, { "cell_type": "markdown", - "id": "2fbf4ca3", + "id": "7c765fde", "metadata": { "editable": true }, @@ -1396,7 +1396,7 @@ }, { "cell_type": "markdown", - "id": "0682ff5b", + "id": "c0482ebb", "metadata": { "editable": true }, @@ -1408,7 +1408,7 @@ }, { "cell_type": "markdown", - "id": "dea76841", + "id": "476de327", "metadata": { "editable": true }, @@ -1419,7 +1419,7 @@ }, { "cell_type": "markdown", - "id": "7210c491", + "id": "c0278469", "metadata": { "editable": true }, @@ -1431,7 +1431,7 @@ }, { "cell_type": "markdown", - "id": "d70fad2e", + "id": "94a9c1de", "metadata": { "editable": true }, @@ -1441,7 +1441,7 @@ }, { "cell_type": "markdown", - "id": "9d31a2dc", + "id": "2d897dcc", "metadata": { "editable": true }, @@ -1453,7 +1453,7 @@ }, { "cell_type": "markdown", - "id": "bce0c379", + "id": "3b2a72b5", "metadata": { "editable": true }, @@ -1463,7 +1463,7 @@ }, { "cell_type": "markdown", - "id": "3f0f29b1", + "id": "6edc2d6e", "metadata": { "editable": true }, @@ -1475,7 +1475,7 @@ }, { "cell_type": "markdown", - "id": "6f8d45e4", + "id": "1d2c4fc6", "metadata": { "editable": true }, @@ -1495,7 +1495,7 @@ }, { "cell_type": "markdown", - "id": "164f114d", + "id": "29c94693", "metadata": { "editable": true }, @@ -1507,7 +1507,7 @@ }, { "cell_type": "markdown", - "id": "6168a5fa", + "id": "84f0f60c", "metadata": { "editable": true }, @@ -1517,7 +1517,7 @@ }, { "cell_type": "markdown", - "id": "084ad8db", + "id": "ef7d2a86", "metadata": { "editable": true }, @@ -1529,7 +1529,7 @@ }, { "cell_type": "markdown", - "id": "e1883221", + "id": "70c74483", "metadata": { "editable": true }, @@ -1539,7 +1539,7 @@ }, { "cell_type": "markdown", - "id": "075a780a", + "id": "3c6c125d", "metadata": { "editable": true }, @@ -1550,7 +1550,7 @@ }, { "cell_type": "markdown", - "id": "91c1abb6", + "id": "f5289963", "metadata": { "editable": true }, @@ -1562,7 +1562,7 @@ }, { "cell_type": "markdown", - "id": "7005733a", + "id": "51fcdca8", "metadata": { "editable": true }, @@ -1574,7 +1574,7 @@ }, { "cell_type": "markdown", - "id": "3865bead", + "id": "983a3c9f", "metadata": { "editable": true }, @@ -1586,7 +1586,7 @@ }, { "cell_type": "markdown", - "id": "6953c5f9", + "id": "3f719f47", "metadata": { "editable": true }, @@ -1599,7 +1599,7 @@ }, { "cell_type": "markdown", - "id": "f234b054", + "id": "79491e0f", "metadata": { "editable": true }, @@ -1610,7 +1610,7 @@ }, { "cell_type": "markdown", - "id": "e84398d8", + "id": "4ae40784", "metadata": { "editable": true }, @@ -1622,7 +1622,7 @@ }, { "cell_type": "markdown", - "id": "0de3dca1", + "id": "a2914381", "metadata": { "editable": true }, @@ -1638,7 +1638,7 @@ }, { "cell_type": "markdown", - "id": "216fb6c6", + "id": "efbf8133", "metadata": { "editable": true }, @@ -1650,7 +1650,7 @@ }, { "cell_type": "markdown", - "id": "f614bb20", + "id": "3bde74a6", "metadata": { "editable": true }, @@ -1661,7 +1661,7 @@ }, { "cell_type": "markdown", - "id": "9c78ff37", + "id": "bce26f0f", "metadata": { "editable": true }, @@ -1673,7 +1673,7 @@ }, { "cell_type": "markdown", - "id": "c39b751a", + "id": "5f38d7ec", "metadata": { "editable": true }, @@ -1683,7 +1683,7 @@ }, { "cell_type": "markdown", - "id": "2d1e7293", + "id": "9e81c2c1", "metadata": { "editable": true }, @@ -1695,7 +1695,7 @@ }, { "cell_type": "markdown", - "id": "5f73e45c", + "id": "6be2137c", "metadata": { "editable": true }, @@ -1705,7 +1705,7 @@ }, { "cell_type": "markdown", - "id": "685cb5be", + "id": "561bd5e2", "metadata": { "editable": true }, @@ -1717,7 +1717,7 @@ }, { "cell_type": "markdown", - "id": "60937263", + "id": "065640bd", "metadata": { "editable": true }, @@ -1727,7 +1727,7 @@ }, { "cell_type": "markdown", - "id": "d3a9ed73", + "id": "f3996317", "metadata": { "editable": true }, @@ -1739,7 +1739,7 @@ }, { "cell_type": "markdown", - "id": "2241adf9", + "id": "7ac551bb", "metadata": { "editable": true }, @@ -1763,7 +1763,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "77cabb13", + "id": "a8d5c582", "metadata": { "collapsed": false, "editable": true @@ -1776,7 +1776,7 @@ }, { "cell_type": "markdown", - "id": "8c1860a9", + "id": "c744ea5e", "metadata": { "editable": true }, @@ -1787,7 +1787,7 @@ }, { "cell_type": "markdown", - "id": "5f48be08", + "id": "82608a26", "metadata": { "editable": true }, @@ -1799,7 +1799,7 @@ }, { "cell_type": "markdown", - "id": "8cf4e546", + "id": "442e9467", "metadata": { "editable": true }, @@ -1809,7 +1809,7 @@ }, { "cell_type": "markdown", - "id": "b5139d61", + "id": "8ad46313", "metadata": { "editable": true }, @@ -1821,7 +1821,7 @@ }, { "cell_type": "markdown", - "id": "81ccbcdb", + "id": "e22ec716", "metadata": { "editable": true }, @@ -1835,7 +1835,7 @@ }, { "cell_type": "markdown", - "id": "d6159d58", + "id": "37d9cf9c", "metadata": { "editable": true }, @@ -1851,7 +1851,7 @@ }, { "cell_type": "markdown", - "id": "9b504eb5", + "id": "84918c4d", "metadata": { "editable": true }, @@ -1861,7 +1861,7 @@ }, { "cell_type": "markdown", - "id": "2524eb92", + "id": "dea330f5", "metadata": { "editable": true }, @@ -1873,7 +1873,7 @@ }, { "cell_type": "markdown", - "id": "eebbdaac", + "id": "1631c887", "metadata": { "editable": true }, @@ -1883,7 +1883,7 @@ }, { "cell_type": "markdown", - "id": "5dd05bbd", + "id": "d15acc27", "metadata": { "editable": true }, @@ -1895,7 +1895,7 @@ }, { "cell_type": "markdown", - "id": "d7a0de59", + "id": "1518cedf", "metadata": { "editable": true }, @@ -1909,7 +1909,7 @@ }, { "cell_type": "markdown", - "id": "6b27fc77", + "id": "a53a28bc", "metadata": { "editable": true }, @@ -1919,7 +1919,7 @@ }, { "cell_type": "markdown", - "id": "5aa8a7fd", + "id": "dc5fe31a", "metadata": { "editable": true }, @@ -1930,7 +1930,7 @@ }, { "cell_type": "markdown", - "id": "e3d47199", + "id": "ce10953f", "metadata": { "editable": true }, @@ -1945,7 +1945,7 @@ }, { "cell_type": "markdown", - "id": "c2824b7e", + "id": "9787a2d9", "metadata": { "editable": true }, @@ -1955,7 +1955,7 @@ }, { "cell_type": "markdown", - "id": "a514ed13", + "id": "0901e914", "metadata": { "editable": true }, @@ -1967,7 +1967,7 @@ }, { "cell_type": "markdown", - "id": "b262f130", + "id": "96a3e2ee", "metadata": { "editable": true }, @@ -1979,7 +1979,7 @@ }, { "cell_type": "markdown", - "id": "a3058264", + "id": "3d2b9277", "metadata": { "editable": true }, @@ -1994,7 +1994,7 @@ }, { "cell_type": "markdown", - "id": "d5ddb569", + "id": "8941ae44", "metadata": { "editable": true }, @@ -2007,7 +2007,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "70d399eb", + "id": "c732c9f2", "metadata": { "collapsed": false, "editable": true @@ -2064,7 +2064,7 @@ }, { "cell_type": "markdown", - "id": "e3d8f7df", + "id": "e177b538", "metadata": { "editable": true }, @@ -2075,7 +2075,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "8a1ba802", + "id": "369db91f", "metadata": { "collapsed": false, "editable": true @@ -2102,7 +2102,7 @@ }, { "cell_type": "markdown", - "id": "48eda1cc", + "id": "4d0ef8cb", "metadata": { "editable": true }, @@ -2114,7 +2114,7 @@ }, { "cell_type": "markdown", - "id": "abf2a3dd", + "id": "2e216f33", "metadata": { "editable": true }, @@ -2126,7 +2126,7 @@ }, { "cell_type": "markdown", - "id": "719e68d8", + "id": "83e09ead", "metadata": { "editable": true }, @@ -2136,7 +2136,7 @@ }, { "cell_type": "markdown", - "id": "1597f251", + "id": "01e43670", "metadata": { "editable": true }, @@ -2150,7 +2150,7 @@ }, { "cell_type": "markdown", - "id": "5dde41b4", + "id": "610047ec", "metadata": { "editable": true }, @@ -2160,7 +2160,7 @@ }, { "cell_type": "markdown", - "id": "982b0a69", + "id": "5dc793f1", "metadata": { "editable": true }, @@ -2172,7 +2172,7 @@ }, { "cell_type": "markdown", - "id": "16b0fb72", + "id": "8473e8e5", "metadata": { "editable": true }, @@ -2183,7 +2183,7 @@ }, { "cell_type": "markdown", - "id": "8c3e10f3", + "id": "dab49112", "metadata": { "editable": true }, @@ -2198,7 +2198,7 @@ }, { "cell_type": "markdown", - "id": "b643d525", + "id": "45942fb4", "metadata": { "editable": true }, @@ -2212,7 +2212,7 @@ }, { "cell_type": "markdown", - "id": "ad4765f6", + "id": "14955730", "metadata": { "editable": true }, @@ -2223,7 +2223,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "da0f3027", + "id": "e6f82b17", "metadata": { "collapsed": false, "editable": true @@ -2284,7 +2284,7 @@ }, { "cell_type": "markdown", - "id": "8e118b9d", + "id": "c21a5ac3", "metadata": { "editable": true }, @@ -2306,7 +2306,7 @@ }, { "cell_type": "markdown", - "id": "43e561d4", + "id": "8df86e97", "metadata": { "editable": true }, @@ -2319,7 +2319,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "be29dc07", + "id": "dc582705", "metadata": { "collapsed": false, "editable": true @@ -2385,7 +2385,7 @@ }, { "cell_type": "markdown", - "id": "22fdea1b", + "id": "fe2da0a9", "metadata": { "editable": true }, @@ -2396,7 +2396,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "27b0632b", + "id": "36a30290", "metadata": { "collapsed": false, "editable": true @@ -2470,7 +2470,7 @@ }, { "cell_type": "markdown", - "id": "2a6267ce", + "id": "f57b1a17", "metadata": { "editable": true }, @@ -2482,7 +2482,7 @@ }, { "cell_type": "markdown", - "id": "1a10c6d4", + "id": "9fde30f7", "metadata": { "editable": true }, @@ -2503,7 +2503,7 @@ }, { "cell_type": "markdown", - "id": "3c68aebd", + "id": "d07714cf", "metadata": { "editable": true }, @@ -2535,7 +2535,7 @@ }, { "cell_type": "markdown", - "id": "8283d20b", + "id": "6f5b98fc", "metadata": { "editable": true }, @@ -2552,7 +2552,7 @@ }, { "cell_type": "markdown", - "id": "fbdca45c", + "id": "577ba46e", "metadata": { "editable": true }, @@ -2565,7 +2565,7 @@ }, { "cell_type": "markdown", - "id": "f1f1f068", + "id": "ac4cd6c3", "metadata": { "editable": true }, @@ -2578,7 +2578,7 @@ }, { "cell_type": "markdown", - "id": "66a52755", + "id": "baf1f036", "metadata": { "editable": true }, @@ -2591,7 +2591,7 @@ }, { "cell_type": "markdown", - "id": "f47e74f0", + "id": "ee0eb339", "metadata": { "editable": true }, @@ -2605,7 +2605,7 @@ }, { "cell_type": "markdown", - "id": "0aeadf13", + "id": "cc4aa5ea", "metadata": { "editable": true }, @@ -2627,7 +2627,7 @@ }, { "cell_type": "markdown", - "id": "49d99f49", + "id": "8a26b84f", "metadata": { "editable": true }, @@ -2642,7 +2642,7 @@ }, { "cell_type": "markdown", - "id": "959a2b24", + "id": "2fd3bfa0", "metadata": { "editable": true }, @@ -2654,7 +2654,7 @@ }, { "cell_type": "markdown", - "id": "affa536f", + "id": "20c9898d", "metadata": { "editable": true }, @@ -2667,7 +2667,7 @@ }, { "cell_type": "markdown", - "id": "bcfde5f4", + "id": "bb7ed388", "metadata": { "editable": true }, @@ -2681,7 +2681,7 @@ }, { "cell_type": "markdown", - "id": "f308a718", + "id": "6b9f476b", "metadata": { "editable": true }, @@ -2692,7 +2692,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "b31d32cd", + "id": "861be06c", "metadata": { "collapsed": false, "editable": true @@ -2717,7 +2717,7 @@ }, { "cell_type": "markdown", - "id": "228dae78", + "id": "6d546149", "metadata": { "editable": true }, @@ -2733,7 +2733,7 @@ }, { "cell_type": "markdown", - "id": "31c0b03e", + "id": "92fe18e6", "metadata": { "editable": true }, @@ -2754,7 +2754,7 @@ }, { "cell_type": "markdown", - "id": "32b211da", + "id": "9614b80f", "metadata": { "editable": true }, @@ -2774,7 +2774,7 @@ }, { "cell_type": "markdown", - "id": "f794af58", + "id": "b333ddc0", "metadata": { "editable": true }, @@ -2793,7 +2793,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "72c1bec2", + "id": "43e2526f", "metadata": { "collapsed": false, "editable": true @@ -2828,7 +2828,7 @@ }, { "cell_type": "markdown", - "id": "611a188b", + "id": "1f8467fc", "metadata": { "editable": true }, @@ -2841,7 +2841,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "7be674cb", + "id": "ddba2679", "metadata": { "collapsed": false, "editable": true @@ -2918,7 +2918,7 @@ }, { "cell_type": "markdown", - "id": "3525ebc9", + "id": "8993069e", "metadata": { "editable": true }, @@ -2933,7 +2933,7 @@ }, { "cell_type": "markdown", - "id": "72f723b6", + "id": "a6f26f92", "metadata": { "editable": true }, @@ -2948,7 +2948,7 @@ }, { "cell_type": "markdown", - "id": "16ac1388", + "id": "a0b83fff", "metadata": { "editable": true }, @@ -2960,7 +2960,7 @@ }, { "cell_type": "markdown", - "id": "08d79365", + "id": "5530c40e", "metadata": { "editable": true }, @@ -2978,7 +2978,7 @@ }, { "cell_type": "markdown", - "id": "ffdb814c", + "id": "b85bef60", "metadata": { "editable": true }, @@ -2997,7 +2997,7 @@ }, { "cell_type": "markdown", - "id": "5e114fda", + "id": "fe8da4c4", "metadata": { "editable": true }, @@ -3009,7 +3009,7 @@ }, { "cell_type": "markdown", - "id": "8293d9ba", + "id": "91299ffe", "metadata": { "editable": true }, @@ -3019,7 +3019,7 @@ }, { "cell_type": "markdown", - "id": "de7d6799", + "id": "19ae1341", "metadata": { "editable": true }, @@ -3035,7 +3035,7 @@ }, { "cell_type": "markdown", - "id": "ba2bc11b", + "id": "05d461e5", "metadata": { "editable": true }, @@ -3047,7 +3047,7 @@ }, { "cell_type": "markdown", - "id": "08adfe31", + "id": "5d405644", "metadata": { "editable": true }, @@ -3057,7 +3057,7 @@ }, { "cell_type": "markdown", - "id": "5f9de2b1", + "id": "6ebfbe1c", "metadata": { "editable": true }, @@ -3069,7 +3069,7 @@ }, { "cell_type": "markdown", - "id": "cbd227bd", + "id": "7c62d8c5", "metadata": { "editable": true }, @@ -3079,7 +3079,7 @@ }, { "cell_type": "markdown", - "id": "a6eac778", + "id": "c3998019", "metadata": { "editable": true }, @@ -3091,7 +3091,7 @@ }, { "cell_type": "markdown", - "id": "1a81ccb9", + "id": "6547cfb0", "metadata": { "editable": true }, @@ -3107,7 +3107,7 @@ }, { "cell_type": "markdown", - "id": "ff8d8514", + "id": "566451ea", "metadata": { "editable": true }, @@ -3119,7 +3119,7 @@ }, { "cell_type": "markdown", - "id": "64ca1cbe", + "id": "238fbe95", "metadata": { "editable": true }, @@ -3152,7 +3152,7 @@ }, { "cell_type": "markdown", - "id": "000a6174", + "id": "aa84105f", "metadata": { "editable": true }, @@ -3164,7 +3164,7 @@ }, { "cell_type": "markdown", - "id": "91c16167", + "id": "3f44d0b4", "metadata": { "editable": true }, @@ -3182,7 +3182,7 @@ }, { "cell_type": "markdown", - "id": "44a6513e", + "id": "0f0129a7", "metadata": { "editable": true }, @@ -3192,7 +3192,7 @@ }, { "cell_type": "markdown", - "id": "4d3f46ab", + "id": "77e3d42f", "metadata": { "editable": true }, @@ -3223,7 +3223,7 @@ }, { "cell_type": "markdown", - "id": "3ab65226", + "id": "c79841ef", "metadata": { "editable": true }, @@ -3238,7 +3238,7 @@ }, { "cell_type": "markdown", - "id": "63af5a3d", + "id": "1385d44b", "metadata": { "editable": true }, @@ -3256,7 +3256,7 @@ }, { "cell_type": "markdown", - "id": "aa72ec54", + "id": "abbf8209", "metadata": { "editable": true }, @@ -3268,7 +3268,7 @@ }, { "cell_type": "markdown", - "id": "d20282e6", + "id": "3455b657", "metadata": { "editable": true }, @@ -3280,7 +3280,7 @@ }, { "cell_type": "markdown", - "id": "a23484c9", + "id": "964542c9", "metadata": { "editable": true }, @@ -3298,17 +3298,23 @@ }, { "cell_type": "markdown", - "id": "c6fdd601", + "id": "66ff6606", "metadata": { "editable": true }, "source": [ "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", "\n", - "A related algorithm is the ADAM optimizer. In [ADAM](https://arxiv.org/abs/1412.6980), we keep a running\n", - "average of both the first and second moment of the gradient and use\n", - "this information to adaptively change the learning rate for different\n", - "parameters. In addition to keeping a running average of the first and\n", + "A related algorithm is the ADAM optimizer. In\n", + "[ADAM](https://arxiv.org/abs/1412.6980), we keep a running average of\n", + "both the first and second moment of the gradient and use this\n", + "information to adaptively change the learning rate for different\n", + "parameters. The method isefficient when working with large\n", + "problems involving lots data and/or parameters. It is a combination of the\n", + "gradient descent with momentum algorithm and the RMSprop algorithm\n", + "discussed above.\n", + "\n", + "In addition to keeping a running average of the first and\n", "second moments of the gradient\n", "(i.e. $\\mathbf{m}_t=\\mathbb{E}[\\mathbf{g}_t]$ and\n", "$\\mathbf{s}_t=\\mathbb{E}[\\mathbf{g}^2_t]$, respectively), ADAM\n", @@ -3321,7 +3327,7 @@ }, { "cell_type": "markdown", - "id": "922c7fbd", + "id": "eb3a50c2", "metadata": { "editable": true }, @@ -3339,7 +3345,7 @@ }, { "cell_type": "markdown", - "id": "e5c76084", + "id": "fe4b2300", "metadata": { "editable": true }, @@ -3351,7 +3357,7 @@ }, { "cell_type": "markdown", - "id": "6ace1786", + "id": "910e0f94", "metadata": { "editable": true }, @@ -3363,7 +3369,7 @@ }, { "cell_type": "markdown", - "id": "0b4a86ec", + "id": "55d4e408", "metadata": { "editable": true }, @@ -3375,7 +3381,7 @@ }, { "cell_type": "markdown", - "id": "c6d9336a", + "id": "9e590482", "metadata": { "editable": true }, @@ -3387,7 +3393,7 @@ }, { "cell_type": "markdown", - "id": "4cbddf1a", + "id": "22660f5e", "metadata": { "editable": true }, @@ -3399,7 +3405,7 @@ }, { "cell_type": "markdown", - "id": "8a666e9f", + "id": "6cf31f55", "metadata": { "editable": true }, @@ -3416,7 +3422,7 @@ }, { "cell_type": "markdown", - "id": "a4cccc1e", + "id": "38eb2a6a", "metadata": { "editable": true }, @@ -3435,7 +3441,7 @@ }, { "cell_type": "markdown", - "id": "15812461", + "id": "a9f1a129", "metadata": { "editable": true }, @@ -3447,7 +3453,21 @@ }, { "cell_type": "markdown", - "id": "3e5cc6d5", + "id": "15bb1241", + "metadata": { + "editable": true + }, + "source": [ + "## Algorithms and codes for Adagrad, RMSprop and Adam\n", + "\n", + "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n", + "\n", + "The codes which implement these algorithms are discussed after our presentation of automatic differentiation." + ] + }, + { + "cell_type": "markdown", + "id": "7977bb32", "metadata": { "editable": true }, @@ -3467,7 +3487,7 @@ }, { "cell_type": "markdown", - "id": "eda86286", + "id": "25684316", "metadata": { "editable": true }, @@ -3505,7 +3525,7 @@ }, { "cell_type": "markdown", - "id": "be3e7165", + "id": "0acff299", "metadata": { "editable": true }, @@ -3517,7 +3537,7 @@ }, { "cell_type": "markdown", - "id": "26ca3832", + "id": "363cba19", "metadata": { "editable": true }, @@ -3527,7 +3547,7 @@ }, { "cell_type": "markdown", - "id": "7474a5bc", + "id": "d0e2eecb", "metadata": { "editable": true }, @@ -3539,7 +3559,7 @@ }, { "cell_type": "markdown", - "id": "b893478c", + "id": "057f24ca", "metadata": { "editable": true }, @@ -3550,7 +3570,7 @@ { "cell_type": "code", "execution_count": 15, - "id": "3d148b31", + "id": "1f7c200a", "metadata": { "collapsed": false, "editable": true @@ -3595,7 +3615,7 @@ }, { "cell_type": "markdown", - "id": "00451e68", + "id": "7a951fec", "metadata": { "editable": true }, @@ -3612,7 +3632,7 @@ { "cell_type": "code", "execution_count": 16, - "id": "28053212", + "id": "d6182b53", "metadata": { "collapsed": false, "editable": true @@ -3640,7 +3660,7 @@ }, { "cell_type": "markdown", - "id": "ed2f8b44", + "id": "09ae2932", "metadata": { "editable": true }, @@ -3655,7 +3675,7 @@ { "cell_type": "code", "execution_count": 17, - "id": "ad29df73", + "id": "97d76ab7", "metadata": { "collapsed": false, "editable": true @@ -3699,7 +3719,7 @@ }, { "cell_type": "markdown", - "id": "0819f320", + "id": "25e997f8", "metadata": { "editable": true }, @@ -3709,7 +3729,7 @@ }, { "cell_type": "markdown", - "id": "1d7ccac2", + "id": "c0317908", "metadata": { "editable": true }, @@ -3720,7 +3740,7 @@ { "cell_type": "code", "execution_count": 18, - "id": "bb9e4228", + "id": "9c31e5e4", "metadata": { "collapsed": false, "editable": true @@ -3748,7 +3768,7 @@ }, { "cell_type": "markdown", - "id": "be4e6d99", + "id": "48d62537", "metadata": { "editable": true }, @@ -3763,7 +3783,7 @@ }, { "cell_type": "markdown", - "id": "ba08cf51", + "id": "833b1c88", "metadata": { "editable": true }, @@ -3774,7 +3794,7 @@ { "cell_type": "code", "execution_count": 19, - "id": "2d175328", + "id": "67c295c6", "metadata": { "collapsed": false, "editable": true @@ -3802,7 +3822,7 @@ }, { "cell_type": "markdown", - "id": "aff28e3a", + "id": "2dd6483a", "metadata": { "editable": true }, @@ -3813,7 +3833,7 @@ { "cell_type": "code", "execution_count": 20, - "id": "edabb1dd", + "id": "ce65e6e4", "metadata": { "collapsed": false, "editable": true @@ -3838,7 +3858,7 @@ }, { "cell_type": "markdown", - "id": "af563ee3", + "id": "299a9b1d", "metadata": { "editable": true }, @@ -3849,7 +3869,7 @@ { "cell_type": "code", "execution_count": 21, - "id": "ac3ad080", + "id": "e0ce00ec", "metadata": { "collapsed": false, "editable": true @@ -3885,7 +3905,7 @@ { "cell_type": "code", "execution_count": 22, - "id": "a5beeab6", + "id": "29b82fe4", "metadata": { "collapsed": false, "editable": true @@ -3905,7 +3925,7 @@ }, { "cell_type": "markdown", - "id": "528ba061", + "id": "5ab98e85", "metadata": { "editable": true }, @@ -3916,7 +3936,7 @@ { "cell_type": "code", "execution_count": 23, - "id": "8237c400", + "id": "5ec23bfc", "metadata": { "collapsed": false, "editable": true @@ -3954,7 +3974,7 @@ }, { "cell_type": "markdown", - "id": "94a469eb", + "id": "f224a5ba", "metadata": { "editable": true }, @@ -3964,7 +3984,7 @@ }, { "cell_type": "markdown", - "id": "53771be6", + "id": "4fc0e83e", "metadata": { "editable": true }, @@ -3978,7 +3998,7 @@ { "cell_type": "code", "execution_count": 24, - "id": "22df1bf3", + "id": "ee9c6265", "metadata": { "collapsed": false, "editable": true @@ -4000,7 +4020,7 @@ }, { "cell_type": "markdown", - "id": "a5ca665d", + "id": "0f42fe39", "metadata": { "editable": true }, @@ -4010,7 +4030,7 @@ }, { "cell_type": "markdown", - "id": "8c6ab4ca", + "id": "59eb26f6", "metadata": { "editable": true }, @@ -4021,7 +4041,7 @@ { "cell_type": "code", "execution_count": 25, - "id": "66cdb9e9", + "id": "7fad7b56", "metadata": { "collapsed": false, "editable": true @@ -4043,7 +4063,7 @@ }, { "cell_type": "markdown", - "id": "a99da8ee", + "id": "74dc67b1", "metadata": { "editable": true }, @@ -4056,7 +4076,7 @@ { "cell_type": "code", "execution_count": 26, - "id": "4190a01e", + "id": "cb4cd73b", "metadata": { "collapsed": false, "editable": true @@ -4081,7 +4101,7 @@ }, { "cell_type": "markdown", - "id": "ce7085b3", + "id": "4306ba8f", "metadata": { "editable": true }, @@ -4093,7 +4113,7 @@ { "cell_type": "code", "execution_count": 27, - "id": "bd35ddaa", + "id": "53112d0a", "metadata": { "collapsed": false, "editable": true @@ -4108,7 +4128,7 @@ }, { "cell_type": "markdown", - "id": "d35d2531", + "id": "22f27155", "metadata": { "editable": true }, @@ -4123,7 +4143,7 @@ { "cell_type": "code", "execution_count": 28, - "id": "5163580e", + "id": "21cc23e7", "metadata": { "collapsed": false, "editable": true @@ -4183,7 +4203,7 @@ }, { "cell_type": "markdown", - "id": "4068063b", + "id": "72d7044f", "metadata": { "editable": true }, @@ -4194,7 +4214,7 @@ { "cell_type": "code", "execution_count": 29, - "id": "86743170", + "id": "7738f520", "metadata": { "collapsed": false, "editable": true @@ -4258,7 +4278,7 @@ }, { "cell_type": "markdown", - "id": "ca006785", + "id": "2917b811", "metadata": { "editable": true }, @@ -4269,7 +4289,7 @@ { "cell_type": "code", "execution_count": 30, - "id": "99c1bf17", + "id": "e8e2bcc8", "metadata": { "collapsed": false, "editable": true @@ -4318,7 +4338,7 @@ }, { "cell_type": "markdown", - "id": "b6e354fb", + "id": "33af042d", "metadata": { "editable": true }, @@ -4330,7 +4350,7 @@ { "cell_type": "code", "execution_count": 31, - "id": "92100e28", + "id": "d39fd015", "metadata": { "collapsed": false, "editable": true @@ -4414,7 +4434,7 @@ }, { "cell_type": "markdown", - "id": "04a93f00", + "id": "ad76fab7", "metadata": { "editable": true }, @@ -4425,7 +4445,7 @@ { "cell_type": "code", "execution_count": 32, - "id": "cf2d793b", + "id": "c93f7f15", "metadata": { "collapsed": false, "editable": true @@ -4503,7 +4523,7 @@ }, { "cell_type": "markdown", - "id": "54ff0cfd", + "id": "0a079dcb", "metadata": { "editable": true }, @@ -4514,7 +4534,7 @@ { "cell_type": "code", "execution_count": 33, - "id": "bc5fc421", + "id": "d3891f6b", "metadata": { "collapsed": false, "editable": true @@ -4578,7 +4598,7 @@ }, { "cell_type": "markdown", - "id": "5a4e61b4", + "id": "e97f18ba", "metadata": { "editable": true }, @@ -4588,7 +4608,7 @@ }, { "cell_type": "markdown", - "id": "c06228d6", + "id": "66f93128", "metadata": { "editable": true }, @@ -4599,7 +4619,7 @@ { "cell_type": "code", "execution_count": 34, - "id": "ad77ac5d", + "id": "f001dd7d", "metadata": { "collapsed": false, "editable": true @@ -4668,7 +4688,7 @@ }, { "cell_type": "markdown", - "id": "3840836c", + "id": "c4fcfa2e", "metadata": { "editable": true }, @@ -4679,7 +4699,7 @@ { "cell_type": "code", "execution_count": 35, - "id": "b340e498", + "id": "2655ba09", "metadata": { "collapsed": false, "editable": true @@ -4723,7 +4743,7 @@ }, { "cell_type": "markdown", - "id": "71a25972", + "id": "77682985", "metadata": { "editable": true }, @@ -4742,7 +4762,7 @@ { "cell_type": "code", "execution_count": 36, - "id": "a5b15751", + "id": "76ee1c4b", "metadata": { "collapsed": false, "editable": true @@ -4762,7 +4782,7 @@ }, { "cell_type": "markdown", - "id": "ab953a89", + "id": "a00f85da", "metadata": { "editable": true }, diff --git a/doc/src/week39/adagradSGD.py b/doc/src/week39/codes/AdamSGD.py similarity index 71% rename from doc/src/week39/adagradSGD.py rename to doc/src/week39/codes/AdamSGD.py index 4aa6688cd..c3bf6ddde 100644 --- a/doc/src/week39/adagradSGD.py +++ b/doc/src/week39/codes/AdamSGD.py @@ -1,4 +1,4 @@ -# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent +# Using Autograd to calculate gradients using Adam and Stochastic Gradient descent # OLS example from random import random, seed import numpy as np @@ -32,19 +32,30 @@ theta = np.random.randn(3,1) # Value for learning rate eta = 0.01 +rho1 = 0.9 +rho2 = 0.99 # Including AdaGrad parameter to avoid possible division by zero delta = 1e-8 for epoch in range(n_epochs): Giter = np.zeros(shape=(3,3)) + t = 0 + s = np.zeros(shape=(3,1)) for i in range(m): random_index = M*np.random.randint(m) xi = X[random_index:random_index+M] yi = y[random_index:random_index+M] gradients = (1.0/M)*training_gradient(yi, xi, theta) + t += 1 + Previous = Giter Giter +=gradients @ gradients.T - Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Giter)))] - update = np.multiply(Ginverse,gradients) - theta -= update -print("theta from own AdaGrad") + Gnew = (rho2*Previous+(1-rho2)*Giter) + Gnew = Gnew#/(1.0-rho2*t) + Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Gnew)))] + s += rho1*s+(1-rho1)*gradients +# snew = s/(1.0-rho1*t) + theta -= Ginverse.T @ s +print("theta from own Adam") print(theta) + +from math import sqrt diff --git a/doc/src/week39/rmspropSGD.py b/doc/src/week39/codes/rmspropSGD.py similarity index 93% rename from doc/src/week39/rmspropSGD.py rename to doc/src/week39/codes/rmspropSGD.py index eda585e25..6a4b28d88 100644 --- a/doc/src/week39/rmspropSGD.py +++ b/doc/src/week39/codes/rmspropSGD.py @@ -1,4 +1,4 @@ -# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent +# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent # OLS example from random import random, seed import numpy as np @@ -48,5 +48,5 @@ for epoch in range(n_epochs): Ginverse = np.c_[eta/(delta+np.sqrt(np.diagonal(Gnew)))] update = np.multiply(Ginverse,gradients) theta -= update -print("theta from own AdaGrad") +print("theta from own RMSprop") print(theta) diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index d0c197d2c..4b4d5c9aa 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -1676,10 +1676,16 @@ learning rate for flat directions. !split ===== "ADAM optimizer":"https://arxiv.org/abs/1412.6980" ===== -A related algorithm is the ADAM optimizer. In "ADAM":"https://arxiv.org/abs/1412.6980", we keep a running -average of both the first and second moment of the gradient and use -this information to adaptively change the learning rate for different -parameters. In addition to keeping a running average of the first and +A related algorithm is the ADAM optimizer. In +"ADAM":"https://arxiv.org/abs/1412.6980", we keep a running average of +both the first and second moment of the gradient and use this +information to adaptively change the learning rate for different +parameters. The method isefficient when working with large +problems involving lots data and/or parameters. It is a combination of the +gradient descent with momentum algorithm and the RMSprop algorithm +discussed above. + +In addition to keeping a running average of the first and second moments of the gradient (i.e. $\mathbf{m}_t=\mathbb{E}[\mathbf{g}_t]$ and $\mathbf{s}_t=\mathbb{E}[\mathbf{g}^2_t]$, respectively), ADAM @@ -1717,7 +1723,12 @@ update rule for this parameter is given by \] !et +!split +===== Algorithms and codes for Adagrad, RMSprop and Adam ===== +The algorithms we have implemented are well described in the text by "Goodfellow, Bengio and Courville, chapter 8":"https://www.deeplearningbook.org/contents/optimization.html". + +The codes which implement these algorithms are discussed after our presentation of automatic differentiation. !split