Update week39.do.txt

This commit is contained in:
Morten Hjorth-Jensen
2022-09-27 11:47:39 +02:00
parent b47ee084ea
commit 4e7bc1767a
+3 -3
View File
@@ -1508,7 +1508,7 @@ Hessians.
Recently, a number of methods have been introduced that accomplish
this by tracking not only the gradient, but also the second moment of
the gradient. These methods include AdaGrad, AdaDelta, Root Mean Squared Propagation (RMS-Prop), and
ADAM.
"ADAM":"https://arxiv.org/abs/1412.6980".
!split
===== RMS prop =====
@@ -1538,9 +1538,9 @@ learning rate for flat directions.
!split
===== ADAM optimizer =====
===== "ADAM optimizer":"https://arxiv.org/abs/1412.6980" =====
A related algorithm is the ADAM optimizer. In ADAM, we keep a running
A related algorithm is the ADAM optimizer. In "ADAM":"https://arxiv.org/abs/1412.6980", we keep a running
average of both the first and second moment of the gradient and use
this information to adaptively change the learning rate for different
parameters. In addition to keeping a running average of the first and