Update week39.do.txt

doc/src/week39/week39.do.txt

@@ -2636,3 +2636,42 @@ print(derivative_fn(x_small))
 * Try to run the above codes and implement the stochastic gradient descent with the ADAM. Here you can use as examples the Adagrad and the RMSprop algorithms.
 * Add a more complicated function and study the rate of convergence for the derivatives as function of the different methods
 * Extend from linear regression to logistic regression.
+
+
+
+AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011.[24] Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse. This strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative. Examples of such applications include natural language processing and image recognition.[24] It still has a base learning rate η, but this is multiplied with the elements of a vector {Gj,j} which is the diagonal of the outer product matrix
+
+{\displaystyle G=\sum _{\tau =1}^{t}g_{\tau }g_{\tau }^{\mathsf {T}}}G=\sum _{\tau =1}^{t}g_{\tau }g_{\tau }^{\mathsf {T}}
+where {\displaystyle g_{\tau }=\nabla Q_{i}(w)}g_{\tau }=\nabla Q_{i}(w), the gradient, at iteration τ. The diagonal is given by
+
+{\displaystyle G_{j,j}=\sum _{\tau =1}^{t}g_{\tau ,j}^{2}}G_{j,j}=\sum _{\tau =1}^{t}g_{\tau ,j}^{2}.
+This vector is updated after every iteration. The formula for an update is now
+
+{\displaystyle w:=w-\eta \,\mathrm {diag} (G)^{-{\frac {1}{2}}}\odot g}{\displaystyle w:=w-\eta \,\mathrm {diag} (G)^{-{\frac {1}{2}}}\odot g}[a]
+or, written as per-parameter updates,
+
+{\displaystyle w_{j}:=w_{j}-{\frac {\eta }{\sqrt {G_{j,j}}}}g_{j}.}w_{j}:=w_{j}-{\frac {\eta }{\sqrt {G_{j,j}}}}g_{j}.
+Each {G(i,i)} gives rise to a scaling factor for the learning rate that applies to a single parameter wi. Since the denominator in this factor, {\displaystyle {\sqrt {G_{i}}}={\sqrt {\sum _{\tau =1}^{t}g_{\tau }^{2}}}}{\sqrt {G_{i}}}={\sqrt {\sum _{\tau =1}^{t}g_{\tau }^{2}}} is the ℓ2 norm of previous derivatives, extreme parameter updates get dampened, while parameters that get few or small updates receive higher learning rates.[21]
+
+While designed for convex problems, AdaGrad has been successfully applied to non-convex optimization.[25]
+
+RMSProp
+RMSProp (for Root Mean Square Propagation) is also a method in which the learning rate is adapted for each of the parameters. The idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight.[26] So, first the running average is calculated in terms of means square,
+
+{\displaystyle v(w,t):=\gamma v(w,t-1)+(1-\gamma )(\nabla Q_{i}(w))^{2}}{\displaystyle v(w,t):=\gamma v(w,t-1)+(1-\gamma )(\nabla Q_{i}(w))^{2}}
+where, {\displaystyle \gamma }\gamma  is the forgetting factor.
+
+And the parameters are updated as,
+
+{\displaystyle w:=w-{\frac {\eta }{\sqrt {v(w,t)}}}\nabla Q_{i}(w)}{\displaystyle w:=w-{\frac {\eta }{\sqrt {v(w,t)}}}\nabla Q_{i}(w)}
+RMSProp has shown good adaptation of learning rate in different applications. RMSProp can be seen as a generalization of Rprop and is capable to work with mini-batches as well opposed to only full-batches.[27]
+
+Adam
+Adam[28] (short for Adaptive Moment Estimation) is an update to the RMSProp optimizer. In this optimization algorithm, running averages of both the gradients and the second moments of the gradients are used. Given parameters {\displaystyle w^{(t)}}{\displaystyle w^{(t)}} and a loss function {\displaystyle L^{(t)}}{\displaystyle L^{(t)}}, where {\displaystyle t}t indexes the current training iteration (indexed at {\displaystyle 0}{\displaystyle  0 }), Adam's parameter update is given by:
+
+{\displaystyle m_{w}^{(t+1)}\leftarrow \beta _{1}m_{w}^{(t)}+(1-\beta _{1})\nabla _{w}L^{(t)}}{\displaystyle m_{w}^{(t+1)}\leftarrow \beta _{1}m_{w}^{(t)}+(1-\beta _{1})\nabla _{w}L^{(t)}}
+{\displaystyle v_{w}^{(t+1)}\leftarrow \beta _{2}v_{w}^{(t)}+(1-\beta _{2})(\nabla _{w}L^{(t)})^{2}}{\displaystyle v_{w}^{(t+1)}\leftarrow \beta _{2}v_{w}^{(t)}+(1-\beta _{2})(\nabla _{w}L^{(t)})^{2}}
+{\displaystyle {\hat {m}}_{w}={\frac {m_{w}^{(t+1)}}{1-\beta _{1}^{t}}}}{\displaystyle {\hat {m}}_{w}={\frac {m_{w}^{(t+1)}}{1-\beta _{1}^{t}}}}
+{\displaystyle {\hat {v}}_{w}={\frac {v_{w}^{(t+1)}}{1-\beta _{2}^{t}}}}{\displaystyle {\hat {v}}_{w}={\frac {v_{w}^{(t+1)}}{1-\beta _{2}^{t}}}}
+{\displaystyle w^{(t+1)}\leftarrow w^{(t)}-\eta {\frac {{\hat {m}}_{w}}{{\sqrt {{\hat {v}}_{w}}}+\epsilon }}}{\displaystyle w^{(t+1)}\leftarrow w^{(t)}-\eta {\frac {{\hat {m}}_{w}}{{\sqrt {{\hat {v}}_{w}}}+\epsilon }}}
+where {\displaystyle \epsilon }\epsilon  is a small scalar (e.g. {\displaystyle 10^{-8}}10^{{-8}}) used to prevent division by 0, and {\displaystyle \beta _{1}}\beta _{1} (e.g. 0.9) and {\displaystyle \beta _{2}}\beta _{2} (e.g. 0.999) are the forgetting factors for gradients and second moments of gradients, respectively. Squaring and square-rooting is done element-wise.