lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

udpate

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-10-04 22:15:33 +02:00
parent a9c46b5306
commit 1331fab795
7 changed files with 409 additions and 319 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week39/html/week39-bs.html
+4 -2
View File
@@ -267,7 +267,8 @@ doconce format html week39.do.txt --html_style=bootstrap --pygments_html_style=d
2,
None,
'introducing-jax-https-jax-readthedocs-io-en-latest'),
('Weekend challenge', 2, None, 'weekend-challenge')]}
('Weekend challenge', 2, None, 'weekend-challenge'),
('algorithms', 2, None, 'algorithms')]}
end of tocinfo -->
<body>
@@ -389,6 +390,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week39-bs085.html#and-logistic-regression" style="font-size: 80%;">And Logistic Regression</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs086.html#introducing-jax-https-jax-readthedocs-io-en-latest" style="font-size: 80%;">Introducing "JAX":"https://jax.readthedocs.io/en/latest/"</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs087.html#weekend-challenge" style="font-size: 80%;">Weekend challenge</a></li>
<!-- navigation toc: --> <li><a href="._week39-bs088.html#algorithms" style="font-size: 80%;">algorithms</a></li>
</ul>
</li>
@@ -443,7 +445,7 @@ MathJax.Hub.Config({
<li><a href="._week39-bs008.html">9</a></li>
<li><a href="._week39-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week39-bs087.html">88</a></li>
<li><a href="._week39-bs088.html">89</a></li>
<li><a href="._week39-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week39/html/week39-reveal.html
+24
View File
@@ -3781,6 +3781,30 @@ derivative_fn = grad(sum_logistic)
</ul>
</section>
<section>
<h2 id="algorithms">algorithms </h2>
<p>\begin{algorithm}
\caption{An algorithm with caption}
\KwData{\( n \geq 0 \)}
\KwResult{\( y = x^n \)}
$y \gets 1$\;
$X \gets x$\;
$N \gets n$\;
\While{\( N \neq 0 \)}{
\eIf{\( N \) is even}{
$X \gets X \times X$\;
\( N \gets \frac{N}{2} \) \Comment*[r]{This is a comment}
}{\If{\( N \) is odd}{
$y \gets y \times X$\;
$N \gets N - 1$\;
}
}
}
\end{algorithm}
</p>
</section>
</div> <!-- class="slides" -->
doc/pub/week39/html/week39-solarized.html
+25 -1
View File
@@ -294,7 +294,8 @@ div.toc p,a {
2,
None,
'introducing-jax-https-jax-readthedocs-io-en-latest'),
('Weekend challenge', 2, None, 'weekend-challenge')]}
('Weekend challenge', 2, None, 'weekend-challenge'),
('algorithms', 2, None, 'algorithms')]}
end of tocinfo -->
<body>
@@ -3712,6 +3713,29 @@ derivative_fn = grad(sum_logistic)
<li> Add a more complicated function and study the rate of convergence for the derivatives as function of the different methods</li>
<li> Extend from linear regression to logistic regression.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="algorithms">algorithms </h2>
<p>\begin{algorithm}
\caption{An algorithm with caption}
\KwData{\( n \geq 0 \)}
\KwResult{\( y = x^n \)}
$y \gets 1$\;
$X \gets x$\;
$N \gets n$\;
\While{\( N \neq 0 \)}{
\eIf{\( N \) is even}{
$X \gets X \times X$\;
\( N \gets \frac{N}{2} \) \Comment*[r]{This is a comment}
}{\If{\( N \) is odd}{
$y \gets y \times X$\;
$N \gets N - 1$\;
}
}
}
\end{algorithm}
</p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
doc/pub/week39/html/week39.html
+25 -1
View File
@@ -371,7 +371,8 @@ div.toc p,a {
2,
None,
'introducing-jax-https-jax-readthedocs-io-en-latest'),
('Weekend challenge', 2, None, 'weekend-challenge')]}
('Weekend challenge', 2, None, 'weekend-challenge'),
('algorithms', 2, None, 'algorithms')]}
end of tocinfo -->
<body>
@@ -3789,6 +3790,29 @@ derivative_fn <span style="color: #666666">=</span> grad(sum_logistic)
<li> Add a more complicated function and study the rate of convergence for the derivatives as function of the different methods</li>
<li> Extend from linear regression to logistic regression.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="algorithms">algorithms </h2>
<p>\begin{algorithm}
\caption{An algorithm with caption}
\KwData{\( n \geq 0 \)}
\KwResult{\( y = x^n \)}
$y \gets 1$\;
$X \gets x$\;
$N \gets n$\;
\While{\( N \neq 0 \)}{
\eIf{\( N \) is even}{
$X \gets X \times X$\;
\( N \gets \frac{N}{2} \) \Comment*[r]{This is a comment}
}{\If{\( N \) is odd}{
$y \gets y \times X$\;
$N \gets N - 1$\;
}
}
}
\end{algorithm}
</p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2022, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week39/ipynb/week39.ipynb
+311 -282
View File
File diff suppressed because it is too large Load Diff
doc/src/week39/week39.do.txt
+20 -33
View File
@@ -2638,40 +2638,27 @@ print(derivative_fn(x_small))
* Extend from linear regression to logistic regression.
!split
===== algorithms =====
AdaGrad (for adaptive gradient algorithm) is a modified stochastic gradient descent algorithm with per-parameter learning rate, first published in 2011.[24] Informally, this increases the learning rate for sparser parameters and decreases the learning rate for ones that are less sparse. This strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative. Examples of such applications include natural language processing and image recognition.[24] It still has a base learning rate η, but this is multiplied with the elements of a vector {Gj,j} which is the diagonal of the outer product matrix
\begin{algorithm}
\caption{An algorithm with caption}
\KwData{$n \geq 0$}
\KwResult{$y = x^n$}
$y \gets 1$\;
$X \gets x$\;
$N \gets n$\;
\While{$N \neq 0$}{
\eIf{$N$ is even}{
$X \gets X \times X$\;
$N \gets \frac{N}{2}$ \Comment*[r]{This is a comment}
}{\If{$N$ is odd}{
$y \gets y \times X$\;
$N \gets N - 1$\;
}
}
}
\end{algorithm}
{\displaystyle G=\sum _{\tau =1}^{t}g_{\tau }g_{\tau }^{\mathsf {T}}}G=\sum _{\tau =1}^{t}g_{\tau }g_{\tau }^{\mathsf {T}}
where {\displaystyle g_{\tau }=\nabla Q_{i}(w)}g_{\tau }=\nabla Q_{i}(w), the gradient, at iteration τ. The diagonal is given by
{\displaystyle G_{j,j}=\sum _{\tau =1}^{t}g_{\tau ,j}^{2}}G_{j,j}=\sum _{\tau =1}^{t}g_{\tau ,j}^{2}.
This vector is updated after every iteration. The formula for an update is now
{\displaystyle w:=w-\eta \,\mathrm {diag} (G)^{-{\frac {1}{2}}}\odot g}{\displaystyle w:=w-\eta \,\mathrm {diag} (G)^{-{\frac {1}{2}}}\odot g}[a]
or, written as per-parameter updates,
{\displaystyle w_{j}:=w_{j}-{\frac {\eta }{\sqrt {G_{j,j}}}}g_{j}.}w_{j}:=w_{j}-{\frac {\eta }{\sqrt {G_{j,j}}}}g_{j}.
Each {G(i,i)} gives rise to a scaling factor for the learning rate that applies to a single parameter wi. Since the denominator in this factor, {\displaystyle {\sqrt {G_{i}}}={\sqrt {\sum _{\tau =1}^{t}g_{\tau }^{2}}}}{\sqrt {G_{i}}}={\sqrt {\sum _{\tau =1}^{t}g_{\tau }^{2}}} is the ℓ2 norm of previous derivatives, extreme parameter updates get dampened, while parameters that get few or small updates receive higher learning rates.[21]
While designed for convex problems, AdaGrad has been successfully applied to non-convex optimization.[25]
RMSProp
RMSProp (for Root Mean Square Propagation) is also a method in which the learning rate is adapted for each of the parameters. The idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight.[26] So, first the running average is calculated in terms of means square,
{\displaystyle v(w,t):=\gamma v(w,t-1)+(1-\gamma )(\nabla Q_{i}(w))^{2}}{\displaystyle v(w,t):=\gamma v(w,t-1)+(1-\gamma )(\nabla Q_{i}(w))^{2}}
where, {\displaystyle \gamma }\gamma is the forgetting factor.
And the parameters are updated as,
{\displaystyle w:=w-{\frac {\eta }{\sqrt {v(w,t)}}}\nabla Q_{i}(w)}{\displaystyle w:=w-{\frac {\eta }{\sqrt {v(w,t)}}}\nabla Q_{i}(w)}
RMSProp has shown good adaptation of learning rate in different applications. RMSProp can be seen as a generalization of Rprop and is capable to work with mini-batches as well opposed to only full-batches.[27]
Adam
Adam[28] (short for Adaptive Moment Estimation) is an update to the RMSProp optimizer. In this optimization algorithm, running averages of both the gradients and the second moments of the gradients are used. Given parameters {\displaystyle w^{(t)}}{\displaystyle w^{(t)}} and a loss function {\displaystyle L^{(t)}}{\displaystyle L^{(t)}}, where {\displaystyle t}t indexes the current training iteration (indexed at {\displaystyle 0}{\displaystyle 0 }), Adam's parameter update is given by:
{\displaystyle m_{w}^{(t+1)}\leftarrow \beta _{1}m_{w}^{(t)}+(1-\beta _{1})\nabla _{w}L^{(t)}}{\displaystyle m_{w}^{(t+1)}\leftarrow \beta _{1}m_{w}^{(t)}+(1-\beta _{1})\nabla _{w}L^{(t)}}
{\displaystyle v_{w}^{(t+1)}\leftarrow \beta _{2}v_{w}^{(t)}+(1-\beta _{2})(\nabla _{w}L^{(t)})^{2}}{\displaystyle v_{w}^{(t+1)}\leftarrow \beta _{2}v_{w}^{(t)}+(1-\beta _{2})(\nabla _{w}L^{(t)})^{2}}
{\displaystyle {\hat {m}}_{w}={\frac {m_{w}^{(t+1)}}{1-\beta _{1}^{t}}}}{\displaystyle {\hat {m}}_{w}={\frac {m_{w}^{(t+1)}}{1-\beta _{1}^{t}}}}
{\displaystyle {\hat {v}}_{w}={\frac {v_{w}^{(t+1)}}{1-\beta _{2}^{t}}}}{\displaystyle {\hat {v}}_{w}={\frac {v_{w}^{(t+1)}}{1-\beta _{2}^{t}}}}
{\displaystyle w^{(t+1)}\leftarrow w^{(t)}-\eta {\frac {{\hat {m}}_{w}}{{\sqrt {{\hat {v}}_{w}}}+\epsilon }}}{\displaystyle w^{(t+1)}\leftarrow w^{(t)}-\eta {\frac {{\hat {m}}_{w}}{{\sqrt {{\hat {v}}_{w}}}+\epsilon }}}
where {\displaystyle \epsilon }\epsilon is a small scalar (e.g. {\displaystyle 10^{-8}}10^{{-8}}) used to prevent division by 0, and {\displaystyle \beta _{1}}\beta _{1} (e.g. 0.9) and {\displaystyle \beta _{2}}\beta _{2} (e.g. 0.999) are the forgetting factors for gradients and second moments of gradients, respectively. Squaring and square-rooting is done element-wise.