mhjensen
179 lines
5.4 KiB
HTML
179 lines
5.4 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Recurrent neural networks">
|
|
|
|
<title>Data Analysis and Machine Learning: Recurrent neural networks</title>
|
|
|
|
|
|
<style type="text/css">
|
|
/* bloodish style */
|
|
|
|
body {
|
|
font-family: Helvetica, Verdana, Arial, Sans-serif;
|
|
color: #404040;
|
|
background: #ffffff;
|
|
}
|
|
h1 { font-size: 1.8em; color: #8A0808; }
|
|
h2 { font-size: 1.6em; color: #8A0808; }
|
|
h3 { font-size: 1.4em; color: #8A0808; }
|
|
h4 { color: #8A0808; }
|
|
a { color: #8A0808; text-decoration:none; }
|
|
tt { font-family: "Courier New", Courier; }
|
|
/* pre style removed because it will interfer with pygments */
|
|
p { text-indent: 0px; }
|
|
hr { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
p.caption { width: 80%; font-style: normal; text-align: left; }
|
|
hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
|
|
|
|
div { text-align: justify; text-justify: inter-word; }
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Recurrent neural networks: Overarching view',
|
|
2,
|
|
None,
|
|
'___sec0'),
|
|
('Set up of an RNN', 2, None, '___sec1'),
|
|
('Solving differential equations and eigenvalue problems with '
|
|
'RNNs',
|
|
2,
|
|
None,
|
|
'___sec2'),
|
|
('Long-Short Time Memory', 2, None, '___sec3')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
|
|
<script type="text/x-mathjax-config">
|
|
MathJax.Hub.Config({
|
|
TeX: {
|
|
equationNumbers: { autoNumber: "AMS" },
|
|
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
|
}
|
|
});
|
|
</script>
|
|
<script type="text/javascript" async
|
|
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
|
</script>
|
|
|
|
|
|
|
|
|
|
<!-- ------------------- main content ---------------------- -->
|
|
|
|
|
|
|
|
<center><h1>Data Analysis and Machine Learning: Recurrent neural networks</h1></center> <!-- document title -->
|
|
|
|
<p>
|
|
<!-- author(s): Morten Hjorth-Jensen -->
|
|
|
|
<center>
|
|
<b>Morten Hjorth-Jensen</b> [1, 2]
|
|
</center>
|
|
|
|
<p>
|
|
<!-- institution(s) -->
|
|
|
|
<center>[1] <b>Department of Physics, University of Oslo</b></center>
|
|
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
|
<br>
|
|
<p>
|
|
<center><h4>Dec 17, 2019</h4></center> <!-- date -->
|
|
<br>
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec0">Recurrent neural networks: Overarching view </h2>
|
|
|
|
<p>
|
|
Till now our focus has been, including convolutional neural networks
|
|
as well, on feedforward neural networks. The output or the activations
|
|
flow only in one direction, from the input layer to the output layer.
|
|
|
|
<p>
|
|
A recurrent neural network (RNN) looks very much like a feedforward
|
|
neural network, except that it also has connections pointing
|
|
backward.
|
|
|
|
<p>
|
|
RNNs are used to analyze time series data such as stock prices, and
|
|
tell you when to buy or sell. In autonomous driving systems, they can
|
|
anticipate car trajectories and help avoid accidents. More generally,
|
|
they can work on sequences of arbitrary lengths, rather than on
|
|
fixed-sized inputs like all the nets we have discussed so far. For
|
|
example, they can take sentences, documents, or audio samples as
|
|
input, making them extremely useful for natural language processing
|
|
systems such as automatic translation and speech-to-text.
|
|
|
|
<p>
|
|
<b>The text here is under development</b>. Planned finished mid Jan 2020.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec1">Set up of an RNN </h2>
|
|
|
|
<p>
|
|
The figure here displays a simple example of an RNN, with inputs \( x_t \)
|
|
at a given time \( t \) and outputs \( y_t \). Introducing time as a variable
|
|
offers an intutitive way of understanding these networks. In addition
|
|
to the inputs \( x_t \), the layer at a time \( t \) receives also as input
|
|
the output from the previous layer \( t-1 \), that is \( y_{t1} \).
|
|
|
|
<p>
|
|
This means also that we need to have weights that link both the inputs
|
|
\( x_t \) to the outputs \( y_t \) as well as weights that link the output
|
|
from the previous time \( y_{t-1} \) and \( y_t \). The figure here shows an
|
|
example of a simple RNN.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec2">Solving differential equations and eigenvalue problems with RNNs </h2>
|
|
|
|
<p>
|
|
In our discussions of ordinary differential equations and partial
|
|
differential equations using neural networks. Here we will discuss how
|
|
we can solve say ordinary differential equations and eigenvalue
|
|
problems using RNNs. Eigenvalue problems can be solved using RNNs by
|
|
rewriting such a problems as a non-linear differential equation.
|
|
|
|
<p>
|
|
Instead of starting with a well-known ordinary differential equation,
|
|
we start directly with an eigenvaule problem.
|
|
|
|
<p>
|
|
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
|
|
|
<h2 id="___sec3">Long-Short Time Memory </h2>
|
|
|
|
<p>
|
|
Discussions about dynamic unrolling through time. discuss memory cells, input and output
|
|
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright --> © 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|