reordering RNNs

2019-12-16 12:04:11 +01:00
parent 3fb3749494
commit aa177fbcba
12 changed files with 170 additions and 58 deletions
@@ -45,7 +45,13 @@ Automatically generated HTML file from DocOnce source
               2,
               None,
               '___sec0'),
-              ('Set up of an RNN', 2, None, '___sec1')]}
+              ('Set up of an RNN', 2, None, '___sec1'),
+              ('Solving differential equations and eigenvalue problems with '
+               'RNNs',
+               2,
+               None,
+               '___sec2'),
+              ('Long-Short Time Memory', 2, None, '___sec3')]}
 end of tocinfo -->

 <body>
@@ -85,6 +91,8 @@ MathJax.Hub.Config({
        <ul class="dropdown-menu">
     <!-- navigation toc: --> <li><a href="._Recurrent-bs001.html#___sec0" style="font-size: 80%;">Recurrent neural networks: Overarching view</a></li>
     <!-- navigation toc: --> <li><a href="._Recurrent-bs002.html#___sec1" style="font-size: 80%;">Set up of an RNN</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs003.html#___sec2" style="font-size: 80%;">Solving differential equations and eigenvalue problems with RNNs</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs004.html#___sec3" style="font-size: 80%;">Long-Short Time Memory</a></li>

        </ul>
      </li>
@@ -119,7 +127,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 8, 2019</h4></center> <!-- date -->
+<center><h4>Dec 16, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -135,6 +143,8 @@ MathJax.Hub.Config({
  <li class="active"><a href="._Recurrent-bs000.html">1</a></li>
  <li><a href="._Recurrent-bs001.html">2</a></li>
  <li><a href="._Recurrent-bs002.html">3</a></li>
+  <li><a href="._Recurrent-bs003.html">4</a></li>
+  <li><a href="._Recurrent-bs004.html">5</a></li>
  <li><a href="._Recurrent-bs001.html">&raquo;</a></li>
 </ul>
 <!-- ------------------- end of main content --------------- -->
@@ -45,7 +45,13 @@ Automatically generated HTML file from DocOnce source
               2,
               None,
               '___sec0'),
-              ('Set up of an RNN', 2, None, '___sec1')]}
+              ('Set up of an RNN', 2, None, '___sec1'),
+              ('Solving differential equations and eigenvalue problems with '
+               'RNNs',
+               2,
+               None,
+               '___sec2'),
+              ('Long-Short Time Memory', 2, None, '___sec3')]}
 end of tocinfo -->

 <body>
@@ -85,6 +91,8 @@ MathJax.Hub.Config({
        <ul class="dropdown-menu">
     <!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Recurrent neural networks: Overarching view</a></li>
     <!-- navigation toc: --> <li><a href="._Recurrent-bs002.html#___sec1" style="font-size: 80%;">Set up of an RNN</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs003.html#___sec2" style="font-size: 80%;">Solving differential equations and eigenvalue problems with RNNs</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs004.html#___sec3" style="font-size: 80%;">Long-Short Time Memory</a></li>

        </ul>
      </li>
@@ -123,6 +131,9 @@ example, they can take sentences, documents, or audio samples as
 input, making them extremely useful for natural language processing
 systems such as automatic translation and speech-to-text.

+<p>
+<b>The text here is under development</b>. Planned finished Jan 2020.
+
 <p>
 <p>
 <!-- navigation buttons at the bottom of the page -->
@@ -131,6 +142,8 @@ systems such as automatic translation and speech-to-text.
  <li><a href="._Recurrent-bs000.html">1</a></li>
  <li class="active"><a href="._Recurrent-bs001.html">2</a></li>
  <li><a href="._Recurrent-bs002.html">3</a></li>
+  <li><a href="._Recurrent-bs003.html">4</a></li>
+  <li><a href="._Recurrent-bs004.html">5</a></li>
  <li><a href="._Recurrent-bs002.html">&raquo;</a></li>
 </ul>
 <!-- ------------------- end of main content --------------- -->
@@ -45,7 +45,13 @@ Automatically generated HTML file from DocOnce source
               2,
               None,
               '___sec0'),
-              ('Set up of an RNN', 2, None, '___sec1')]}
+              ('Set up of an RNN', 2, None, '___sec1'),
+              ('Solving differential equations and eigenvalue problems with '
+               'RNNs',
+               2,
+               None,
+               '___sec2'),
+              ('Long-Short Time Memory', 2, None, '___sec3')]}
 end of tocinfo -->

 <body>
@@ -85,6 +91,8 @@ MathJax.Hub.Config({
        <ul class="dropdown-menu">
     <!-- navigation toc: --> <li><a href="._Recurrent-bs001.html#___sec0" style="font-size: 80%;">Recurrent neural networks: Overarching view</a></li>
     <!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Set up of an RNN</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs003.html#___sec2" style="font-size: 80%;">Solving differential equations and eigenvalue problems with RNNs</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs004.html#___sec3" style="font-size: 80%;">Long-Short Time Memory</a></li>

        </ul>
      </li>
@@ -113,6 +121,10 @@ the output from the previous layer \( t-1 \), that is \( y_{t1} \).
 This means also that we need to have weights that link both the inputs \( x_t \) to the outputs \( y_t \) as well as weights that link
 the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here shows an example of a simple RNN.

+<p>
+Add figures and bring up equations.
+
+<p>
 <p>
 <!-- navigation buttons at the bottom of the page -->
 <ul class="pagination">
@@ -120,6 +132,9 @@ the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here
  <li><a href="._Recurrent-bs000.html">1</a></li>
  <li><a href="._Recurrent-bs001.html">2</a></li>
  <li class="active"><a href="._Recurrent-bs002.html">3</a></li>
+  <li><a href="._Recurrent-bs003.html">4</a></li>
+  <li><a href="._Recurrent-bs004.html">5</a></li>
+  <li><a href="._Recurrent-bs003.html">&raquo;</a></li>
 </ul>
 <!-- ------------------- end of main content --------------- -->

@@ -45,7 +45,13 @@ Automatically generated HTML file from DocOnce source
               2,
               None,
               '___sec0'),
-              ('Set up of an RNN', 2, None, '___sec1')]}
+              ('Set up of an RNN', 2, None, '___sec1'),
+              ('Solving differential equations and eigenvalue problems with '
+               'RNNs',
+               2,
+               None,
+               '___sec2'),
+              ('Long-Short Time Memory', 2, None, '___sec3')]}
 end of tocinfo -->

 <body>
@@ -85,6 +91,8 @@ MathJax.Hub.Config({
        <ul class="dropdown-menu">
     <!-- navigation toc: --> <li><a href="._Recurrent-bs001.html#___sec0" style="font-size: 80%;">Recurrent neural networks: Overarching view</a></li>
     <!-- navigation toc: --> <li><a href="._Recurrent-bs002.html#___sec1" style="font-size: 80%;">Set up of an RNN</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs003.html#___sec2" style="font-size: 80%;">Solving differential equations and eigenvalue problems with RNNs</a></li>
+     <!-- navigation toc: --> <li><a href="._Recurrent-bs004.html#___sec3" style="font-size: 80%;">Long-Short Time Memory</a></li>

        </ul>
      </li>
@@ -119,7 +127,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 8, 2019</h4></center> <!-- date -->
+<center><h4>Dec 16, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -135,6 +143,8 @@ MathJax.Hub.Config({
  <li class="active"><a href="._Recurrent-bs000.html">1</a></li>
  <li><a href="._Recurrent-bs001.html">2</a></li>
  <li><a href="._Recurrent-bs002.html">3</a></li>
+  <li><a href="._Recurrent-bs003.html">4</a></li>
+  <li><a href="._Recurrent-bs004.html">5</a></li>
  <li><a href="._Recurrent-bs001.html">&raquo;</a></li>
 </ul>
 <!-- ------------------- end of main content --------------- -->
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>&nbsp;<br>
-<center><h4>Jan 8, 2019</h4></center> <!-- date -->
+<center><h4>Dec 16, 2019</h4></center> <!-- date -->
 <br>
 <p>

@@ -181,6 +181,9 @@ fixed-sized inputs like all the nets we have discussed so far. For
 example, they can take sentences, documents, or audio samples as
 input, making them extremely useful for natural language processing
 systems such as automatic translation and speech-to-text.
+
+<p>
+<b>The text here is under development</b>. Planned finished Jan 2020.
 </section>


@@ -197,6 +200,25 @@ the output from the previous layer \( t-1 \), that is \( y_{t1} \).
 <p>
 This means also that we need to have weights that link both the inputs \( x_t \) to the outputs \( y_t \) as well as weights that link
 the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here shows an example of a simple RNN.
+
+<p>
+Add figures and bring up equations.
+</section>
+
+
+<section>
+<h2 id="___sec2">Solving differential equations and eigenvalue problems with RNNs </h2>
+
+<p>
+Have example with eigenvalues solvers as well.
+</section>
+
+
+<section>
+<h2 id="___sec3">Long-Short Time Memory </h2>
+
+<p>
+Discussions about dynamic unrolling through time. discuss memory cells, input and output
 </section>


@@ -39,7 +39,13 @@ div { text-align: justify; text-justify: inter-word; }
               2,
               None,
               '___sec0'),
-              ('Set up of an RNN', 2, None, '___sec1')]}
+              ('Set up of an RNN', 2, None, '___sec1'),
+              ('Solving differential equations and eigenvalue problems with '
+               'RNNs',
+               2,
+               None,
+               '___sec2'),
+              ('Long-Short Time Memory', 2, None, '___sec3')]}
 end of tocinfo -->

 <body>
@@ -81,7 +87,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 8, 2019</h4></center> <!-- date -->
+<center><h4>Dec 16, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -109,6 +115,9 @@ example, they can take sentences, documents, or audio samples as
 input, making them extremely useful for natural language processing
 systems such as automatic translation and speech-to-text.

+<p>
+<b>The text here is under development</b>. Planned finished Jan 2020.
+
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

@@ -125,6 +134,25 @@ the output from the previous layer \( t-1 \), that is \( y_{t1} \).
 This means also that we need to have weights that link both the inputs \( x_t \) to the outputs \( y_t \) as well as weights that link
 the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here shows an example of a simple RNN.

+<p>
+Add figures and bring up equations.
+
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec2">Solving differential equations and eigenvalue problems with RNNs </h2>
+
+<p>
+Have example with eigenvalues solvers as well.
+
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec3">Long-Short Time Memory </h2>
+
+<p>
+Discussions about dynamic unrolling through time. discuss memory cells, input and output
+
 <!-- ------------------- end of main content --------------- -->


@@ -44,7 +44,13 @@ div { text-align: justify; text-justify: inter-word; }
               2,
               None,
               '___sec0'),
-              ('Set up of an RNN', 2, None, '___sec1')]}
+              ('Set up of an RNN', 2, None, '___sec1'),
+              ('Solving differential equations and eigenvalue problems with '
+               'RNNs',
+               2,
+               None,
+               '___sec2'),
+              ('Long-Short Time Memory', 2, None, '___sec3')]}
 end of tocinfo -->

 <body>
@@ -86,7 +92,7 @@ MathJax.Hub.Config({
 <center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
 <br>
 <p>
-<center><h4>Jan 8, 2019</h4></center> <!-- date -->
+<center><h4>Dec 16, 2019</h4></center> <!-- date -->
 <br>
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -114,6 +120,9 @@ example, they can take sentences, documents, or audio samples as
 input, making them extremely useful for natural language processing
 systems such as automatic translation and speech-to-text.

+<p>
+<b>The text here is under development</b>. Planned finished Jan 2020.
+
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>

@@ -130,6 +139,25 @@ the output from the previous layer \( t-1 \), that is \( y_{t1} \).
 This means also that we need to have weights that link both the inputs \( x_t \) to the outputs \( y_t \) as well as weights that link
 the output from the previous time \( y_{t-1} \) and \( y_t \).   The figure here shows an example of a simple RNN.

+<p>
+Add figures and bring up equations.
+
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec2">Solving differential equations and eigenvalue problems with RNNs </h2>
+
+<p>
+Have example with eigenvalues solvers as well.
+
+<p>
+<!-- !split --><br><br><br><br><br><br><br><br><br><br>
+
+<h2 id="___sec3">Long-Short Time Memory </h2>
+
+<p>
+Discussions about dynamic unrolling through time. discuss memory cells, input and output
+
 <!-- ------------------- end of main content --------------- -->


@@ -10,7 +10,7 @@
    "<!-- Author: -->  \n",
    "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
    "\n",
-    "Date: **Jan 8, 2019**\n",
+    "Date: **Dec 16, 2019**\n",
    "\n",
    "Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
    "\n",
@@ -37,6 +37,8 @@
    "input, making them extremely useful for natural language processing\n",
    "systems such as automatic translation and speech-to-text.\n",
    "\n",
+    "**The text here is under development**. Planned finished Jan 2020.\n",
+    "\n",
    "\n",
    "## Set up of an RNN\n",
    "\n",
@@ -47,7 +49,21 @@
    "the output from the previous layer $t-1$, that is $y_{t1}$.\n",
    "\n",
    "This means also that we need to have weights that link both the inputs $x_t$ to the outputs $y_t$ as well as weights that link\n",
-    "the output from the previous time $y_{t-1}$ and $y_t$.   The figure here shows an example of a simple RNN."
+    "the output from the previous time $y_{t-1}$ and $y_t$.   The figure here shows an example of a simple RNN.\n",
+    "\n",
+    "\n",
+    "Add figures and bring up equations. \n",
+    "\n",
+    "\n",
+    "\n",
+    "## Solving differential equations and eigenvalue problems with RNNs\n",
+    "\n",
+    "Have example with eigenvalues solvers as well.\n",
+    "\n",
+    "\n",
+    "## Long-Short Time Memory\n",
+    "\n",
+    "Discussions about dynamic unrolling through time. discuss memory cells, input and output"
   ]
  }
 ],
@@ -24,6 +24,8 @@ example, they can take sentences, documents, or audio samples as
 input, making them extremely useful for natural language processing
 systems such as automatic translation and speech-to-text.

+_The text here is under development_. Planned finished Jan 2020.
+

 !split
 =====  Set up of an RNN =====
@@ -38,8 +40,17 @@ This means also that we need to have weights that link both the inputs $x_t$ to
 the output from the previous time $y_{t-1}$ and $y_t$.   The figure here shows an example of a simple RNN.


-Add figures and bring up equations. Have example with eigenvalues solvers as well.
-Discussions about dynamic unrolling through time. discuss memory cells, input and output
+Add figures and bring up equations. 

-Add about LSTM cell.
-Should add autoencoders? 
+
+
+!split
+===== Solving differential equations and eigenvalue problems with RNNs =====
+
+Have example with eigenvalues solvers as well.
+
+
+!split
+===== Long-Short Time Memory =====
+
+Discussions about dynamic unrolling through time. discuss memory cells, input and output
@@ -1,41 +0,0 @@
-TITLE: Data Analysis and Machine Learning: Recurrent neural networks
-AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-DATE: today
-
-
-!split
-===== Recurrent neural networks: Overarching view =====
-
-Till now our focus has been, including convolutional neural networks as well, 
-on feedforward neural networks. The output or the
-activations flow only in one direction, from the input layer to the
-output layer.
-
-A recurrent neural network (RNN) looks very much like a feedforward
-neural network, except that it also has connections pointing
-backward. 
-
-RNNs are used to analyze time series data such as stock prices, and
-tell you when to buy or sell. In autonomous driving systems, they can
-anticipate car trajectories and help avoid accidents. More generally,
-they can work on sequences of arbitrary lengths, rather than on
-fixed-sized inputs like all the nets we have discussed so far. For
-example, they can take sentences, documents, or audio samples as
-input, making them extremely useful for natural language processing
-systems such as automatic translation and speech-to-text.
-
-
-!split
-=====  Set up of an RNN =====
-
-The figure here displays a simple example of an RNN, with inputs $x_t$
-at a given time $t$ and outputs $y_t$.  Introducing time as a variable
-offers an intutitive way of understanding these networks. In addition
-to the inputs $x_t$, the layer at a time $t$ receives also as input
-the output from the previous layer $t-1$, that is $y_{t1}$.
-
-This means also that we need to have weights that link both the inputs $x_t$ to the outputs $y_t$ as well as weights that link
-the output from the previous time $y_{t-1}$ and $y_t$.   The figure here shows an example of a simple RNN.
-
-
-Add figures and bring up equations. Have example with eigenvalues solvers