mhjensen
82 lines
3.6 KiB
Plaintext
82 lines
3.6 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!-- dom:TITLE: Data Analysis and Machine Learning: Recurrent neural networks -->\n",
|
|
"# Data Analysis and Machine Learning: Recurrent neural networks\n",
|
|
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
|
"<!-- Author: --> \n",
|
|
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
|
"\n",
|
|
"Date: **Dec 17, 2019**\n",
|
|
"\n",
|
|
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"**The text here is under development**. Planned finished mid Jan 2020.\n",
|
|
"\n",
|
|
"\n",
|
|
"## Set up of an RNN\n",
|
|
"\n",
|
|
"The figure here displays a simple example of an RNN, with inputs $x_t$\n",
|
|
"at a given time $t$ and outputs $y_t$. Introducing time as a variable\n",
|
|
"offers an intutitive way of understanding these networks. In addition\n",
|
|
"to the inputs $x_t$, the layer at a time $t$ receives also as input\n",
|
|
"the output from the previous layer $t-1$, that is $y_{t1}$.\n",
|
|
"\n",
|
|
"This means also that we need to have weights that link both the inputs\n",
|
|
"$x_t$ to the outputs $y_t$ as well as weights that link the output\n",
|
|
"from the previous time $y_{t-1}$ and $y_t$. The figure here shows an\n",
|
|
"example of a simple RNN.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Solving differential equations and eigenvalue problems with RNNs\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"In our discussions of ordinary differential equations and partial\n",
|
|
"differential equations using neural networks. Here we will discuss how\n",
|
|
"we can solve say ordinary differential equations and eigenvalue\n",
|
|
"problems using RNNs. Eigenvalue problems can be solved using RNNs by\n",
|
|
"rewriting such a problems as a non-linear differential equation.\n",
|
|
"\n",
|
|
"Instead of starting with a well-known ordinary differential equation,\n",
|
|
"we start directly with an eigenvaule problem.\n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"## Long-Short Time Memory\n",
|
|
"\n",
|
|
"Discussions about dynamic unrolling through time. discuss memory cells, input and output"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|