diff --git a/doc/pub/week45/html/._week45-bs000.html b/doc/pub/week45/html/._week45-bs000.html index d11bdcd92..f34386285 100644 --- a/doc/pub/week45/html/._week45-bs000.html +++ b/doc/pub/week45/html/._week45-bs000.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs001.html b/doc/pub/week45/html/._week45-bs001.html index 1657ae349..9f4bc1156 100644 --- a/doc/pub/week45/html/._week45-bs001.html +++ b/doc/pub/week45/html/._week45-bs001.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs002.html b/doc/pub/week45/html/._week45-bs002.html index eaf4a388c..a72aa2955 100644 --- a/doc/pub/week45/html/._week45-bs002.html +++ b/doc/pub/week45/html/._week45-bs002.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs003.html b/doc/pub/week45/html/._week45-bs003.html index 3ebd0632e..5cd637f58 100644 --- a/doc/pub/week45/html/._week45-bs003.html +++ b/doc/pub/week45/html/._week45-bs003.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs004.html b/doc/pub/week45/html/._week45-bs004.html index 2383dc302..0fd46acd0 100644 --- a/doc/pub/week45/html/._week45-bs004.html +++ b/doc/pub/week45/html/._week45-bs004.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs005.html b/doc/pub/week45/html/._week45-bs005.html index 941abd027..2c1d4a674 100644 --- a/doc/pub/week45/html/._week45-bs005.html +++ b/doc/pub/week45/html/._week45-bs005.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs006.html b/doc/pub/week45/html/._week45-bs006.html index 946db4f92..8998943cf 100644 --- a/doc/pub/week45/html/._week45-bs006.html +++ b/doc/pub/week45/html/._week45-bs006.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs007.html b/doc/pub/week45/html/._week45-bs007.html index a4cbe3770..9c16e665e 100644 --- a/doc/pub/week45/html/._week45-bs007.html +++ b/doc/pub/week45/html/._week45-bs007.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs008.html b/doc/pub/week45/html/._week45-bs008.html index d0a0665c4..21f1a9a56 100644 --- a/doc/pub/week45/html/._week45-bs008.html +++ b/doc/pub/week45/html/._week45-bs008.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs009.html b/doc/pub/week45/html/._week45-bs009.html index cf7c1ecb1..6498ae1f5 100644 --- a/doc/pub/week45/html/._week45-bs009.html +++ b/doc/pub/week45/html/._week45-bs009.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs010.html b/doc/pub/week45/html/._week45-bs010.html index 72ff75674..fccaead1d 100644 --- a/doc/pub/week45/html/._week45-bs010.html +++ b/doc/pub/week45/html/._week45-bs010.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs011.html b/doc/pub/week45/html/._week45-bs011.html index a5d378090..4ab99abde 100644 --- a/doc/pub/week45/html/._week45-bs011.html +++ b/doc/pub/week45/html/._week45-bs011.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs012.html b/doc/pub/week45/html/._week45-bs012.html index 19c813969..2940d9dc7 100644 --- a/doc/pub/week45/html/._week45-bs012.html +++ b/doc/pub/week45/html/._week45-bs012.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • @@ -294,58 +278,6 @@ plt.show() -

    Memoryless models

    - -

    Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using delay taps.

    - -
    -
    - -

    These generalize autoregressive -models by using one or more -layers of non-linear hidden units. -

    -
    -
    - - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -
    1. It can store information in its hidden state for a long time.
      2. -
    2. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
      3. -
    3. The best we can do is to infer a probability distribution over the
      4. -
    -

    space of hidden state vectors.

    - -

    This inference is only tractable for two types of hidden state model.

    -

    Linear dynamical model

    - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -
    1. It can store information in its hidden state for a long time. -
        -
      1. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
        2. -
      -
    2. The best we can do is to infer a probability distribution over the space of hidden state vectors.
      3. -
    -

    Hidden Markov Models

    -

    Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states -are stochastic and controlled by a transition -matrix. The outputs produced by a state are -stochastic. -

    - -

    HMMs have efficient algorithms for inference and learning

    RNNs

    RNNs are very powerful, because they @@ -357,48 +289,8 @@ combine two properties:

    With enough neurons and time, RNNs can compute anything that can be -computed by your computer. +computed by your computer!

    -

    Do generative models need to be stochastic?

    - -
    -
    - - -

    But the posterior probability -distribution over their -hidden states given the -observed data so far is a -deterministic function of the -data. -

    -
    -
    - - -
    -
    - -

    Think of the hidden state -of an RNN as the -equivalent of the -deterministic probability -distribution over hidden -states in a linear dynamical -system or hidden Markov -model. -

    -
    -
    - -

    What kinds of behaviour can RNNs exhibit?

    -
      -
    1. They can oscillate.
      2. -
    2. They can settle to point attractors.
      3. -
    3. They can behave chaotically.
      4. -
    4. RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.
      5. -
    -

    But the computational power of RNNs makes them very hard to train.

    diff --git a/doc/pub/week45/html/._week45-bs013.html b/doc/pub/week45/html/._week45-bs013.html index 7731c8f08..c5a4614bd 100644 --- a/doc/pub/week45/html/._week45-bs013.html +++ b/doc/pub/week45/html/._week45-bs013.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({

  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs014.html b/doc/pub/week45/html/._week45-bs014.html index 18b4b8855..0c8ab042a 100644 --- a/doc/pub/week45/html/._week45-bs014.html +++ b/doc/pub/week45/html/._week45-bs014.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs015.html b/doc/pub/week45/html/._week45-bs015.html index 7f5474f72..678f21c3f 100644 --- a/doc/pub/week45/html/._week45-bs015.html +++ b/doc/pub/week45/html/._week45-bs015.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs016.html b/doc/pub/week45/html/._week45-bs016.html index fdbf33db0..e7f0848e2 100644 --- a/doc/pub/week45/html/._week45-bs016.html +++ b/doc/pub/week45/html/._week45-bs016.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs017.html b/doc/pub/week45/html/._week45-bs017.html index e7d658fab..c26c38a23 100644 --- a/doc/pub/week45/html/._week45-bs017.html +++ b/doc/pub/week45/html/._week45-bs017.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs018.html b/doc/pub/week45/html/._week45-bs018.html index e67ef5015..82b7aee0b 100644 --- a/doc/pub/week45/html/._week45-bs018.html +++ b/doc/pub/week45/html/._week45-bs018.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs019.html b/doc/pub/week45/html/._week45-bs019.html index 95908b523..f1bf94914 100644 --- a/doc/pub/week45/html/._week45-bs019.html +++ b/doc/pub/week45/html/._week45-bs019.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/._week45-bs020.html b/doc/pub/week45/html/._week45-bs020.html index 6ad6af390..6daef1c51 100644 --- a/doc/pub/week45/html/._week45-bs020.html +++ b/doc/pub/week45/html/._week45-bs020.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/week45-bs.html b/doc/pub/week45/html/week45-bs.html index d11bdcd92..f34386285 100644 --- a/doc/pub/week45/html/week45-bs.html +++ b/doc/pub/week45/html/week45-bs.html @@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -177,12 +166,7 @@ MathJax.Hub.Config({
  • Material for Lecture Thursday November 9
  • Recurrent neural networks (RNNs): Overarching view
  • A simple example
    • -
  •    Memoryless models
    • -
  •    Linear dynamical model
    • -
  •    Hidden Markov Models
  •    RNNs
    • -
  •    Do generative models need to be stochastic?
    • -
  •    What kinds of behaviour can RNNs exhibit?
  • Basic layout
  •    We need to specify the initial activity state of all the hidden and output units
  •    We can specify inputs in several ways
    • diff --git a/doc/pub/week45/html/week45-reveal.html b/doc/pub/week45/html/week45-reveal.html index 77a8ba32f..e0452ef3b 100644 --- a/doc/pub/week45/html/week45-reveal.html +++ b/doc/pub/week45/html/week45-reveal.html @@ -843,60 +843,6 @@ plt.show() -

    Memoryless models

    - -

    Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using delay taps.

    - -
    -Feed-forward neural networks -

    -

    These generalize autoregressive -models by using one or more -layers of non-linear hidden units. -

    -
    - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -

    1. It can store information in its hidden state for a long time.
      2. -

    2. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
      3. -

    3. The best we can do is to infer a probability distribution over the
      4. -
    -

    -

    space of hidden state vectors.

    - -

    This inference is only tractable for two types of hidden state model.

    -

    Linear dynamical model

    - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -

    1. It can store information in its hidden state for a long time. -
        -

      1. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
        2. -
      -

      -

    2. The best we can do is to infer a probability distribution over the space of hidden state vectors.
      3. -
    -

    -

    Hidden Markov Models

    -

    Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states -are stochastic and controlled by a transition -matrix. The outputs produced by a state are -stochastic. -

    - -

    -

    HMMs have efficient algorithms for inference and learning

    RNNs

    RNNs are very powerful, because they @@ -909,46 +855,8 @@ combine two properties:

    With enough neurons and time, RNNs can compute anything that can be -computed by your computer. +computed by your computer!

    -

    Do generative models need to be stochastic?

    - -
    -Linear dynamical systems and hidden Markov models are stochastic models. -

    - -

    But the posterior probability -distribution over their -hidden states given the -observed data so far is a -deterministic function of the -data. -

    -
    - - -
    -Recurrent neural networks are deterministic. -

    -

    Think of the hidden state -of an RNN as the -equivalent of the -deterministic probability -distribution over hidden -states in a linear dynamical -system or hidden Markov -model. -

    -
    -

    What kinds of behaviour can RNNs exhibit?

    -
      -

    1. They can oscillate.
      2. -

    2. They can settle to point attractors.
      3. -

    3. They can behave chaotically.
      4. -

    4. RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.
      5. -
    -

    -

    But the computational power of RNNs makes them very hard to train.

    diff --git a/doc/pub/week45/html/week45-solarized.html b/doc/pub/week45/html/week45-solarized.html index b9aa6f339..21e1e98b4 100644 --- a/doc/pub/week45/html/week45-solarized.html +++ b/doc/pub/week45/html/week45-solarized.html @@ -97,18 +97,7 @@ div.toc p,a { None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -828,57 +817,6 @@ plt.show() -

    Memoryless models

    - -

    Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using delay taps.

    - -
    -Feed-forward neural networks -

    -

    These generalize autoregressive -models by using one or more -layers of non-linear hidden units. -

    -
    - - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -
    1. It can store information in its hidden state for a long time.
      2. -
    2. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
      3. -
    3. The best we can do is to infer a probability distribution over the
      4. -
    -

    space of hidden state vectors.

    - -

    This inference is only tractable for two types of hidden state model.

    -

    Linear dynamical model

    - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -
    1. It can store information in its hidden state for a long time. -
        -
      1. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
        2. -
      -
    2. The best we can do is to infer a probability distribution over the space of hidden state vectors.
      3. -
    -

    Hidden Markov Models

    -

    Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states -are stochastic and controlled by a transition -matrix. The outputs produced by a state are -stochastic. -

    - -

    HMMs have efficient algorithms for inference and learning

    RNNs

    RNNs are very powerful, because they @@ -890,46 +828,8 @@ combine two properties:

    With enough neurons and time, RNNs can compute anything that can be -computed by your computer. +computed by your computer!

    -

    Do generative models need to be stochastic?

    - -
    -Linear dynamical systems and hidden Markov models are stochastic models. -

    - -

    But the posterior probability -distribution over their -hidden states given the -observed data so far is a -deterministic function of the -data. -

    -
    - - -
    -Recurrent neural networks are deterministic. -

    -

    Think of the hidden state -of an RNN as the -equivalent of the -deterministic probability -distribution over hidden -states in a linear dynamical -system or hidden Markov -model. -

    -
    - -

    What kinds of behaviour can RNNs exhibit?

    -
      -
    1. They can oscillate.
      2. -
    2. They can settle to point attractors.
      3. -
    3. They can behave chaotically.
      4. -
    4. RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.
      5. -
    -

    But the computational power of RNNs makes them very hard to train.











    Basic layout

    diff --git a/doc/pub/week45/html/week45.html b/doc/pub/week45/html/week45.html index 9b9ec5553..c0709a0ef 100644 --- a/doc/pub/week45/html/week45.html +++ b/doc/pub/week45/html/week45.html @@ -174,18 +174,7 @@ div.toc p,a { None, 'recurrent-neural-networks-rnns-overarching-view'), ('A simple example', 2, None, 'a-simple-example'), - ('Memoryless models', 3, None, 'memoryless-models'), - ('Linear dynamical model', 3, None, 'linear-dynamical-model'), - ('Hidden Markov Models', 3, None, 'hidden-markov-models'), ('RNNs', 3, None, 'rnns'), - ('Do generative models need to be stochastic?', - 3, - None, - 'do-generative-models-need-to-be-stochastic'), - ('What kinds of behaviour can RNNs exhibit?', - 3, - None, - 'what-kinds-of-behaviour-can-rnns-exhibit'), ('Basic layout', 2, None, 'basic-layout'), ('We need to specify the initial activity state of all the ' 'hidden and output units', @@ -905,57 +894,6 @@ plt.show() -

    Memoryless models

    - -

    Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using delay taps.

    - -
    -Feed-forward neural networks -

    -

    These generalize autoregressive -models by using one or more -layers of non-linear hidden units. -

    -
    - - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -
    1. It can store information in its hidden state for a long time.
      2. -
    2. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
      3. -
    3. The best we can do is to infer a probability distribution over the
      4. -
    -

    space of hidden state vectors.

    - -

    This inference is only tractable for two types of hidden state model.

    -

    Linear dynamical model

    - -

    If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -

    -
      -
    1. It can store information in its hidden state for a long time. -
        -
      1. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
        2. -
      -
    2. The best we can do is to infer a probability distribution over the space of hidden state vectors.
      3. -
    -

    Hidden Markov Models

    -

    Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states -are stochastic and controlled by a transition -matrix. The outputs produced by a state are -stochastic. -

    - -

    HMMs have efficient algorithms for inference and learning

    RNNs

    RNNs are very powerful, because they @@ -967,46 +905,8 @@ combine two properties:

    With enough neurons and time, RNNs can compute anything that can be -computed by your computer. +computed by your computer!

    -

    Do generative models need to be stochastic?

    - -
    -Linear dynamical systems and hidden Markov models are stochastic models. -

    - -

    But the posterior probability -distribution over their -hidden states given the -observed data so far is a -deterministic function of the -data. -

    -
    - - -
    -Recurrent neural networks are deterministic. -

    -

    Think of the hidden state -of an RNN as the -equivalent of the -deterministic probability -distribution over hidden -states in a linear dynamical -system or hidden Markov -model. -

    -
    - -

    What kinds of behaviour can RNNs exhibit?

    -
      -
    1. They can oscillate.
      2. -
    2. They can settle to point attractors.
      3. -
    3. They can behave chaotically.
      4. -
    4. RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.
      5. -
    -

    But the computational power of RNNs makes them very hard to train.











    Basic layout

    diff --git a/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz b/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz index 176ba89cb..54689d7ff 100644 Binary files a/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz and b/doc/pub/week45/ipynb/ipynb-week45-src.tar.gz differ diff --git a/doc/pub/week45/ipynb/week45.ipynb b/doc/pub/week45/ipynb/week45.ipynb index 4786ae8b8..6a8c0abea 100644 --- a/doc/pub/week45/ipynb/week45.ipynb +++ b/doc/pub/week45/ipynb/week45.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "69171598", + "id": "38177bc8", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "ebbc8fdd", + "id": "a816084f", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "b433ff40", + "id": "f62a8799", "metadata": { "editable": true }, @@ -67,7 +67,7 @@ }, { "cell_type": "markdown", - "id": "ea411c22", + "id": "2a50d9d2", "metadata": { "editable": true }, @@ -77,7 +77,7 @@ }, { "cell_type": "markdown", - "id": "8fbbadbf", + "id": "30ed4f8b", "metadata": { "editable": true }, @@ -96,7 +96,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "893502f0", + "id": "b6060ea0", "metadata": { "collapsed": false, "editable": true @@ -153,7 +153,7 @@ }, { "cell_type": "markdown", - "id": "b643c0f8", + "id": "02a16352", "metadata": { "editable": true }, @@ -165,7 +165,7 @@ }, { "cell_type": "markdown", - "id": "4123686a", + "id": "d15635eb", "metadata": { "editable": true }, @@ -180,7 +180,7 @@ { "cell_type": "code", "execution_count": 2, - "id": "209ceb02", + "id": "3587f6a6", "metadata": { "collapsed": false, "editable": true @@ -233,7 +233,7 @@ }, { "cell_type": "markdown", - "id": "fd60a719", + "id": "e5c4df1d", "metadata": { "editable": true }, @@ -248,7 +248,7 @@ }, { "cell_type": "markdown", - "id": "570fa984", + "id": "334af746", "metadata": { "editable": true }, @@ -268,7 +268,7 @@ { "cell_type": "code", "execution_count": 3, - "id": "7b54dcac", + "id": "04ba2582", "metadata": { "collapsed": false, "editable": true @@ -322,7 +322,7 @@ }, { "cell_type": "markdown", - "id": "8bf97196", + "id": "e713b0d3", "metadata": { "editable": true }, @@ -337,7 +337,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "73e30852", + "id": "a0304d86", "metadata": { "collapsed": false, "editable": true @@ -364,7 +364,7 @@ }, { "cell_type": "markdown", - "id": "93d360f0", + "id": "a67cf20f", "metadata": { "editable": true }, @@ -378,7 +378,7 @@ { "cell_type": "code", "execution_count": 5, - "id": "fc66576c", + "id": "78a673cf", "metadata": { "collapsed": false, "editable": true @@ -423,7 +423,7 @@ }, { "cell_type": "markdown", - "id": "08ab6f58", + "id": "e533f0d8", "metadata": { "editable": true }, @@ -448,7 +448,7 @@ { "cell_type": "code", "execution_count": 6, - "id": "1388f3f6", + "id": "88a99e3b", "metadata": { "collapsed": false, "editable": true @@ -460,7 +460,7 @@ }, { "cell_type": "markdown", - "id": "03180a67", + "id": "e1095373", "metadata": { "editable": true }, @@ -471,7 +471,7 @@ { "cell_type": "code", "execution_count": 7, - "id": "188de2a7", + "id": "580dfa45", "metadata": { "collapsed": false, "editable": true @@ -483,7 +483,7 @@ }, { "cell_type": "markdown", - "id": "d506a495", + "id": "1f67b639", "metadata": { "editable": true }, @@ -496,7 +496,7 @@ }, { "cell_type": "markdown", - "id": "9443edde", + "id": "a42f0ef1", "metadata": { "editable": true }, @@ -507,7 +507,7 @@ { "cell_type": "code", "execution_count": 8, - "id": "f75ec2d4", + "id": "286c60f7", "metadata": { "collapsed": false, "editable": true @@ -550,7 +550,7 @@ }, { "cell_type": "markdown", - "id": "fc240f33", + "id": "5f126f5b", "metadata": { "editable": true }, @@ -560,7 +560,7 @@ }, { "cell_type": "markdown", - "id": "4b7fe66b", + "id": "bf47aea8", "metadata": { "editable": true }, @@ -587,7 +587,7 @@ }, { "cell_type": "markdown", - "id": "212fa502", + "id": "43911e80", "metadata": { "editable": true }, @@ -598,7 +598,7 @@ { "cell_type": "code", "execution_count": 9, - "id": "a7dfd983", + "id": "93a5e85c", "metadata": { "collapsed": false, "editable": true @@ -671,79 +671,7 @@ }, { "cell_type": "markdown", - "id": "658a2163", - "metadata": { - "editable": true - }, - "source": [ - "### Memoryless models\n", - "\n", - "Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using **delay taps**.\n", - "\n", - "**Feed-forward neural networks.**\n", - "\n", - "These generalize autoregressive\n", - "models by using one or more\n", - "layers of non-linear hidden units.\n", - "\n", - "If we give our generative model some hidden state, and if we give\n", - "this hidden state its own internal dynamics, we get a much more\n", - "interesting kind of model.\n", - "1. It can store information in its hidden state for a long time.\n", - "\n", - "2. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.\n", - "\n", - "3. The best we can do is to infer a probability distribution over the\n", - "\n", - "space of hidden state vectors.\n", - "\n", - "This inference is only tractable for two types of hidden state model." - ] - }, - { - "cell_type": "markdown", - "id": "f53098a5", - "metadata": { - "editable": true - }, - "source": [ - "### Linear dynamical model\n", - "\n", - "If we give our generative model some hidden state, and if we give\n", - "this hidden state its own internal dynamics, we get a much more\n", - "interesting kind of model.\n", - "1. It can store information in its hidden state for a long time.\n", - "\n", - "a. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.\n", - "\n", - "2. The best we can do is to infer a probability distribution over the space of hidden state vectors." - ] - }, - { - "cell_type": "markdown", - "id": "9c6db868", - "metadata": { - "editable": true - }, - "source": [ - "### Hidden Markov Models\n", - "\n", - "Hidden Markov Models have a discrete oneof-$N$ hidden state. Transitions between states\n", - "are stochastic and controlled by a transition\n", - "matrix. The outputs produced by a state are\n", - "stochastic.\n", - "* We cannot be sure which state produced a given output. So the state is “hidden”.\n", - "\n", - "* It is easy to represent a probability distribution across N states with N numbers.\n", - "\n", - "* To predict the next output we need to infer the probability distribution over hidden states.\n", - "\n", - "HMMs have efficient algorithms for inference and learning" - ] - }, - { - "cell_type": "markdown", - "id": "c41a29b4", + "id": "15c0abdc", "metadata": { "editable": true }, @@ -758,70 +686,12 @@ "\n", "With enough neurons and time, RNNs\n", "can compute anything that can be\n", - "computed by your computer." + "computed by your computer!" ] }, { "cell_type": "markdown", - "id": "9442be08", - "metadata": { - "editable": true - }, - "source": [ - "### Do generative models need to be stochastic?\n", - "\n", - "**Linear dynamical systems and hidden Markov models are stochastic models.**\n", - "\n", - "But the posterior probability\n", - "distribution over their\n", - "hidden states given the\n", - "observed data so far is a\n", - "deterministic function of the\n", - "data." - ] - }, - { - "cell_type": "markdown", - "id": "25be0826", - "metadata": { - "editable": true - }, - "source": [ - "**Recurrent neural networks are deterministic.**\n", - "\n", - "Think of the hidden state\n", - "of an RNN as the\n", - "equivalent of the\n", - "deterministic probability\n", - "distribution over hidden\n", - "states in a linear dynamical\n", - "system or hidden Markov\n", - "model." - ] - }, - { - "cell_type": "markdown", - "id": "2ac411a1", - "metadata": { - "editable": true - }, - "source": [ - "### What kinds of behaviour can RNNs exhibit?\n", - "\n", - "1. They can oscillate. \n", - "\n", - "2. They can settle to point attractors.\n", - "\n", - "3. They can behave chaotically.\n", - "\n", - "4. RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.\n", - "\n", - "But the computational power of RNNs makes them very hard to train." - ] - }, - { - "cell_type": "markdown", - "id": "cf5863ca", + "id": "e2c9969b", "metadata": { "editable": true }, @@ -837,7 +707,7 @@ }, { "cell_type": "markdown", - "id": "71024067", + "id": "bc82b4e6", "metadata": { "editable": true }, @@ -859,7 +729,7 @@ }, { "cell_type": "markdown", - "id": "26ab640a", + "id": "c1dfa318", "metadata": { "editable": true }, @@ -877,7 +747,7 @@ }, { "cell_type": "markdown", - "id": "e1645578", + "id": "bce88bd1", "metadata": { "editable": true }, @@ -923,7 +793,7 @@ }, { "cell_type": "markdown", - "id": "33704039", + "id": "630c5765", "metadata": { "editable": true }, @@ -944,7 +814,7 @@ }, { "cell_type": "markdown", - "id": "66c99132", + "id": "74a05e7a", "metadata": { "editable": true }, @@ -1005,7 +875,7 @@ }, { "cell_type": "markdown", - "id": "5b7d3f8d", + "id": "ba29a00f", "metadata": { "editable": true }, @@ -1030,7 +900,7 @@ }, { "cell_type": "markdown", - "id": "7179eb43", + "id": "4fb99b23", "metadata": { "editable": true }, @@ -1049,7 +919,7 @@ }, { "cell_type": "markdown", - "id": "69429d28", + "id": "640d0ffd", "metadata": { "editable": true }, @@ -1073,7 +943,7 @@ }, { "cell_type": "markdown", - "id": "a4f15f01", + "id": "0aa61a82", "metadata": { "editable": true }, @@ -1155,7 +1025,7 @@ }, { "cell_type": "markdown", - "id": "dc571c8c", + "id": "05d59f36", "metadata": { "editable": true }, @@ -1171,7 +1041,7 @@ { "cell_type": "code", "execution_count": 10, - "id": "967eb9a4", + "id": "b1bf0cab", "metadata": { "collapsed": false, "editable": true @@ -1210,7 +1080,7 @@ }, { "cell_type": "markdown", - "id": "d05f3007", + "id": "c6095262", "metadata": { "editable": true }, @@ -1254,7 +1124,7 @@ { "cell_type": "code", "execution_count": 11, - "id": "b0162a0e", + "id": "9eebf739", "metadata": { "collapsed": false, "editable": true @@ -1337,7 +1207,7 @@ }, { "cell_type": "markdown", - "id": "bc485b10", + "id": "b1f61819", "metadata": { "editable": true }, @@ -1348,7 +1218,7 @@ { "cell_type": "code", "execution_count": 12, - "id": "7af19f47", + "id": "eed2466a", "metadata": { "collapsed": false, "editable": true @@ -1450,7 +1320,7 @@ }, { "cell_type": "markdown", - "id": "b7557e22", + "id": "71855b2c", "metadata": { "editable": true }, @@ -1472,7 +1342,7 @@ { "cell_type": "code", "execution_count": 13, - "id": "acd0f31b", + "id": "d1650ea8", "metadata": { "collapsed": false, "editable": true @@ -1569,7 +1439,7 @@ }, { "cell_type": "markdown", - "id": "cfb91ca0", + "id": "32c4c59c", "metadata": { "editable": true }, @@ -1595,7 +1465,7 @@ { "cell_type": "code", "execution_count": 14, - "id": "49c4b8ac", + "id": "bbb48a15", "metadata": { "collapsed": false, "editable": true diff --git a/doc/src/week45/week45.do.txt b/doc/src/week45/week45.do.txt index 3de2a4b1e..679214c91 100644 --- a/doc/src/week45/week45.do.txt +++ b/doc/src/week45/week45.do.txt @@ -460,48 +460,6 @@ plt.show() -=== Memoryless models === - -Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using _delay taps_. - -!bblock Feed-forward neural networks -These generalize autoregressive -models by using one or more -layers of non-linear hidden units. -!eblock - -If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -o It can store information in its hidden state for a long time. -o If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state. -o The best we can do is to infer a probability distribution over the -space of hidden state vectors. - - -This inference is only tractable for two types of hidden state model. - - -=== Linear dynamical model === - -If we give our generative model some hidden state, and if we give -this hidden state its own internal dynamics, we get a much more -interesting kind of model. -o It can store information in its hidden state for a long time. -o If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state. -o The best we can do is to infer a probability distribution over the space of hidden state vectors. - - -=== Hidden Markov Models === -Hidden Markov Models have a discrete oneof-$N$ hidden state. Transitions between states -are stochastic and controlled by a transition -matrix. The outputs produced by a state are -stochastic. -* We cannot be sure which state produced a given output. So the state is “hidden”. -* It is easy to represent a probability distribution across N states with N numbers. -* To predict the next output we need to infer the probability distribution over hidden states. - -HMMs have efficient algorithms for inference and learning === RNNs === @@ -512,41 +470,9 @@ o Non-linear dynamics that allows them to update their hidden state in complicat With enough neurons and time, RNNs can compute anything that can be -computed by your computer. +computed by your computer! -=== Do generative models need to be stochastic? === - -!bblock Linear dynamical systems and hidden Markov models are stochastic models. - -But the posterior probability -distribution over their -hidden states given the -observed data so far is a -deterministic function of the -data. -!eblock - - -!bblock Recurrent neural networks are deterministic. -Think of the hidden state -of an RNN as the -equivalent of the -deterministic probability -distribution over hidden -states in a linear dynamical -system or hidden Markov -model. -!eblock - -=== What kinds of behaviour can RNNs exhibit? === -o They can oscillate. -o They can settle to point attractors. -o They can behave chaotically. -o RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects. - -But the computational power of RNNs makes them very hard to train. - !split ===== Basic layout =====