update week 45

doc/src/week45/week45.do.txt

@@ -460,48 +460,6 @@ plt.show()
 
 
 
-=== Memoryless models ===
-
-Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using _delay taps_.
-
-!bblock Feed-forward neural networks
-These generalize autoregressive
-models by using one or more
-layers of non-linear hidden units. 
-!eblock
-
-If we give our generative model some hidden state, and if we give
-this hidden state its own internal dynamics, we get a much more
-interesting kind of model.
-o It can store information in its hidden state for a long time.
-o If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
-o The best we can do is to infer a probability distribution over the
-space of hidden state vectors.
-
-
-This inference is only tractable for two types of hidden state model.
-
-
-=== Linear dynamical model ===
-
-If we give our generative model some hidden state, and if we give
-this hidden state its own internal dynamics, we get a much more
-interesting kind of model.
-o It can store information in its hidden state for a long time.
-o  If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
-o The best we can do is to infer a probability distribution over the space of hidden state vectors.
-
-
-=== Hidden Markov Models ===
-Hidden Markov Models have a discrete oneof-$N$ hidden state. Transitions between states
-are stochastic and controlled by a transition
-matrix. The outputs produced by a state are
-stochastic.
-* We cannot be sure which state produced a given output. So the state is “hidden”.
-* It is easy to represent a probability distribution across N states with N numbers.
-* To predict the next output we need to infer the probability distribution over hidden states.
-
-HMMs have efficient algorithms for inference and learning
 
 === RNNs ===
 
@@ -512,41 +470,9 @@ o Non-linear dynamics that allows them to update their hidden state in complicat
 
 With enough neurons and time, RNNs
 can compute anything that can be
-computed by your computer.
+computed by your computer!
 
 
-=== Do generative models need to be stochastic? ===
-
-!bblock Linear dynamical systems and hidden Markov models are stochastic models.
-
-But the posterior probability
-distribution over their
-hidden states given the
-observed data so far is a
-deterministic function of the
-data.
-!eblock 
-
-
-!bblock Recurrent neural networks are deterministic.
-Think of the hidden state
-of an RNN as the
-equivalent of the
-deterministic probability
-distribution over hidden
-states in a linear dynamical
-system or hidden Markov
-model.
-!eblock
-
-=== What kinds of behaviour can RNNs exhibit? ===
-o They can oscillate. 
-o They can settle to point attractors.
-o They can behave chaotically.
-o RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.
-
-But the computational power of RNNs makes them very hard to train.
-
 !split
 ===== Basic layout =====