update week 45
This commit is contained in:
Morten Hjorth-Jensen
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
@@ -294,58 +278,6 @@ plt<span style="color: #666666">.</span>show()
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<h3 id="memoryless-models" class="anchor">Memoryless models </h3>
|
||||
|
||||
<p>Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using <b>delay taps</b>.</p>
|
||||
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>These generalize autoregressive
|
||||
models by using one or more
|
||||
layers of non-linear hidden units.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<li> It can store information in its hidden state for a long time.</li>
|
||||
<li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
<li> The best we can do is to infer a probability distribution over the</li>
|
||||
</ol>
|
||||
<p>space of hidden state vectors.</p>
|
||||
|
||||
<p>This inference is only tractable for two types of hidden state model.</p>
|
||||
<h3 id="linear-dynamical-model" class="anchor">Linear dynamical model </h3>
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<li> It can store information in its hidden state for a long time.
|
||||
<ol type="a"></li>
|
||||
<li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
</ol>
|
||||
<li> The best we can do is to infer a probability distribution over the space of hidden state vectors.</li>
|
||||
</ol>
|
||||
<h3 id="hidden-markov-models" class="anchor">Hidden Markov Models </h3>
|
||||
<p>Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states
|
||||
are stochastic and controlled by a transition
|
||||
matrix. The outputs produced by a state are
|
||||
stochastic.
|
||||
</p>
|
||||
<ul>
|
||||
<li> We cannot be sure which state produced a given output. So the state is “hidden”.</li>
|
||||
<li> It is easy to represent a probability distribution across N states with N numbers.</li>
|
||||
<li> To predict the next output we need to infer the probability distribution over hidden states.</li>
|
||||
</ul>
|
||||
<p>HMMs have efficient algorithms for inference and learning</p>
|
||||
<h3 id="rnns" class="anchor">RNNs </h3>
|
||||
|
||||
<p>RNNs are very powerful, because they
|
||||
@@ -357,48 +289,8 @@ combine two properties:
|
||||
</ol>
|
||||
<p>With enough neurons and time, RNNs
|
||||
can compute anything that can be
|
||||
computed by your computer.
|
||||
computed by your computer!
|
||||
</p>
|
||||
<h3 id="do-generative-models-need-to-be-stochastic" class="anchor">Do generative models need to be stochastic? </h3>
|
||||
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
|
||||
<p>But the posterior probability
|
||||
distribution over their
|
||||
hidden states given the
|
||||
observed data so far is a
|
||||
deterministic function of the
|
||||
data.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>Think of the hidden state
|
||||
of an RNN as the
|
||||
equivalent of the
|
||||
deterministic probability
|
||||
distribution over hidden
|
||||
states in a linear dynamical
|
||||
system or hidden Markov
|
||||
model.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3 id="what-kinds-of-behaviour-can-rnns-exhibit" class="anchor">What kinds of behaviour can RNNs exhibit? </h3>
|
||||
<ol>
|
||||
<li> They can oscillate.</li>
|
||||
<li> They can settle to point attractors.</li>
|
||||
<li> They can behave chaotically.</li>
|
||||
<li> RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.</li>
|
||||
</ol>
|
||||
<p>But the computational power of RNNs makes them very hard to train.</p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -70,18 +70,7 @@ doconce format html week45.do.txt --html_style=bootstrap --pygments_html_style=d
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -177,12 +166,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs010.html#material-for-lecture-thursday-november-9" style="font-size: 80%;"><b>Material for Lecture Thursday November 9</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs011.html#recurrent-neural-networks-rnns-overarching-view" style="font-size: 80%;"><b>Recurrent neural networks (RNNs): Overarching view</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#a-simple-example" style="font-size: 80%;"><b>A simple example</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#memoryless-models" style="font-size: 80%;"> Memoryless models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#linear-dynamical-model" style="font-size: 80%;"> Linear dynamical model</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#hidden-markov-models" style="font-size: 80%;"> Hidden Markov Models</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#rnns" style="font-size: 80%;"> RNNs</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#do-generative-models-need-to-be-stochastic" style="font-size: 80%;"> Do generative models need to be stochastic?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs012.html#what-kinds-of-behaviour-can-rnns-exhibit" style="font-size: 80%;"> What kinds of behaviour can RNNs exhibit?</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#basic-layout" style="font-size: 80%;"><b>Basic layout</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" style="font-size: 80%;"> We need to specify the initial activity state of all the hidden and output units</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week45-bs013.html#we-can-specify-inputs-in-several-ways" style="font-size: 80%;"> We can specify inputs in several ways</a></li>
|
||||
|
||||
@@ -843,60 +843,6 @@ plt.show()
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<h3 id="memoryless-models">Memoryless models </h3>
|
||||
|
||||
<p>Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using <b>delay taps</b>.</p>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Feed-forward neural networks</b>
|
||||
<p>
|
||||
<p>These generalize autoregressive
|
||||
models by using one or more
|
||||
layers of non-linear hidden units.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<p><li> It can store information in its hidden state for a long time.</li>
|
||||
<p><li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
<p><li> The best we can do is to infer a probability distribution over the</li>
|
||||
</ol>
|
||||
<p>
|
||||
<p>space of hidden state vectors.</p>
|
||||
|
||||
<p>This inference is only tractable for two types of hidden state model.</p>
|
||||
<h3 id="linear-dynamical-model">Linear dynamical model </h3>
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<p><li> It can store information in its hidden state for a long time.
|
||||
<ol type="a"></li>
|
||||
<p><li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
</ol>
|
||||
<p>
|
||||
<p><li> The best we can do is to infer a probability distribution over the space of hidden state vectors.</li>
|
||||
</ol>
|
||||
<p>
|
||||
<h3 id="hidden-markov-models">Hidden Markov Models </h3>
|
||||
<p>Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states
|
||||
are stochastic and controlled by a transition
|
||||
matrix. The outputs produced by a state are
|
||||
stochastic.
|
||||
</p>
|
||||
<ul>
|
||||
<p><li> We cannot be sure which state produced a given output. So the state is “hidden”.</li>
|
||||
<p><li> It is easy to represent a probability distribution across N states with N numbers.</li>
|
||||
<p><li> To predict the next output we need to infer the probability distribution over hidden states.</li>
|
||||
</ul>
|
||||
<p>
|
||||
<p>HMMs have efficient algorithms for inference and learning</p>
|
||||
<h3 id="rnns">RNNs </h3>
|
||||
|
||||
<p>RNNs are very powerful, because they
|
||||
@@ -909,46 +855,8 @@ combine two properties:
|
||||
<p>
|
||||
<p>With enough neurons and time, RNNs
|
||||
can compute anything that can be
|
||||
computed by your computer.
|
||||
computed by your computer!
|
||||
</p>
|
||||
<h3 id="do-generative-models-need-to-be-stochastic">Do generative models need to be stochastic? </h3>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Linear dynamical systems and hidden Markov models are stochastic models.</b>
|
||||
<p>
|
||||
|
||||
<p>But the posterior probability
|
||||
distribution over their
|
||||
hidden states given the
|
||||
observed data so far is a
|
||||
deterministic function of the
|
||||
data.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Recurrent neural networks are deterministic.</b>
|
||||
<p>
|
||||
<p>Think of the hidden state
|
||||
of an RNN as the
|
||||
equivalent of the
|
||||
deterministic probability
|
||||
distribution over hidden
|
||||
states in a linear dynamical
|
||||
system or hidden Markov
|
||||
model.
|
||||
</p>
|
||||
</div>
|
||||
<h3 id="what-kinds-of-behaviour-can-rnns-exhibit">What kinds of behaviour can RNNs exhibit? </h3>
|
||||
<ol>
|
||||
<p><li> They can oscillate.</li>
|
||||
<p><li> They can settle to point attractors.</li>
|
||||
<p><li> They can behave chaotically.</li>
|
||||
<p><li> RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.</li>
|
||||
</ol>
|
||||
<p>
|
||||
<p>But the computational power of RNNs makes them very hard to train.</p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
|
||||
@@ -97,18 +97,7 @@ div.toc p,a {
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -828,57 +817,6 @@ plt.show()
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<h3 id="memoryless-models">Memoryless models </h3>
|
||||
|
||||
<p>Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using <b>delay taps</b>.</p>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Feed-forward neural networks</b>
|
||||
<p>
|
||||
<p>These generalize autoregressive
|
||||
models by using one or more
|
||||
layers of non-linear hidden units.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<li> It can store information in its hidden state for a long time.</li>
|
||||
<li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
<li> The best we can do is to infer a probability distribution over the</li>
|
||||
</ol>
|
||||
<p>space of hidden state vectors.</p>
|
||||
|
||||
<p>This inference is only tractable for two types of hidden state model.</p>
|
||||
<h3 id="linear-dynamical-model">Linear dynamical model </h3>
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<li> It can store information in its hidden state for a long time.
|
||||
<ol type="a"></li>
|
||||
<li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
</ol>
|
||||
<li> The best we can do is to infer a probability distribution over the space of hidden state vectors.</li>
|
||||
</ol>
|
||||
<h3 id="hidden-markov-models">Hidden Markov Models </h3>
|
||||
<p>Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states
|
||||
are stochastic and controlled by a transition
|
||||
matrix. The outputs produced by a state are
|
||||
stochastic.
|
||||
</p>
|
||||
<ul>
|
||||
<li> We cannot be sure which state produced a given output. So the state is “hidden”.</li>
|
||||
<li> It is easy to represent a probability distribution across N states with N numbers.</li>
|
||||
<li> To predict the next output we need to infer the probability distribution over hidden states.</li>
|
||||
</ul>
|
||||
<p>HMMs have efficient algorithms for inference and learning</p>
|
||||
<h3 id="rnns">RNNs </h3>
|
||||
|
||||
<p>RNNs are very powerful, because they
|
||||
@@ -890,46 +828,8 @@ combine two properties:
|
||||
</ol>
|
||||
<p>With enough neurons and time, RNNs
|
||||
can compute anything that can be
|
||||
computed by your computer.
|
||||
computed by your computer!
|
||||
</p>
|
||||
<h3 id="do-generative-models-need-to-be-stochastic">Do generative models need to be stochastic? </h3>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Linear dynamical systems and hidden Markov models are stochastic models.</b>
|
||||
<p>
|
||||
|
||||
<p>But the posterior probability
|
||||
distribution over their
|
||||
hidden states given the
|
||||
observed data so far is a
|
||||
deterministic function of the
|
||||
data.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Recurrent neural networks are deterministic.</b>
|
||||
<p>
|
||||
<p>Think of the hidden state
|
||||
of an RNN as the
|
||||
equivalent of the
|
||||
deterministic probability
|
||||
distribution over hidden
|
||||
states in a linear dynamical
|
||||
system or hidden Markov
|
||||
model.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<h3 id="what-kinds-of-behaviour-can-rnns-exhibit">What kinds of behaviour can RNNs exhibit? </h3>
|
||||
<ol>
|
||||
<li> They can oscillate.</li>
|
||||
<li> They can settle to point attractors.</li>
|
||||
<li> They can behave chaotically.</li>
|
||||
<li> RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.</li>
|
||||
</ol>
|
||||
<p>But the computational power of RNNs makes them very hard to train.</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="basic-layout">Basic layout </h2>
|
||||
|
||||
@@ -174,18 +174,7 @@ div.toc p,a {
|
||||
None,
|
||||
'recurrent-neural-networks-rnns-overarching-view'),
|
||||
('A simple example', 2, None, 'a-simple-example'),
|
||||
('Memoryless models', 3, None, 'memoryless-models'),
|
||||
('Linear dynamical model', 3, None, 'linear-dynamical-model'),
|
||||
('Hidden Markov Models', 3, None, 'hidden-markov-models'),
|
||||
('RNNs', 3, None, 'rnns'),
|
||||
('Do generative models need to be stochastic?',
|
||||
3,
|
||||
None,
|
||||
'do-generative-models-need-to-be-stochastic'),
|
||||
('What kinds of behaviour can RNNs exhibit?',
|
||||
3,
|
||||
None,
|
||||
'what-kinds-of-behaviour-can-rnns-exhibit'),
|
||||
('Basic layout', 2, None, 'basic-layout'),
|
||||
('We need to specify the initial activity state of all the '
|
||||
'hidden and output units',
|
||||
@@ -905,57 +894,6 @@ plt<span style="color: #666666">.</span>show()
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<h3 id="memoryless-models">Memoryless models </h3>
|
||||
|
||||
<p>Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using <b>delay taps</b>.</p>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Feed-forward neural networks</b>
|
||||
<p>
|
||||
<p>These generalize autoregressive
|
||||
models by using one or more
|
||||
layers of non-linear hidden units.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<li> It can store information in its hidden state for a long time.</li>
|
||||
<li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
<li> The best we can do is to infer a probability distribution over the</li>
|
||||
</ol>
|
||||
<p>space of hidden state vectors.</p>
|
||||
|
||||
<p>This inference is only tractable for two types of hidden state model.</p>
|
||||
<h3 id="linear-dynamical-model">Linear dynamical model </h3>
|
||||
|
||||
<p>If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
</p>
|
||||
<ol>
|
||||
<li> It can store information in its hidden state for a long time.
|
||||
<ol type="a"></li>
|
||||
<li> If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.</li>
|
||||
</ol>
|
||||
<li> The best we can do is to infer a probability distribution over the space of hidden state vectors.</li>
|
||||
</ol>
|
||||
<h3 id="hidden-markov-models">Hidden Markov Models </h3>
|
||||
<p>Hidden Markov Models have a discrete oneof-\( N \) hidden state. Transitions between states
|
||||
are stochastic and controlled by a transition
|
||||
matrix. The outputs produced by a state are
|
||||
stochastic.
|
||||
</p>
|
||||
<ul>
|
||||
<li> We cannot be sure which state produced a given output. So the state is “hidden”.</li>
|
||||
<li> It is easy to represent a probability distribution across N states with N numbers.</li>
|
||||
<li> To predict the next output we need to infer the probability distribution over hidden states.</li>
|
||||
</ul>
|
||||
<p>HMMs have efficient algorithms for inference and learning</p>
|
||||
<h3 id="rnns">RNNs </h3>
|
||||
|
||||
<p>RNNs are very powerful, because they
|
||||
@@ -967,46 +905,8 @@ combine two properties:
|
||||
</ol>
|
||||
<p>With enough neurons and time, RNNs
|
||||
can compute anything that can be
|
||||
computed by your computer.
|
||||
computed by your computer!
|
||||
</p>
|
||||
<h3 id="do-generative-models-need-to-be-stochastic">Do generative models need to be stochastic? </h3>
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Linear dynamical systems and hidden Markov models are stochastic models.</b>
|
||||
<p>
|
||||
|
||||
<p>But the posterior probability
|
||||
distribution over their
|
||||
hidden states given the
|
||||
observed data so far is a
|
||||
deterministic function of the
|
||||
data.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b>Recurrent neural networks are deterministic.</b>
|
||||
<p>
|
||||
<p>Think of the hidden state
|
||||
of an RNN as the
|
||||
equivalent of the
|
||||
deterministic probability
|
||||
distribution over hidden
|
||||
states in a linear dynamical
|
||||
system or hidden Markov
|
||||
model.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<h3 id="what-kinds-of-behaviour-can-rnns-exhibit">What kinds of behaviour can RNNs exhibit? </h3>
|
||||
<ol>
|
||||
<li> They can oscillate.</li>
|
||||
<li> They can settle to point attractors.</li>
|
||||
<li> They can behave chaotically.</li>
|
||||
<li> RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.</li>
|
||||
</ol>
|
||||
<p>But the computational power of RNNs makes them very hard to train.</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="basic-layout">Basic layout </h2>
|
||||
|
||||
Binary file not shown.
@@ -2,7 +2,7 @@
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "69171598",
|
||||
"id": "38177bc8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -14,7 +14,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ebbc8fdd",
|
||||
"id": "a816084f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -27,7 +27,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b433ff40",
|
||||
"id": "f62a8799",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -67,7 +67,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ea411c22",
|
||||
"id": "2a50d9d2",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -77,7 +77,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8fbbadbf",
|
||||
"id": "30ed4f8b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -96,7 +96,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "893502f0",
|
||||
"id": "b6060ea0",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -153,7 +153,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b643c0f8",
|
||||
"id": "02a16352",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -165,7 +165,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4123686a",
|
||||
"id": "d15635eb",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -180,7 +180,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "209ceb02",
|
||||
"id": "3587f6a6",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -233,7 +233,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd60a719",
|
||||
"id": "e5c4df1d",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -248,7 +248,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "570fa984",
|
||||
"id": "334af746",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -268,7 +268,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "7b54dcac",
|
||||
"id": "04ba2582",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -322,7 +322,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8bf97196",
|
||||
"id": "e713b0d3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -337,7 +337,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "73e30852",
|
||||
"id": "a0304d86",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -364,7 +364,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "93d360f0",
|
||||
"id": "a67cf20f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -378,7 +378,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"id": "fc66576c",
|
||||
"id": "78a673cf",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -423,7 +423,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "08ab6f58",
|
||||
"id": "e533f0d8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -448,7 +448,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "1388f3f6",
|
||||
"id": "88a99e3b",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -460,7 +460,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "03180a67",
|
||||
"id": "e1095373",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -471,7 +471,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "188de2a7",
|
||||
"id": "580dfa45",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -483,7 +483,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d506a495",
|
||||
"id": "1f67b639",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -496,7 +496,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9443edde",
|
||||
"id": "a42f0ef1",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -507,7 +507,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "f75ec2d4",
|
||||
"id": "286c60f7",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -550,7 +550,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fc240f33",
|
||||
"id": "5f126f5b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -560,7 +560,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4b7fe66b",
|
||||
"id": "bf47aea8",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -587,7 +587,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "212fa502",
|
||||
"id": "43911e80",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -598,7 +598,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "a7dfd983",
|
||||
"id": "93a5e85c",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -671,79 +671,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "658a2163",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Memoryless models\n",
|
||||
"\n",
|
||||
"Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using **delay taps**.\n",
|
||||
"\n",
|
||||
"**Feed-forward neural networks.**\n",
|
||||
"\n",
|
||||
"These generalize autoregressive\n",
|
||||
"models by using one or more\n",
|
||||
"layers of non-linear hidden units.\n",
|
||||
"\n",
|
||||
"If we give our generative model some hidden state, and if we give\n",
|
||||
"this hidden state its own internal dynamics, we get a much more\n",
|
||||
"interesting kind of model.\n",
|
||||
"1. It can store information in its hidden state for a long time.\n",
|
||||
"\n",
|
||||
"2. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.\n",
|
||||
"\n",
|
||||
"3. The best we can do is to infer a probability distribution over the\n",
|
||||
"\n",
|
||||
"space of hidden state vectors.\n",
|
||||
"\n",
|
||||
"This inference is only tractable for two types of hidden state model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f53098a5",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Linear dynamical model\n",
|
||||
"\n",
|
||||
"If we give our generative model some hidden state, and if we give\n",
|
||||
"this hidden state its own internal dynamics, we get a much more\n",
|
||||
"interesting kind of model.\n",
|
||||
"1. It can store information in its hidden state for a long time.\n",
|
||||
"\n",
|
||||
"a. If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.\n",
|
||||
"\n",
|
||||
"2. The best we can do is to infer a probability distribution over the space of hidden state vectors."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9c6db868",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Hidden Markov Models\n",
|
||||
"\n",
|
||||
"Hidden Markov Models have a discrete oneof-$N$ hidden state. Transitions between states\n",
|
||||
"are stochastic and controlled by a transition\n",
|
||||
"matrix. The outputs produced by a state are\n",
|
||||
"stochastic.\n",
|
||||
"* We cannot be sure which state produced a given output. So the state is “hidden”.\n",
|
||||
"\n",
|
||||
"* It is easy to represent a probability distribution across N states with N numbers.\n",
|
||||
"\n",
|
||||
"* To predict the next output we need to infer the probability distribution over hidden states.\n",
|
||||
"\n",
|
||||
"HMMs have efficient algorithms for inference and learning"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c41a29b4",
|
||||
"id": "15c0abdc",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -758,70 +686,12 @@
|
||||
"\n",
|
||||
"With enough neurons and time, RNNs\n",
|
||||
"can compute anything that can be\n",
|
||||
"computed by your computer."
|
||||
"computed by your computer!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "9442be08",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### Do generative models need to be stochastic?\n",
|
||||
"\n",
|
||||
"**Linear dynamical systems and hidden Markov models are stochastic models.**\n",
|
||||
"\n",
|
||||
"But the posterior probability\n",
|
||||
"distribution over their\n",
|
||||
"hidden states given the\n",
|
||||
"observed data so far is a\n",
|
||||
"deterministic function of the\n",
|
||||
"data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "25be0826",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**Recurrent neural networks are deterministic.**\n",
|
||||
"\n",
|
||||
"Think of the hidden state\n",
|
||||
"of an RNN as the\n",
|
||||
"equivalent of the\n",
|
||||
"deterministic probability\n",
|
||||
"distribution over hidden\n",
|
||||
"states in a linear dynamical\n",
|
||||
"system or hidden Markov\n",
|
||||
"model."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ac411a1",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"### What kinds of behaviour can RNNs exhibit?\n",
|
||||
"\n",
|
||||
"1. They can oscillate. \n",
|
||||
"\n",
|
||||
"2. They can settle to point attractors.\n",
|
||||
"\n",
|
||||
"3. They can behave chaotically.\n",
|
||||
"\n",
|
||||
"4. RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.\n",
|
||||
"\n",
|
||||
"But the computational power of RNNs makes them very hard to train."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cf5863ca",
|
||||
"id": "e2c9969b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -837,7 +707,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "71024067",
|
||||
"id": "bc82b4e6",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -859,7 +729,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "26ab640a",
|
||||
"id": "c1dfa318",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -877,7 +747,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e1645578",
|
||||
"id": "bce88bd1",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -923,7 +793,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "33704039",
|
||||
"id": "630c5765",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -944,7 +814,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "66c99132",
|
||||
"id": "74a05e7a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1005,7 +875,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5b7d3f8d",
|
||||
"id": "ba29a00f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1030,7 +900,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7179eb43",
|
||||
"id": "4fb99b23",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1049,7 +919,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "69429d28",
|
||||
"id": "640d0ffd",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1073,7 +943,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a4f15f01",
|
||||
"id": "0aa61a82",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1155,7 +1025,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dc571c8c",
|
||||
"id": "05d59f36",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1171,7 +1041,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "967eb9a4",
|
||||
"id": "b1bf0cab",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1210,7 +1080,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d05f3007",
|
||||
"id": "c6095262",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1254,7 +1124,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "b0162a0e",
|
||||
"id": "9eebf739",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1337,7 +1207,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "bc485b10",
|
||||
"id": "b1f61819",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1348,7 +1218,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "7af19f47",
|
||||
"id": "eed2466a",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1450,7 +1320,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7557e22",
|
||||
"id": "71855b2c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1472,7 +1342,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "acd0f31b",
|
||||
"id": "d1650ea8",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
@@ -1569,7 +1439,7 @@
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "cfb91ca0",
|
||||
"id": "32c4c59c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
@@ -1595,7 +1465,7 @@
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "49c4b8ac",
|
||||
"id": "bbb48a15",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
|
||||
@@ -460,48 +460,6 @@ plt.show()
|
||||
|
||||
|
||||
|
||||
=== Memoryless models ===
|
||||
|
||||
Autoregressive models Predict the next term in a sequence from a fixed number of previous terms using _delay taps_.
|
||||
|
||||
!bblock Feed-forward neural networks
|
||||
These generalize autoregressive
|
||||
models by using one or more
|
||||
layers of non-linear hidden units.
|
||||
!eblock
|
||||
|
||||
If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
o It can store information in its hidden state for a long time.
|
||||
o If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
|
||||
o The best we can do is to infer a probability distribution over the
|
||||
space of hidden state vectors.
|
||||
|
||||
|
||||
This inference is only tractable for two types of hidden state model.
|
||||
|
||||
|
||||
=== Linear dynamical model ===
|
||||
|
||||
If we give our generative model some hidden state, and if we give
|
||||
this hidden state its own internal dynamics, we get a much more
|
||||
interesting kind of model.
|
||||
o It can store information in its hidden state for a long time.
|
||||
o If the dynamics is noisy and the way it generates outputs from its hidden state is noisy, we can never know its exact hidden state.
|
||||
o The best we can do is to infer a probability distribution over the space of hidden state vectors.
|
||||
|
||||
|
||||
=== Hidden Markov Models ===
|
||||
Hidden Markov Models have a discrete oneof-$N$ hidden state. Transitions between states
|
||||
are stochastic and controlled by a transition
|
||||
matrix. The outputs produced by a state are
|
||||
stochastic.
|
||||
* We cannot be sure which state produced a given output. So the state is “hidden”.
|
||||
* It is easy to represent a probability distribution across N states with N numbers.
|
||||
* To predict the next output we need to infer the probability distribution over hidden states.
|
||||
|
||||
HMMs have efficient algorithms for inference and learning
|
||||
|
||||
=== RNNs ===
|
||||
|
||||
@@ -512,41 +470,9 @@ o Non-linear dynamics that allows them to update their hidden state in complicat
|
||||
|
||||
With enough neurons and time, RNNs
|
||||
can compute anything that can be
|
||||
computed by your computer.
|
||||
computed by your computer!
|
||||
|
||||
|
||||
=== Do generative models need to be stochastic? ===
|
||||
|
||||
!bblock Linear dynamical systems and hidden Markov models are stochastic models.
|
||||
|
||||
But the posterior probability
|
||||
distribution over their
|
||||
hidden states given the
|
||||
observed data so far is a
|
||||
deterministic function of the
|
||||
data.
|
||||
!eblock
|
||||
|
||||
|
||||
!bblock Recurrent neural networks are deterministic.
|
||||
Think of the hidden state
|
||||
of an RNN as the
|
||||
equivalent of the
|
||||
deterministic probability
|
||||
distribution over hidden
|
||||
states in a linear dynamical
|
||||
system or hidden Markov
|
||||
model.
|
||||
!eblock
|
||||
|
||||
=== What kinds of behaviour can RNNs exhibit? ===
|
||||
o They can oscillate.
|
||||
o They can settle to point attractors.
|
||||
o They can behave chaotically.
|
||||
o RNNs could potentially learn to implement lots of small programs that each capture a nugget of knowledge and run in parallel, interacting to produce very complicated effects.
|
||||
|
||||
But the computational power of RNNs makes them very hard to train.
|
||||
|
||||
!split
|
||||
===== Basic layout =====
|
||||
|
||||
|
||||
Reference in New Issue
Block a user