lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
0e5076dfbfc8cf7946a5cec60d742ecccdbbcd3e
FYS-STK4155/doc/pub/week43/html/._week43-bs012.html
T
mhjensen 328e39e288 update lectures
2021-10-29 18:05:40 +02:00

503 lines
38 KiB
HTML
Raw Blame History

<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis">
<title>Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Plans for week 43', 2, None, '___sec0'),
('Reading Recommendations', 2, None, '___sec1'),
('Summary on Deep Learning Methods', 2, None, '___sec2'),
('CNNs in brief', 2, None, '___sec3'),
('Recurrent neural networks: Overarching view',
2,
None,
'___sec4'),
('Set up of an RNN', 2, None, '___sec5'),
('A simple example', 2, None, '___sec6'),
('An extrapolation example', 2, None, '___sec7'),
('Formatting the Data', 2, None, '___sec8'),
('Predicting New Points With A Trained Recurrent Neural Network',
2,
None,
'___sec9'),
('Other Things to Try', 2, None, '___sec10'),
('Other Types of Recurrent Neural Networks', 2, None, '___sec11'),
('Generative Models', 2, None, '___sec12'),
('Generative Adversarial Networks', 2, None, '___sec13'),
('Discriminator', 2, None, '___sec14'),
('Learning Process', 2, None, '___sec15'),
('More about the Learning Process', 2, None, '___sec16'),
('Additional References', 2, None, '___sec17'),
('Writing Our First Generative Adversarial Network',
2,
None,
'___sec18'),
('MNIST and GANs', 2, None, '___sec19'),
('Other Models', 2, None, '___sec20'),
('Training Step', 2, None, '___sec21'),
('Checkpoints', 2, None, '___sec22'),
('Exploring the Latent Space', 2, None, '___sec23'),
('Getting Results', 2, None, '___sec24'),
('Interpolating Between MNIST Digits', 2, None, '___sec25'),
('Basic ideas of the Principal Component Analysis (PCA)',
2,
None,
'___sec26'),
('Introducing the Covariance and Correlation functions',
2,
None,
'___sec27'),
('More on the covariance', 2, None, '___sec28'),
('Reminding ourselves about Linear Regression',
2,
None,
'___sec29'),
('Simple Example', 2, None, '___sec30'),
('The Correlation Matrix', 2, None, '___sec31'),
('Numpy Functionality', 2, None, '___sec32'),
('Correlation Matrix again', 2, None, '___sec33'),
('Using Pandas', 2, None, '___sec34'),
('And then the Franke Function', 2, None, '___sec35'),
('Lnks with the Design Matrix', 2, None, '___sec36'),
('Computing the Expectation Values', 2, None, '___sec37'),
('Towards the PCA theorem', 2, None, '___sec38'),
('More on the PCA Theorem', 2, None, '___sec39'),
('The Algorithm before the Theorem', 2, None, '___sec40'),
('Writing our own PCA code', 2, None, '___sec41'),
('Implementing it', 2, None, '___sec42'),
('First Step', 2, None, '___sec43'),
('Scaling', 2, None, '___sec44'),
('Centered Data', 2, None, '___sec45'),
('Exploring', 2, None, '___sec46'),
('Diagonalize the sample covariance matrix to obtain the '
'principal components',
2,
None,
'___sec47'),
('Collecting all Steps', 2, None, '___sec48'),
('Classical PCA Theorem', 2, None, '___sec49'),
('The PCA Theorem', 2, None, '___sec50'),
('Geometric Interpretation and link with Singular Value '
'Decomposition',
2,
None,
'___sec51'),
('PCA and scikit-learn', 2, None, '___sec52'),
('Back to the Cancer Data', 2, None, '___sec53'),
('Incremental PCA', 2, None, '___sec54'),
('Randomized PCA', 3, None, '___sec55'),
('Kernel PCA', 3, None, '___sec56'),
('Other techniques', 2, None, '___sec57')]}
end of tocinfo -->
<body>
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
TeX: {
equationNumbers: { autoNumber: "none" },
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
}
});
</script>
<script type="text/javascript" async
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
</script>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week43-bs.html">Week 43: Deep Learning: Recurrent Neural Networks and other Deep Learning Methods. Principal Component analysis</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week43-bs001.html#___sec0" style="font-size: 80%;"><b>Plans for week 43</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs002.html#___sec1" style="font-size: 80%;"><b>Reading Recommendations</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs003.html#___sec2" style="font-size: 80%;"><b>Summary on Deep Learning Methods</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs004.html#___sec3" style="font-size: 80%;"><b>CNNs in brief</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs005.html#___sec4" style="font-size: 80%;"><b>Recurrent neural networks: Overarching view</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs006.html#___sec5" style="font-size: 80%;"><b>Set up of an RNN</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs007.html#___sec6" style="font-size: 80%;"><b>A simple example</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs008.html#___sec7" style="font-size: 80%;"><b>An extrapolation example</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs009.html#___sec8" style="font-size: 80%;"><b>Formatting the Data</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs010.html#___sec9" style="font-size: 80%;"><b>Predicting New Points With A Trained Recurrent Neural Network</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs011.html#___sec10" style="font-size: 80%;"><b>Other Things to Try</b></a></li>
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;"><b>Other Types of Recurrent Neural Networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs013.html#___sec12" style="font-size: 80%;"><b>Generative Models</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs014.html#___sec13" style="font-size: 80%;"><b>Generative Adversarial Networks</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs015.html#___sec14" style="font-size: 80%;"><b>Discriminator</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs016.html#___sec15" style="font-size: 80%;"><b>Learning Process</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs017.html#___sec16" style="font-size: 80%;"><b>More about the Learning Process</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs018.html#___sec17" style="font-size: 80%;"><b>Additional References</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs019.html#___sec18" style="font-size: 80%;"><b>Writing Our First Generative Adversarial Network</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs020.html#___sec19" style="font-size: 80%;"><b>MNIST and GANs</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs021.html#___sec20" style="font-size: 80%;"><b>Other Models</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs022.html#___sec21" style="font-size: 80%;"><b>Training Step</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs023.html#___sec22" style="font-size: 80%;"><b>Checkpoints</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs024.html#___sec23" style="font-size: 80%;"><b>Exploring the Latent Space</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs025.html#___sec24" style="font-size: 80%;"><b>Getting Results</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs026.html#___sec25" style="font-size: 80%;"><b>Interpolating Between MNIST Digits</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs027.html#___sec26" style="font-size: 80%;"><b>Basic ideas of the Principal Component Analysis (PCA)</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs028.html#___sec27" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs029.html#___sec28" style="font-size: 80%;"><b>More on the covariance</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs030.html#___sec29" style="font-size: 80%;"><b>Reminding ourselves about Linear Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs031.html#___sec30" style="font-size: 80%;"><b>Simple Example</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs032.html#___sec31" style="font-size: 80%;"><b>The Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs033.html#___sec32" style="font-size: 80%;"><b>Numpy Functionality</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs034.html#___sec33" style="font-size: 80%;"><b>Correlation Matrix again</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs035.html#___sec34" style="font-size: 80%;"><b>Using Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs036.html#___sec35" style="font-size: 80%;"><b>And then the Franke Function</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs037.html#___sec36" style="font-size: 80%;"><b>Lnks with the Design Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs038.html#___sec37" style="font-size: 80%;"><b>Computing the Expectation Values</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs039.html#___sec38" style="font-size: 80%;"><b>Towards the PCA theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs040.html#___sec39" style="font-size: 80%;"><b>More on the PCA Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs041.html#___sec40" style="font-size: 80%;"><b>The Algorithm before the Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs042.html#___sec41" style="font-size: 80%;"><b>Writing our own PCA code</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs043.html#___sec42" style="font-size: 80%;"><b>Implementing it</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs044.html#___sec43" style="font-size: 80%;"><b>First Step</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs045.html#___sec44" style="font-size: 80%;"><b>Scaling</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs046.html#___sec45" style="font-size: 80%;"><b>Centered Data</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs047.html#___sec46" style="font-size: 80%;"><b>Exploring</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs048.html#___sec47" style="font-size: 80%;"><b>Diagonalize the sample covariance matrix to obtain the principal components</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs049.html#___sec48" style="font-size: 80%;"><b>Collecting all Steps</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs050.html#___sec49" style="font-size: 80%;"><b>Classical PCA Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs051.html#___sec50" style="font-size: 80%;"><b>The PCA Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs052.html#___sec51" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs053.html#___sec52" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs054.html#___sec53" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs055.html#___sec54" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
<!-- navigation toc: --> <li><a href="._week43-bs055.html#___sec55" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._week43-bs055.html#___sec56" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._week43-bs056.html#___sec57" style="font-size: 80%;"><b>Other techniques</b></a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0012"></a>
<!-- !split -->
<h2 id="___sec11" class="anchor">Other Types of Recurrent Neural Networks </h2>
<p>
Besides a simple recurrent neural network layer, there are two other
commonly used types of recurrent neural network layers: Long Short
Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short
introduction to these layers see <a href="https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b" target="_self"><tt>https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b</tt></a>
and <a href="https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b" target="_self"><tt>https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b</tt></a>.
<p>
The first network created below is similar to the previous network,
but it replaces the SimpleRNN layers with LSTM layers. The second
network below has two hidden layers made up of GRUs, which are
preceeded by two dense (feeddorward) neural network layers. These
dense layers "preprocess" the data before it reaches the recurrent
layers. This architecture has been shown to improve the performance
of recurrent neural networks (see the link above and also
<a href="https://arxiv.org/pdf/1807.02857.pdf" target="_self"><tt>https://arxiv.org/pdf/1807.02857.pdf</tt></a>.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">lstm_2layers</span>(length_of_sequences, batch_size <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>, stateful <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">False</span>):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Inputs:</span>
<span style="color: #BA2121; font-style: italic"> length_of_sequences (an int): the number of y values in &quot;x data&quot;. This is determined</span>
<span style="color: #BA2121; font-style: italic"> when the data is formatted</span>
<span style="color: #BA2121; font-style: italic"> batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.</span>
<span style="color: #BA2121; font-style: italic"> stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.</span>
<span style="color: #BA2121; font-style: italic"> Returns:</span>
<span style="color: #BA2121; font-style: italic"> model (a Keras model): The recurrent neural network that is built and compiled by this</span>
<span style="color: #BA2121; font-style: italic"> method</span>
<span style="color: #BA2121; font-style: italic"> Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic"># Number of neurons on the input/output layer and the number of neurons in the hidden layer</span>
in_out_neurons <span style="color: #666666">=</span> <span style="color: #666666">1</span>
hidden_neurons <span style="color: #666666">=</span> <span style="color: #666666">250</span>
<span style="color: #408080; font-style: italic"># Input Layer</span>
inp <span style="color: #666666">=</span> Input(batch_shape<span style="color: #666666">=</span>(batch_size,
length_of_sequences,
in_out_neurons))
<span style="color: #408080; font-style: italic"># Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)</span>
rnn<span style="color: #666666">=</span> LSTM(hidden_neurons,
return_sequences<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,
stateful <span style="color: #666666">=</span> stateful,
name<span style="color: #666666">=</span><span style="color: #BA2121">&quot;RNN&quot;</span>, use_bias<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;tanh&#39;</span>)(inp)
rnn1 <span style="color: #666666">=</span> LSTM(hidden_neurons,
return_sequences<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>,
stateful <span style="color: #666666">=</span> stateful,
name<span style="color: #666666">=</span><span style="color: #BA2121">&quot;RNN1&quot;</span>, use_bias<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;tanh&#39;</span>)(rnn)
<span style="color: #408080; font-style: italic"># Output layer</span>
dens <span style="color: #666666">=</span> Dense(in_out_neurons,name<span style="color: #666666">=</span><span style="color: #BA2121">&quot;dense&quot;</span>)(rnn1)
<span style="color: #408080; font-style: italic"># Define the midel</span>
model <span style="color: #666666">=</span> Model(inputs<span style="color: #666666">=</span>[inp],outputs<span style="color: #666666">=</span>[dens])
<span style="color: #408080; font-style: italic"># Compile the model</span>
model<span style="color: #666666">.</span>compile(loss<span style="color: #666666">=</span><span style="color: #BA2121">&#39;mean_squared_error&#39;</span>, optimizer<span style="color: #666666">=</span><span style="color: #BA2121">&#39;adam&#39;</span>)
<span style="color: #408080; font-style: italic"># Return the model</span>
<span style="color: #008000; font-weight: bold">return</span> model
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">dnn2_gru2</span>(length_of_sequences, batch_size <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">None</span>, stateful <span style="color: #666666">=</span> <span style="color: #008000; font-weight: bold">False</span>):
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic"> Inputs:</span>
<span style="color: #BA2121; font-style: italic"> length_of_sequences (an int): the number of y values in &quot;x data&quot;. This is determined</span>
<span style="color: #BA2121; font-style: italic"> when the data is formatted</span>
<span style="color: #BA2121; font-style: italic"> batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.</span>
<span style="color: #BA2121; font-style: italic"> stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.</span>
<span style="color: #BA2121; font-style: italic"> Returns:</span>
<span style="color: #BA2121; font-style: italic"> model (a Keras model): The recurrent neural network that is built and compiled by this</span>
<span style="color: #BA2121; font-style: italic"> method</span>
<span style="color: #BA2121; font-style: italic"> Builds and compiles a recurrent neural network with four hidden layers (two dense followed by</span>
<span style="color: #BA2121; font-style: italic"> two GRU layers) and returns the model.</span>
<span style="color: #BA2121; font-style: italic"> &quot;&quot;&quot;</span>
<span style="color: #408080; font-style: italic"># Number of neurons on the input/output layers and hidden layers</span>
in_out_neurons <span style="color: #666666">=</span> <span style="color: #666666">1</span>
hidden_neurons <span style="color: #666666">=</span> <span style="color: #666666">250</span>
<span style="color: #408080; font-style: italic"># Input layer</span>
inp <span style="color: #666666">=</span> Input(batch_shape<span style="color: #666666">=</span>(batch_size,
length_of_sequences,
in_out_neurons))
<span style="color: #408080; font-style: italic"># Hidden Dense (feedforward) layers</span>
dnn <span style="color: #666666">=</span> Dense(hidden_neurons<span style="color: #666666">/2</span>, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>, name<span style="color: #666666">=</span><span style="color: #BA2121">&#39;dnn&#39;</span>)(inp)
dnn1 <span style="color: #666666">=</span> Dense(hidden_neurons<span style="color: #666666">/2</span>, activation<span style="color: #666666">=</span><span style="color: #BA2121">&#39;relu&#39;</span>, name<span style="color: #666666">=</span><span style="color: #BA2121">&#39;dnn1&#39;</span>)(dnn)
<span style="color: #408080; font-style: italic"># Hidden GRU layers</span>
rnn1 <span style="color: #666666">=</span> GRU(hidden_neurons,
return_sequences<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,
stateful <span style="color: #666666">=</span> stateful,
name<span style="color: #666666">=</span><span style="color: #BA2121">&quot;RNN1&quot;</span>, use_bias<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)(dnn1)
rnn <span style="color: #666666">=</span> GRU(hidden_neurons,
return_sequences<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>,
stateful <span style="color: #666666">=</span> stateful,
name<span style="color: #666666">=</span><span style="color: #BA2121">&quot;RNN&quot;</span>, use_bias<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)(rnn1)
<span style="color: #408080; font-style: italic"># Output layer</span>
dens <span style="color: #666666">=</span> Dense(in_out_neurons,name<span style="color: #666666">=</span><span style="color: #BA2121">&quot;dense&quot;</span>)(rnn)
<span style="color: #408080; font-style: italic"># Define the model</span>
model <span style="color: #666666">=</span> Model(inputs<span style="color: #666666">=</span>[inp],outputs<span style="color: #666666">=</span>[dens])
<span style="color: #408080; font-style: italic"># Compile the mdoel</span>
model<span style="color: #666666">.</span>compile(loss<span style="color: #666666">=</span><span style="color: #BA2121">&#39;mean_squared_error&#39;</span>, optimizer<span style="color: #666666">=</span><span style="color: #BA2121">&#39;adam&#39;</span>)
<span style="color: #408080; font-style: italic"># Return the model</span>
<span style="color: #008000; font-weight: bold">return</span> model
<span style="color: #408080; font-style: italic"># Check to make sure the data set is complete</span>
<span style="color: #008000; font-weight: bold">assert</span> <span style="color: #008000">len</span>(X_tot) <span style="color: #666666">==</span> <span style="color: #008000">len</span>(y_tot)
<span style="color: #408080; font-style: italic"># This is the number of points that will be used in as the training data</span>
dim<span style="color: #666666">=12</span>
<span style="color: #408080; font-style: italic"># Separate the training data from the whole data set</span>
X_train <span style="color: #666666">=</span> X_tot[:dim]
y_train <span style="color: #666666">=</span> y_tot[:dim]
<span style="color: #408080; font-style: italic"># Generate the training data for the RNN, using a sequence of 2</span>
rnn_input, rnn_training <span style="color: #666666">=</span> format_data(y_train, <span style="color: #666666">2</span>)
<span style="color: #408080; font-style: italic"># Create a recurrent neural network in Keras and produce a summary of the </span>
<span style="color: #408080; font-style: italic"># machine learning model</span>
<span style="color: #408080; font-style: italic"># Change the method name to reflect which network you want to use</span>
model <span style="color: #666666">=</span> dnn2_gru2(length_of_sequences <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
model<span style="color: #666666">.</span>summary()
<span style="color: #408080; font-style: italic"># Start the timer. Want to time training+testing</span>
start <span style="color: #666666">=</span> timer()
<span style="color: #408080; font-style: italic"># Fit the model using the training data genenerated above using 150 training iterations and a 5%</span>
<span style="color: #408080; font-style: italic"># validation split. Setting verbose to True prints information about each training iteration.</span>
hist <span style="color: #666666">=</span> model<span style="color: #666666">.</span>fit(rnn_input, rnn_training, batch_size<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>, epochs<span style="color: #666666">=150</span>,
verbose<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,validation_split<span style="color: #666666">=0.05</span>)
<span style="color: #408080; font-style: italic"># This section plots the training loss and the validation loss as a function of training iteration.</span>
<span style="color: #408080; font-style: italic"># This is not required for analyzing the couple cluster data but can help determine if the network is</span>
<span style="color: #408080; font-style: italic"># being overtrained.</span>
<span style="color: #008000; font-weight: bold">for</span> label <span style="color: #AA22FF; font-weight: bold">in</span> [<span style="color: #BA2121">&quot;loss&quot;</span>,<span style="color: #BA2121">&quot;val_loss&quot;</span>]:
plt<span style="color: #666666">.</span>plot(hist<span style="color: #666666">.</span>history[label],label<span style="color: #666666">=</span>label)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;loss&quot;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;epoch&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;The final validation loss: </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">.</span>format(hist<span style="color: #666666">.</span>history[<span style="color: #BA2121">&quot;val_loss&quot;</span>][<span style="color: #666666">-1</span>]))
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic"># Use the trained neural network to predict more points of the data set</span>
test_rnn(X_tot, y_tot, X_tot[<span style="color: #666666">0</span>], X_tot[dim<span style="color: #666666">-1</span>])
<span style="color: #408080; font-style: italic"># Stop the timer and calculate the total time needed.</span>
end <span style="color: #666666">=</span> timer()
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Time: &#39;</span>, end<span style="color: #666666">-</span>start)
<span style="color: #408080; font-style: italic"># ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)</span>
<span style="color: #408080; font-style: italic"># </span>
<span style="color: #408080; font-style: italic"># Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.</span>
<span style="color: #408080; font-style: italic"># Check to make sure the data set is complete</span>
<span style="color: #008000; font-weight: bold">assert</span> <span style="color: #008000">len</span>(X_tot) <span style="color: #666666">==</span> <span style="color: #008000">len</span>(y_tot)
<span style="color: #408080; font-style: italic"># This is the number of points that will be used in as the training data</span>
dim<span style="color: #666666">=12</span>
<span style="color: #408080; font-style: italic"># Separate the training data from the whole data set</span>
X_train <span style="color: #666666">=</span> X_tot[:dim]
y_train <span style="color: #666666">=</span> y_tot[:dim]
<span style="color: #408080; font-style: italic"># Reshape the data for Keras specifications</span>
X_train <span style="color: #666666">=</span> X_train<span style="color: #666666">.</span>reshape((dim, <span style="color: #666666">1</span>))
y_train <span style="color: #666666">=</span> y_train<span style="color: #666666">.</span>reshape((dim, <span style="color: #666666">1</span>))
<span style="color: #408080; font-style: italic"># Create a recurrent neural network in Keras and produce a summary of the </span>
<span style="color: #408080; font-style: italic"># machine learning model</span>
<span style="color: #408080; font-style: italic"># Set the sequence length to 1 for regular data formatting </span>
model <span style="color: #666666">=</span> rnn(length_of_sequences <span style="color: #666666">=</span> <span style="color: #666666">1</span>)
model<span style="color: #666666">.</span>summary()
<span style="color: #408080; font-style: italic"># Start the timer. Want to time training+testing</span>
start <span style="color: #666666">=</span> timer()
<span style="color: #408080; font-style: italic"># Fit the model using the training data genenerated above using 150 training iterations and a 5%</span>
<span style="color: #408080; font-style: italic"># validation split. Setting verbose to True prints information about each training iteration.</span>
hist <span style="color: #666666">=</span> model<span style="color: #666666">.</span>fit(X_train, y_train, batch_size<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">None</span>, epochs<span style="color: #666666">=150</span>,
verbose<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>,validation_split<span style="color: #666666">=0.05</span>)
<span style="color: #408080; font-style: italic"># This section plots the training loss and the validation loss as a function of training iteration.</span>
<span style="color: #408080; font-style: italic"># This is not required for analyzing the couple cluster data but can help determine if the network is</span>
<span style="color: #408080; font-style: italic"># being overtrained.</span>
<span style="color: #008000; font-weight: bold">for</span> label <span style="color: #AA22FF; font-weight: bold">in</span> [<span style="color: #BA2121">&quot;loss&quot;</span>,<span style="color: #BA2121">&quot;val_loss&quot;</span>]:
plt<span style="color: #666666">.</span>plot(hist<span style="color: #666666">.</span>history[label],label<span style="color: #666666">=</span>label)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&quot;loss&quot;</span>)
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&quot;epoch&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;The final validation loss: </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">.</span>format(hist<span style="color: #666666">.</span>history[<span style="color: #BA2121">&quot;val_loss&quot;</span>][<span style="color: #666666">-1</span>]))
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic"># Use the trained neural network to predict the remaining data points</span>
X_pred <span style="color: #666666">=</span> X_tot[dim:]
X_pred <span style="color: #666666">=</span> X_pred<span style="color: #666666">.</span>reshape((<span style="color: #008000">len</span>(X_pred), <span style="color: #666666">1</span>))
y_model <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(X_pred)
y_pred <span style="color: #666666">=</span> np<span style="color: #666666">.</span>concatenate((y_tot[:dim], y_model<span style="color: #666666">.</span>flatten()))
<span style="color: #408080; font-style: italic"># Plot the known data set and the predicted data set. The red box represents the region that was used</span>
<span style="color: #408080; font-style: italic"># for the training data.</span>
fig, ax <span style="color: #666666">=</span> plt<span style="color: #666666">.</span>subplots()
ax<span style="color: #666666">.</span>plot(X_tot, y_tot, label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;true&quot;</span>, linewidth<span style="color: #666666">=3</span>)
ax<span style="color: #666666">.</span>plot(X_tot, y_pred, <span style="color: #BA2121">&#39;g-.&#39;</span>,label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;predicted&quot;</span>, linewidth<span style="color: #666666">=4</span>)
ax<span style="color: #666666">.</span>legend()
<span style="color: #408080; font-style: italic"># Created a red region to represent the points used in the training data.</span>
ax<span style="color: #666666">.</span>axvspan(X_tot[<span style="color: #666666">0</span>], X_tot[dim], alpha<span style="color: #666666">=0.25</span>, color<span style="color: #666666">=</span><span style="color: #BA2121">&#39;red&#39;</span>)
plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic"># Stop the timer and calculate the total time needed.</span>
end <span style="color: #666666">=</span> timer()
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Time: &#39;</span>, end<span style="color: #666666">-</span>start)
</pre></div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._week43-bs011.html">&laquo;</a></li>
<li><a href="._week43-bs000.html">1</a></li>
<li><a href="">...</a></li>
<li><a href="._week43-bs004.html">5</a></li>
<li><a href="._week43-bs005.html">6</a></li>
<li><a href="._week43-bs006.html">7</a></li>
<li><a href="._week43-bs007.html">8</a></li>
<li><a href="._week43-bs008.html">9</a></li>
<li><a href="._week43-bs009.html">10</a></li>
<li><a href="._week43-bs010.html">11</a></li>
<li><a href="._week43-bs011.html">12</a></li>
<li class="active"><a href="._week43-bs012.html">13</a></li>
<li><a href="._week43-bs013.html">14</a></li>
<li><a href="._week43-bs014.html">15</a></li>
<li><a href="._week43-bs015.html">16</a></li>
<li><a href="._week43-bs016.html">17</a></li>
<li><a href="._week43-bs017.html">18</a></li>
<li><a href="._week43-bs018.html">19</a></li>
<li><a href="._week43-bs019.html">20</a></li>
<li><a href="._week43-bs020.html">21</a></li>
<li><a href="._week43-bs021.html">22</a></li>
<li><a href="">...</a></li>
<li><a href="._week43-bs056.html">57</a></li>
<li><a href="._week43-bs013.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink