lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e0b90b192cd5ac1bbd2b79a13b6efbe7319f6cda
FYS-STK4155/doc/LectureNotes/_build/html/week45.html
T
Morten Hjorth-Jensen c0b89fb7cd update
2024-05-24 09:51:19 +02:00

2954 lines
236 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Week 45, Recurrent Neural Networks &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet">
<link href="_static/css/index.ff1ffe594081f20da1ef19478df9384b.css" rel="stylesheet">
<link rel="stylesheet"
href="_static/vendor/fontawesome/5.13.0/css/all.min.css">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-solid-900.woff2">
<link rel="preload" as="font" type="font/woff2" crossorigin
href="_static/vendor/fontawesome/5.13.0/webfonts/fa-brands-400.woff2">
<link rel="stylesheet" type="text/css" href="_static/pygments.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-book-theme.css?digest=c3fdc42140077d1ad13ad2f1588a4309" />
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
<link rel="preload" as="script" href="_static/js/index.be7d3bbb2ef33a8344ce.js">
<script data-url_root="./" id="documentation_options" src="_static/documentation_options.js"></script>
<script src="_static/jquery.js"></script>
<script src="_static/underscore.js"></script>
<script src="_static/doctools.js"></script>
<script src="_static/clipboard.min.js"></script>
<script src="_static/copybutton.js"></script>
<script>let toggleHintShow = 'Click to show';</script>
<script>let toggleHintHide = 'Click to hide';</script>
<script>let toggleOpenOnPrint = 'true';</script>
<script src="_static/togglebutton.js"></script>
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
<script src="_static/sphinx-book-theme.d59cb220de22ca1c485ebbdc042f0030.js"></script>
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"
const thebe_selector = ".thebe,.cell"
const thebe_selector_input = "pre"
const thebe_selector_output = ".output, .cell_output"
</script>
<script async="async" src="_static/sphinx-thebe.js"></script>
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Week 46: Decision Trees, Ensemble methods and Random Forests" href="week46.html" />
<link rel="prev" title="Week 44, Convolutional Neural Networks (CNN)" href="week44.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="None">
<!-- Google Analytics -->
</head>
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
<div class="container-fluid" id="banner"></div>
<div class="container-xl">
<div class="row">
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
<div class="navbar-brand-box">
<a class="navbar-brand text-wrap" href="index.html">
<!-- `logo` is deprecated in Sphinx 4.0, so remove this when we stop supporting 3 -->
<img src="_static/logo.png" class="logo" alt="logo">
<h1 class="site-logo" id="site-title">Applied Data Analysis and Machine Learning</h1>
</a>
</div><form class="bd-search d-flex align-items-center" action="search.html" method="get">
<i class="icon fas fa-search"></i>
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
</form><nav class="bd-links" id="bd-docs-nav" aria-label="Main">
<div class="bd-toc-item active">
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="intro.html">
Applied Data Analysis and Machine Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
About the course
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="schedule.html">
Teaching schedule with links to material
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="teachers.html">
Teachers and Grading
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="textbooks.html">
Textbooks
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Review of Statistics with Resampling Techniques and Linear Algebra
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="statistics.html">
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="linalg.html">
2. Linear Algebra, Handling of Arrays and more Python Features
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
From Regression to Support Vector Machines
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Decision Trees, Ensemble Methods and Boosting
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Dimensionality Reduction
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter8.html">
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="clustering.html">
12. Clustering and Unsupervised Learning
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Deep Learning Methods
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
14. Building a Feed Forward Neural Network
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter11.html">
15. Solving Differential Equations with Deep Learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter12.html">
16. Convolutional Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter13.html">
17. Recurrent neural networks: Overarching view
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Weekly material, notes and exercises
</span>
</p>
<ul class="current nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek34.html">
Exercises week 34
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week34.html">
Week 34: Introduction to the course, Logistics and Practicalities
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek35.html">
Exercises week 35
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week35.html">
Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek36.html">
Exercises week 36
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week36.html">
Week 36: Statistical interpretation of Linear Regression and Resampling techniques
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek37.html">
Exercises week 37
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week37.html">
Week 37: Statistical interpretations and Resampling Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek38.html">
Exercises week 38
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week38.html">
Week 38: Logistic Regression and Optimization
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek39.html">
Exercises week 39
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week39.html">
Week 39: Optimization and Gradient Methods
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week40.html">
Week 40: Gradient descent methods (continued) and start Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek41.html">
Exercises week 41
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week41.html">
Week 41 Neural networks and constructing a neural network code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek42.html">
Exercises week 42
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week42.html">
Week 42 Constructing a Neural Network code with introduction to Tensor flow
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek43.html">
Exercises weeks 43 and 44
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week43.html">
Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week44.html">
Week 44, Convolutional Neural Networks (CNN)
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
Week 45, Recurrent Neural Networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week46.html">
Week 46: Decision Trees, Ensemble methods and Random Forests
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week47.html">
Week 47: From Decision Trees to Ensemble Methods, Random Forests and Boosting Methods and Summary of Course
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek47.html">
Exercise week 47
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
Projects
</span>
</p>
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="project1.html">
Project 1 on Machine Learning, deadline October 9 (midnight), 2023
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project2.html">
Project 2 on Machine Learning, deadline November 17 (Midnight)
</a>
</li>
</ul>
</div>
</nav> <!-- To handle the deprecated key -->
<div class="navbar_extra_footer">
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
</div>
</div>
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
<div class="topbar container-xl fixed-top">
<div class="topbar-contents row">
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show"></div>
<div class="col pl-md-4 topbar-main">
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
<i class="fas fa-bars"></i>
<i class="fas fa-arrow-left"></i>
<i class="fas fa-arrow-up"></i>
</button>
<div class="dropdown-buttons-trigger">
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
class="fas fa-download"></i></button>
<div class="dropdown-buttons">
<!-- ipynb file if we had a myst markdown file -->
<!-- Download raw file -->
<a class="dropdown-buttons" href="_sources/week45.ipynb"><button type="button"
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
data-placement="left">.ipynb</button></a>
<!-- Download PDF via print -->
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
onclick="printPdf(this)" data-toggle="tooltip" data-placement="left">.pdf</button>
</div>
</div>
<!-- Source interaction buttons -->
<!-- Full screen (wrap in <a> to have style consistency -->
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
data-placement="bottom" onclick="toggleFullScreen()" aria-label="Fullscreen mode"
title="Fullscreen mode"><i
class="fas fa-expand"></i></button></a>
<!-- Launch buttons -->
</div>
<!-- Table of contents -->
<div class="d-none d-md-block col-md-2 bd-toc show noprint">
<div class="tocsection onthispage pt-5 pb-3">
<i class="fas fa-list"></i> Contents
</div>
<nav id="bd-toc-nav" aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#plan-for-week-45">
Plan for week 45
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#material-for-the-lab-sessions-additional-ways-to-present-classification-results-and-other-practicalities">
Material for the lab sessions, additional ways to present classification results and other practicalities
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#searching-for-optimal-regularization-parameters-lambda">
Searching for Optimal Regularization Parameters
<span class="math notranslate nohighlight">
\(\lambda\)
</span>
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#grid-search">
Grid Search
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#randomized-grid-search">
Randomized Grid Search
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#wisconsin-cancer-data">
Wisconsin Cancer Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-the-correlation-matrix">
Using the correlation matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#discussing-the-correlation-data">
Discussing the correlation data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-ways-of-presenting-a-classification-problem">
Other ways of presenting a classification problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#combinations-of-classification-results">
Combinations of classification results
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#positive-and-negative-prediction-values">
Positive and negative prediction values
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-quantities">
Other quantities
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#f-1-score">
<span class="math notranslate nohighlight">
\(F_1\)
</span>
score
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#roc-curve">
ROC curve
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cumulative-gain-curve">
Cumulative gain curve
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-measures-in-classification-studies-cancer-data-again">
Other measures in classification studies: Cancer Data again
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#material-for-lecture-thursday-november-9">
Material for Lecture Thursday November 9
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#recurrent-neural-networks-rnns-overarching-view">
Recurrent neural networks (RNNs): Overarching view
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-simple-example">
A simple example
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#rnns">
RNNs
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-layout">
Basic layout
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units">
We need to specify the initial activity state of all the hidden and output units
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#we-can-specify-inputs-in-several-ways">
We can specify inputs in several ways
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#we-can-specify-targets-in-several-ways">
We can specify targets in several ways
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#backpropagation-through-time">
Backpropagation through time
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-backward-pass-is-linear">
The backward pass is linear
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-problem-of-exploding-or-vanishing-gradients">
The problem of exploding or vanishing gradients
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#four-effective-ways-to-learn-an-rnn">
Four effective ways to learn an RNN
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#long-short-term-memory-lstm">
Long Short Term Memory (LSTM)
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#implementing-a-memory-cell-in-a-neural-network">
Implementing a memory cell in a neural network
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#an-extrapolation-example">
An extrapolation example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#formatting-the-data">
Formatting the Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#predicting-new-points-with-a-trained-recurrent-neural-network">
Predicting New Points With A Trained Recurrent Neural Network
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-things-to-try">
Other Things to Try
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-types-of-recurrent-neural-networks">
Other Types of Recurrent Neural Networks
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div id="main-content" class="row">
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
<!-- Table of contents that is only displayed when printing the page -->
<div id="jb-print-docs-body" class="onlyprint">
<h1>Week 45, Recurrent Neural Networks</h1>
<!-- Table of contents -->
<div id="print-main-content">
<div id="jb-print-toc">
<div>
<h2> Contents </h2>
</div>
<nav aria-label="Page">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#plan-for-week-45">
Plan for week 45
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#material-for-the-lab-sessions-additional-ways-to-present-classification-results-and-other-practicalities">
Material for the lab sessions, additional ways to present classification results and other practicalities
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#searching-for-optimal-regularization-parameters-lambda">
Searching for Optimal Regularization Parameters
<span class="math notranslate nohighlight">
\(\lambda\)
</span>
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#grid-search">
Grid Search
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#randomized-grid-search">
Randomized Grid Search
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#wisconsin-cancer-data">
Wisconsin Cancer Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#using-the-correlation-matrix">
Using the correlation matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#discussing-the-correlation-data">
Discussing the correlation data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-ways-of-presenting-a-classification-problem">
Other ways of presenting a classification problem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#combinations-of-classification-results">
Combinations of classification results
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#positive-and-negative-prediction-values">
Positive and negative prediction values
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-quantities">
Other quantities
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#f-1-score">
<span class="math notranslate nohighlight">
\(F_1\)
</span>
score
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#roc-curve">
ROC curve
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#cumulative-gain-curve">
Cumulative gain curve
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-measures-in-classification-studies-cancer-data-again">
Other measures in classification studies: Cancer Data again
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#material-for-lecture-thursday-november-9">
Material for Lecture Thursday November 9
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#recurrent-neural-networks-rnns-overarching-view">
Recurrent neural networks (RNNs): Overarching view
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#a-simple-example">
A simple example
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#rnns">
RNNs
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-layout">
Basic layout
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units">
We need to specify the initial activity state of all the hidden and output units
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#we-can-specify-inputs-in-several-ways">
We can specify inputs in several ways
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#we-can-specify-targets-in-several-ways">
We can specify targets in several ways
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#backpropagation-through-time">
Backpropagation through time
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-backward-pass-is-linear">
The backward pass is linear
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-problem-of-exploding-or-vanishing-gradients">
The problem of exploding or vanishing gradients
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#four-effective-ways-to-learn-an-rnn">
Four effective ways to learn an RNN
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#long-short-term-memory-lstm">
Long Short Term Memory (LSTM)
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#implementing-a-memory-cell-in-a-neural-network">
Implementing a memory cell in a neural network
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#an-extrapolation-example">
An extrapolation example
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#formatting-the-data">
Formatting the Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#predicting-new-points-with-a-trained-recurrent-neural-network">
Predicting New Points With A Trained Recurrent Neural Network
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-things-to-try">
Other Things to Try
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-types-of-recurrent-neural-networks">
Other Types of Recurrent Neural Networks
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div>
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html week45.do.txt --no_mako -->
<!-- dom:TITLE: Week 45, Recurrent Neural Networks --><div class="tex2jax_ignore mathjax_ignore section" id="week-45-recurrent-neural-networks">
<h1>Week 45, Recurrent Neural Networks<a class="headerlink" href="#week-45-recurrent-neural-networks" title="Permalink to this headline"></a></h1>
<p><strong>Morten Hjorth-Jensen</strong>, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</p>
<p>Date: <strong>November 6-10</strong></p>
<div class="section" id="plan-for-week-45">
<h2>Plan for week 45<a class="headerlink" href="#plan-for-week-45" title="Permalink to this headline"></a></h2>
<p><strong>Material for the active learning sessions on Tuesday and Wednesday.</strong></p>
<ul class="simple">
<li><p>Discussion of project 2</p></li>
<li><p><a class="reference external" href="https://youtu.be/Ia6wwDLxqtM">Video of lab session from week 43</a></p></li>
<li><p><a class="reference external" href="https://youtu.be/EajWMW__k0I">Video of lab session from week 44</a></p></li>
<li><p><a class="reference external" href="https://youtu.be/tgkj0KAEtZo">Video of lab session from week 45</a></p></li>
<li><p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/Exercisesweek44.pdf">See also whiteboard notes from lab session week 44</a></p></li>
</ul>
<p><strong>Material for the lecture on Thursday November 9, 2023.</strong></p>
<ul class="simple">
<li><p>Short repetition on Convolutional Neural Networks</p></li>
<li><p>Recurrent Neural Networks (RNNs)</p></li>
<li><p>Readings and Videos:</p>
<ul>
<li><p>These lecture notes</p></li>
<li><p><a class="reference external" href="https://youtu.be/z0x-vgyAZUk">Video of lecture</a></p></li>
<li><p><a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesNov9.pdf">Whiteboard notes</a></p></li>
<li><p>For a more in depth discussion on neural networks we recommend Goodfellow et al chapter 10. See also chapter 11 and 12 on practicalities and applications</p></li>
<li><p>Reading suggestions for implementation of RNNs: <a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/TensorflowML.pdf">Aurelien Geron’s chapter 14</a>.</p></li>
<li><p><a class="reference external" href="https://www.youtube.com/watch?v=SEnXr6v2ifU&amp;ab_channel=AlexanderAmini">Video on Recurrent Neural Networks from MIT</a></p></li>
<li><p><a class="reference external" href="https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi">Video on Deep Learning</a></p></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="material-for-the-lab-sessions-additional-ways-to-present-classification-results-and-other-practicalities">
<h2>Material for the lab sessions, additional ways to present classification results and other practicalities<a class="headerlink" href="#material-for-the-lab-sessions-additional-ways-to-present-classification-results-and-other-practicalities" title="Permalink to this headline"></a></h2>
</div>
<div class="section" id="searching-for-optimal-regularization-parameters-lambda">
<h2>Searching for Optimal Regularization Parameters <span class="math notranslate nohighlight">\(\lambda\)</span><a class="headerlink" href="#searching-for-optimal-regularization-parameters-lambda" title="Permalink to this headline"></a></h2>
<p>In project 1, when using Ridge and Lasso regression, we end up
searching for the optimal parameter <span class="math notranslate nohighlight">\(\lambda\)</span> which minimizes our
selected scores (MSE or <span class="math notranslate nohighlight">\(R2\)</span> values for example). The brute force
approach, as discussed in the code here for Ridge regression, consists
in evaluating the MSE as function of different <span class="math notranslate nohighlight">\(\lambda\)</span> values.
Based on these calculations, one tries then to determine the value of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>
which results in optimal scores (for example the smallest MSE or an <span class="math notranslate nohighlight">\(R2=1\)</span>).</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn</span> <span class="kn">import</span> <span class="n">linear_model</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># A seed just to ensure that the random numbers are the same for every run.</span>
<span class="c1"># Useful for eventual debugging.</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">Maxpolydegree</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="n">Maxpolydegree</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">Maxpolydegree</span><span class="p">):</span> <span class="c1">#No intercept column</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">degree</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">degree</span><span class="p">)</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">500</span>
<span class="n">MSERidgePredict</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">)</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">nlambdas</span><span class="p">):</span>
<span class="n">lmb</span> <span class="o">=</span> <span class="n">lambdas</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">RegRidge</span> <span class="o">=</span> <span class="n">linear_model</span><span class="o">.</span><span class="n">Ridge</span><span class="p">(</span><span class="n">lmb</span><span class="p">)</span>
<span class="n">RegRidge</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span><span class="n">y_train</span><span class="p">)</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">RegRidge</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">MSERidgePredict</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span>
<span class="c1"># Now plot the results</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log10</span><span class="p">(</span><span class="n">lambdas</span><span class="p">),</span> <span class="n">MSERidgePredict</span><span class="p">,</span> <span class="s1">&#39;g--&#39;</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s1">&#39;MSE SL Ridge Test&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;log10(lambda)&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;MSE&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week45_5_0.png" src="_images/week45_5_0.png" />
</div>
</div>
<p>Here we have performed a rather data greedy calculation as function of the regularization parameter <span class="math notranslate nohighlight">\(\lambda\)</span>. There is no resampling here. The latter can easily be added by employing the function <strong>RidgeCV</strong> instead of just calling the <strong>Ridge</strong> function. For <strong>RidgeCV</strong> we need to pass the array of <span class="math notranslate nohighlight">\(\lambda\)</span> values.
By inspecting the figure we can in turn determine which is the optimal regularization parameter.
This becomes however less functional in the long run.</p>
</div>
<div class="section" id="grid-search">
<h2>Grid Search<a class="headerlink" href="#grid-search" title="Permalink to this headline"></a></h2>
<p>An alternative is to use the so-called grid search functionality
included with the library <strong>Scikit-Learn</strong>, as demonstrated for the same
example here.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">Ridge</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">GridSearchCV</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># A seed just to ensure that the random numbers are the same for every run.</span>
<span class="c1"># Useful for eventual debugging.</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">Maxpolydegree</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="n">Maxpolydegree</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">Maxpolydegree</span><span class="p">):</span> <span class="c1">#No intercept column</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">degree</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">degree</span><span class="p">)</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># Decide which values of lambda to use</span>
<span class="n">nlambdas</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">lambdas</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">logspace</span><span class="p">(</span><span class="o">-</span><span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="n">nlambdas</span><span class="p">)</span>
<span class="c1"># create and fit a ridge regression model, testing each alpha</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Ridge</span><span class="p">()</span>
<span class="n">gridsearch</span> <span class="o">=</span> <span class="n">GridSearchCV</span><span class="p">(</span><span class="n">estimator</span><span class="o">=</span><span class="n">model</span><span class="p">,</span> <span class="n">param_grid</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span><span class="n">alpha</span><span class="o">=</span><span class="n">lambdas</span><span class="p">))</span>
<span class="n">gridsearch</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">gridsearch</span><span class="p">)</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">gridsearch</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># summarize the results of the grid search</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Best estimated lambda-value: </span><span class="si">{</span><span class="n">gridsearch</span><span class="o">.</span><span class="n">best_estimator_</span><span class="o">.</span><span class="n">alpha</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;MSE score: </span><span class="si">{</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;R2 score: </span><span class="si">{</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>GridSearchCV(estimator=Ridge(),
param_grid={&#39;alpha&#39;: array([1.00000000e-04, 4.64158883e-04, 2.15443469e-03, 1.00000000e-02,
4.64158883e-02, 2.15443469e-01, 1.00000000e+00, 4.64158883e+00,
2.15443469e+01, 1.00000000e+02])})
Best estimated lambda-value: 100.0
MSE score: 1.0892144853354966
R2 score: -0.0038332550504751595
</pre></div>
</div>
</div>
</div>
<p>By default the grid search function includes cross validation with
five folds. The <a class="reference external" href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html#sklearn.model_selection.GridSearchCV">Scikit-Learn
documentation</a>
contains more information on how to set the different parameters.</p>
<p>If we take out the random noise, running the above codes results in <span class="math notranslate nohighlight">\(\lambda=0\)</span> yielding the best fit.</p>
</div>
<div class="section" id="randomized-grid-search">
<h2>Randomized Grid Search<a class="headerlink" href="#randomized-grid-search" title="Permalink to this headline"></a></h2>
<p>An alternative to the above manual grid set up, is to use a random
search where the parameters are tuned from a random distribution
(uniform below) for a fixed number of iterations. A model is
constructed and evaluated for each combination of chosen parameters.
We repeat the previous example but now with a random search. Note
that values of <span class="math notranslate nohighlight">\(\lambda\)</span> are now limited to be within <span class="math notranslate nohighlight">\(x\in
[0,1]\)</span>. This domain may not be the most relevant one for the specific
case under study.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">Ridge</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">GridSearchCV</span>
<span class="kn">from</span> <span class="nn">scipy.stats</span> <span class="kn">import</span> <span class="n">uniform</span> <span class="k">as</span> <span class="n">randuniform</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">RandomizedSearchCV</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="c1"># A seed just to ensure that the random numbers are the same for every run.</span>
<span class="c1"># Useful for eventual debugging.</span>
<span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">2021</span><span class="p">)</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">)</span>
<span class="n">Maxpolydegree</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n</span><span class="p">,</span><span class="n">Maxpolydegree</span><span class="o">-</span><span class="mi">1</span><span class="p">))</span>
<span class="k">for</span> <span class="n">degree</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">Maxpolydegree</span><span class="p">):</span> <span class="c1">#No intercept column</span>
<span class="n">X</span><span class="p">[:,</span><span class="n">degree</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="p">(</span><span class="n">degree</span><span class="p">)</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="n">param_grid</span> <span class="o">=</span> <span class="p">{</span><span class="s1">&#39;alpha&#39;</span><span class="p">:</span> <span class="n">randuniform</span><span class="p">()}</span>
<span class="c1"># create and fit a ridge regression model, testing each alpha</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Ridge</span><span class="p">()</span>
<span class="n">gridsearch</span> <span class="o">=</span> <span class="n">RandomizedSearchCV</span><span class="p">(</span><span class="n">estimator</span><span class="o">=</span><span class="n">model</span><span class="p">,</span> <span class="n">param_distributions</span><span class="o">=</span><span class="n">param_grid</span><span class="p">,</span> <span class="n">n_iter</span><span class="o">=</span><span class="mi">100</span><span class="p">)</span>
<span class="n">gridsearch</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">gridsearch</span><span class="p">)</span>
<span class="n">ypredictRidge</span> <span class="o">=</span> <span class="n">gridsearch</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="c1"># summarize the results of the grid search</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;Best estimated lambda-value: </span><span class="si">{</span><span class="n">gridsearch</span><span class="o">.</span><span class="n">best_estimator_</span><span class="o">.</span><span class="n">alpha</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;MSE score: </span><span class="si">{</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;R2 score: </span><span class="si">{</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredictRidge</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>RandomizedSearchCV(estimator=Ridge(), n_iter=100,
param_distributions={&#39;alpha&#39;: &lt;scipy.stats._distn_infrastructure.rv_frozen object at 0x1646da640&gt;})
Best estimated lambda-value: 0.9849967686928113
MSE score: 1.0853136633465326
R2 score: -0.0002382102844775691
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="wisconsin-cancer-data">
<h2>Wisconsin Cancer Data<a class="headerlink" href="#wisconsin-cancer-data" title="Permalink to this headline"></a></h2>
<p>We show here how we can use a simple regression case on the breast
cancer data using Logistic regression as our algorithm for
classification.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_breast_cancer</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="c1"># Load the data</span>
<span class="n">cancer</span> <span class="o">=</span> <span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">,</span><span class="n">cancer</span><span class="o">.</span><span class="n">target</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_test</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="c1"># Logistic Regression</span>
<span class="n">logreg</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">solver</span><span class="o">=</span><span class="s1">&#39;lbfgs&#39;</span><span class="p">)</span>
<span class="n">logreg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test set accuracy with Logistic Regression: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">logreg</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
(143, 30)
Test set accuracy with Logistic Regression: 0.94
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="using-the-correlation-matrix">
<h2>Using the correlation matrix<a class="headerlink" href="#using-the-correlation-matrix" title="Permalink to this headline"></a></h2>
<p>In addition to the above scores, we could also study the covariance (and the correlation matrix).
We use <strong>Pandas</strong> to compute the correlation matrix.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_breast_cancer</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="n">cancer</span> <span class="o">=</span> <span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="c1"># Making a data frame</span>
<span class="n">cancerpd</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">cancer</span><span class="o">.</span><span class="n">feature_names</span><span class="p">)</span>
<span class="n">fig</span><span class="p">,</span> <span class="n">axes</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">(</span><span class="mi">15</span><span class="p">,</span><span class="mi">2</span><span class="p">,</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">10</span><span class="p">,</span><span class="mi">20</span><span class="p">))</span>
<span class="n">malignant</span> <span class="o">=</span> <span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="n">cancer</span><span class="o">.</span><span class="n">target</span> <span class="o">==</span> <span class="mi">0</span><span class="p">]</span>
<span class="n">benign</span> <span class="o">=</span> <span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="n">cancer</span><span class="o">.</span><span class="n">target</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">ax</span> <span class="o">=</span> <span class="n">axes</span><span class="o">.</span><span class="n">ravel</span><span class="p">()</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">30</span><span class="p">):</span>
<span class="n">_</span><span class="p">,</span> <span class="n">bins</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">histogram</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">[:,</span><span class="n">i</span><span class="p">],</span> <span class="n">bins</span> <span class="o">=</span><span class="mi">50</span><span class="p">)</span>
<span class="n">ax</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">malignant</span><span class="p">[:,</span><span class="n">i</span><span class="p">],</span> <span class="n">bins</span> <span class="o">=</span> <span class="n">bins</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mf">0.5</span><span class="p">)</span>
<span class="n">ax</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">benign</span><span class="p">[:,</span><span class="n">i</span><span class="p">],</span> <span class="n">bins</span> <span class="o">=</span> <span class="n">bins</span><span class="p">,</span> <span class="n">alpha</span> <span class="o">=</span> <span class="mf">0.5</span><span class="p">)</span>
<span class="n">ax</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">set_title</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">feature_names</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="n">ax</span><span class="p">[</span><span class="n">i</span><span class="p">]</span><span class="o">.</span><span class="n">set_yticks</span><span class="p">(())</span>
<span class="n">ax</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s2">&quot;Feature magnitude&quot;</span><span class="p">)</span>
<span class="n">ax</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="s2">&quot;Frequency&quot;</span><span class="p">)</span>
<span class="n">ax</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">legend</span><span class="p">([</span><span class="s2">&quot;Malignant&quot;</span><span class="p">,</span> <span class="s2">&quot;Benign&quot;</span><span class="p">],</span> <span class="n">loc</span> <span class="o">=</span><span class="s2">&quot;best&quot;</span><span class="p">)</span>
<span class="n">fig</span><span class="o">.</span><span class="n">tight_layout</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="kn">import</span> <span class="nn">seaborn</span> <span class="k">as</span> <span class="nn">sns</span>
<span class="n">correlation_matrix</span> <span class="o">=</span> <span class="n">cancerpd</span><span class="o">.</span><span class="n">corr</span><span class="p">()</span><span class="o">.</span><span class="n">round</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="c1"># use the heatmap function from seaborn to plot the correlation matrix</span>
<span class="c1"># annot = True to print the values inside the square</span>
<span class="n">plt</span><span class="o">.</span><span class="n">figure</span><span class="p">(</span><span class="n">figsize</span><span class="o">=</span><span class="p">(</span><span class="mi">15</span><span class="p">,</span><span class="mi">8</span><span class="p">))</span>
<span class="n">sns</span><span class="o">.</span><span class="n">heatmap</span><span class="p">(</span><span class="n">data</span><span class="o">=</span><span class="n">correlation_matrix</span><span class="p">,</span> <span class="n">annot</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/week45_15_0.png" src="_images/week45_15_0.png" />
<img alt="_images/week45_15_1.png" src="_images/week45_15_1.png" />
</div>
</div>
</div>
<div class="section" id="discussing-the-correlation-data">
<h2>Discussing the correlation data<a class="headerlink" href="#discussing-the-correlation-data" title="Permalink to this headline"></a></h2>
<p>In the above example we note two things. In the first plot we display
the overlap of benign and malignant tumors as functions of the various
features in the Wisconsing breast cancer data set. We see that for
some of the features we can distinguish clearly the benign and
malignant cases while for other features we cannot. This can point to
us which features may be of greater interest when we wish to classify
a benign or not benign tumour.</p>
<p>In the second figure we have computed the so-called correlation
matrix, which in our case with thirty features becomes a <span class="math notranslate nohighlight">\(30\times 30\)</span>
matrix.</p>
<p>We constructed this matrix using <strong>pandas</strong> via the statements</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">cancerpd</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">columns</span><span class="o">=</span><span class="n">cancer</span><span class="o">.</span><span class="n">feature_names</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>and then</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">correlation_matrix</span> <span class="o">=</span> <span class="n">cancerpd</span><span class="o">.</span><span class="n">corr</span><span class="p">()</span><span class="o">.</span><span class="n">round</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
<p>Diagonalizing this matrix we can in turn say something about which
features are of relevance and which are not. This leads us to
the classical Principal Component Analysis (PCA) theorem with
applications. This will be discussed later this semester (<a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/pub/week43/html/week43-bs.html">week 43</a>).</p>
</div>
<div class="section" id="other-ways-of-presenting-a-classification-problem">
<h2>Other ways of presenting a classification problem<a class="headerlink" href="#other-ways-of-presenting-a-classification-problem" title="Permalink to this headline"></a></h2>
<p>For a binary classifcation matrix, the so-called <strong>confusion matrix</strong>, is often used. It can also be extended to more catgeories/classes as well.
The following quantities are then used</p>
<ol class="simple">
<li><p>positive condition number <span class="math notranslate nohighlight">\(P\)</span>, which represents the number of real positive cases in the data (output one/true etc)</p></li>
<li><p>The condition negative number <span class="math notranslate nohighlight">\(N\)</span> which is the number of negative cases (ouput zero/false etc)</p></li>
<li><p>The true positive number <span class="math notranslate nohighlight">\(TP\)</span> which represents whether a positive test result has been correctly classified (the application of our trained model on a test data set)</p></li>
<li><p>The true negative <span class="math notranslate nohighlight">\(TN\)</span> number which represents whether a negative test has been correctly classified</p></li>
<li><p>The false positive <span class="math notranslate nohighlight">\(FP\)</span> number, a so-called type I error which tells us about the fraction of positive test result which are wrongly classified</p></li>
<li><p>A false negative <span class="math notranslate nohighlight">\(FN\)</span> number, a so-called type II error which, should be pretty obvious, indicates if a negative test has been wrongly classified.</p></li>
</ol>
<p>It is is easy to think in terms of illness. You could think of the above as</p>
<ol class="simple">
<li><p>True positive: Sick people correctly identified as sick</p></li>
<li><p>False positive: Healthy people incorrectly identified as sick</p></li>
<li><p>True negative: Healthy people correctly identified as healthy</p></li>
<li><p>False negative: Sick people incorrectly identified as healthy</p></li>
</ol>
</div>
<div class="section" id="combinations-of-classification-results">
<h2>Combinations of classification results<a class="headerlink" href="#combinations-of-classification-results" title="Permalink to this headline"></a></h2>
<p>It is common in the literature to define various combinations the above numbers. The most commonly used are</p>
<p><strong>Sensitivity, recall, hit rate, or true positive rate <span class="math notranslate nohighlight">\(TPR\)</span>. It is the probability of a positive test result, conditioned on the individual truly being positive.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {TPR} ={\frac {\mathrm {TP} }{\mathrm {P} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }}=1-\mathrm {FNR} }
\]</div>
<p>The <span class="math notranslate nohighlight">\(TPR\)</span> defines how many correct positive results occur among all positive samples available during the test</p>
<p><strong>Miss rate or false negative rate <span class="math notranslate nohighlight">\(FNR\)</span>.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {FNR} ={\frac {\mathrm {FN} }{\mathrm {P} }}={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TP} }} }
\]</div>
<p><strong>Specificity, selectivity or true negative rate <span class="math notranslate nohighlight">\(TNR\)</span>. It is the probability of a negative test result, conditioned on the individual truly being negative.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {TNR} ={\frac {\mathrm {TN} }{\mathrm {N} }}={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FP} }}=1-\mathrm {FPR} }
\]</div>
<p>with the fall-out false positive rate</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {FPR} ={\frac {\mathrm {FP} }{\mathrm {N} }}={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TN} }}=1-\mathrm {TNR} }
\]</div>
<p>The <span class="math notranslate nohighlight">\(FPR\)</span> defines how many incorrect positive results occur among
all negative samples available during the test.</p>
</div>
<div class="section" id="positive-and-negative-prediction-values">
<h2>Positive and negative prediction values<a class="headerlink" href="#positive-and-negative-prediction-values" title="Permalink to this headline"></a></h2>
<p>The positive and negative predictive values
are the proportions of positive and negative results in statistics and
diagnostic tests that are true positive and true negative results,
respectively.[1] The PPV and NPV describe the performance of a
diagnostic test or other statistical measure. A high result can be
interpreted as indicating the accuracy of such a statistic.</p>
<p><strong>Precision or positive predictive value <span class="math notranslate nohighlight">\(PPV\)</span>.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {PPV} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FP} }}=1-\mathrm {FDR} }
\]</div>
<p><strong>Negative predictive value <span class="math notranslate nohighlight">\(NPV\)</span>.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {NPV} ={\frac {\mathrm {TN} }{\mathrm {TN} +\mathrm {FN} }}=1-\mathrm {FOR} }
\]</div>
</div>
<div class="section" id="other-quantities">
<h2>Other quantities<a class="headerlink" href="#other-quantities" title="Permalink to this headline"></a></h2>
<p><strong>False discovery rate <span class="math notranslate nohighlight">\(FDR\)</span>.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {FDR} ={\frac {\mathrm {FP} }{\mathrm {FP} +\mathrm {TP} }}=1-\mathrm {PPV} }
\]</div>
<p><strong>False omission rate <span class="math notranslate nohighlight">\(FOR\)</span>.</strong></p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {FOR} ={\frac {\mathrm {FN} }{\mathrm {FN} +\mathrm {TN} }}=1-\mathrm {NPV} }
\]</div>
</div>
<div class="section" id="f-1-score">
<h2><span class="math notranslate nohighlight">\(F_1\)</span> score<a class="headerlink" href="#f-1-score" title="Permalink to this headline"></a></h2>
<p>In statistical analysis of binary classification, the F-score or
F-measure is a measure of a test’s accuracy. It is calculated from the
precision and recall of the test, where the precision is the number of
true positive results divided by the number of all positive results,
including those not identified correctly, and the recall is the number
of true positive results divided by the number of all samples that
should have been identified as positive. Precision is also known as
positive predictive value, and recall is also known as sensitivity in
diagnostic binary classification.</p>
<p>The F1 score is the harmonic mean of the precision and recall. It thus
symmetrically represents both precision and recall in one metric. The
highest possible value of an F-score is 1.0, indicating perfect
precision and recall, and the lowest possible value is 0, if either
precision or recall are zero.</p>
<p>It is defined as</p>
<div class="math notranslate nohighlight">
\[
{\displaystyle \mathrm {F} _{1}=2\times {\frac {\mathrm {PPV} \times \mathrm {TPR} }{\mathrm {PPV} +\mathrm {TPR} }}={\frac {2\mathrm {TP} }{2\mathrm {TP} +\mathrm {FP} +\mathrm {FN} }}}
\]</div>
</div>
<div class="section" id="roc-curve">
<h2>ROC curve<a class="headerlink" href="#roc-curve" title="Permalink to this headline"></a></h2>
<p>A receiver operating characteristic curve, or ROC curve, is a
graphical plot that illustrates the performance of a binary classifier
model at varying threshold values.</p>
<p>The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold setting.</p>
<p>To draw a ROC curve, only the true positive rate (TPR) and false
positive rate (FPR) are needed (as functions of some classifier
parameter). The TPR defines how many correct positive results occur
among all positive samples available during the test. FPR, on the
other hand, defines how many incorrect positive results occur among
all negative samples available during the test.</p>
<p>See <a class="reference external" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">https://en.wikipedia.org/wiki/Receiver_operating_characteristic</a> for more discussions.</p>
</div>
<div class="section" id="cumulative-gain-curve">
<h2>Cumulative gain curve<a class="headerlink" href="#cumulative-gain-curve" title="Permalink to this headline"></a></h2>
<p>The cumulative gain curve is a performance evaluation used typically for binary classification problems.
It plots the <span class="math notranslate nohighlight">\(TPR\)</span> True Positive Rate or Sensitivity (which represents the
fraction of examples correctly classified
against Predictive Positive Rate, which represents
the fraction of positively predicted examples.</p>
<p>The examples below show the confusion matrix, the ROC curve and the cumulative gain for the Wisconsin cancer data.</p>
</div>
<div class="section" id="other-measures-in-classification-studies-cancer-data-again">
<h2>Other measures in classification studies: Cancer Data again<a class="headerlink" href="#other-measures-in-classification-studies-cancer-data-again" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="kn">from</span> <span class="nn">sklearn.datasets</span> <span class="kn">import</span> <span class="n">load_breast_cancer</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LogisticRegression</span>
<span class="c1"># Load the data</span>
<span class="n">cancer</span> <span class="o">=</span> <span class="n">load_breast_cancer</span><span class="p">()</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">cancer</span><span class="o">.</span><span class="n">data</span><span class="p">,</span><span class="n">cancer</span><span class="o">.</span><span class="n">target</span><span class="p">,</span><span class="n">random_state</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">X_test</span><span class="o">.</span><span class="n">shape</span><span class="p">)</span>
<span class="c1"># Logistic Regression</span>
<span class="n">logreg</span> <span class="o">=</span> <span class="n">LogisticRegression</span><span class="p">(</span><span class="n">solver</span><span class="o">=</span><span class="s1">&#39;lbfgs&#39;</span><span class="p">)</span>
<span class="n">logreg</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">LabelEncoder</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">cross_validate</span>
<span class="c1">#Cross validation</span>
<span class="n">accuracy</span> <span class="o">=</span> <span class="n">cross_validate</span><span class="p">(</span><span class="n">logreg</span><span class="p">,</span><span class="n">X_test</span><span class="p">,</span><span class="n">y_test</span><span class="p">,</span><span class="n">cv</span><span class="o">=</span><span class="mi">10</span><span class="p">)[</span><span class="s1">&#39;test_score&#39;</span><span class="p">]</span>
<span class="nb">print</span><span class="p">(</span><span class="n">accuracy</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test set accuracy with Logistic Regression: </span><span class="si">{:.2f}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">logreg</span><span class="o">.</span><span class="n">score</span><span class="p">(</span><span class="n">X_test</span><span class="p">,</span><span class="n">y_test</span><span class="p">)))</span>
<span class="kn">import</span> <span class="nn">scikitplot</span> <span class="k">as</span> <span class="nn">skplt</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">logreg</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_confusion_matrix</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="n">normalize</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">y_probas</span> <span class="o">=</span> <span class="n">logreg</span><span class="o">.</span><span class="n">predict_proba</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_roc</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_probas</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="n">skplt</span><span class="o">.</span><span class="n">metrics</span><span class="o">.</span><span class="n">plot_cumulative_gain</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_probas</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(426, 30)
(143, 30)
[1. 0.86666667 1. 0.92857143 1. 0.85714286
1. 0.92857143 0.92857143 1. ]
Test set accuracy with Logistic Regression: 0.94
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.
Increase the number of iterations (max_iter) or scale the data as shown in:
https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
n_iter_i = _check_optimize_result(
</pre></div>
</div>
<img alt="_images/week45_44_2.png" src="_images/week45_44_2.png" />
<img alt="_images/week45_44_3.png" src="_images/week45_44_3.png" />
<img alt="_images/week45_44_4.png" src="_images/week45_44_4.png" />
</div>
</div>
</div>
<div class="section" id="material-for-lecture-thursday-november-9">
<h2>Material for Lecture Thursday November 9<a class="headerlink" href="#material-for-lecture-thursday-november-9" title="Permalink to this headline"></a></h2>
</div>
<div class="section" id="recurrent-neural-networks-rnns-overarching-view">
<h2>Recurrent neural networks (RNNs): Overarching view<a class="headerlink" href="#recurrent-neural-networks-rnns-overarching-view" title="Permalink to this headline"></a></h2>
<p>Till now our focus has been, including convolutional neural networks
as well, on feedforward neural networks. The output or the activations
flow only in one direction, from the input layer to the output layer.</p>
<p>A recurrent neural network (RNN) looks very much like a feedforward
neural network, except that it also has connections pointing
backward.</p>
<p>RNNs are used to analyze time series data such as stock prices, and
tell you when to buy or sell. In autonomous driving systems, they can
anticipate car trajectories and help avoid accidents. More generally,
they can work on sequences of arbitrary lengths, rather than on
fixed-sized inputs like all the nets we have discussed so far. For
example, they can take sentences, documents, or audio samples as
input, making them extremely useful for natural language processing
systems such as automatic translation and speech-to-text.</p>
</div>
<div class="section" id="a-simple-example">
<h2>A simple example<a class="headerlink" href="#a-simple-example" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Start importing packages</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras</span> <span class="kn">import</span> <span class="n">datasets</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.layers</span> <span class="kn">import</span> <span class="n">Input</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.models</span> <span class="kn">import</span> <span class="n">Model</span><span class="p">,</span> <span class="n">Sequential</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span><span class="p">,</span> <span class="n">SimpleRNN</span><span class="p">,</span> <span class="n">LSTM</span><span class="p">,</span> <span class="n">GRU</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras</span> <span class="kn">import</span> <span class="n">optimizers</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras</span> <span class="kn">import</span> <span class="n">regularizers</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.utils</span> <span class="kn">import</span> <span class="n">to_categorical</span>
<span class="c1"># convert into dataset matrix</span>
<span class="k">def</span> <span class="nf">convertToMatrix</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">step</span><span class="p">):</span>
<span class="n">X</span><span class="p">,</span> <span class="n">Y</span> <span class="o">=</span><span class="p">[],</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="o">-</span><span class="n">step</span><span class="p">):</span>
<span class="n">d</span><span class="o">=</span><span class="n">i</span><span class="o">+</span><span class="n">step</span>
<span class="n">X</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">d</span><span class="p">,])</span>
<span class="n">Y</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">d</span><span class="p">,])</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">X</span><span class="p">),</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">Y</span><span class="p">)</span>
<span class="n">step</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">Tp</span> <span class="o">=</span> <span class="mi">800</span>
<span class="n">t</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="n">N</span><span class="p">)</span>
<span class="n">x</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">sin</span><span class="p">(</span><span class="mf">0.02</span><span class="o">*</span><span class="n">t</span><span class="p">)</span><span class="o">+</span><span class="mi">2</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">N</span><span class="p">)</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">df</span><span class="o">.</span><span class="n">head</span><span class="p">()</span>
<span class="n">values</span><span class="o">=</span><span class="n">df</span><span class="o">.</span><span class="n">values</span>
<span class="n">train</span><span class="p">,</span><span class="n">test</span> <span class="o">=</span> <span class="n">values</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="n">Tp</span><span class="p">,:],</span> <span class="n">values</span><span class="p">[</span><span class="n">Tp</span><span class="p">:</span><span class="n">N</span><span class="p">,:]</span>
<span class="c1"># add step elements into train and test</span>
<span class="n">test</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">test</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">repeat</span><span class="p">(</span><span class="n">test</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">,],</span><span class="n">step</span><span class="p">))</span>
<span class="n">train</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">train</span><span class="p">,</span><span class="n">np</span><span class="o">.</span><span class="n">repeat</span><span class="p">(</span><span class="n">train</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">,],</span><span class="n">step</span><span class="p">))</span>
<span class="n">trainX</span><span class="p">,</span><span class="n">trainY</span> <span class="o">=</span><span class="n">convertToMatrix</span><span class="p">(</span><span class="n">train</span><span class="p">,</span><span class="n">step</span><span class="p">)</span>
<span class="n">testX</span><span class="p">,</span><span class="n">testY</span> <span class="o">=</span><span class="n">convertToMatrix</span><span class="p">(</span><span class="n">test</span><span class="p">,</span><span class="n">step</span><span class="p">)</span>
<span class="n">trainX</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="p">(</span><span class="n">trainX</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="mi">1</span><span class="p">,</span> <span class="n">trainX</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">testX</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">testX</span><span class="p">,</span> <span class="p">(</span><span class="n">testX</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="mi">1</span><span class="p">,</span> <span class="n">testX</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Sequential</span><span class="p">()</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">SimpleRNN</span><span class="p">(</span><span class="n">units</span><span class="o">=</span><span class="mi">32</span><span class="p">,</span> <span class="n">input_shape</span><span class="o">=</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="n">step</span><span class="p">),</span> <span class="n">activation</span><span class="o">=</span><span class="s2">&quot;relu&quot;</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">8</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s2">&quot;relu&quot;</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">Dense</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s1">&#39;mean_squared_error&#39;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s1">&#39;rmsprop&#39;</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
<span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span><span class="n">trainY</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="n">trainPredict</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
<span class="n">testPredict</span><span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
<span class="n">predicted</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">((</span><span class="n">trainPredict</span><span class="p">,</span><span class="n">testPredict</span><span class="p">),</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">trainScore</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">evaluate</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span> <span class="n">trainY</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">trainScore</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">predicted</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Metal device set to: Apple M1
Model: &quot;sequential&quot;
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>_________________________________________________________________
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> Layer (type) Output Shape Param #
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>=================================================================
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> simple_rnn (SimpleRNN) (None, 32) 1184
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> dense (Dense) (None, 8) 264
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> dense_1 (Dense) (None, 1) 9
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>=================================================================
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Total params: 1,457
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Trainable params: 1,457
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Non-trainable params: 0
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>_________________________________________________________________
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 1/100
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2023-11-21 06:15:36.594141: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 1s - loss: 0.6155 - 1s/epoch - 20ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 2/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4372 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 3/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4090 - 443ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 4/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.4011 - 441ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 5/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3967 - 440ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 6/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3961 - 439ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 7/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3966 - 443ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 8/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3936 - 442ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 9/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3915 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 10/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3908 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 11/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3897 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 12/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3897 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 13/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3896 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 14/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3881 - 445ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 15/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3876 - 445ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 16/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3861 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 17/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3860 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 18/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3831 - 448ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 19/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3857 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 20/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3854 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 21/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3856 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 22/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3845 - 448ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 23/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3831 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 24/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3839 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 25/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3827 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 26/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3814 - 448ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 27/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3829 - 445ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 28/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3814 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 29/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3810 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 30/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3803 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 31/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3805 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 32/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3812 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 33/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3812 - 447ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 34/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3794 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 35/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3778 - 447ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 36/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3787 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 37/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3802 - 445ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 38/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3758 - 447ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 39/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3795 - 445ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 40/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3788 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 41/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3701 - 447ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 42/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3802 - 443ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 43/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3770 - 444ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 44/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3758 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 45/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3763 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 46/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3768 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 47/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3745 - 445ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 48/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3773 - 443ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 49/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3753 - 447ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 50/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3747 - 446ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 51/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3750 - 440ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 52/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3740 - 448ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 53/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3767 - 448ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 54/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3745 - 455ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 55/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3750 - 452ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 56/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3730 - 452ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 57/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3728 - 452ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 58/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3736 - 451ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 59/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3727 - 450ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 60/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3732 - 453ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 61/100
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>50/50 - 0s - loss: 0.3705 - 451ms/epoch - 9ms/step
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Epoch 62/100
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">KeyboardInterrupt</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="nn">Input In [9],</span> in <span class="ni">&lt;cell line: 53&gt;</span><span class="nt">()</span>
<span class="g g-Whitespace"> </span><span class="mi">50</span> <span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s1">&#39;mean_squared_error&#39;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s1">&#39;rmsprop&#39;</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">51</span> <span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
<span class="ne">---&gt; </span><span class="mi">53</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">trainX</span><span class="p">,</span><span class="n">trainY</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="mi">16</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">54</span> <span class="n">trainPredict</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">trainX</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">55</span> <span class="n">testPredict</span><span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">testX</span><span class="p">)</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/utils/traceback_utils.py:64,</span> in <span class="ni">filter_traceback.&lt;locals&gt;.error_handler</span><span class="nt">(*args, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">62</span> <span class="n">filtered_tb</span> <span class="o">=</span> <span class="kc">None</span>
<span class="g g-Whitespace"> </span><span class="mi">63</span> <span class="k">try</span><span class="p">:</span>
<span class="ne">---&gt; </span><span class="mi">64</span> <span class="k">return</span> <span class="n">fn</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">65</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span> <span class="c1"># pylint: disable=broad-except</span>
<span class="g g-Whitespace"> </span><span class="mi">66</span> <span class="n">filtered_tb</span> <span class="o">=</span> <span class="n">_process_traceback_frames</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">__traceback__</span><span class="p">)</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/engine/training.py:1384,</span> in <span class="ni">Model.fit</span><span class="nt">(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)</span>
<span class="g g-Whitespace"> </span><span class="mi">1377</span> <span class="k">with</span> <span class="n">tf</span><span class="o">.</span><span class="n">profiler</span><span class="o">.</span><span class="n">experimental</span><span class="o">.</span><span class="n">Trace</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">1378</span> <span class="s1">&#39;train&#39;</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">1379</span> <span class="n">epoch_num</span><span class="o">=</span><span class="n">epoch</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">1380</span> <span class="n">step_num</span><span class="o">=</span><span class="n">step</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">1381</span> <span class="n">batch_size</span><span class="o">=</span><span class="n">batch_size</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">1382</span> <span class="n">_r</span><span class="o">=</span><span class="mi">1</span><span class="p">):</span>
<span class="g g-Whitespace"> </span><span class="mi">1383</span> <span class="n">callbacks</span><span class="o">.</span><span class="n">on_train_batch_begin</span><span class="p">(</span><span class="n">step</span><span class="p">)</span>
<span class="ne">-&gt; </span><span class="mi">1384</span> <span class="n">tmp_logs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">train_function</span><span class="p">(</span><span class="n">iterator</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">1385</span> <span class="k">if</span> <span class="n">data_handler</span><span class="o">.</span><span class="n">should_sync</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">1386</span> <span class="n">context</span><span class="o">.</span><span class="n">async_wait</span><span class="p">()</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150,</span> in <span class="ni">filter_traceback.&lt;locals&gt;.error_handler</span><span class="nt">(*args, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">148</span> <span class="n">filtered_tb</span> <span class="o">=</span> <span class="kc">None</span>
<span class="g g-Whitespace"> </span><span class="mi">149</span> <span class="k">try</span><span class="p">:</span>
<span class="ne">--&gt; </span><span class="mi">150</span> <span class="k">return</span> <span class="n">fn</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">151</span> <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">152</span> <span class="n">filtered_tb</span> <span class="o">=</span> <span class="n">_process_traceback_frames</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">__traceback__</span><span class="p">)</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:915,</span> in <span class="ni">Function.__call__</span><span class="nt">(self, *args, **kwds)</span>
<span class="g g-Whitespace"> </span><span class="mi">912</span> <span class="n">compiler</span> <span class="o">=</span> <span class="s2">&quot;xla&quot;</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_jit_compile</span> <span class="k">else</span> <span class="s2">&quot;nonXla&quot;</span>
<span class="g g-Whitespace"> </span><span class="mi">914</span> <span class="k">with</span> <span class="n">OptionalXlaContext</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_jit_compile</span><span class="p">):</span>
<span class="ne">--&gt; </span><span class="mi">915</span> <span class="n">result</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_call</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">917</span> <span class="n">new_tracing_count</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">experimental_get_tracing_count</span><span class="p">()</span>
<span class="g g-Whitespace"> </span><span class="mi">918</span> <span class="n">without_tracing</span> <span class="o">=</span> <span class="p">(</span><span class="n">tracing_count</span> <span class="o">==</span> <span class="n">new_tracing_count</span><span class="p">)</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:947,</span> in <span class="ni">Function._call</span><span class="nt">(self, *args, **kwds)</span>
<span class="g g-Whitespace"> </span><span class="mi">944</span> <span class="bp">self</span><span class="o">.</span><span class="n">_lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="g g-Whitespace"> </span><span class="mi">945</span> <span class="c1"># In this case we have created variables on the first call, so we run the</span>
<span class="g g-Whitespace"> </span><span class="mi">946</span> <span class="c1"># defunned version which is guaranteed to never create variables.</span>
<span class="ne">--&gt; </span><span class="mi">947</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_stateless_fn</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwds</span><span class="p">)</span> <span class="c1"># pylint: disable=not-callable</span>
<span class="g g-Whitespace"> </span><span class="mi">948</span> <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_stateful_fn</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">949</span> <span class="c1"># Release the lock early so that multiple threads can perform the call</span>
<span class="g g-Whitespace"> </span><span class="mi">950</span> <span class="c1"># in parallel.</span>
<span class="g g-Whitespace"> </span><span class="mi">951</span> <span class="bp">self</span><span class="o">.</span><span class="n">_lock</span><span class="o">.</span><span class="n">release</span><span class="p">()</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2956,</span> in <span class="ni">Function.__call__</span><span class="nt">(self, *args, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">2953</span> <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">_lock</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">2954</span> <span class="p">(</span><span class="n">graph_function</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">2955</span> <span class="n">filtered_flat_args</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_maybe_define_function</span><span class="p">(</span><span class="n">args</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">)</span>
<span class="ne">-&gt; </span><span class="mi">2956</span> <span class="k">return</span> <span class="n">graph_function</span><span class="o">.</span><span class="n">_call_flat</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">2957</span> <span class="n">filtered_flat_args</span><span class="p">,</span> <span class="n">captured_inputs</span><span class="o">=</span><span class="n">graph_function</span><span class="o">.</span><span class="n">captured_inputs</span><span class="p">)</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1853,</span> in <span class="ni">ConcreteFunction._call_flat</span><span class="nt">(self, args, captured_inputs, cancellation_manager)</span>
<span class="g g-Whitespace"> </span><span class="mi">1849</span> <span class="n">possible_gradient_type</span> <span class="o">=</span> <span class="n">gradients_util</span><span class="o">.</span><span class="n">PossibleTapeGradientTypes</span><span class="p">(</span><span class="n">args</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">1850</span> <span class="k">if</span> <span class="p">(</span><span class="n">possible_gradient_type</span> <span class="o">==</span> <span class="n">gradients_util</span><span class="o">.</span><span class="n">POSSIBLE_GRADIENT_TYPES_NONE</span>
<span class="g g-Whitespace"> </span><span class="mi">1851</span> <span class="ow">and</span> <span class="n">executing_eagerly</span><span class="p">):</span>
<span class="g g-Whitespace"> </span><span class="mi">1852</span> <span class="c1"># No tape is watching; skip to running the function.</span>
<span class="ne">-&gt; </span><span class="mi">1853</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_build_call_outputs</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_inference_function</span><span class="o">.</span><span class="n">call</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">1854</span> <span class="n">ctx</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">cancellation_manager</span><span class="o">=</span><span class="n">cancellation_manager</span><span class="p">))</span>
<span class="g g-Whitespace"> </span><span class="mi">1855</span> <span class="n">forward_backward</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_select_forward_and_backward_functions</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">1856</span> <span class="n">args</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">1857</span> <span class="n">possible_gradient_type</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">1858</span> <span class="n">executing_eagerly</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">1859</span> <span class="n">forward_function</span><span class="p">,</span> <span class="n">args_with_tangents</span> <span class="o">=</span> <span class="n">forward_backward</span><span class="o">.</span><span class="n">forward</span><span class="p">()</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:499,</span> in <span class="ni">_EagerDefinedFunction.call</span><span class="nt">(self, ctx, args, cancellation_manager)</span>
<span class="g g-Whitespace"> </span><span class="mi">497</span> <span class="k">with</span> <span class="n">_InterpolateFunctionError</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="g g-Whitespace"> </span><span class="mi">498</span> <span class="k">if</span> <span class="n">cancellation_manager</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="ne">--&gt; </span><span class="mi">499</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">execute</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">500</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">signature</span><span class="o">.</span><span class="n">name</span><span class="p">),</span>
<span class="g g-Whitespace"> </span><span class="mi">501</span> <span class="n">num_outputs</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_num_outputs</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">502</span> <span class="n">inputs</span><span class="o">=</span><span class="n">args</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">503</span> <span class="n">attrs</span><span class="o">=</span><span class="n">attrs</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">504</span> <span class="n">ctx</span><span class="o">=</span><span class="n">ctx</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">505</span> <span class="k">else</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">506</span> <span class="n">outputs</span> <span class="o">=</span> <span class="n">execute</span><span class="o">.</span><span class="n">execute_with_cancellation</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">507</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">signature</span><span class="o">.</span><span class="n">name</span><span class="p">),</span>
<span class="g g-Whitespace"> </span><span class="mi">508</span> <span class="n">num_outputs</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">_num_outputs</span><span class="p">,</span>
<span class="p">(</span><span class="o">...</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">511</span> <span class="n">ctx</span><span class="o">=</span><span class="n">ctx</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">512</span> <span class="n">cancellation_manager</span><span class="o">=</span><span class="n">cancellation_manager</span><span class="p">)</span>
<span class="nn">File ~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54,</span> in <span class="ni">quick_execute</span><span class="nt">(op_name, num_outputs, inputs, attrs, ctx, name)</span>
<span class="g g-Whitespace"> </span><span class="mi">52</span> <span class="k">try</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">53</span> <span class="n">ctx</span><span class="o">.</span><span class="n">ensure_initialized</span><span class="p">()</span>
<span class="ne">---&gt; </span><span class="mi">54</span> <span class="n">tensors</span> <span class="o">=</span> <span class="n">pywrap_tfe</span><span class="o">.</span><span class="n">TFE_Py_Execute</span><span class="p">(</span><span class="n">ctx</span><span class="o">.</span><span class="n">_handle</span><span class="p">,</span> <span class="n">device_name</span><span class="p">,</span> <span class="n">op_name</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">55</span> <span class="n">inputs</span><span class="p">,</span> <span class="n">attrs</span><span class="p">,</span> <span class="n">num_outputs</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">56</span> <span class="k">except</span> <span class="n">core</span><span class="o">.</span><span class="n">_NotOkStatusException</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">57</span> <span class="k">if</span> <span class="n">name</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="ne">KeyboardInterrupt</span>:
</pre></div>
</div>
</div>
</div>
<div class="section" id="rnns">
<h3>RNNs<a class="headerlink" href="#rnns" title="Permalink to this headline"></a></h3>
<p>RNNs are very powerful, because they
combine two properties:</p>
<ol class="simple">
<li><p>Distributed hidden state that allows them to store a lot of information about the past efficiently.</p></li>
<li><p>Non-linear dynamics that allows them to update their hidden state in complicated ways.</p></li>
</ol>
<p>With enough neurons and time, RNNs
can compute anything that can be
computed by your computer!</p>
</div>
</div>
<div class="section" id="basic-layout">
<h2>Basic layout<a class="headerlink" href="#basic-layout" title="Permalink to this headline"></a></h2>
<!-- dom:FIGURE: [figslides/RNN1.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN1.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --><div class="section" id="we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units">
<h3>We need to specify the initial activity state of all the hidden and output units<a class="headerlink" href="#we-need-to-specify-the-initial-activity-state-of-all-the-hidden-and-output-units" title="Permalink to this headline"></a></h3>
<ol class="simple">
<li><p>We could just fix these initial states to have some default value like 0.5.</p></li>
<li><p>But it is better to treat the initial states as learned parameters.</p></li>
<li><p>We learn them in the same way as we learn the weights.</p></li>
</ol>
<ul class="simple">
<li><p>Start off with an initial random guess for the initial states.</p></li>
</ul>
<p>a. At the end of each training sequence, backpropagate through time all the way to the initial states to get the gradient of the error function with respect to each initial state.</p>
<p>b. Adjust the initial states by following the negative gradient.</p>
</div>
<div class="section" id="we-can-specify-inputs-in-several-ways">
<h3>We can specify inputs in several ways<a class="headerlink" href="#we-can-specify-inputs-in-several-ways" title="Permalink to this headline"></a></h3>
<ol class="simple">
<li><p>Specify the initial states of all the units.</p></li>
<li><p>Specify the initial states of a subset of the units.</p></li>
<li><p>Specify the states of the same subset of the units at every time step.</p></li>
</ol>
<p>This is the natural way to model most sequential data.</p>
</div>
<div class="section" id="we-can-specify-targets-in-several-ways">
<h3>We can specify targets in several ways<a class="headerlink" href="#we-can-specify-targets-in-several-ways" title="Permalink to this headline"></a></h3>
<ol class="simple">
<li><p>Specify desired final activities of all the units</p></li>
<li><p>Specify desired activities of all units for the last few steps</p></li>
</ol>
<ul class="simple">
<li><p>Good for learning attractors</p></li>
<li><p>It is easy to add in extra error derivatives as we backpropagate.</p>
<ul>
<li><p>Specify the desired activity of a subset of the units.</p></li>
</ul>
</li>
<li><p>The other units are input or hidden units.</p></li>
</ul>
<!-- dom:FIGURE: [figslides/RNN2.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN2.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN3.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN3.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN4.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN4.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN5.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN5.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></div>
<div class="section" id="backpropagation-through-time">
<h3>Backpropagation through time<a class="headerlink" href="#backpropagation-through-time" title="Permalink to this headline"></a></h3>
<p>We can think of the recurrent net as a layered, feed-forward
net with shared weights and then train the feed-forward net
with weight constraints.</p>
<p>We can also think of this training algorithm in the time domain:</p>
<ol class="simple">
<li><p>The forward pass builds up a stack of the activities of all the units at each time step.</p></li>
<li><p>The backward pass peels activities off the stack to compute the error derivatives at each time step.</p></li>
<li><p>After the backward pass we add together the derivatives at all the different times for each weight.</p></li>
</ol>
</div>
<div class="section" id="the-backward-pass-is-linear">
<h3>The backward pass is linear<a class="headerlink" href="#the-backward-pass-is-linear" title="Permalink to this headline"></a></h3>
<ol class="simple">
<li><p>There is a big difference between the forward and backward passes.</p></li>
<li><p>In the forward pass we use squashing functions (like the logistic) to prevent the activity vectors from exploding.</p></li>
<li><p>The backward pass, is completely linear. If you double the error derivatives at the final layer, all the error derivatives will double.</p></li>
</ol>
<p>The forward pass determines the slope of the linear function used for
backpropagating through each neuron</p>
<!-- dom:FIGURE: [figslides/RNN6.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN6.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN7.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN7.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN8.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN8.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN9.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN9.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN10.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN10.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN11.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN11.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN12.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN12.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></div>
</div>
<div class="section" id="the-problem-of-exploding-or-vanishing-gradients">
<h2>The problem of exploding or vanishing gradients<a class="headerlink" href="#the-problem-of-exploding-or-vanishing-gradients" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li><p>What happens to the magnitude of the gradients as we backpropagate through many layers?</p></li>
</ul>
<p>a. If the weights are small, the gradients shrink exponentially.</p>
<p>b. If the weights are big the gradients grow exponentially.</p>
<ul class="simple">
<li><p>Typical feed-forward neural nets can cope with these exponential effects because they only have a few hidden layers.</p></li>
<li><p>In an RNN trained on long sequences (e.g. 100 time steps) the gradients can easily explode or vanish.</p></li>
</ul>
<p>a. We can avoid this by initializing the weights very carefully.</p>
<ul class="simple">
<li><p>Even with good initial weights, its very hard to detect that the current target output depends on an input from many time-steps ago.</p></li>
</ul>
<p>RNNs have difficulty dealing with long-range dependencies.</p>
</div>
<div class="section" id="four-effective-ways-to-learn-an-rnn">
<h2>Four effective ways to learn an RNN<a class="headerlink" href="#four-effective-ways-to-learn-an-rnn" title="Permalink to this headline"></a></h2>
<ol class="simple">
<li><p>Long Short Term Memory Make the RNN out of little modules that are designed to remember values for a long time.</p></li>
<li><p>Hessian Free Optimization: Deal with the vanishing gradients problem by using a fancy optimizer that can detect directions with a tiny gradient but even smaller curvature.</p></li>
<li><p>Echo State Networks: Initialize the input a hidden and hidden-hidden and output-hidden connections very carefully so that the hidden state has a huge reservoir of weakly coupled oscillators which can be selectively driven by the input.</p></li>
</ol>
<ul class="simple">
<li><p>ESNs only need to learn the hidden-output connections.</p></li>
</ul>
<ol class="simple">
<li><p>Good initialization with momentum Initialize like in Echo State Networks, but then learn all of the connections using momentum</p></li>
</ol>
<div class="section" id="long-short-term-memory-lstm">
<h3>Long Short Term Memory (LSTM)<a class="headerlink" href="#long-short-term-memory-lstm" title="Permalink to this headline"></a></h3>
<p>LSTM uses a memory cell for
modeling long-range dependencies and avoid vanishing gradient
problems.</p>
<ol class="simple">
<li><p>Introduced by Hochreiter and Schmidhuber (1997) who solved the problem of getting an RNN to remember things for a long time (like hundreds of time steps).</p></li>
<li><p>They designed a memory cell using logistic and linear units with multiplicative interactions.</p></li>
<li><p>Information gets into the cell whenever its “write” gate is on.</p></li>
<li><p>The information stays in the cell so long as its <strong>keep</strong> gate is on.</p></li>
<li><p>Information can be read from the cell by turning on its <strong>read</strong> gate.</p></li>
</ol>
</div>
<div class="section" id="implementing-a-memory-cell-in-a-neural-network">
<h3>Implementing a memory cell in a neural network<a class="headerlink" href="#implementing-a-memory-cell-in-a-neural-network" title="Permalink to this headline"></a></h3>
<p>To preserve information for a long time in
the activities of an RNN, we use a circuit
that implements an analog memory cell.</p>
<ol class="simple">
<li><p>A linear unit that has a self-link with a weight of 1 will maintain its state.</p></li>
<li><p>Information is stored in the cell by activating its write gate.</p></li>
<li><p>Information is retrieved by activating the read gate.</p></li>
<li><p>We can backpropagate through this circuit because logistics are have nice derivatives.</p></li>
</ol>
<!-- dom:FIGURE: [figslides/RNN13.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN13.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN14.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN14.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN15.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN15.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN16.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN16.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN17.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN17.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN18.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN18.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN19.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN19.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN20.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN20.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN21.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN21.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure -->
<!-- dom:FIGURE: [figslides/RNN22.png, width=700 frac=0.9] -->
<!-- begin figure -->
<p><img src="figslides/RNN22.png" width="700"><p style="font-size: 0.9em"><i>Figure 1: </i></p></p>
<!-- end figure --></div>
</div>
<div class="section" id="an-extrapolation-example">
<h2>An extrapolation example<a class="headerlink" href="#an-extrapolation-example" title="Permalink to this headline"></a></h2>
<p>The following code provides an example of how recurrent neural
networks can be used to extrapolate to unknown values of physics data
sets. Specifically, the data sets used in this program come from
a quantum mechanical many-body calculation of energies as functions of the number of particles.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># For matrices and calculations</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="c1"># For machine learning (backend for keras)</span>
<span class="kn">import</span> <span class="nn">tensorflow</span> <span class="k">as</span> <span class="nn">tf</span>
<span class="c1"># User-friendly machine learning library</span>
<span class="c1"># Front end for TensorFlow</span>
<span class="kn">import</span> <span class="nn">tensorflow.keras</span>
<span class="c1"># Different methods from Keras needed to create an RNN</span>
<span class="c1"># This is not necessary but it shortened function calls </span>
<span class="c1"># that need to be used in the code.</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras</span> <span class="kn">import</span> <span class="n">datasets</span><span class="p">,</span> <span class="n">layers</span><span class="p">,</span> <span class="n">models</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.layers</span> <span class="kn">import</span> <span class="n">Input</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras</span> <span class="kn">import</span> <span class="n">regularizers</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.models</span> <span class="kn">import</span> <span class="n">Model</span><span class="p">,</span> <span class="n">Sequential</span>
<span class="kn">from</span> <span class="nn">tensorflow.keras.layers</span> <span class="kn">import</span> <span class="n">Dense</span><span class="p">,</span> <span class="n">SimpleRNN</span><span class="p">,</span> <span class="n">LSTM</span><span class="p">,</span> <span class="n">GRU</span>
<span class="c1"># For timing the code</span>
<span class="kn">from</span> <span class="nn">timeit</span> <span class="kn">import</span> <span class="n">default_timer</span> <span class="k">as</span> <span class="n">timer</span>
<span class="c1"># For plotting</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="c1"># The data set</span>
<span class="n">datatype</span><span class="o">=</span><span class="s1">&#39;VaryDimension&#39;</span>
<span class="n">X_tot</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">42</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">y_tot</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="o">-</span><span class="mf">0.03077640549</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.08336233266</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.1446729567</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.2116753732</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.2830637392</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.3581341341</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.436462435</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.5177783846</span><span class="p">,</span>
<span class="o">-</span><span class="mf">0.6019067271</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.6887363571</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.7782028952</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.8702784034</span><span class="p">,</span> <span class="o">-</span><span class="mf">0.9649652536</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.062292565</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.16231451</span><span class="p">,</span>
<span class="o">-</span><span class="mf">1.265109911</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.370782966</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.479465113</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.591317992</span><span class="p">,</span> <span class="o">-</span><span class="mf">1.70653767</span><span class="p">])</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="formatting-the-data">
<h2>Formatting the Data<a class="headerlink" href="#formatting-the-data" title="Permalink to this headline"></a></h2>
<p>The way the recurrent neural networks are trained in this program
differs from how machine learning algorithms are usually trained.
Typically a machine learning algorithm is trained by learning the
relationship between the x data and the y data. In this program, the
recurrent neural network will be trained to recognize the relationship
in a sequence of y values. This is type of data formatting is
typically used time series forcasting, but it can also be used in any
extrapolation (time series forecasting is just a specific type of
extrapolation along the time axis). This method of data formatting
does not use the x data and assumes that the y data are evenly spaced.</p>
<p>For a standard machine learning algorithm, the training data has the
form of (x,y) so the machine learning algorithm learns to assiciate a
y value with a given x value. This is useful when the test data has x
values within the same range as the training data. However, for this
application, the x values of the test data are outside of the x values
of the training data and the traditional method of training a machine
learning algorithm does not work as well. For this reason, the
recurrent neural network is trained on sequences of y values of the
form ((y1, y2), y3), so that the network is concerned with learning
the pattern of the y data and not the relation between the x and y
data. As long as the pattern of y data outside of the training region
stays relatively stable compared to what was inside the training
region, this method of training can produce accurate extrapolations to
y values far removed from the training data set.</p>
<!-- -->
<!-- The idea behind formatting the data in this way comes from [this resource](https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/) and [this one](https://fairyonice.github.io/Understand-Keras%27s-RNN-behind-the-scenes-with-a-sin-wave-example.html). -->
<!-- -->
<!-- The following method takes in a y data set and formats it so the "x data" are of the form (y1, y2) and the "y data" are of the form y3, with extra brackets added in to make the resulting arrays compatable with both Keras and Tensorflow. -->
<!-- -->
<!-- Note: Using a sequence length of two is not required for time series forecasting so any lenght of sequence could be used (for example instead of ((y1, y2) y3) you could change the length of sequence to be 4 and the resulting data points would have the form ((y1, y2, y3, y4), y5)). While the following method can be used to create a data set of any sequence length, the remainder of the code expects the length of sequence to be 2. This is because the data sets are very small and the higher the lenght of the sequence the less resulting data points. --><div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># FORMAT_DATA</span>
<span class="k">def</span> <span class="nf">format_data</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">length_of_sequence</span> <span class="o">=</span> <span class="mi">2</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Inputs:</span>
<span class="sd"> data(a numpy array): the data that will be the inputs to the recurrent neural</span>
<span class="sd"> network</span>
<span class="sd"> length_of_sequence (an int): the number of elements in one iteration of the</span>
<span class="sd"> sequence patter. For a function approximator use length_of_sequence = 2.</span>
<span class="sd"> Returns:</span>
<span class="sd"> rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its</span>
<span class="sd"> dimensions are length of data - length of sequence, length of sequence, </span>
<span class="sd"> dimnsion of data</span>
<span class="sd"> rnn_output (a numpy array): the training data for the neural network</span>
<span class="sd"> Formats data to be used in a recurrent neural network.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="n">X</span><span class="p">,</span> <span class="n">Y</span> <span class="o">=</span> <span class="p">[],</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="o">-</span><span class="n">length_of_sequence</span><span class="p">):</span>
<span class="c1"># Get the next length_of_sequence elements</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">:</span><span class="n">i</span><span class="o">+</span><span class="n">length_of_sequence</span><span class="p">]</span>
<span class="c1"># Get the element that immediately follows that</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="n">length_of_sequence</span><span class="p">]</span>
<span class="c1"># Reshape so that each data point is contained in its own array</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">reshape</span> <span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="mi">1</span><span class="p">))</span>
<span class="n">X</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="n">Y</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="n">rnn_input</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">X</span><span class="p">)</span>
<span class="n">rnn_output</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">(</span><span class="n">Y</span><span class="p">)</span>
<span class="k">return</span> <span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_output</span>
<span class="c1"># ## Defining the Recurrent Neural Network Using Keras</span>
<span class="c1"># </span>
<span class="c1"># The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.</span>
<span class="k">def</span> <span class="nf">rnn</span><span class="p">(</span><span class="n">length_of_sequences</span><span class="p">,</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">stateful</span> <span class="o">=</span> <span class="kc">False</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Inputs:</span>
<span class="sd"> length_of_sequences (an int): the number of y values in &quot;x data&quot;. This is determined</span>
<span class="sd"> when the data is formatted</span>
<span class="sd"> batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.</span>
<span class="sd"> stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.</span>
<span class="sd"> Returns:</span>
<span class="sd"> model (a Keras model): The recurrent neural network that is built and compiled by this</span>
<span class="sd"> method</span>
<span class="sd"> Builds and compiles a recurrent neural network with one hidden layer and returns the model.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Number of neurons in the input and output layers</span>
<span class="n">in_out_neurons</span> <span class="o">=</span> <span class="mi">1</span>
<span class="c1"># Number of neurons in the hidden layer</span>
<span class="n">hidden_neurons</span> <span class="o">=</span> <span class="mi">200</span>
<span class="c1"># Define the input layer</span>
<span class="n">inp</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">batch_shape</span><span class="o">=</span><span class="p">(</span><span class="n">batch_size</span><span class="p">,</span>
<span class="n">length_of_sequences</span><span class="p">,</span>
<span class="n">in_out_neurons</span><span class="p">))</span>
<span class="c1"># Define the hidden layer as a simple RNN layer with a set number of neurons and add it to </span>
<span class="c1"># the network immediately after the input layer</span>
<span class="n">rnn</span> <span class="o">=</span> <span class="n">SimpleRNN</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN&quot;</span><span class="p">)(</span><span class="n">inp</span><span class="p">)</span>
<span class="c1"># Define the output layer as a dense neural network layer (standard neural network layer)</span>
<span class="c1">#and add it to the network immediately after the hidden layer.</span>
<span class="n">dens</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">in_out_neurons</span><span class="p">,</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;dense&quot;</span><span class="p">)(</span><span class="n">rnn</span><span class="p">)</span>
<span class="c1"># Create the machine learning model starting with the input layer and ending with the </span>
<span class="c1"># output layer</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Model</span><span class="p">(</span><span class="n">inputs</span><span class="o">=</span><span class="p">[</span><span class="n">inp</span><span class="p">],</span><span class="n">outputs</span><span class="o">=</span><span class="p">[</span><span class="n">dens</span><span class="p">])</span>
<span class="c1"># Compile the machine learning model using the mean squared error function as the loss </span>
<span class="c1"># function and an Adams optimizer.</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s2">&quot;mean_squared_error&quot;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s2">&quot;adam&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">model</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="predicting-new-points-with-a-trained-recurrent-neural-network">
<h2>Predicting New Points With A Trained Recurrent Neural Network<a class="headerlink" href="#predicting-new-points-with-a-trained-recurrent-neural-network" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">test_rnn</span> <span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y_test</span><span class="p">,</span> <span class="n">plot_min</span><span class="p">,</span> <span class="n">plot_max</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Inputs:</span>
<span class="sd"> x1 (a list or numpy array): The complete x component of the data set</span>
<span class="sd"> y_test (a list or numpy array): The complete y component of the data set</span>
<span class="sd"> plot_min (an int or float): the smallest x value used in the training data</span>
<span class="sd"> plot_max (an int or float): the largest x valye used in the training data</span>
<span class="sd"> Returns:</span>
<span class="sd"> None.</span>
<span class="sd"> Uses a trained recurrent neural network model to predict future points in the </span>
<span class="sd"> series. Computes the MSE of the predicted data set from the true data set, saves</span>
<span class="sd"> the predicted data set to a csv file, and plots the predicted and true data sets w</span>
<span class="sd"> while also displaying the data range used for training.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Add the training data as the first dim points in the predicted data array as these</span>
<span class="c1"># are known values.</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">y_test</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span><span class="o">.</span><span class="n">tolist</span><span class="p">()</span>
<span class="c1"># Generate the first input to the trained recurrent neural network using the last two </span>
<span class="c1"># points of the training data. Based on how the network was trained this means that it</span>
<span class="c1"># will predict the first point in the data set after the training data. All of the </span>
<span class="c1"># brackets are necessary for Tensorflow.</span>
<span class="n">next_input</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[[</span><span class="n">y_test</span><span class="p">[</span><span class="n">dim</span><span class="o">-</span><span class="mi">2</span><span class="p">]],</span> <span class="p">[</span><span class="n">y_test</span><span class="p">[</span><span class="n">dim</span><span class="o">-</span><span class="mi">1</span><span class="p">]]]])</span>
<span class="c1"># Save the very last point in the training data set. This will be used later.</span>
<span class="n">last</span> <span class="o">=</span> <span class="p">[</span><span class="n">y_test</span><span class="p">[</span><span class="n">dim</span><span class="o">-</span><span class="mi">1</span><span class="p">]]</span>
<span class="c1"># Iterate until the complete data set is created.</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span> <span class="p">(</span><span class="n">dim</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">y_test</span><span class="p">)):</span>
<span class="c1"># Predict the next point in the data set using the previous two points.</span>
<span class="nb">next</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">next_input</span><span class="p">)</span>
<span class="c1"># Append just the number of the predicted data set</span>
<span class="n">y_pred</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="nb">next</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span>
<span class="c1"># Create the input that will be used to predict the next data point in the data set.</span>
<span class="n">next_input</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([[</span><span class="n">last</span><span class="p">,</span> <span class="nb">next</span><span class="p">[</span><span class="mi">0</span><span class="p">]]],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">float64</span><span class="p">)</span>
<span class="n">last</span> <span class="o">=</span> <span class="nb">next</span>
<span class="c1"># Print the mean squared error between the known data set and the predicted data set.</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;MSE: &#39;</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">square</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">subtract</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">))</span><span class="o">.</span><span class="n">mean</span><span class="p">())</span>
<span class="c1"># Save the predicted data set as a csv file for later use</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">datatype</span> <span class="o">+</span> <span class="s1">&#39;Predicted&#39;</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">dim</span><span class="p">)</span><span class="o">+</span><span class="s1">&#39;.csv&#39;</span>
<span class="n">np</span><span class="o">.</span><span class="n">savetxt</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">&#39;,&#39;</span><span class="p">)</span>
<span class="c1"># Plot the known data set and the predicted data set. The red box represents the region that was used</span>
<span class="c1"># for the training data.</span>
<span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y_test</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;true&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x1</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="s1">&#39;g-.&#39;</span><span class="p">,</span><span class="n">label</span><span class="o">=</span><span class="s2">&quot;predicted&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="c1"># Created a red region to represent the points used in the training data.</span>
<span class="n">ax</span><span class="o">.</span><span class="n">axvspan</span><span class="p">(</span><span class="n">plot_min</span><span class="p">,</span> <span class="n">plot_max</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.25</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s1">&#39;red&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Check to make sure the data set is complete</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">X_tot</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">y_tot</span><span class="p">)</span>
<span class="c1"># This is the number of points that will be used in as the training data</span>
<span class="n">dim</span><span class="o">=</span><span class="mi">12</span>
<span class="c1"># Separate the training data from the whole data set</span>
<span class="n">X_train</span> <span class="o">=</span> <span class="n">X_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="n">y_train</span> <span class="o">=</span> <span class="n">y_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="c1"># Generate the training data for the RNN, using a sequence of 2</span>
<span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_training</span> <span class="o">=</span> <span class="n">format_data</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="c1"># Create a recurrent neural network in Keras and produce a summary of the </span>
<span class="c1"># machine learning model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">rnn</span><span class="p">(</span><span class="n">length_of_sequences</span> <span class="o">=</span> <span class="n">rnn_input</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
<span class="c1"># Start the timer. Want to time training+testing</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="c1"># Fit the model using the training data genenerated above using 150 training iterations and a 5%</span>
<span class="c1"># validation split. Setting verbose to True prints information about each training iteration.</span>
<span class="n">hist</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_training</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">150</span><span class="p">,</span>
<span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span><span class="n">validation_split</span><span class="o">=</span><span class="mf">0.05</span><span class="p">)</span>
<span class="k">for</span> <span class="n">label</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">&quot;loss&quot;</span><span class="p">,</span><span class="s2">&quot;val_loss&quot;</span><span class="p">]:</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="n">label</span><span class="p">],</span><span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;loss&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;epoch&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;The final validation loss: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="s2">&quot;val_loss&quot;</span><span class="p">][</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Use the trained neural network to predict more points of the data set</span>
<span class="n">test_rnn</span><span class="p">(</span><span class="n">X_tot</span><span class="p">,</span> <span class="n">y_tot</span><span class="p">,</span> <span class="n">X_tot</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">X_tot</span><span class="p">[</span><span class="n">dim</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c1"># Stop the timer and calculate the total time needed.</span>
<span class="n">end</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Time: &#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="other-things-to-try">
<h2>Other Things to Try<a class="headerlink" href="#other-things-to-try" title="Permalink to this headline"></a></h2>
<p>Changing the size of the recurrent neural network and its parameters
can drastically change the results you get from the model. The below
code takes the simple recurrent neural network from above and adds a
second hidden layer, changes the number of neurons in the hidden
layer, and explicitly declares the activation function of the hidden
layers to be a sigmoid function. The loss function and optimizer can
also be changed but are kept the same as the above network. These
parameters can be tuned to provide the optimal result from the
network. For some ideas on how to improve the performance of a
<a class="reference external" href="https://danijar.com/tips-for-training-recurrent-neural-networks">recurrent neural network</a>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">rnn_2layers</span><span class="p">(</span><span class="n">length_of_sequences</span><span class="p">,</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">stateful</span> <span class="o">=</span> <span class="kc">False</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Inputs:</span>
<span class="sd"> length_of_sequences (an int): the number of y values in &quot;x data&quot;. This is determined</span>
<span class="sd"> when the data is formatted</span>
<span class="sd"> batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.</span>
<span class="sd"> stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.</span>
<span class="sd"> Returns:</span>
<span class="sd"> model (a Keras model): The recurrent neural network that is built and compiled by this</span>
<span class="sd"> method</span>
<span class="sd"> Builds and compiles a recurrent neural network with two hidden layers and returns the model.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Number of neurons in the input and output layers</span>
<span class="n">in_out_neurons</span> <span class="o">=</span> <span class="mi">1</span>
<span class="c1"># Number of neurons in the hidden layer, increased from the first network</span>
<span class="n">hidden_neurons</span> <span class="o">=</span> <span class="mi">500</span>
<span class="c1"># Define the input layer</span>
<span class="n">inp</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">batch_shape</span><span class="o">=</span><span class="p">(</span><span class="n">batch_size</span><span class="p">,</span>
<span class="n">length_of_sequences</span><span class="p">,</span>
<span class="n">in_out_neurons</span><span class="p">))</span>
<span class="c1"># Create two hidden layers instead of one hidden layer. Explicitly set the activation</span>
<span class="c1"># function to be the sigmoid function (the default value is hyperbolic tangent)</span>
<span class="n">rnn1</span> <span class="o">=</span> <span class="n">SimpleRNN</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="c1"># This needs to be True if another hidden layer is to follow</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="s1">&#39;sigmoid&#39;</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN1&quot;</span><span class="p">)(</span><span class="n">inp</span><span class="p">)</span>
<span class="n">rnn2</span> <span class="o">=</span> <span class="n">SimpleRNN</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">activation</span> <span class="o">=</span> <span class="s1">&#39;sigmoid&#39;</span><span class="p">,</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN2&quot;</span><span class="p">)(</span><span class="n">rnn1</span><span class="p">)</span>
<span class="c1"># Define the output layer as a dense neural network layer (standard neural network layer)</span>
<span class="c1">#and add it to the network immediately after the hidden layer.</span>
<span class="n">dens</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">in_out_neurons</span><span class="p">,</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;dense&quot;</span><span class="p">)(</span><span class="n">rnn2</span><span class="p">)</span>
<span class="c1"># Create the machine learning model starting with the input layer and ending with the </span>
<span class="c1"># output layer</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Model</span><span class="p">(</span><span class="n">inputs</span><span class="o">=</span><span class="p">[</span><span class="n">inp</span><span class="p">],</span><span class="n">outputs</span><span class="o">=</span><span class="p">[</span><span class="n">dens</span><span class="p">])</span>
<span class="c1"># Compile the machine learning model using the mean squared error function as the loss </span>
<span class="c1"># function and an Adams optimizer.</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s2">&quot;mean_squared_error&quot;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s2">&quot;adam&quot;</span><span class="p">)</span>
<span class="k">return</span> <span class="n">model</span>
<span class="c1"># Check to make sure the data set is complete</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">X_tot</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">y_tot</span><span class="p">)</span>
<span class="c1"># This is the number of points that will be used in as the training data</span>
<span class="n">dim</span><span class="o">=</span><span class="mi">12</span>
<span class="c1"># Separate the training data from the whole data set</span>
<span class="n">X_train</span> <span class="o">=</span> <span class="n">X_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="n">y_train</span> <span class="o">=</span> <span class="n">y_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="c1"># Generate the training data for the RNN, using a sequence of 2</span>
<span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_training</span> <span class="o">=</span> <span class="n">format_data</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="c1"># Create a recurrent neural network in Keras and produce a summary of the </span>
<span class="c1"># machine learning model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">rnn_2layers</span><span class="p">(</span><span class="n">length_of_sequences</span> <span class="o">=</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
<span class="c1"># Start the timer. Want to time training+testing</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="c1"># Fit the model using the training data genenerated above using 150 training iterations and a 5%</span>
<span class="c1"># validation split. Setting verbose to True prints information about each training iteration.</span>
<span class="n">hist</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_training</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">150</span><span class="p">,</span>
<span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span><span class="n">validation_split</span><span class="o">=</span><span class="mf">0.05</span><span class="p">)</span>
<span class="c1"># This section plots the training loss and the validation loss as a function of training iteration.</span>
<span class="c1"># This is not required for analyzing the couple cluster data but can help determine if the network is</span>
<span class="c1"># being overtrained.</span>
<span class="k">for</span> <span class="n">label</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">&quot;loss&quot;</span><span class="p">,</span><span class="s2">&quot;val_loss&quot;</span><span class="p">]:</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="n">label</span><span class="p">],</span><span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;loss&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;epoch&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;The final validation loss: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="s2">&quot;val_loss&quot;</span><span class="p">][</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Use the trained neural network to predict more points of the data set</span>
<span class="n">test_rnn</span><span class="p">(</span><span class="n">X_tot</span><span class="p">,</span> <span class="n">y_tot</span><span class="p">,</span> <span class="n">X_tot</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">X_tot</span><span class="p">[</span><span class="n">dim</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c1"># Stop the timer and calculate the total time needed.</span>
<span class="n">end</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Time: &#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="other-types-of-recurrent-neural-networks">
<h2>Other Types of Recurrent Neural Networks<a class="headerlink" href="#other-types-of-recurrent-neural-networks" title="Permalink to this headline"></a></h2>
<p>Besides a simple recurrent neural network layer, there are two other
commonly used types of recurrent neural network layers: Long Short
Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short
introduction to these layers see <a class="reference external" href="https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b">https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b</a>
and <a class="reference external" href="https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b">https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b</a>.</p>
<p>The first network created below is similar to the previous network,
but it replaces the SimpleRNN layers with LSTM layers. The second
network below has two hidden layers made up of GRUs, which are
preceeded by two dense (feeddorward) neural network layers. These
dense layers “preprocess” the data before it reaches the recurrent
layers. This architecture has been shown to improve the performance
of recurrent neural networks (see the link above and also
<a class="reference external" href="https://arxiv.org/pdf/1807.02857.pdf">https://arxiv.org/pdf/1807.02857.pdf</a>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">lstm_2layers</span><span class="p">(</span><span class="n">length_of_sequences</span><span class="p">,</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">stateful</span> <span class="o">=</span> <span class="kc">False</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Inputs:</span>
<span class="sd"> length_of_sequences (an int): the number of y values in &quot;x data&quot;. This is determined</span>
<span class="sd"> when the data is formatted</span>
<span class="sd"> batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.</span>
<span class="sd"> stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.</span>
<span class="sd"> Returns:</span>
<span class="sd"> model (a Keras model): The recurrent neural network that is built and compiled by this</span>
<span class="sd"> method</span>
<span class="sd"> Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Number of neurons on the input/output layer and the number of neurons in the hidden layer</span>
<span class="n">in_out_neurons</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">hidden_neurons</span> <span class="o">=</span> <span class="mi">250</span>
<span class="c1"># Input Layer</span>
<span class="n">inp</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">batch_shape</span><span class="o">=</span><span class="p">(</span><span class="n">batch_size</span><span class="p">,</span>
<span class="n">length_of_sequences</span><span class="p">,</span>
<span class="n">in_out_neurons</span><span class="p">))</span>
<span class="c1"># Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)</span>
<span class="n">rnn</span><span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN&quot;</span><span class="p">,</span> <span class="n">use_bias</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;tanh&#39;</span><span class="p">)(</span><span class="n">inp</span><span class="p">)</span>
<span class="n">rnn1</span> <span class="o">=</span> <span class="n">LSTM</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN1&quot;</span><span class="p">,</span> <span class="n">use_bias</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;tanh&#39;</span><span class="p">)(</span><span class="n">rnn</span><span class="p">)</span>
<span class="c1"># Output layer</span>
<span class="n">dens</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">in_out_neurons</span><span class="p">,</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;dense&quot;</span><span class="p">)(</span><span class="n">rnn1</span><span class="p">)</span>
<span class="c1"># Define the midel</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Model</span><span class="p">(</span><span class="n">inputs</span><span class="o">=</span><span class="p">[</span><span class="n">inp</span><span class="p">],</span><span class="n">outputs</span><span class="o">=</span><span class="p">[</span><span class="n">dens</span><span class="p">])</span>
<span class="c1"># Compile the model</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s1">&#39;mean_squared_error&#39;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s1">&#39;adam&#39;</span><span class="p">)</span>
<span class="c1"># Return the model</span>
<span class="k">return</span> <span class="n">model</span>
<span class="k">def</span> <span class="nf">dnn2_gru2</span><span class="p">(</span><span class="n">length_of_sequences</span><span class="p">,</span> <span class="n">batch_size</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">stateful</span> <span class="o">=</span> <span class="kc">False</span><span class="p">):</span>
<span class="w"> </span><span class="sd">&quot;&quot;&quot;</span>
<span class="sd"> Inputs:</span>
<span class="sd"> length_of_sequences (an int): the number of y values in &quot;x data&quot;. This is determined</span>
<span class="sd"> when the data is formatted</span>
<span class="sd"> batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.</span>
<span class="sd"> stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.</span>
<span class="sd"> Returns:</span>
<span class="sd"> model (a Keras model): The recurrent neural network that is built and compiled by this</span>
<span class="sd"> method</span>
<span class="sd"> Builds and compiles a recurrent neural network with four hidden layers (two dense followed by</span>
<span class="sd"> two GRU layers) and returns the model.</span>
<span class="sd"> &quot;&quot;&quot;</span>
<span class="c1"># Number of neurons on the input/output layers and hidden layers</span>
<span class="n">in_out_neurons</span> <span class="o">=</span> <span class="mi">1</span>
<span class="n">hidden_neurons</span> <span class="o">=</span> <span class="mi">250</span>
<span class="c1"># Input layer</span>
<span class="n">inp</span> <span class="o">=</span> <span class="n">Input</span><span class="p">(</span><span class="n">batch_shape</span><span class="o">=</span><span class="p">(</span><span class="n">batch_size</span><span class="p">,</span>
<span class="n">length_of_sequences</span><span class="p">,</span>
<span class="n">in_out_neurons</span><span class="p">))</span>
<span class="c1"># Hidden Dense (feedforward) layers</span>
<span class="n">dnn</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="o">/</span><span class="mi">2</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s1">&#39;dnn&#39;</span><span class="p">)(</span><span class="n">inp</span><span class="p">)</span>
<span class="n">dnn1</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="o">/</span><span class="mi">2</span><span class="p">,</span> <span class="n">activation</span><span class="o">=</span><span class="s1">&#39;relu&#39;</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s1">&#39;dnn1&#39;</span><span class="p">)(</span><span class="n">dnn</span><span class="p">)</span>
<span class="c1"># Hidden GRU layers</span>
<span class="n">rnn1</span> <span class="o">=</span> <span class="n">GRU</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN1&quot;</span><span class="p">,</span> <span class="n">use_bias</span><span class="o">=</span><span class="kc">True</span><span class="p">)(</span><span class="n">dnn1</span><span class="p">)</span>
<span class="n">rnn</span> <span class="o">=</span> <span class="n">GRU</span><span class="p">(</span><span class="n">hidden_neurons</span><span class="p">,</span>
<span class="n">return_sequences</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span>
<span class="n">stateful</span> <span class="o">=</span> <span class="n">stateful</span><span class="p">,</span>
<span class="n">name</span><span class="o">=</span><span class="s2">&quot;RNN&quot;</span><span class="p">,</span> <span class="n">use_bias</span><span class="o">=</span><span class="kc">True</span><span class="p">)(</span><span class="n">rnn1</span><span class="p">)</span>
<span class="c1"># Output layer</span>
<span class="n">dens</span> <span class="o">=</span> <span class="n">Dense</span><span class="p">(</span><span class="n">in_out_neurons</span><span class="p">,</span><span class="n">name</span><span class="o">=</span><span class="s2">&quot;dense&quot;</span><span class="p">)(</span><span class="n">rnn</span><span class="p">)</span>
<span class="c1"># Define the model</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">Model</span><span class="p">(</span><span class="n">inputs</span><span class="o">=</span><span class="p">[</span><span class="n">inp</span><span class="p">],</span><span class="n">outputs</span><span class="o">=</span><span class="p">[</span><span class="n">dens</span><span class="p">])</span>
<span class="c1"># Compile the mdoel</span>
<span class="n">model</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="n">loss</span><span class="o">=</span><span class="s1">&#39;mean_squared_error&#39;</span><span class="p">,</span> <span class="n">optimizer</span><span class="o">=</span><span class="s1">&#39;adam&#39;</span><span class="p">)</span>
<span class="c1"># Return the model</span>
<span class="k">return</span> <span class="n">model</span>
<span class="c1"># Check to make sure the data set is complete</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">X_tot</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">y_tot</span><span class="p">)</span>
<span class="c1"># This is the number of points that will be used in as the training data</span>
<span class="n">dim</span><span class="o">=</span><span class="mi">12</span>
<span class="c1"># Separate the training data from the whole data set</span>
<span class="n">X_train</span> <span class="o">=</span> <span class="n">X_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="n">y_train</span> <span class="o">=</span> <span class="n">y_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="c1"># Generate the training data for the RNN, using a sequence of 2</span>
<span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_training</span> <span class="o">=</span> <span class="n">format_data</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="c1"># Create a recurrent neural network in Keras and produce a summary of the </span>
<span class="c1"># machine learning model</span>
<span class="c1"># Change the method name to reflect which network you want to use</span>
<span class="n">model</span> <span class="o">=</span> <span class="n">dnn2_gru2</span><span class="p">(</span><span class="n">length_of_sequences</span> <span class="o">=</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
<span class="c1"># Start the timer. Want to time training+testing</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="c1"># Fit the model using the training data genenerated above using 150 training iterations and a 5%</span>
<span class="c1"># validation split. Setting verbose to True prints information about each training iteration.</span>
<span class="n">hist</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">rnn_input</span><span class="p">,</span> <span class="n">rnn_training</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">150</span><span class="p">,</span>
<span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span><span class="n">validation_split</span><span class="o">=</span><span class="mf">0.05</span><span class="p">)</span>
<span class="c1"># This section plots the training loss and the validation loss as a function of training iteration.</span>
<span class="c1"># This is not required for analyzing the couple cluster data but can help determine if the network is</span>
<span class="c1"># being overtrained.</span>
<span class="k">for</span> <span class="n">label</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">&quot;loss&quot;</span><span class="p">,</span><span class="s2">&quot;val_loss&quot;</span><span class="p">]:</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="n">label</span><span class="p">],</span><span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;loss&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;epoch&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;The final validation loss: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="s2">&quot;val_loss&quot;</span><span class="p">][</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Use the trained neural network to predict more points of the data set</span>
<span class="n">test_rnn</span><span class="p">(</span><span class="n">X_tot</span><span class="p">,</span> <span class="n">y_tot</span><span class="p">,</span> <span class="n">X_tot</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">X_tot</span><span class="p">[</span><span class="n">dim</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
<span class="c1"># Stop the timer and calculate the total time needed.</span>
<span class="n">end</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Time: &#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">)</span>
<span class="c1"># ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)</span>
<span class="c1"># </span>
<span class="c1"># Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.</span>
<span class="c1"># Check to make sure the data set is complete</span>
<span class="k">assert</span> <span class="nb">len</span><span class="p">(</span><span class="n">X_tot</span><span class="p">)</span> <span class="o">==</span> <span class="nb">len</span><span class="p">(</span><span class="n">y_tot</span><span class="p">)</span>
<span class="c1"># This is the number of points that will be used in as the training data</span>
<span class="n">dim</span><span class="o">=</span><span class="mi">12</span>
<span class="c1"># Separate the training data from the whole data set</span>
<span class="n">X_train</span> <span class="o">=</span> <span class="n">X_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="n">y_train</span> <span class="o">=</span> <span class="n">y_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">]</span>
<span class="c1"># Reshape the data for Keras specifications</span>
<span class="n">X_train</span> <span class="o">=</span> <span class="n">X_train</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">dim</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
<span class="n">y_train</span> <span class="o">=</span> <span class="n">y_train</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="n">dim</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
<span class="c1"># Create a recurrent neural network in Keras and produce a summary of the </span>
<span class="c1"># machine learning model</span>
<span class="c1"># Set the sequence length to 1 for regular data formatting </span>
<span class="n">model</span> <span class="o">=</span> <span class="n">rnn</span><span class="p">(</span><span class="n">length_of_sequences</span> <span class="o">=</span> <span class="mi">1</span><span class="p">)</span>
<span class="n">model</span><span class="o">.</span><span class="n">summary</span><span class="p">()</span>
<span class="c1"># Start the timer. Want to time training+testing</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="c1"># Fit the model using the training data genenerated above using 150 training iterations and a 5%</span>
<span class="c1"># validation split. Setting verbose to True prints information about each training iteration.</span>
<span class="n">hist</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">batch_size</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">epochs</span><span class="o">=</span><span class="mi">150</span><span class="p">,</span>
<span class="n">verbose</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span><span class="n">validation_split</span><span class="o">=</span><span class="mf">0.05</span><span class="p">)</span>
<span class="c1"># This section plots the training loss and the validation loss as a function of training iteration.</span>
<span class="c1"># This is not required for analyzing the couple cluster data but can help determine if the network is</span>
<span class="c1"># being overtrained.</span>
<span class="k">for</span> <span class="n">label</span> <span class="ow">in</span> <span class="p">[</span><span class="s2">&quot;loss&quot;</span><span class="p">,</span><span class="s2">&quot;val_loss&quot;</span><span class="p">]:</span>
<span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="n">label</span><span class="p">],</span><span class="n">label</span><span class="o">=</span><span class="n">label</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s2">&quot;loss&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s2">&quot;epoch&quot;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">title</span><span class="p">(</span><span class="s2">&quot;The final validation loss: </span><span class="si">{}</span><span class="s2">&quot;</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hist</span><span class="o">.</span><span class="n">history</span><span class="p">[</span><span class="s2">&quot;val_loss&quot;</span><span class="p">][</span><span class="o">-</span><span class="mi">1</span><span class="p">]))</span>
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Use the trained neural network to predict the remaining data points</span>
<span class="n">X_pred</span> <span class="o">=</span> <span class="n">X_tot</span><span class="p">[</span><span class="n">dim</span><span class="p">:]</span>
<span class="n">X_pred</span> <span class="o">=</span> <span class="n">X_pred</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">X_pred</span><span class="p">),</span> <span class="mi">1</span><span class="p">))</span>
<span class="n">y_model</span> <span class="o">=</span> <span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_pred</span><span class="p">)</span>
<span class="n">y_pred</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">((</span><span class="n">y_tot</span><span class="p">[:</span><span class="n">dim</span><span class="p">],</span> <span class="n">y_model</span><span class="o">.</span><span class="n">flatten</span><span class="p">()))</span>
<span class="c1"># Plot the known data set and the predicted data set. The red box represents the region that was used</span>
<span class="c1"># for the training data.</span>
<span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X_tot</span><span class="p">,</span> <span class="n">y_tot</span><span class="p">,</span> <span class="n">label</span><span class="o">=</span><span class="s2">&quot;true&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">3</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">X_tot</span><span class="p">,</span> <span class="n">y_pred</span><span class="p">,</span> <span class="s1">&#39;g-.&#39;</span><span class="p">,</span><span class="n">label</span><span class="o">=</span><span class="s2">&quot;predicted&quot;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span>
<span class="n">ax</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
<span class="c1"># Created a red region to represent the points used in the training data.</span>
<span class="n">ax</span><span class="o">.</span><span class="n">axvspan</span><span class="p">(</span><span class="n">X_tot</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">X_tot</span><span class="p">[</span><span class="n">dim</span><span class="p">],</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.25</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="s1">&#39;red&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
<span class="c1"># Stop the timer and calculate the total time needed.</span>
<span class="n">end</span> <span class="o">=</span> <span class="n">timer</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">&#39;Time: &#39;</span><span class="p">,</span> <span class="n">end</span><span class="o">-</span><span class="n">start</span><span class="p">)</span>
</pre></div>
</div>
</div>
</div>
</div>
</div>
<script type="text/x-thebe-config">
{
requestKernel: true,
binderOptions: {
repo: "binder-examples/jupyter-stacks-datascience",
ref: "master",
},
codeMirrorConfig: {
theme: "abcdef",
mode: "python"
},
kernelOptions: {
kernelName: "python3",
path: "./."
},
predefinedOutput: true
}
</script>
<script>kernelName = 'python3'</script>
</div>
<!-- Previous / next buttons -->
<div class='prev-next-area'>
<a class='left-prev' id="prev-link" href="week44.html" title="previous page">
<i class="fas fa-angle-left"></i>
<div class="prev-next-info">
<p class="prev-next-subtitle">previous</p>
<p class="prev-next-title">Week 44, Convolutional Neural Networks (CNN)</p>
</div>
</a>
<a class='right-next' id="next-link" href="week46.html" title="next page">
<div class="prev-next-info">
<p class="prev-next-subtitle">next</p>
<p class="prev-next-title">Week 46: Decision Trees, Ensemble methods and Random Forests</p>
</div>
<i class="fas fa-angle-right"></i>
</a>
</div>
</div>
</div>
<footer class="footer">
<p>
By Morten Hjorth-Jensen<br/>
&copy; Copyright 2021.<br/>
</p>
</footer>
</main>
</div>
</div>
<script src="_static/js/index.be7d3bbb2ef33a8344ce.js"></script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink