added files
This commit is contained in:
mhjensen
@@ -0,0 +1 @@
|
||||
{'holder': ['Morten Hjorth-Jensen'], 'year': '1999-2020', 'license': 'Released under CC Attribution-NonCommercial 4.0 license', 'cite doconce': False}
|
||||
@@ -0,0 +1,20 @@
|
||||
-3.077640548999999864e-02
|
||||
-8.336233265999999642e-02
|
||||
-1.446729566999999939e-01
|
||||
-2.116753731999999888e-01
|
||||
-2.830637391999999974e-01
|
||||
-3.581341341000000011e-01
|
||||
-4.364624349999999819e-01
|
||||
-5.177783846000000301e-01
|
||||
-6.019067271000000385e-01
|
||||
-6.887363571000000295e-01
|
||||
-7.782028951999999666e-01
|
||||
-8.702784033999999558e-01
|
||||
-9.671434760093688965e-01
|
||||
-1.065430045127868652e+00
|
||||
-1.166525721549987793e+00
|
||||
-1.269154906272888184e+00
|
||||
-1.373405933380126953e+00
|
||||
-1.478558778762817383e+00
|
||||
-1.584262609481811523e+00
|
||||
-1.689914822578430176e+00
|
||||
|
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 5.0 KiB After Width: | Height: | Size: 5.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 10 KiB After Width: | Height: | Size: 9.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 9.2 KiB After Width: | Height: | Size: 9.2 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 22 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 27 KiB |
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -124,6 +124,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,899 @@
|
||||
|
||||
|
||||
<!DOCTYPE html>
|
||||
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>11. Data Analysis and Machine Learning: — Applied Machine Learning and Data Analysis</title>
|
||||
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/all.min.css" integrity="sha384-KA6wR/X5RY4zFAHpv/CnoG2UW1uogYfdnP67Uv7eULvTveboZJg0qUpmJZb5VqzN" crossorigin="anonymous">
|
||||
<link href="_static/css/index.css" rel="stylesheet">
|
||||
<link rel="stylesheet" href="_static/sphinx-book-theme.css" type="text/css" />
|
||||
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/copybutton.css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/mystnb.css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/panels-main.c949a650a448cc0ae9fd3441c0e17fb0.css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/panels-variables.06eb56fa6e07937060861dad626602ad.css" />
|
||||
<script id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script>
|
||||
<script src="_static/jquery.js"></script>
|
||||
<script src="_static/underscore.js"></script>
|
||||
<script src="_static/doctools.js"></script>
|
||||
<script src="_static/language_data.js"></script>
|
||||
<script src="_static/togglebutton.js"></script>
|
||||
<script src="_static/clipboard.min.js"></script>
|
||||
<script src="_static/copybutton.js"></script>
|
||||
<script src="_static/sphinx-book-theme.js"></script>
|
||||
<script >var togglebuttonSelector = '.toggle, .admonition.dropdown, .tag_hide_input div.cell_input, .tag_hide-input div.cell_input, .tag_hide_output div.cell_output, .tag_hide-output div.cell_output, .tag_hide_cell.cell, .tag_hide-cell.cell';</script>
|
||||
<script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
|
||||
<script type="text/x-mathjax-config">MathJax.Hub.Config({"tex2jax": {"inlineMath": [["\\(", "\\)"]], "displayMath": [["\\[", "\\]"]], "processRefs": false, "processEnvironments": false}})</script>
|
||||
<script async="async" src="https://unpkg.com/thebelab@latest/lib/index.js"></script>
|
||||
<script >
|
||||
const thebe_selector = ".thebe"
|
||||
const thebe_selector_input = "pre"
|
||||
const thebe_selector_output = ".output"
|
||||
</script>
|
||||
<script async="async" src="_static/sphinx-thebe.js"></script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="prev" title="9. Recurrent Neural Networks" href="chapter10.html" />
|
||||
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="docsearch:language" content="en">
|
||||
|
||||
|
||||
|
||||
</head>
|
||||
<body data-spy="scroll" data-target="#bd-toc-nav" data-offset="80">
|
||||
|
||||
|
||||
<div class="container-xl">
|
||||
<div class="row">
|
||||
|
||||
<div class="col-12 col-md-3 bd-sidebar site-navigation show" id="site-navigation">
|
||||
|
||||
<div class="navbar-brand-box">
|
||||
<a class="navbar-brand text-wrap" href="index.html">
|
||||
|
||||
<img src="_static/Picture.png" class="logo" alt="logo">
|
||||
|
||||
|
||||
<h1 class="site-logo" id="site-title">Applied Machine Learning and Data Analysis</h1>
|
||||
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<form class="bd-search d-flex align-items-center" action="search.html" method="get">
|
||||
<i class="icon fas fa-search"></i>
|
||||
<input type="search" class="form-control" name="q" id="search-input" placeholder="Search this book..." aria-label="Search this book..." autocomplete="off" >
|
||||
</form>
|
||||
|
||||
<nav class="bd-links" id="bd-docs-nav" aria-label="Main navigation">
|
||||
<ul class="nav sidenav_l1">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter1.html">
|
||||
Introduction to Applied Data Analysis and Machine Learning
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
<p class="caption">
|
||||
<span class="caption-text">
|
||||
Supervised Learning
|
||||
</span>
|
||||
</p>
|
||||
<ul class="current nav sidenav_l1">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter2.html">
|
||||
1. Elements of Probability Theory and Statistical Data Analysis
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter3.html">
|
||||
2. Getting started, our first data and Machine Learning encounters
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter4.html">
|
||||
3. Linear Regression and more Advanced Regression Analysis
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
4. Logistic Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter6.html">
|
||||
5. Neural networks, from the simple perceptron to deep learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter7.html">
|
||||
6. Support Vector Machines, overarching aims
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter8.html">
|
||||
7. Dimensionality Reduction
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter9.html">
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1 current active">
|
||||
<a class="current reference internal" href="#">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
<!-- To handle the deprecated key -->
|
||||
|
||||
<div class="navbar_extra_footer">
|
||||
Powered by <a href="https://jupyterbook.org">Jupyter Book</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<main class="col py-md-3 pl-md-4 bd-content overflow-auto" role="main">
|
||||
|
||||
<div class="row topbar fixed-top container-xl">
|
||||
<div class="col-12 col-md-3 bd-topbar-whitespace site-navigation show">
|
||||
</div>
|
||||
<div class="col pl-2 topbar-main">
|
||||
|
||||
<button id="navbar-toggler" class="navbar-toggler ml-0" type="button" data-toggle="collapse"
|
||||
data-toggle="tooltip" data-placement="bottom" data-target=".site-navigation" aria-controls="navbar-menu"
|
||||
aria-expanded="true" aria-label="Toggle navigation" aria-controls="site-navigation"
|
||||
title="Toggle navigation" data-toggle="tooltip" data-placement="left">
|
||||
<i class="fas fa-bars"></i>
|
||||
<i class="fas fa-arrow-left"></i>
|
||||
<i class="fas fa-arrow-up"></i>
|
||||
</button>
|
||||
|
||||
<div class="dropdown-buttons-trigger">
|
||||
<button id="dropdown-buttons-trigger" class="btn btn-secondary topbarbtn" aria-label="Download this page"><i
|
||||
class="fas fa-download"></i></button>
|
||||
|
||||
|
||||
<div class="dropdown-buttons">
|
||||
<!-- ipynb file if we had a myst markdown file -->
|
||||
|
||||
<!-- Download raw file -->
|
||||
<a class="dropdown-buttons" href="_sources/chapter11.ipynb"><button type="button"
|
||||
class="btn btn-secondary topbarbtn" title="Download source file" data-toggle="tooltip"
|
||||
data-placement="left">.ipynb</button></a>
|
||||
<!-- Download PDF via print -->
|
||||
<button type="button" id="download-print" class="btn btn-secondary topbarbtn" title="Print to PDF"
|
||||
onClick="window.print()" data-toggle="tooltip" data-placement="left">.pdf</button>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
<!-- Source interaction buttons -->
|
||||
|
||||
|
||||
<!-- Full screen (wrap in <a> to have style consistency -->
|
||||
<a class="full-screen-button"><button type="button" class="btn btn-secondary topbarbtn" data-toggle="tooltip"
|
||||
data-placement="bottom" onclick="toggleFullScreen()" title="Fullscreen mode"><i
|
||||
class="fas fa-expand"></i></button></a>
|
||||
|
||||
<!-- Launch buttons -->
|
||||
|
||||
</div>
|
||||
|
||||
<!-- Table of contents -->
|
||||
<div class="d-none d-md-block col-md-2 bd-toc show">
|
||||
<div class="tocsection onthispage pt-5 pb-3">
|
||||
<i class="fas fa-list"></i> Contents
|
||||
</div>
|
||||
<nav id="bd-toc-nav">
|
||||
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
<div id="main-content" class="row">
|
||||
<div class="col-12 col-md-9 pl-md-3 pr-md-0">
|
||||
|
||||
<div>
|
||||
|
||||
<!-- dom:TITLE: Data Analysis and Machine Learning: -->
|
||||
<div class="section" id="data-analysis-and-machine-learning">
|
||||
<h1><span class="section-number">11. </span>Data Analysis and Machine Learning:<a class="headerlink" href="#data-analysis-and-machine-learning" title="Permalink to this headline">¶</a></h1>
|
||||
<!-- dom:AUTHOR: Christian Forssén at Department of Physics, Chalmers University of Technology, Sweden -->
|
||||
<!-- Author: -->
|
||||
<p><strong>Christian Forssén</strong>, Department of Physics, Chalmers University of Technology, Sweden</p>
|
||||
<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->
|
||||
<!-- Author: --> **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
|
||||
<p>Date: <strong>Dec 23, 2020</strong></p>
|
||||
<p>Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license</p>
|
||||
</div>
|
||||
<div class="section" id="elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
<h1><span class="section-number">12. </span>Elements of Bayesian theory and Bayesian Neural Networks<a class="headerlink" href="#elements-of-bayesian-theory-and-bayesian-neural-networks" title="Permalink to this headline">¶</a></h1>
|
||||
<div class="section" id="why-bayesian-statistics">
|
||||
<h2><span class="section-number">12.1. </span>Why Bayesian Statistics?<a class="headerlink" href="#why-bayesian-statistics" title="Permalink to this headline">¶</a></h2>
|
||||
<p>We have already made ourselves familiar with elements of a statistical
|
||||
data analysis via quantities like the bias-variance tradeoff as well
|
||||
as some central distribution functions such as the Normal
|
||||
distribution, the binomial distribution and other probability
|
||||
distribution functions.</p>
|
||||
<p>In essentially all the Machine Learning algorithms we have studied,
|
||||
our focus has been on a so-called <strong>frequentist approach</strong>, where
|
||||
knowledge of an underlying likelihood function has not been
|
||||
emphasized. Our data, whether we had a classification or a regression
|
||||
problem, have been our central points of departure.</p>
|
||||
<p>Here we wish to merge this approach with the derivation of a
|
||||
likelihood function which can be used to make prediction on how our
|
||||
system under study evolves. We will venture into the realm of what is
|
||||
called Bayesian Neural Networks. To get an overarching view on what
|
||||
this entails, the following figure conveys the essential differences
|
||||
between a standard Neural network that we have met earlier and a
|
||||
Bayesian Neural Network. In order to get there, we need to present
|
||||
some of the basic elements of Bayesian statistics, starting with the
|
||||
product rule and Bayes’ theorem.</p>
|
||||
</div>
|
||||
<div class="section" id="inference">
|
||||
<h2><span class="section-number">12.2. </span>Inference<a class="headerlink" href="#inference" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Inference:
|
||||
:<br />
|
||||
“the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former” (Webster)
|
||||
Do premises <span class="math notranslate nohighlight">\(A, B, \ldots \to\)</span> hypothesis, <span class="math notranslate nohighlight">\(H\)</span>?</p>
|
||||
<p>Deductive inference:
|
||||
:<br />
|
||||
Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
|
||||
<span class="math notranslate nohighlight">\(B(H|A,B,...) = 0\)</span> or <span class="math notranslate nohighlight">\(1\)</span></p>
|
||||
<p>Inductive inference:
|
||||
:<br />
|
||||
Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
|
||||
<span class="math notranslate nohighlight">\(A, B, C, D\)</span> share properties <span class="math notranslate nohighlight">\(x, y, z\)</span>; <span class="math notranslate nohighlight">\(E\)</span> has properties <span class="math notranslate nohighlight">\(x, y\)</span>
|
||||
<span class="math notranslate nohighlight">\(\to\)</span> <span class="math notranslate nohighlight">\(E\)</span> probably has property <span class="math notranslate nohighlight">\(z\)</span>.</p>
|
||||
</div>
|
||||
<div class="section" id="statistical-inference">
|
||||
<h2><span class="section-number">12.3. </span>Statistical Inference<a class="headerlink" href="#statistical-inference" title="Permalink to this headline">¶</a></h2>
|
||||
<ul class="simple">
|
||||
<li><p>Quantify the strength of inductive inferences from facts, in the form of data (<span class="math notranslate nohighlight">\(D\)</span>), and other premises, e.g. models, to hypotheses about the phenomena producing the data.</p></li>
|
||||
<li><p>Quantify via probabilities, or averages calculated using probabilities. Frequentists (<span class="math notranslate nohighlight">\(\mathcal{F}\)</span>) and Bayesians (<span class="math notranslate nohighlight">\(\mathcal{B}\)</span>) use probabilities very differently for this.</p></li>
|
||||
<li><p>To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a <em>degree-of-belief</em> or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.</p></li>
|
||||
<li><p>To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the <em>long run relative frequency</em> with which an event occurred, given (infinitely) many repeated (experimental) trials.</p></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="some-history">
|
||||
<h2><span class="section-number">12.4. </span>Some history<a class="headerlink" href="#some-history" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Adapted from D.S. Sivia<a class="footnote-reference brackets" href="#sivia" id="id1">1</a>:</p>
|
||||
<blockquote>
|
||||
<div><p>Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2].</p>
|
||||
</div></blockquote>
|
||||
<!-- dom:FIGURE: [fig/sivia_fig_1_2.png, width=700 frac=0.9] -->
|
||||
<!-- begin figure -->
|
||||
<p></p>
|
||||
<img src="fig/sivia_fig_1_2.png" width=700>
|
||||
<!-- end figure -->
|
||||
<blockquote>
|
||||
<div><p>To Laplace, the (shaded) area under the posterior pdf curve between <span class="math notranslate nohighlight">\(m_1\)</span> and <span class="math notranslate nohighlight">\(m_2\)</span> was a measure of how much he believed that the mass of Saturn lay in the range <span class="math notranslate nohighlight">\(m_1 \le M \le m_2\)</span>. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ … it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%!</p>
|
||||
</div></blockquote>
|
||||
<blockquote>
|
||||
<div><p>According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.</p>
|
||||
<p>If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn.</p>
|
||||
</div></blockquote>
|
||||
<blockquote>
|
||||
<div><p>As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.
|
||||
For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.</p>
|
||||
</div></blockquote>
|
||||
<blockquote>
|
||||
<div><p>Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.</p>
|
||||
</div></blockquote>
|
||||
</div>
|
||||
<div class="section" id="the-bayesian-recipe">
|
||||
<h2><span class="section-number">12.5. </span>The Bayesian recipe<a class="headerlink" href="#the-bayesian-recipe" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Assess hypotheses by calculating their probabilities <span class="math notranslate nohighlight">\(p(H_i | \ldots)\)</span> conditional on known and/or presumed information using the rules of probability theory.</p>
|
||||
<p>Probability Theory Axioms:
|
||||
Product (AND) rule :
|
||||
:<br />
|
||||
<span class="math notranslate nohighlight">\(p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)\)</span>
|
||||
Should read <span class="math notranslate nohighlight">\(p(A,B|I)\)</span> as the probability for propositions <span class="math notranslate nohighlight">\(A\)</span> AND <span class="math notranslate nohighlight">\(B\)</span> being true given that <span class="math notranslate nohighlight">\(I\)</span> is true.</p>
|
||||
<p>Sum (OR) rule:
|
||||
:<br />
|
||||
<span class="math notranslate nohighlight">\(p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)\)</span>
|
||||
<span class="math notranslate nohighlight">\(p(A+B|I)\)</span> is the probability that proposition <span class="math notranslate nohighlight">\(A\)</span> OR <span class="math notranslate nohighlight">\(B\)</span> is true given that <span class="math notranslate nohighlight">\(I\)</span> is true.</p>
|
||||
<p>Normalization:
|
||||
:<br />
|
||||
<span class="math notranslate nohighlight">\(p(A|I) + p(\bar{A}|I) = 1\)</span>
|
||||
<span class="math notranslate nohighlight">\(\bar{A}\)</span> denotes the proposition that <span class="math notranslate nohighlight">\(A\)</span> is false.</p>
|
||||
</div>
|
||||
<div class="section" id="bayes-theorem">
|
||||
<h2><span class="section-number">12.6. </span>Bayes’ theorem<a class="headerlink" href="#bayes-theorem" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Bayes’ theorem follows directly from the product rule</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\]</div>
|
||||
<p>p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}.
|
||||
$<span class="math notranslate nohighlight">\(
|
||||
\)</span>$</p>
|
||||
<p>The importance of this property to data analysis becomes apparent if we replace <span class="math notranslate nohighlight">\(A\)</span> and <span class="math notranslate nohighlight">\(B\)</span> by hypothesis(<span class="math notranslate nohighlight">\(H\)</span>) and data(<span class="math notranslate nohighlight">\(D\)</span>):</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="eq:bayes"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}.
|
||||
\label{eq:bayes} \tag{1}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.</p>
|
||||
<p>The various terms in Bayes’ theorem have formal names.</p>
|
||||
<ul class="simple">
|
||||
<li><p>The quantity on the far right, <span class="math notranslate nohighlight">\(p(H|I)\)</span>, is called the <em>prior</em> probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data.</p></li>
|
||||
<li><p>This is modified by the experimental measurements through <span class="math notranslate nohighlight">\(p(D|H,I)\)</span>, the <em>likelihood</em> function,</p></li>
|
||||
<li><p>The denominator <span class="math notranslate nohighlight">\(p(D|I)\)</span> is called the <em>evidence</em>. It does not depend on the hypothesis and can be regarded as a normalization constant.</p></li>
|
||||
<li><p>Together, these yield the <em>posterior</em> probability, <span class="math notranslate nohighlight">\(p(H|D, I )\)</span>, representing our state of knowledge about the truth of the hypothesis in the light of the data.</p></li>
|
||||
</ul>
|
||||
<p>In a sense, Bayes’ theorem encapsulates the process of learning.</p>
|
||||
</div>
|
||||
<div class="section" id="the-friends-of-bayes-theorem">
|
||||
<h2><span class="section-number">12.7. </span>The friends of Bayes’ theorem<a class="headerlink" href="#the-friends-of-bayes-theorem" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Normalization:
|
||||
:<br />
|
||||
<span class="math notranslate nohighlight">\(\sum_i p(H_i|\ldots) = 1\)</span>.</p>
|
||||
<p>Marginalization:
|
||||
:<br />
|
||||
<span class="math notranslate nohighlight">\(\sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I)\)</span>.</p>
|
||||
<p>Marginalization (continuum limit):
|
||||
:<br />
|
||||
<span class="math notranslate nohighlight">\(\int dx p(A,H(x)|I) = p(A|I)\)</span>.</p>
|
||||
<p>In the above, <span class="math notranslate nohighlight">\(H_i\)</span> is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then <span class="math notranslate nohighlight">\(H_1\)</span> could be the proposition that the first candidate will win, and so on. The probability that <span class="math notranslate nohighlight">\(A\)</span> is true, for example that unemployment will be lower in a year’s time (given all relevant information <span class="math notranslate nohighlight">\(I\)</span>, but irrespective of whoever becomes president) is then given by <span class="math notranslate nohighlight">\(\sum_i p(A,H_i|I)\)</span>.</p>
|
||||
<p>In the continuum limit of propositions we must understand <span class="math notranslate nohighlight">\(p(\ldots)\)</span> as a pdf (probability density function).</p>
|
||||
<p>Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.</p>
|
||||
</div>
|
||||
<div class="section" id="inference-with-parametric-models">
|
||||
<h2><span class="section-number">12.8. </span>Inference With Parametric Models<a class="headerlink" href="#inference-with-parametric-models" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Inductive inference with parametric models is a very important tool in the natural sciences.</p>
|
||||
<ul class="simple">
|
||||
<li><p>Consider <span class="math notranslate nohighlight">\(N\)</span> different models <span class="math notranslate nohighlight">\(M_i\)</span> (<span class="math notranslate nohighlight">\(i = 1, \ldots, N\)</span>), each with parameters <span class="math notranslate nohighlight">\(\boldsymbol{\alpha}_i\)</span>. Each of them implies a sampling distribution (conditional predictive distribution for possible data)</p></li>
|
||||
</ul>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\]</div>
|
||||
<p>p(D|\boldsymbol{\alpha}_i, M_i)
|
||||
$<span class="math notranslate nohighlight">\(
|
||||
\)</span>$</p>
|
||||
<ul class="simple">
|
||||
<li><p>The <span class="math notranslate nohighlight">\(\boldsymbol{\alpha}_i\)</span> dependence when we fix attention on the actual, observed data (<span class="math notranslate nohighlight">\(D_\mathrm{obs}\)</span>) is the likelihood function:</p></li>
|
||||
</ul>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\]</div>
|
||||
<p>\mathcal{L}_i (\boldsymbol{\alpha}<em>i) \equiv p(D</em>\mathrm{obs}|\boldsymbol{\alpha}_i, M_i)
|
||||
$<span class="math notranslate nohighlight">\(
|
||||
\)</span>$</p>
|
||||
<ul class="simple">
|
||||
<li><p>We may be uncertain about <span class="math notranslate nohighlight">\(i\)</span> (model uncertainty),</p></li>
|
||||
<li><p>or uncertain about <span class="math notranslate nohighlight">\(\boldsymbol{\alpha}_i\)</span> (parameter uncertainty).</p></li>
|
||||
</ul>
|
||||
<p>Parameter Estimation:
|
||||
:<br />
|
||||
Premise = choice of model (pick specific <span class="math notranslate nohighlight">\(i\)</span>)
|
||||
<span class="math notranslate nohighlight">\(\Rightarrow\)</span> What can we say about <span class="math notranslate nohighlight">\(\boldsymbol{\alpha}_i\)</span>?</p>
|
||||
<p>Model comparison:
|
||||
:<br />
|
||||
Premise = <span class="math notranslate nohighlight">\(\{M_i\}\)</span>
|
||||
<span class="math notranslate nohighlight">\(\Rightarrow\)</span> What can we say about <span class="math notranslate nohighlight">\(i\)</span>?</p>
|
||||
<p>Model adequacy:
|
||||
:<br />
|
||||
Premise = <span class="math notranslate nohighlight">\(M_1\)</span>
|
||||
<span class="math notranslate nohighlight">\(\Rightarrow\)</span> Is <span class="math notranslate nohighlight">\(M_1\)</span> adequate?</p>
|
||||
<p>Hybrid Uncertainty:
|
||||
:<br />
|
||||
Models share some common params: <span class="math notranslate nohighlight">\(\boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\}\)</span>
|
||||
<span class="math notranslate nohighlight">\(\Rightarrow\)</span> What can we say about <span class="math notranslate nohighlight">\(\boldsymbol{\varphi}\)</span>? (Systematic error is an example)</p>
|
||||
</div>
|
||||
<div class="section" id="illustrative-examples-with-python-code">
|
||||
<h2><span class="section-number">12.9. </span>Illustrative examples with python code<a class="headerlink" href="#illustrative-examples-with-python-code" title="Permalink to this headline">¶</a></h2>
|
||||
<ul class="simple">
|
||||
<li><p>Is this a fair coin? (analytical)</p></li>
|
||||
<li><p>Flux from a star (single parameter, MCMC)</p></li>
|
||||
<li><p>The lighthouse problem (two parameters, MCMC)</p></li>
|
||||
<li><p>Linear fit with outliers (nuisance parameters)</p></li>
|
||||
<li><p>…</p></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="example-is-this-a-fair-coin">
|
||||
<h2><span class="section-number">12.10. </span>Example: Is this a fair coin?<a class="headerlink" href="#example-is-this-a-fair-coin" title="Permalink to this headline">¶</a></h2>
|
||||
<p>Let us begin with the analysis of data from a simple coin-tossing experiment.
|
||||
Given that we had observed 6 heads in 8 flips, would you think it was a fair coin? By fair, we mean that we would be prepared to lay an even 1 : 1 bet on the outcome of a flip being a head or a tail. If we decide that the coin was fair, the question which follows naturally is how sure are we that this was so; if it was not fair, how unfair do we think it was? Furthermore, if we were to continue collecting data for this particular coin, observing the outcomes of additional flips, how would we update our belief on the fairness of the coin?</p>
|
||||
<p>A sensible way of formulating this problem is to consider a large number of hypotheses about the range in which the bias-weighting of the coin might lie. If we denote the bias-weighting by <span class="math notranslate nohighlight">\(H\)</span>, then <span class="math notranslate nohighlight">\(H = 0\)</span> and <span class="math notranslate nohighlight">\(H = 1\)</span> can represent a coin which produces a tail or a head on every flip, respectively. There is a continuum of possibilities for the value of H between these limits, with <span class="math notranslate nohighlight">\(H = 0.5\)</span> indicating a fair coin. Our state of knowledge about the fairness, or the degree of unfairness, of the coin is then completely summarized by specifying how much we believe these various propositions to be true.</p>
|
||||
<p>Let us perform a computer simulation of a coin-tossing experiment. This provides the data that we will be analysing.</p>
|
||||
<p>0</p>
|
||||
<p><
|
||||
<
|
||||
<
|
||||
!
|
||||
!
|
||||
C
|
||||
O
|
||||
D
|
||||
E
|
||||
_
|
||||
B
|
||||
L
|
||||
O
|
||||
C
|
||||
K</p>
|
||||
<p>p
|
||||
y
|
||||
c
|
||||
o
|
||||
d</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">999</span><span class="p">)</span> <span class="c1"># for reproducibility</span>
|
||||
<span class="n">a</span><span class="o">=</span><span class="mf">0.6</span> <span class="c1"># biased coin</span>
|
||||
<span class="n">flips</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">12</span><span class="p">)</span> <span class="c1"># simulates 4096 coin flips</span>
|
||||
<span class="n">heads</span><span class="o">=</span><span class="n">flips</span><span class="o"><</span><span class="n">a</span> <span class="c1"># boolean array, heads[i]=True if flip i is heads</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">NameError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="o"><</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">1</span><span class="o">-</span><span class="mi">3</span><span class="n">f1158901148</span><span class="o">></span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
|
||||
<span class="ne">----> </span><span class="mi">1</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">999</span><span class="p">)</span> <span class="c1"># for reproducibility</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">2</span> <span class="n">a</span><span class="o">=</span><span class="mf">0.6</span> <span class="c1"># biased coin</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">3</span> <span class="n">flips</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">12</span><span class="p">)</span> <span class="c1"># simulates 4096 coin flips</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">4</span> <span class="n">heads</span><span class="o">=</span><span class="n">flips</span><span class="o"><</span><span class="n">a</span> <span class="c1"># boolean array, heads[i]=True if flip i is heads</span>
|
||||
|
||||
<span class="ne">NameError</span>: name 'np' is not defined
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>In the light of this data, our inference about the fairness of this coin is summarized by the conditional pdf: <span class="math notranslate nohighlight">\(p(H|D,I)\)</span>. This is, of course, shorthand for the limiting case of a continuum of propositions for the value of <span class="math notranslate nohighlight">\(H\)</span>; that is to say, the probability that <span class="math notranslate nohighlight">\(H\)</span> lies in an infinitesimally narrow range is given by <span class="math notranslate nohighlight">\(p(H|D,I) dH\)</span>.</p>
|
||||
<p>To estimate this posterior pdf, we need to use Bayes’ theorem (<a class="reference external" href="#eq:bayes">1</a>). We will ignore the denominator <span class="math notranslate nohighlight">\(p(D|I)\)</span> as it does not involve bias-weighting explicitly, and it will therefore not affect the shape of the desired pdf. At the end we can evaluate the missing constant subsequently from the normalization condition</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="eq:coin_posterior_norm"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
\int_0^1 p(H|D,I) dH = 1.
|
||||
\label{eq:coin_posterior_norm} \tag{2}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>The prior pdf, <span class="math notranslate nohighlight">\(p(H|I)\)</span>, represents what we know about the coin given only the information <span class="math notranslate nohighlight">\(I\)</span> that we are dealing with a ‘strange coin’. We could keep a very open mind about the nature of the coin; a simple probability assignment which reflects this is a uniform, or flat, prior</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="eq:coin_prior_uniform"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\begin{equation}
|
||||
p(H|I) = \left\{ \begin{array}{ll}
|
||||
1 & 0 \le H \le 1, \\
|
||||
0 & \mathrm{otherwise}.
|
||||
\end{array} \right.
|
||||
\label{eq:coin_prior_uniform} \tag{3}
|
||||
\end{equation}
|
||||
\end{split}\]</div>
|
||||
<p>We will get back later to the choice of prior and its effect on the analysis.</p>
|
||||
<p>This prior state of knowledge, or ignorance, is modified by the data through the likelihood function <span class="math notranslate nohighlight">\(p(D|H,I)\)</span>. It is a measure of the chance that we would have obtained the data that we actually observed, if the value of the bias-weighting was given (as known). If, in the conditioning information <span class="math notranslate nohighlight">\(I\)</span>, we assume that the flips of the coin were independent events, so that the outcome of one did not influence that of another, then the probability of obtaining the data `R heads in N tosses’ is given by the binomial distribution (we leave a formal definition of this to a statistics textbook)</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto1"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
p(D|H,I) \propto H^R (1-H)^{N-R}.
|
||||
\label{_auto1} \tag{4}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>It seems reasonable because <span class="math notranslate nohighlight">\(H\)</span> is the chance of obtaining a head on any flip, and there were <span class="math notranslate nohighlight">\(R\)</span> of them, and <span class="math notranslate nohighlight">\(1-H\)</span> is the corresponding probability for a tail, of which there were <span class="math notranslate nohighlight">\(N-R\)</span>. We note that this binomial distribution also contains a normalization factor, but we will ignore it since it does not depend explicitly on <span class="math notranslate nohighlight">\(H\)</span>, the quantity of interest. It will be absorbed by the normalization condition (<a class="reference external" href="#eq:coin_posterior_norm">2</a>).</p>
|
||||
<p>We perform the setup of this Bayesian framework on the computer.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">prior</span><span class="p">(</span><span class="n">H</span><span class="p">):</span>
|
||||
<span class="n">p</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">zeros_like</span><span class="p">(</span><span class="n">H</span><span class="p">)</span>
|
||||
<span class="n">p</span><span class="p">[(</span><span class="mi">0</span><span class="o"><=</span><span class="n">x</span><span class="p">)</span><span class="o">&</span><span class="p">(</span><span class="n">x</span><span class="o"><=</span><span class="mi">1</span><span class="p">)]</span><span class="o">=</span><span class="mi">1</span> <span class="c1"># allowed range: 0<=H<=1</span>
|
||||
<span class="k">return</span> <span class="n">p</span> <span class="c1"># uniform prior</span>
|
||||
<span class="k">def</span> <span class="nf">likelihood</span><span class="p">(</span><span class="n">H</span><span class="p">,</span><span class="n">data</span><span class="p">):</span>
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
|
||||
<span class="n">no_of_heads</span> <span class="o">=</span> <span class="nb">sum</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
|
||||
<span class="n">no_of_tails</span> <span class="o">=</span> <span class="n">N</span> <span class="o">-</span> <span class="n">no_of_heads</span>
|
||||
<span class="k">return</span> <span class="n">H</span><span class="o">**</span><span class="n">no_of_heads</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1</span><span class="o">-</span><span class="n">H</span><span class="p">)</span><span class="o">**</span><span class="n">no_of_tails</span>
|
||||
<span class="k">def</span> <span class="nf">posterior</span><span class="p">(</span><span class="n">H</span><span class="p">,</span><span class="n">data</span><span class="p">):</span>
|
||||
<span class="n">p</span><span class="o">=</span><span class="n">prior</span><span class="p">(</span><span class="n">H</span><span class="p">)</span><span class="o">*</span><span class="n">likelihood</span><span class="p">(</span><span class="n">H</span><span class="p">,</span><span class="n">data</span><span class="p">)</span>
|
||||
<span class="n">norm</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">trapz</span><span class="p">(</span><span class="n">p</span><span class="p">,</span><span class="n">H</span><span class="p">)</span>
|
||||
<span class="k">return</span> <span class="n">p</span><span class="o">/</span><span class="n">norm</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>The next step is to confront this setup with the simulated data. To get a feel for the result, it is instructive to see how the posterior pdf evolves as we obtain more and more data pertaining to the coin. The results of such an analyses is shown in Fig. <a class="reference external" href="#fig:coinflipping">fig:coinflipping</a>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">x</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">,</span><span class="mi">100</span><span class="p">)</span>
|
||||
<span class="n">fig</span><span class="p">,</span> <span class="n">axs</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">(</span><span class="n">nrows</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span><span class="n">ncols</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span><span class="n">sharex</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span><span class="n">sharey</span><span class="o">=</span><span class="s1">'row'</span><span class="p">)</span>
|
||||
<span class="n">axs_vec</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="n">axs</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">axs_vec</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">prior</span><span class="p">(</span><span class="n">x</span><span class="p">))</span>
|
||||
<span class="k">for</span> <span class="n">ndouble</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">11</span><span class="p">):</span>
|
||||
<span class="n">ax</span><span class="o">=</span><span class="n">axs_vec</span><span class="p">[</span><span class="mi">1</span><span class="o">+</span><span class="n">ndouble</span><span class="p">]</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">posterior</span><span class="p">(</span><span class="n">x</span><span class="p">,</span><span class="n">heads</span><span class="p">[:</span><span class="mi">2</span><span class="o">**</span><span class="n">ndouble</span><span class="p">]))</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">text</span><span class="p">(</span><span class="mf">0.1</span><span class="p">,</span> <span class="mf">0.8</span><span class="p">,</span> <span class="s1">'$N=</span><span class="si">{0}</span><span class="s1">$'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="n">ndouble</span><span class="p">),</span> <span class="n">transform</span><span class="o">=</span><span class="n">ax</span><span class="o">.</span><span class="n">transAxes</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">4</span><span class="p">):</span> <span class="n">axs</span><span class="p">[</span><span class="n">row</span><span class="p">,</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="s1">'$p(H|D_\mathrm</span><span class="si">{obs}</span><span class="s1">,I)$'</span><span class="p">)</span>
|
||||
<span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">3</span><span class="p">):</span> <span class="n">axs</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="n">col</span><span class="p">]</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s1">'$H$'</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- dom:FIGURE:[fig/coinflipping_fig_1.png, width=500 frac=0.95] The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis. <div id="fig:coinflipping"></div> -->
|
||||
<!-- begin figure -->
|
||||
<div id="fig:coinflipping"></div>
|
||||
<p>The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis.</p>
|
||||
<img src="fig/coinflipping_fig_1.png" width=500>
|
||||
<!-- end figure -->
|
||||
<p>The panel in the top left-hand corner shows the posterior pdf for <span class="math notranslate nohighlight">\(H\)</span> given no data, i.e., it is the same as the prior pdf of Eq. (<a class="reference external" href="#eq:coin_prior_uniform">3</a>). It indicates that we have no more reason to believe that the coin is fair than we have to think that it is double-headed, double-tailed, or of any other intermediate bias-weighting.</p>
|
||||
<p>The first flip is obviously tails. At this point we have no evidence that the coin has a side with heads, as indicated by the pdf going to zero as <span class="math notranslate nohighlight">\(H \to 1\)</span>. The second flip is obviously heads and we have now excluded both extreme options <span class="math notranslate nohighlight">\(H=0\)</span> (double-tailed) and <span class="math notranslate nohighlight">\(H=1\)</span> (double-headed). We can note that the posterior at this point has the simple form <span class="math notranslate nohighlight">\(p(H|D,I) = H(1-H)\)</span> for <span class="math notranslate nohighlight">\(0 \le H \le 1\)</span>.</p>
|
||||
<p>The remainder of Fig. <a class="reference external" href="#fig:coinflipping">fig:coinflipping</a> shows how the posterior pdf evolves as the number of data analysed becomes larger and larger. We see that the position of the maximum moves around, but that the amount by which it does so decreases with the increasing number of observations. The width of the posterior pdf also becomes narrower with more data, indicating that we are becoming increasingly confident in our estimate of the bias-weighting. For the coin in this example, the best estimate of <span class="math notranslate nohighlight">\(H\)</span> eventually converges to 0.6, which, of course, was the value chosen to simulate the flips.</p>
|
||||
</div>
|
||||
<div class="section" id="a-few-words-on-different-priors">
|
||||
<h2><span class="section-number">12.11. </span>A few words on different priors<a class="headerlink" href="#a-few-words-on-different-priors" title="Permalink to this headline">¶</a></h2>
|
||||
<ul class="simple">
|
||||
<li><p>uniform</p></li>
|
||||
<li><p>Gaussian</p></li>
|
||||
<li><p>Jeffrey’s prior</p></li>
|
||||
</ul>
|
||||
<p>Repeat the coin flipping experiment with other priors.</p>
|
||||
</div>
|
||||
<div class="section" id="bayesian-parameter-estimation-single-parameter">
|
||||
<h2><span class="section-number">12.12. </span>Bayesian parameter estimation (single parameter)<a class="headerlink" href="#bayesian-parameter-estimation-single-parameter" title="Permalink to this headline">¶</a></h2>
|
||||
<p>We will now consider the very important task of model parameter estimation using statistical inference.
|
||||
[CF 1: maybe stress that model parameters are not random variables, and the meaning of parameter estimation is therefore very different between frequentist and bayesian approaches.]</p>
|
||||
<p>Throughout this section we will consider a specific example that involves a model with a single parameter: “Measured flux from a star”.</p>
|
||||
<div class="section" id="example-measured-flux-from-a-star">
|
||||
<h3><span class="section-number">12.12.1. </span>Example: Measured flux from a star<a class="headerlink" href="#example-measured-flux-from-a-star" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Adapted from the blog <a class="reference external" href="http://jakevdp.github.io">Pythonic Perambulations</a> by Jake VanderPlas.</p>
|
||||
<p>Imagine that we point our telescope to the sky, and observe the light coming from a single star. For the time being, we’ll assume that the star’s true flux is constant with time, i.e. that is it has a fixed value <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span> (we’ll also ignore effects like sky noise and other sources of systematic error). We’ll assume that we perform a series of <span class="math notranslate nohighlight">\(N\)</span> measurements with our telescope, where the ith measurement reports the observed photon flux <span class="math notranslate nohighlight">\(F_i\)</span> and error <span class="math notranslate nohighlight">\(e_i\)</span><a class="footnote-reference brackets" href="#errors" id="id2">2</a>.
|
||||
The question is, given this set of measurements <span class="math notranslate nohighlight">\(D = \{F_i, e_i\}\)</span>, what is our best estimate of the true flux <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>?</p>
|
||||
<p>Because the measurements are number counts, a Poisson distribution is a good approximation to the measurement process:</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">seed</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="c1"># for repeatability</span>
|
||||
<span class="n">F_true</span> <span class="o">=</span> <span class="mi">1000</span> <span class="c1"># true flux, say number of photons measured in 1 second</span>
|
||||
<span class="n">N</span> <span class="o">=</span> <span class="mi">50</span> <span class="c1"># number of measurements</span>
|
||||
<span class="n">F</span> <span class="o">=</span> <span class="n">stats</span><span class="o">.</span><span class="n">poisson</span><span class="p">(</span><span class="n">F_true</span><span class="p">)</span><span class="o">.</span><span class="n">rvs</span><span class="p">(</span><span class="n">N</span><span class="p">)</span>
|
||||
<span class="c1"># N measurements of the flux</span>
|
||||
<span class="n">e</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">sqrt</span><span class="p">(</span><span class="n">F</span><span class="p">)</span> <span class="c1"># errors on Poisson counts estimated via square root</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Now let’s make a simple visualization of the “observed” data, see Fig. <a class="reference external" href="#fig:flux">fig:flux</a>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">errorbar</span><span class="p">(</span><span class="n">F</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="n">N</span><span class="p">),</span> <span class="n">xerr</span><span class="o">=</span><span class="n">e</span><span class="p">,</span> <span class="n">fmt</span><span class="o">=</span><span class="s1">'ok'</span><span class="p">,</span> <span class="n">ecolor</span><span class="o">=</span><span class="s1">'gray'</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">vlines</span><span class="p">([</span><span class="n">F_true</span><span class="p">],</span> <span class="mi">0</span><span class="p">,</span> <span class="n">N</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="s2">"Flux"</span><span class="p">);</span><span class="n">ax</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="s2">"measurement number"</span><span class="p">);</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- dom:FIGURE:[fig/singlephotoncount_fig_1.png, width=400 frac=0.8] Single photon counts (flux measurements). <div id="fig:flux"></div> -->
|
||||
<!-- begin figure -->
|
||||
<div id="fig:flux"></div>
|
||||
<p>Single photon counts (flux measurements).</p>
|
||||
<img src="fig/singlephotoncount_fig_1.png" width=400>
|
||||
<!-- end figure -->
|
||||
<p>These measurements each have a different error <span class="math notranslate nohighlight">\(e_i\)</span> which is estimated from Poisson statistics using the standard square-root rule. In this toy example we already know the true flux <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>, but the question is this: given our measurements and errors, what is our best estimate of the true flux?</p>
|
||||
<p>Let’s take a look at the frequentist and Bayesian approaches to solving this.</p>
|
||||
</div>
|
||||
<div class="section" id="simple-photon-counts-frequentist-approach">
|
||||
<h3><span class="section-number">12.12.2. </span>Simple Photon Counts: Frequentist Approach<a class="headerlink" href="#simple-photon-counts-frequentist-approach" title="Permalink to this headline">¶</a></h3>
|
||||
<p>We’ll start with the classical frequentist maximum likelihood approach. Given a single observation <span class="math notranslate nohighlight">\(D_i = (F_i, e_i)\)</span>, we can compute the probability distribution of the measurement given the true flux Ftrue given our assumption of Gaussian errors</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto2"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
p(D_i | F_\mathrm{true}, I) = \frac{1}{\sqrt{2\pi e_i^2}} \exp \left( \frac{-(F_i-F_\mathrm{true})^2}{2e_i^2} \right).
|
||||
\label{_auto2} \tag{5}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>This should be read “the probability of <span class="math notranslate nohighlight">\(D_i\)</span> given <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>
|
||||
equals …”. You should recognize this as a normal distribution with mean <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span> and standard deviation <span class="math notranslate nohighlight">\(e_i\)</span>.</p>
|
||||
<p>We construct the <em>likelihood function</em> by computing the product of the probabilities for each data point</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto3"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
\mathcal{L}(D | F_\mathrm{true}, I) = \prod_{i=1}^N p(D_i | F_\mathrm{true}, I),
|
||||
\label{_auto3} \tag{6}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>here <span class="math notranslate nohighlight">\(D = \{D_i\}\)</span> represents the entire set of measurements. Because the value of the likelihood can become very small, it is often more convenient to instead compute the log-likelihood. Combining the previous two equations and computing the log, we have</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto4"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi e_i^2) + \frac{(F_i-F_\mathrm{true})^2}{e_i^2} \right].
|
||||
\label{_auto4} \tag{7}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>What we’d like to do is determine <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span> such that the likelihood is maximized. For this simple problem, the maximization can be computed analytically (i.e. by setting <span class="math notranslate nohighlight">\(d\log\mathcal{L}/d F_\mathrm{true} = 0\)</span>). This results in the following observed estimate of <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span></p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto5"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
F_\mathrm{est} = \sum_{i=1}^N w_i F_i; \quad w_i = 1/e_i^2.
|
||||
\label{_auto5} \tag{8}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>Notice that in the special case of all errors <span class="math notranslate nohighlight">\(e_i\)</span> being equal, this reduces to</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto6"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
F_\mathrm{est} = \frac{1}{N} \sum_{i=1} F_i.
|
||||
\label{_auto6} \tag{9}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>That is, in agreement with intuition, <span class="math notranslate nohighlight">\(F_\mathrm{est}\)</span> is simply the mean of the observed data when errors are equal.</p>
|
||||
<p>We can go further and ask what the error of our estimate is. In the frequentist approach, this can be accomplished by fitting a Gaussian approximation to the likelihood curve at maximum; in this simple case this can also be solved analytically (the sum of Gaussians is also a Gaussian). It can be shown that the standard deviation of this Gaussian approximation is</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto7"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
\sigma_\mathrm{est} = \sum_{i=1}^N w_i.
|
||||
\label{_auto7} \tag{10}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>These results are fairly simple calculations; let’s evaluate them for our toy dataset:</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">w</span><span class="o">=</span><span class="mf">1.</span><span class="o">/</span><span class="n">e</span><span class="o">**</span><span class="mi">2</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"""</span>
|
||||
<span class="s2">F_true = </span><span class="si">{0}</span><span class="s2"></span>
|
||||
<span class="s2">F_est = </span><span class="si">{1:.0f}</span><span class="s2"> +/- </span><span class="si">{2:.0f}</span><span class="s2"> (based on </span><span class="si">{3}</span><span class="s2"> measurements) """</span>\
|
||||
<span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">F_true</span><span class="p">,</span> <span class="p">(</span><span class="n">w</span> <span class="o">*</span> <span class="n">F</span><span class="p">)</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="o">/</span> <span class="n">w</span><span class="o">.</span><span class="n">sum</span><span class="p">(),</span> <span class="n">w</span><span class="o">.</span><span class="n">sum</span><span class="p">()</span> <span class="o">**</span> <span class="o">-</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">N</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><code class="docutils literal notranslate"><span class="pre">F_true</span> <span class="pre">=</span> <span class="pre">1000</span></code>
|
||||
<code class="docutils literal notranslate"><span class="pre">F_est</span> <span class="pre">=</span> <span class="pre">998</span> <span class="pre">+/-</span> <span class="pre">4</span> <span class="pre">(based</span> <span class="pre">on</span> <span class="pre">50</span> <span class="pre">measurements)</span></code></p>
|
||||
<p>We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.</p>
|
||||
</div>
|
||||
<div class="section" id="simple-photon-counts-bayesian-approach">
|
||||
<h3><span class="section-number">12.12.3. </span>Simple Photon Counts: Bayesian Approach<a class="headerlink" href="#simple-photon-counts-bayesian-approach" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The Bayesian approach, as you might expect, begins and ends with probabilities. Our hypothesis is that the star has a constant flux <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>. It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question given the data and other information (such as our knowledge of uncertainties for the observed values), i.e. in this case, <span class="math notranslate nohighlight">\(p(F_\mathrm{true} | D,I)\)</span>.
|
||||
Note that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>. Nevertheless, within the Bayesian philosophy this is perfectly acceptable.</p>
|
||||
<p>To compute this result, Bayesians next apply Bayes’ Theorem (<a class="reference external" href="#eq:bayes">1</a>).
|
||||
If we set the prior <span class="math notranslate nohighlight">\(p(F_\mathrm{true}|I) \propto 1\)</span> (a flat prior), we find
|
||||
<span class="math notranslate nohighlight">\(p(F_\mathrm{true}|D,I) \propto p(D | F_\mathrm{true},I) \equiv \mathcal{L}(D | F_\mathrm{true},I)\)</span>
|
||||
and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.</p>
|
||||
</div>
|
||||
<div class="section" id="a-note-about-priors">
|
||||
<h3><span class="section-number">12.12.4. </span>A note about priors<a class="headerlink" href="#a-note-about-priors" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The prior allows inclusion of other information into the computation, which becomes very useful in cases where multiple measurement strategies are being combined to constrain a single model. The necessity to specify a prior, however, is one of the more controversial pieces of Bayesian analysis.
|
||||
A frequentist will point out that the prior is problematic when no true prior information is available. Though it might seem straightforward to use a noninformative prior like the flat prior mentioned above, there are some [surprisingly subtleties](http://normaldeviate.wordpress.com/2013/07/13/lost-causes-in-statistics-ii-noninformative- priors/comment-page-1/) involved. It turns out that in many situations, a truly noninformative prior does not exist! Frequentists point out that the subjective choice of a prior which necessarily biases your result has no place in statistical data analysis.
|
||||
A Bayesian would counter that frequentism doesn’t solve this problem, but simply skirts the question. Frequentism can often be viewed as simply a special case of the Bayesian approach for some (implicit) choice of the prior: a Bayesian would say that it’s better to make this implicit choice explicit, even if the choice might include some subjectivity.</p>
|
||||
</div>
|
||||
<div class="section" id="simple-photon-counts-bayesian-approach-in-practice">
|
||||
<h3><span class="section-number">12.12.5. </span>Simple Photon Counts: Bayesian approach in practice<a class="headerlink" href="#simple-photon-counts-bayesian-approach-in-practice" title="Permalink to this headline">¶</a></h3>
|
||||
<p>Leaving these philosophical debates aside for the time being, let’s address how Bayesian results are generally computed in practice. For a one parameter problem like the one considered here, it’s as simple as computing the posterior probability <span class="math notranslate nohighlight">\(p(F_\mathrm{true} | D,I)\)</span> as a function of <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>: this is the distribution reflecting our knowledge of the parameter <span class="math notranslate nohighlight">\(F_\mathrm{true}\)</span>.
|
||||
But as the dimension of the model grows, this direct approach becomes increasingly intractable. For this reason, Bayesian calculations often depend on sampling methods such as Markov Chain Monte Carlo (MCMC). For this practical example, let us apply an MCMC approach using Dan Foreman-Mackey’s <a class="reference external" href="http://dan.iel.fm/emcee/current/">emcee</a> package. Keep in mind here that the goal is to generate a set of points drawn from the posterior probability distribution, and to use those points to determine the answer we seek.
|
||||
To perform this MCMC, we start by defining Python functions for the prior <span class="math notranslate nohighlight">\(p(F_\mathrm{true} | I)\)</span>, the likelihood <span class="math notranslate nohighlight">\(p(D | F_\mathrm{true},I)\)</span>, and the posterior <span class="math notranslate nohighlight">\(p(F_\mathrm{true} | D,I)\)</span>, noting that none of these need be properly normalized. Our model here is one-dimensional, but to handle multi-dimensional models we’ll define the model in terms of an array of parameters <span class="math notranslate nohighlight">\(\boldsymbol{\alpha}\)</span>, which in this case is <span class="math notranslate nohighlight">\(\boldsymbol{\alpha} = [F_\mathrm{true}]\)</span></p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">log_prior</span><span class="p">(</span><span class="n">alpha</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="mi">0</span> <span class="c1"># flat prior</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">log_likelihood</span><span class="p">(</span><span class="n">alpha</span><span class="p">,</span> <span class="n">F</span><span class="p">,</span> <span class="n">e</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="o">-</span><span class="mf">0.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">(</span><span class="n">np</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="mi">2</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">pi</span> <span class="o">*</span> <span class="n">e</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> \
|
||||
<span class="o">+</span> <span class="p">(</span><span class="n">F</span> <span class="o">-</span> <span class="n">alpha</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">**</span> <span class="mi">2</span> <span class="o">/</span> <span class="n">e</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
|
||||
|
||||
<span class="k">def</span> <span class="nf">log_posterior</span><span class="p">(</span><span class="n">alpha</span><span class="p">,</span> <span class="n">F</span><span class="p">,</span> <span class="n">e</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">log_prior</span><span class="p">(</span><span class="n">alpha</span><span class="p">)</span> <span class="o">+</span> <span class="n">log_likelihood</span><span class="p">(</span><span class="n">alpha</span><span class="p">,</span> <span class="n">F</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>Now we set up the problem, including generating some random starting guesses for the multiple chains of points.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">ndim</span> <span class="o">=</span> <span class="mi">1</span> <span class="c1"># number of parameters in the model</span>
|
||||
<span class="n">nwalkers</span> <span class="o">=</span> <span class="mi">50</span> <span class="c1"># number of MCMC walkers</span>
|
||||
<span class="n">nburn</span> <span class="o">=</span> <span class="mi">1000</span> <span class="c1"># "burn-in" period to let chains stabilize</span>
|
||||
<span class="n">nsteps</span> <span class="o">=</span> <span class="mi">2000</span> <span class="c1"># number of MCMC steps to take</span>
|
||||
<span class="c1"># we'll start at random locations between 0 and 2000</span>
|
||||
<span class="n">starting_guesses</span> <span class="o">=</span> <span class="mi">2000</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">nwalkers</span><span class="p">,</span> <span class="n">ndim</span><span class="p">)</span>
|
||||
<span class="n">sampler</span> <span class="o">=</span> <span class="n">emcee</span><span class="o">.</span><span class="n">EnsembleSampler</span><span class="p">(</span><span class="n">nwalkers</span><span class="p">,</span> <span class="n">ndim</span><span class="p">,</span> <span class="n">log_posterior</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">[</span><span class="n">F</span><span class="p">,</span><span class="n">e</span><span class="p">])</span>
|
||||
<span class="n">sampler</span><span class="o">.</span><span class="n">run_mcmc</span><span class="p">(</span><span class="n">starting_guesses</span><span class="p">,</span> <span class="n">nsteps</span><span class="p">)</span>
|
||||
<span class="c1"># Shape of sampler.chain = (nwalkers, nsteps, ndim)</span>
|
||||
<span class="c1"># Flatten the sampler chain and discard burn-in points:</span>
|
||||
<span class="n">samples</span> <span class="o">=</span> <span class="n">sampler</span><span class="o">.</span><span class="n">chain</span><span class="p">[:,</span> <span class="n">nburn</span><span class="p">:,</span> <span class="p">:]</span><span class="o">.</span><span class="n">reshape</span><span class="p">((</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">ndim</span><span class="p">))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>If this all worked correctly, the array sample should contain a series of 50,000 points drawn from the posterior. Let’s plot them and check. See results in Fig. <a class="reference external" href="#fig:flux-bayesian">fig:flux-bayesian</a>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">fig</span><span class="p">,</span> <span class="n">ax</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">subplots</span><span class="p">()</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">samples</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="mi">50</span><span class="p">,</span> <span class="n">histtype</span><span class="o">=</span><span class="s2">"stepfilled"</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.3</span><span class="p">,</span> <span class="n">normed</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">set_xlabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">'$F_\mathrm</span><span class="si">{est}</span><span class="s1">$'</span><span class="p">)</span>
|
||||
<span class="n">ax</span><span class="o">.</span><span class="n">set_ylabel</span><span class="p">(</span><span class="sa">r</span><span class="s1">'$p(F_\mathrm</span><span class="si">{est}</span><span class="s1">|D,I)$'</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- dom:FIGURE:[fig/singlephotoncount_fig_2.png, width=400 frac=0.8] Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements. <div id="fig:flux-bayesian"></div> -->
|
||||
<!-- begin figure -->
|
||||
<div id="fig:flux-bayesian"></div>
|
||||
<p>Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements.</p>
|
||||
<img src="fig/singlephotoncount_fig_2.png" width=400>
|
||||
<!-- end figure -->
|
||||
</div>
|
||||
<div class="section" id="best-estimates-and-confidence-intervals">
|
||||
<h3><span class="section-number">12.12.6. </span>Best estimates and confidence intervals<a class="headerlink" href="#best-estimates-and-confidence-intervals" title="Permalink to this headline">¶</a></h3>
|
||||
<p>The posterior distribution from our Bayesian data analysis is the key quantity that encodes our inference about the values of the model parameters, given the data and the relevant background information. Often, however, we wish to summarize this result with just a few numbers: the best estimate and a measure of its reliability.</p>
|
||||
<p>There are a few different options for this. The choice of the most appropriate one depends mainly on the shape of the posterior distribution:</p>
|
||||
<p><em>Symmetric posterior pdfs</em>: Since the probability (density) associated with any particular value of the parameter is a measure of how much we believe that it lies in the neighbourhood of that point, our best estimate is given by the maximum of the posterior pdf. If we denote the quantity of interest by <span class="math notranslate nohighlight">\(X\)</span>, with a posterior pdf <span class="math notranslate nohighlight">\(P =p(X|D,I)\)</span>, then the best estimate of its value <span class="math notranslate nohighlight">\(X_0\)</span> is given by the condition <span class="math notranslate nohighlight">\(dP/dX|_{X=X_0}=0\)</span>. Strictly speaking, we should also check the sign of the second derivative to ensure that <span class="math notranslate nohighlight">\(X_0\)</span> represents a maximum.</p>
|
||||
<p>To obtain a measure of the reliability of this best estimate, we need to look at the width or spread of the posterior pdf about <span class="math notranslate nohighlight">\(X_0\)</span>. When considering the behaviour of any function in the neighbourhood of a particular point, it is often helpful to carry out a Taylor series expansion; this is simply a standard tool for (locally) approximating a complicated function by a low-order polynomial. The linear term is zero at the maximum and the quadratic term is often the dominating one determining the width of the posterior pdf. Ignoring all the higher-order terms we arrive at the Gaussian approximation</p>
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto8"></div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
p(X|D,I) \approx \frac{1}{\sigma\sqrt{2\pi}} \exp \left[ -\frac{(x-\mu)^2}{2\sigma^2} \right],
|
||||
\label{_auto8} \tag{11}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>where the mean <span class="math notranslate nohighlight">\(\mu = X_0\)</span> and the variance <span class="math notranslate nohighlight">\(\sigma = \left( - \left. \frac{d^2L}{dX^2} \right|_{X_0} \right)^{-1/2}\)</span>, where <span class="math notranslate nohighlight">\(L\)</span> is the logarithm of the posterior <span class="math notranslate nohighlight">\(P\)</span>. Our inference about the quantity of interest is conveyed very concisely, therefore, by the statement <span class="math notranslate nohighlight">\(X = X_0 \pm \sigma\)</span>, and</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\]</div>
|
||||
<p>p(X_0-\sigma < X < X_0+\sigma | D,I) = \int_{X_0-\sigma}^{X_0+\sigma} p(X|D,I) dX \approx 0.67.
|
||||
$<span class="math notranslate nohighlight">\(
|
||||
\)</span>$</p>
|
||||
<p><em>Asymmetric posterior pdfs</em>: While the maximum of the posterior (<span class="math notranslate nohighlight">\(X_0\)</span>) can still be regarded as giving the best estimate, the true value is now more likely to be on one side of this rather than the other. Alternatively one can compute the mean value, <span class="math notranslate nohighlight">\(\langle X \rangle = \int X p(X|D,I) dX\)</span>, although this tends to overemphasise very long tails. The best option is probably a compromise that can be employed when having access to a large sample from the posterior (as provided by an MCMC), namely to give the median of this ensamble.</p>
|
||||
<p>Furthermore, the concept of an error-bar does not seem appropriate in this case, as it implicitly entails the idea of symmetry. A good way of expressing the reliability with which a parameter can be inferred, for an asymmetric posterior pdf, is rather through a <em>confidence interval</em>. Since the area under the posterior pdf between <span class="math notranslate nohighlight">\(X_1\)</span> and <span class="math notranslate nohighlight">\(X_2\)</span> is proportional to how much we believe that <span class="math notranslate nohighlight">\(X\)</span> lies in that range, the shortest interval that encloses 67% of the area represents a sensible measure of the uncertainty of the estimate. Obviously we can choose to provide some other degree-of-belief that we think is relevant for the case at hand. Assuming that the posterior pdf has been normalized, to have unit area, we need to find <span class="math notranslate nohighlight">\(X_1\)</span> and <span class="math notranslate nohighlight">\(X_2\)</span> such that:</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\]</div>
|
||||
<p>p(X_1 < X < X_2 | D,I) = \int_{X_1}^{X_2} p(X|D,I) dX \approx 0.67,
|
||||
$<span class="math notranslate nohighlight">\(
|
||||
\)</span>$</p>
|
||||
<p>where the difference <span class="math notranslate nohighlight">\(X_2 - X_1\)</span> is as small as possible. The region <span class="math notranslate nohighlight">\(X_1 < X < X_2\)</span> is then called the shortest 67% confidence interval.</p>
|
||||
<p><em>Multimodal posterior pdfs</em>: We can sometimes obtain posteriors which are multimodal; i.e. contains several disconnected regions with large probabilities. There is no difficulty when one of the maxima is very much larger than the others: we can simply ignore the subsidiary solutions, to a good approximation, and concentrate on the global maximum. The problem arises when there are several maxima of comparable magnitude. What do we now mean by a best estimate, and how should we quantify its reliability? The idea of a best estimate and an error-bar, or even a confidence interval, is merely an attempt to summarize the posterior with just two or three numbers; sometimes this just can’t be done, and so these concepts are not valid. For the bimodal case we might be able to characterize the posterior in terms of a few numbers: two best estimates and their associated error-bars, or disjoint confidence intervals. For a general multimodal pdf, the most honest thing we can do is just display the posterior itself.</p>
|
||||
</div>
|
||||
<div class="section" id="simple-photon-counts-best-estimates-and-confidence-intervals">
|
||||
<h3><span class="section-number">12.12.7. </span>Simple Photon Counts: Best estimates and confidence intervals<a class="headerlink" href="#simple-photon-counts-best-estimates-and-confidence-intervals" title="Permalink to this headline">¶</a></h3>
|
||||
<p>To compute these numbers for our example, you would run:</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">sampper</span><span class="o">=</span><span class="n">np</span><span class="o">.</span><span class="n">percentile</span><span class="p">(</span><span class="n">samples</span><span class="p">,</span> <span class="p">[</span><span class="mf">2.5</span><span class="p">,</span> <span class="mf">16.5</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="mf">83.5</span><span class="p">,</span> <span class="mf">97.5</span><span class="p">],</span><span class="n">axis</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span><span class="o">.</span><span class="n">flatten</span><span class="p">()</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"""</span>
|
||||
<span class="s2">F_true = </span><span class="si">{0}</span><span class="s2"></span>
|
||||
<span class="s2">Based on </span><span class="si">{1}</span><span class="s2"> measurements the posterior point estimates are:</span>
|
||||
<span class="s2">...F_est = </span><span class="si">{2:.0f}</span><span class="s2"> +/- </span><span class="si">{3:.0f}</span><span class="s2"></span>
|
||||
<span class="s2">or using credible intervals:</span>
|
||||
<span class="s2">...F_est = </span><span class="si">{4:.0f}</span><span class="s2"> (posterior median) </span>
|
||||
<span class="s2">...F_est in [</span><span class="si">{5:.0f}</span><span class="s2">, </span><span class="si">{6:.0f}</span><span class="s2">] (67</span><span class="si">% c</span><span class="s2">redible interval) </span>
|
||||
<span class="s2">...F_est in [</span><span class="si">{7:.0f}</span><span class="s2">, </span><span class="si">{8:.0f}</span><span class="s2">] (95</span><span class="si">% c</span><span class="s2">redible interval) """</span>\
|
||||
<span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">F_true</span><span class="p">,</span> <span class="n">N</span><span class="p">,</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">samples</span><span class="p">),</span> <span class="n">np</span><span class="o">.</span><span class="n">std</span><span class="p">(</span><span class="n">samples</span><span class="p">),</span> \
|
||||
<span class="n">sampper</span><span class="p">[</span><span class="mi">2</span><span class="p">],</span> <span class="n">sampper</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">sampper</span><span class="p">[</span><span class="mi">3</span><span class="p">],</span> <span class="n">sampper</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">sampper</span><span class="p">[</span><span class="mi">4</span><span class="p">]))</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><code class="docutils literal notranslate"><span class="pre">F_true</span> <span class="pre">=</span> <span class="pre">1000</span></code>
|
||||
<code class="docutils literal notranslate"><span class="pre">Based</span> <span class="pre">on</span> <span class="pre">50</span> <span class="pre">measurements</span> <span class="pre">the</span> <span class="pre">posterior</span> <span class="pre">point</span> <span class="pre">estimates</span> <span class="pre">are:</span></code>
|
||||
<code class="docutils literal notranslate"><span class="pre">...F_est</span> <span class="pre">=</span> <span class="pre">998</span> <span class="pre">+/-</span> <span class="pre">4</span></code>
|
||||
<code class="docutils literal notranslate"><span class="pre">or</span> <span class="pre">using</span> <span class="pre">credible</span> <span class="pre">intervals:</span></code>
|
||||
<code class="docutils literal notranslate"><span class="pre">...F_est</span> <span class="pre">=</span> <span class="pre">998</span>          <span class="pre">(posterior</span> <span class="pre">median)</span></code><br />
|
||||
<code class="docutils literal notranslate"><span class="pre">...F_est</span> <span class="pre">in</span> <span class="pre">[993,</span> <span class="pre">1002]</span> <span class="pre">(67%</span> <span class="pre">credible</span> <span class="pre">interval)</span></code><br />
|
||||
<code class="docutils literal notranslate"><span class="pre">...F_est</span> <span class="pre">in</span> <span class="pre">[989,</span> <span class="pre">1006]</span> <span class="pre">(95%</span> <span class="pre">credible</span> <span class="pre">interval)</span></code></p>
|
||||
<p>In this particular example, the posterior pdf is actually a Gaussian (since it is constructed as a product of Gaussians), and the mean and variance from the quadratic approximation will agree exactly with the frequentist approach.</p>
|
||||
<p>From this final result you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an MCMC sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball, but we did this here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible.</p>
|
||||
<p>Furthermore, as data and models grow in complexity, the two approaches can diverge greatly.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="bayesian-parameter-estimation-multiple-parameters-covariance">
|
||||
<h2><span class="section-number">12.13. </span>Bayesian parameter estimation (multiple parameters, covariance)<a class="headerlink" href="#bayesian-parameter-estimation-multiple-parameters-covariance" title="Permalink to this headline">¶</a></h2>
|
||||
<ul class="simple">
|
||||
<li><p>multidimensional posterior pdf:s</p></li>
|
||||
<li><p>nuisance parameters (e.g. background subtraction?)</p></li>
|
||||
<li><p>corner plots, covariance, correlations</p></li>
|
||||
<li><p>best example?</p></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="bayesian-model-selection">
|
||||
<h2><span class="section-number">12.14. </span>Bayesian model selection<a class="headerlink" href="#bayesian-model-selection" title="Permalink to this headline">¶</a></h2>
|
||||
<ul class="simple">
|
||||
<li><p>Bayesian evidence</p></li>
|
||||
<li><p>Occam’s razor</p></li>
|
||||
<li><p>Best example? How many spectral lines are there?</p></li>
|
||||
</ul>
|
||||
<hr class="docutils" />
|
||||
<dl class="footnote brackets">
|
||||
<dt class="label" id="sivia"><span class="brackets"><a class="fn-backref" href="#id1">1</a></span></dt>
|
||||
<dd><p>Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006</p>
|
||||
</dd>
|
||||
<dt class="label" id="errors"><span class="brackets"><a class="fn-backref" href="#id2">2</a></span></dt>
|
||||
<dd><p>We’ll make the reasonable assumption that errors are Gaussian. In a Frequentist perspective, <span class="math notranslate nohighlight">\(e_i\)</span> is the standard deviation of the results of a single measurement event in the limit of repetitions of <em>that event</em>. In the Bayesian perspective, <span class="math notranslate nohighlight">\(e_i\)</span> is the standard deviation of the (Gaussian) probability distribution describing our knowledge of that particular measurement given its observed value.</p>
|
||||
</dd>
|
||||
</dl>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script type="text/x-thebe-config">
|
||||
{
|
||||
requestKernel: true,
|
||||
binderOptions: {
|
||||
repo: "binder-examples/jupyter-stacks-datascience",
|
||||
ref: "master",
|
||||
},
|
||||
codeMirrorConfig: {
|
||||
theme: "abcdef",
|
||||
mode: "python"
|
||||
},
|
||||
kernelOptions: {
|
||||
kernelName: "python3",
|
||||
path: "./."
|
||||
},
|
||||
predefinedOutput: true
|
||||
}
|
||||
</script>
|
||||
<script>kernelName = 'python3'</script>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class='prev-next-bottom'>
|
||||
|
||||
<a class='left-prev' id="prev-link" href="chapter10.html" title="previous page"><span class="section-number">9. </span>Recurrent Neural Networks</a>
|
||||
|
||||
</div>
|
||||
<footer class="footer mt-5 mt-md-0">
|
||||
<div class="container">
|
||||
<p>
|
||||
|
||||
By Morten Hjorth-Jensen<br/>
|
||||
|
||||
© Copyright 2020.<br/>
|
||||
</p>
|
||||
</div>
|
||||
</footer>
|
||||
</main>
|
||||
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="_static/js/index.js"></script>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -1121,27 +1141,27 @@ uncorrelated.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2.1453055662574343
|
||||
[[12.86665228 10.42534378 14.91184964 7.05197852 10.73905189 9.78942024
|
||||
2.99691286 8.42384711 18.16377739 12.166796 ]
|
||||
[10.42534378 8.44724724 12.082487 5.71394166 8.70143262 7.93198333
|
||||
2.42828097 6.82551297 14.71739653 9.85827768]
|
||||
[14.91184964 12.082487 17.28213796 8.17291407 12.44606006 11.34548129
|
||||
3.47328217 9.76284574 21.05097048 14.10074886]
|
||||
[ 7.05197852 5.71394166 8.17291407 3.8650614 5.88587937 5.36540351
|
||||
1.64255353 4.61695766 9.95523663 6.6684 ]
|
||||
[10.73905189 8.70143262 12.44606006 5.88587937 8.96326666 8.17066394
|
||||
2.50135015 7.03089888 15.16025642 10.15492225]
|
||||
[ 9.78942024 7.93198333 11.34548129 5.36540351 8.17066394 7.44814941
|
||||
2.28016105 6.40917136 13.819667 9.25694395]
|
||||
[ 2.99691286 2.42828097 3.47328217 1.64255353 2.50135015 2.28016105
|
||||
0.69804379 1.96209046 4.23072426 2.83390167]
|
||||
[ 8.42384711 6.82551297 9.76284574 4.61695766 7.03089888 6.40917136
|
||||
1.96209046 5.51512534 11.89189544 7.96564848]
|
||||
[18.16377739 14.71739653 21.05097048 9.95523663 15.16025642 13.819667
|
||||
4.23072426 11.89189544 25.64169777 17.17579438]
|
||||
[12.166796 9.85827768 14.10074886 6.6684 10.15492225 9.25694395
|
||||
2.83390167 7.96564848 17.17579438 11.50500702]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>2.599907486910053
|
||||
[[ 0.2319431 1.78702425 2.33555576 1.01612686 0.80805005 1.10075065
|
||||
1.70915039 0.60318841 0.62019593 1.49305147]
|
||||
[ 1.78702425 13.76827179 17.99447691 7.82883097 6.22568649 8.48082175
|
||||
13.16828638 4.64731357 4.77834938 11.5033349 ]
|
||||
[ 2.33555576 17.99447691 23.51792615 10.23191002 8.13667637 11.08403099
|
||||
17.21032448 6.07381797 6.24507555 15.03431202]
|
||||
[ 1.01612686 7.82883097 10.23191002 4.45158225 3.54001199 4.82231328
|
||||
7.48767091 2.64252717 2.71703596 6.54095632]
|
||||
[ 0.80805005 6.22568649 8.13667637 3.54001199 2.81510801 3.83482678
|
||||
5.95438729 2.10140515 2.16065644 5.20153566]
|
||||
[ 1.10075065 8.48082175 11.08403099 4.82231328 3.83482678 5.22391907
|
||||
8.11124963 2.86259878 2.94331271 7.08569196]
|
||||
[ 1.70915039 13.16828638 17.21032448 7.48767091 5.95438729 8.11124963
|
||||
12.59444676 4.44479575 4.57012136 11.00204953]
|
||||
[ 0.60318841 4.64731357 6.07381797 2.64252717 2.10140515 2.86259878
|
||||
4.44479575 1.56864447 1.61287402 3.88281152]
|
||||
[ 0.62019593 4.77834938 6.24507555 2.71703596 2.16065644 2.94331271
|
||||
4.57012136 1.61287402 1.65835067 3.9922914 ]
|
||||
[ 1.49305147 11.5033349 15.03431202 6.54095632 5.20153566 7.08569196
|
||||
11.00204953 3.88281152 3.9922914 9.61098936]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1449,15 +1469,15 @@ more practically oriented methods like the blocking technique.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.021442867969978376
|
||||
4.117524456004779
|
||||
0.06378292739195897
|
||||
0.9888063524440697 9.919191379006188 17.823611314267136
|
||||
2.969136019500522 3.214525177223236 9.842379203904565
|
||||
[[ 0.98880635 2.96913602 3.21452518]
|
||||
[ 2.96913602 9.91919138 9.8423792 ]
|
||||
[ 3.21452518 9.8423792 17.82361131]]
|
||||
[25.24964689 0.08489291 3.39706925]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.09525604390305954
|
||||
4.231031843972674
|
||||
0.04124568972977915
|
||||
0.831883346005529 8.296268771337688 5.337726521457732
|
||||
2.4701318674515136 1.7159602029423742 5.229477584776403
|
||||
[[0.83188335 2.47013187 1.7159602 ]
|
||||
[2.47013187 8.29626877 5.22947758]
|
||||
[1.7159602 5.22947758 5.33772652]]
|
||||
[12.99430696 0.07864499 1.39292669]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1864,7 +1884,7 @@ assumption for approximating <span class="math notranslate nohighlight">\(\sigma
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>-0.04067115014325762 0.9243683507629109
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.054334900100894076 1.0092227336467725
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter2_184_1.png" src="_images/chapter2_184_1.png" />
|
||||
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -650,8 +670,8 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-0.36370241 0.57281693 0.31446735 -1.74797657 -1.47723222 -1.24113195
|
||||
-0.714415 -1.26679543 -0.27301697 0.6793838 ]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-2.57325284 -1.11383985 0.52678895 -0.47530983 -0.1807752 0.52810513
|
||||
-1.50448048 0.33629261 -1.41402417 -0.39264297]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -1756,13 +1776,13 @@ but now splitting the data into a training set and a test set.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
|
||||
0.9999854211447763
|
||||
0.9999863998248437
|
||||
Training MSE
|
||||
6.298065171189058
|
||||
6.805697972778246
|
||||
Test R2
|
||||
0.9999855160996954
|
||||
0.9999773431421539
|
||||
Test MSE
|
||||
7.1872710004915445
|
||||
4.962804899252701
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1893,7 +1913,7 @@ dtype: int64
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><matplotlib.axes._subplots.AxesSubplot at 0x7f9dd7c37eb0>
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><matplotlib.axes._subplots.AxesSubplot at 0x7fe6839db970>
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter4_131_1.png" src="_images/chapter4_131_1.png" />
|
||||
@@ -2179,27 +2199,27 @@ techniques.</p>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
|
||||
R2 score before scaling 0.99
|
||||
Feature min values before scaling:
|
||||
[1.00000000e+00 1.00405815e-03 3.33637629e-03 1.00813277e-06
|
||||
3.34991581e-06 1.11314068e-05 1.01222393e-09 3.36351028e-09
|
||||
1.11765797e-08 3.71385616e-08 1.01633169e-12 3.37715992e-12
|
||||
1.12219360e-11 3.72892756e-11 1.23908217e-10 1.02045611e-15
|
||||
3.39086494e-15 1.12674763e-14 3.74406011e-14 1.24411055e-13
|
||||
4.13404436e-13]
|
||||
[1.00000000e+00 4.73875395e-04 1.62035199e-03 2.24557890e-07
|
||||
7.67844938e-07 2.62554056e-06 1.06412459e-10 3.63862824e-10
|
||||
1.24417907e-09 4.25429987e-09 5.04262460e-14 1.72425639e-13
|
||||
5.89585850e-13 2.01600803e-12 6.89346326e-12 2.38957572e-17
|
||||
8.17082679e-17 2.79390227e-16 9.55336603e-16 3.26664263e-15
|
||||
1.11698369e-14]
|
||||
Feature max values before scaling:
|
||||
[1. 0.99985941 0.99746704 0.99971883 0.99732681 0.99494051
|
||||
0.99957828 0.99718659 0.99480062 0.99242037 0.99943775 0.99704639
|
||||
0.99466076 0.99228084 0.98990661 0.99929723 0.99690622 0.99452092
|
||||
0.99214133 0.98976744 0.98739922]
|
||||
[1. 0.99909628 0.99839582 0.99819339 0.99749355 0.99679421
|
||||
0.9972913 0.9965921 0.9958934 0.99519518 0.99639003 0.99569147
|
||||
0.99499339 0.9942958 0.99359871 0.99548958 0.99479165 0.9940942
|
||||
0.99339724 0.99270078 0.9920048 ]
|
||||
Feature min values after scaling:
|
||||
[ 0. -1.79147165 -1.7862625 -1.1584177 -1.16385875 -1.16903529
|
||||
-0.9111456 -0.91736698 -0.92355665 -0.92969742 -0.77335697 -0.77892437
|
||||
-0.78453895 -0.79019355 -0.79588019 -0.6835654 -0.68819847 -0.69289982
|
||||
-0.69766733 -0.70249839 -0.70738995]
|
||||
[ 0. -1.71950803 -1.74066137 -1.12098766 -1.13008641 -1.13882035
|
||||
-0.88585974 -0.89326197 -0.90038244 -0.90721064 -0.75127414 -0.7579701
|
||||
-0.76446674 -0.77075409 -0.77682281 -0.6609642 -0.66708788 -0.67306513
|
||||
-0.67888673 -0.68454393 -0.69002844]
|
||||
Feature max values after scaling:
|
||||
[0. 1.702894 1.66035996 2.19865243 2.16709986 2.13442274
|
||||
2.59980117 2.57660557 2.55254863 2.52759568 2.94821185 2.93060502
|
||||
2.91238065 2.89350743 2.8739528 3.26264844 3.24876534 3.23444979
|
||||
3.21967888 3.20442837 3.18867266]
|
||||
[0. 1.75646602 1.64607128 2.30045841 2.19871569 2.09765001
|
||||
2.75230652 2.65433203 2.55670018 2.45952257 3.14599634 3.05079367
|
||||
2.95561862 2.86057207 2.76575615 3.49778287 3.40509201 3.31215604
|
||||
3.21906542 3.12591311 3.03279417]
|
||||
MSE after scaling: 0.00
|
||||
R2 score for scaled data: 0.99
|
||||
</pre></div>
|
||||
@@ -2789,10 +2809,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.012893004091480358
|
||||
4.036145788156587
|
||||
[[0.84855665 2.43974599]
|
||||
[2.43974599 8.11534493]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.13175867934658783
|
||||
4.345411944186196
|
||||
[[ 0.99370801 2.9978939 ]
|
||||
[ 2.9978939 10.13547884]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2832,10 +2852,10 @@ a more brute force way. Here we scale the mean values for each column of the des
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.08534074521320528
|
||||
1.494502745126961
|
||||
[[1. 0.54933577]
|
||||
[0.54933577 1. ]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07991186451576213
|
||||
1.435691712612069
|
||||
[[1. 0.69854547]
|
||||
[0.69854547 1. ]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2867,30 +2887,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.01162555 -0.25149407]
|
||||
[ 0.32741231 1.04530907]
|
||||
[-0.5034199 -1.57100552]
|
||||
[ 1.67655824 6.4270278 ]
|
||||
[-2.84730394 -8.72396496]
|
||||
[ 0.71213034 4.19282793]
|
||||
[ 0.6767474 0.58026128]
|
||||
[ 0.05853864 -0.83340221]
|
||||
[-0.2484568 -1.90992384]
|
||||
[ 0.13616818 1.04436453]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.35269644 -0.0359786 ]
|
||||
[ 0.91157022 2.38965729]
|
||||
[-0.68652791 -1.88459927]
|
||||
[ 0.73533968 3.47606953]
|
||||
[-0.59301581 -2.13696995]
|
||||
[-0.37142517 -1.24766758]
|
||||
[-0.12619024 -0.45164224]
|
||||
[-1.74004988 -5.63952222]
|
||||
[ 1.7194963 3.93948143]
|
||||
[ 0.50349923 1.59117161]]
|
||||
0 1
|
||||
0 0.011626 -0.251494
|
||||
1 0.327412 1.045309
|
||||
2 -0.503420 -1.571006
|
||||
3 1.676558 6.427028
|
||||
4 -2.847304 -8.723965
|
||||
5 0.712130 4.192828
|
||||
6 0.676747 0.580261
|
||||
7 0.058539 -0.833402
|
||||
8 -0.248457 -1.909924
|
||||
9 0.136168 1.044365
|
||||
0 1
|
||||
0 1.0000 0.9645
|
||||
1 0.9645 1.0000
|
||||
0 -0.352696 -0.035979
|
||||
1 0.911570 2.389657
|
||||
2 -0.686528 -1.884599
|
||||
3 0.735340 3.476070
|
||||
4 -0.593016 -2.136970
|
||||
5 -0.371425 -1.247668
|
||||
6 -0.126190 -0.451642
|
||||
7 -1.740050 -5.639522
|
||||
8 1.719496 3.939481
|
||||
9 0.503499 1.591172
|
||||
0 1
|
||||
0 1.000000 0.970341
|
||||
1 0.970341 1.000000
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -2950,37 +2970,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
|
||||
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.0 0.067129 0.071920 0.066488 0.067965 0.069289 0.058136 0.058890
|
||||
2 0.0 0.071920 0.077903 0.072255 0.074320 0.076177 0.063662 0.064746
|
||||
3 0.0 0.066488 0.072255 0.070120 0.072252 0.074165 0.064037 0.065174
|
||||
4 0.0 0.067965 0.074320 0.072252 0.074718 0.076940 0.066275 0.067613
|
||||
5 0.0 0.069289 0.076177 0.074165 0.076940 0.079448 0.068289 0.069815
|
||||
6 0.0 0.058136 0.063662 0.064037 0.066275 0.068289 0.060371 0.061607
|
||||
7 0.0 0.058890 0.064746 0.065174 0.067613 0.069815 0.061607 0.062970
|
||||
8 0.0 0.059633 0.065798 0.066265 0.068894 0.071275 0.062784 0.064269
|
||||
9 0.0 0.060376 0.066836 0.067327 0.070137 0.072691 0.063919 0.065523
|
||||
10 0.0 0.049985 0.054935 0.056831 0.058951 0.060864 0.054874 0.056079
|
||||
11 0.0 0.050516 0.055671 0.057601 0.059851 0.061888 0.055712 0.057004
|
||||
12 0.0 0.051063 0.056417 0.058372 0.060747 0.062904 0.056542 0.057918
|
||||
13 0.0 0.051629 0.057177 0.059149 0.061648 0.063923 0.057370 0.058831
|
||||
14 0.0 0.052218 0.057957 0.059939 0.062558 0.064951 0.058203 0.059747
|
||||
1 0.0 0.080229 0.080679 0.087554 0.086417 0.085175 0.083141 0.081602
|
||||
2 0.0 0.080679 0.082259 0.089654 0.089067 0.088307 0.086044 0.084862
|
||||
3 0.0 0.087554 0.089654 0.100625 0.100093 0.099390 0.098651 0.097358
|
||||
4 0.0 0.086417 0.089067 0.100093 0.099970 0.099642 0.098705 0.097743
|
||||
5 0.0 0.085175 0.088307 0.099390 0.099642 0.099662 0.098566 0.097913
|
||||
6 0.0 0.083141 0.086044 0.098651 0.098705 0.098566 0.098847 0.098003
|
||||
7 0.0 0.081602 0.084862 0.097358 0.097743 0.097913 0.098003 0.097449
|
||||
8 0.0 0.080151 0.083732 0.096133 0.096822 0.097279 0.097201 0.096917
|
||||
9 0.0 0.078794 0.082667 0.094987 0.095957 0.096680 0.096453 0.096419
|
||||
10 0.0 0.076701 0.079987 0.093012 0.093523 0.093835 0.094703 0.094277
|
||||
11 0.0 0.075255 0.078808 0.091666 0.092450 0.093019 0.093698 0.093521
|
||||
12 0.0 0.073932 0.077731 0.090440 0.091477 0.092285 0.092793 0.092846
|
||||
13 0.0 0.072727 0.076752 0.089333 0.090604 0.091633 0.091986 0.092253
|
||||
14 0.0 0.071634 0.075869 0.088337 0.089826 0.091062 0.091273 0.091739
|
||||
|
||||
8 9 10 11 12 13 14
|
||||
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.059633 0.060376 0.049985 0.050516 0.051063 0.051629 0.052218
|
||||
2 0.065798 0.066836 0.054935 0.055671 0.056417 0.057177 0.057957
|
||||
3 0.066265 0.067327 0.056831 0.057601 0.058372 0.059149 0.059939
|
||||
4 0.068894 0.070137 0.058951 0.059851 0.060747 0.061648 0.062558
|
||||
5 0.071275 0.072691 0.060864 0.061888 0.062904 0.063923 0.064951
|
||||
6 0.062784 0.063919 0.054874 0.055712 0.056542 0.057370 0.058203
|
||||
7 0.064269 0.065523 0.056079 0.057004 0.057918 0.058831 0.059747
|
||||
8 0.065687 0.067056 0.057224 0.058233 0.059231 0.060226 0.061224
|
||||
9 0.067056 0.068537 0.058326 0.059418 0.060497 0.061573 0.062652
|
||||
10 0.057224 0.058326 0.050807 0.051634 0.052448 0.053259 0.054070
|
||||
11 0.058233 0.059418 0.051634 0.052522 0.053398 0.054270 0.055142
|
||||
12 0.059231 0.060497 0.052448 0.053398 0.054336 0.055268 0.056202
|
||||
13 0.060226 0.061573 0.053259 0.054270 0.055268 0.056262 0.057257
|
||||
14 0.061224 0.062652 0.054070 0.055142 0.056202 0.057257 0.058313
|
||||
1 0.080151 0.078794 0.076701 0.075255 0.073932 0.072727 0.071634
|
||||
2 0.083732 0.082667 0.079987 0.078808 0.077731 0.076752 0.075869
|
||||
3 0.096133 0.094987 0.093012 0.091666 0.090440 0.089333 0.088337
|
||||
4 0.096822 0.095957 0.093523 0.092450 0.091477 0.090604 0.089826
|
||||
5 0.097279 0.096680 0.093835 0.093019 0.092285 0.091633 0.091062
|
||||
6 0.097201 0.096453 0.094703 0.093698 0.092793 0.091986 0.091273
|
||||
7 0.096917 0.096419 0.094277 0.093521 0.092846 0.092253 0.091739
|
||||
8 0.096634 0.096370 0.093871 0.093345 0.092885 0.092491 0.092163
|
||||
9 0.096370 0.096324 0.093496 0.093186 0.092926 0.092718 0.092564
|
||||
10 0.093871 0.093496 0.091863 0.091208 0.090630 0.090128 0.089699
|
||||
11 0.093345 0.093186 0.091208 0.090769 0.090391 0.090075 0.089820
|
||||
12 0.092885 0.092926 0.090630 0.090391 0.090200 0.090057 0.089963
|
||||
13 0.092491 0.092718 0.090128 0.090075 0.090057 0.090076 0.090133
|
||||
14 0.092163 0.092564 0.089699 0.089820 0.089963 0.090133 0.090330
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -3330,10 +3350,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.410282 sec
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.396214 sec
|
||||
Jackknife Statistics :
|
||||
original bias std. error
|
||||
100.139 100.129 0.148994
|
||||
99.9535 99.9435 0.150002
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -3475,10 +3495,10 @@ theorem.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.02233 sec
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.03201 sec
|
||||
Bootstrap Statistics :
|
||||
original bias std. error
|
||||
100.067 14.9691 100.07 0.150582
|
||||
100.07 15.041 100.069 0.149551
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -2464,6 +2484,14 @@ Accuracy score on test set: 0.07777777777777778
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><ipython-input-5-16b8e3cda33a>:4: RuntimeWarning: overflow encountered in exp
|
||||
return 1/(1 + np.exp(-x))
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter6_134_1.png" src="_images/chapter6_134_1.png" />
|
||||
<img alt="_images/chapter6_134_2.png" src="_images/chapter6_134_2.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="scikit-learn-implementation">
|
||||
@@ -2499,6 +2527,335 @@ performance overall.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.18333333333333332
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.18611111111111112
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.13055555555555556
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.01
|
||||
Accuracy score on test set: 0.24444444444444444
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.23333333333333334
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.12777777777777777
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.1527777777777778
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.9111111111111111
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.8888888888888888
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.8722222222222222
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 0.01
|
||||
Accuracy score on test set: 0.8305555555555556
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.8888888888888888
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.8805555555555555
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.0001
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.8944444444444445
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.975
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.9777777777777777
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.9805555555555555
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 0.01
|
||||
Accuracy score on test set: 0.9861111111111112
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.9805555555555555
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.9777777777777777
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.001
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.9444444444444444
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.
|
||||
warnings.warn(
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.9861111111111112
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.9888888888888889
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.9888888888888889
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 0.01
|
||||
Accuracy score on test set: 0.9861111111111112
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.9888888888888889
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.9722222222222222
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.01
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.9527777777777777
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.8777777777777778
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.8388888888888889
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.8916666666666667
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 0.01
|
||||
Accuracy score on test set: 0.9111111111111111
|
||||
|
||||
Learning rate = 0.1
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.9166666666666666
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.9083333333333333
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 0.1
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.7944444444444444
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.09166666666666666
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.11388888888888889
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.07777777777777778
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.01
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Accuracy score on test set: 0.14444444444444443
|
||||
|
||||
Learning rate = 1.0
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.11944444444444445
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.1361111111111111
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1.0
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.13055555555555556
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 1e-05
|
||||
Accuracy score on test set: 0.07777777777777778
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.0001
|
||||
Accuracy score on test set: 0.21388888888888888
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
|
||||
Lambda = 0.001
|
||||
Accuracy score on test set: 0.10555555555555556
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.01
|
||||
Accuracy score on test set: 0.07777777777777778
|
||||
|
||||
Learning rate = 10.0
|
||||
Lambda = 0.1
|
||||
Accuracy score on test set: 0.125
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
|
||||
Lambda = 1.0
|
||||
Accuracy score on test set: 0.1527777777777778
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 10.0
|
||||
Lambda = 10.0
|
||||
Accuracy score on test set: 0.17777777777777778
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="id5">
|
||||
@@ -2542,6 +2899,10 @@ performance overall.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter6_138_0.png" src="_images/chapter6_138_0.png" />
|
||||
<img alt="_images/chapter6_138_1.png" src="_images/chapter6_138_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="building-neural-networks-in-tensorflow-and-keras">
|
||||
@@ -2580,6 +2941,14 @@ how simple solving a machine learning problem can be.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt"> File</span><span class="nn"> "<ipython-input-13-6ea927cc6e88>"</span><span class="gt">, line </span><span class="mi">1</span>
|
||||
<span class="n">pip3</span> <span class="n">install</span> <span class="n">tensorflow</span>
|
||||
<span class="o">^</span>
|
||||
<span class="ne">SyntaxError</span>: invalid syntax
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>and/or if you use <strong>anaconda</strong>, just write (or install from the graphical user interface)</p>
|
||||
<div class="cell docutils container">
|
||||
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -125,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -548,27 +568,27 @@ techniques.</p>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.01
|
||||
R2 score before scaling 0.94
|
||||
Feature min values before scaling:
|
||||
[1.00000000e+00 2.52277631e-04 1.58998839e-03 6.36440033e-08
|
||||
4.01118504e-07 2.52806307e-06 1.60559584e-11 1.01193226e-10
|
||||
6.37773764e-10 4.01959093e-09 4.05055916e-15 2.55287874e-14
|
||||
1.60896055e-13 1.01405288e-12 6.39110289e-12 1.02186547e-18
|
||||
6.44034203e-18 4.05904756e-17 2.55822858e-16 1.61233230e-15
|
||||
1.01617794e-14]
|
||||
[1.00000000e+00 3.86972479e-03 3.66528972e-03 1.49747700e-05
|
||||
1.41836625e-05 1.34343487e-05 5.79482386e-08 5.48868704e-08
|
||||
5.19872323e-08 4.92407803e-08 2.24243736e-10 2.12397083e-10
|
||||
2.01176282e-10 1.90548268e-10 1.80481726e-10 8.67761543e-13
|
||||
8.21918258e-13 7.78496845e-13 7.37369358e-13 6.98414609e-13
|
||||
6.61517814e-13]
|
||||
Feature max values before scaling:
|
||||
[1. 0.99869632 0.999475 0.99739435 0.998172 0.99895027
|
||||
0.99609407 0.99687071 0.99764796 0.99842581 0.99479548 0.99557111
|
||||
0.99634735 0.99712419 0.99790164 0.99349859 0.99427321 0.99504844
|
||||
0.99582426 0.99660069 0.99737773]
|
||||
[1. 0.99791107 0.99418827 0.99582651 0.99211148 0.98841032
|
||||
0.9937463 0.99003903 0.9863456 0.98266595 0.99167043 0.98797091
|
||||
0.9842852 0.98061323 0.97695496 0.9895989 0.98590711 0.98222909
|
||||
0.9785648 0.97491417 0.97127716]
|
||||
Feature min values after scaling:
|
||||
[ 0. -1.70327945 -1.70487638 -1.10269333 -1.11031464 -1.11779074
|
||||
-0.8675412 -0.87387845 -0.88025384 -0.88666051 -0.73621263 -0.74081963
|
||||
-0.74548023 -0.75019465 -0.75496285 -0.65064103 -0.65385832 -0.65711132
|
||||
-0.66040159 -0.66373067 -0.66710007]
|
||||
[ 0. -1.66745992 -1.72823447 -1.09228638 -1.1114811 -1.1313648
|
||||
-0.86506245 -0.87590574 -0.88698187 -0.89829265 -0.73687981 -0.74440021
|
||||
-0.75205856 -0.75985475 -0.76778823 -0.65190491 -0.65755533 -0.66330693
|
||||
-0.66916042 -0.67511627 -0.68117476]
|
||||
Feature max values after scaling:
|
||||
[0. 1.7648157 1.69590827 2.29614193 2.24159129 2.18630067
|
||||
2.72674862 2.67767959 2.62822832 2.57838874 3.10180456 3.05384136
|
||||
3.0057178 2.95743601 2.90899738 3.44157349 3.39262419 3.3436382
|
||||
3.2946235 3.24558749 3.196537 ]
|
||||
[0. 1.76330702 1.70952132 2.29506297 2.25330559 2.21033632
|
||||
2.73427311 2.69911557 2.66307432 2.6261167 3.11835914 3.08796041
|
||||
3.05688927 3.02512212 2.99263434 3.46490261 3.4379033 3.41038453
|
||||
3.38232999 3.35372259 3.32454443]
|
||||
MSE after scaling: 0.00
|
||||
R2 score for scaled data: 0.97
|
||||
</pre></div>
|
||||
@@ -683,9 +703,7 @@ Test set accuracy scaled data with Standar Scaler: 0.96
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Test set accuracy: 0.95
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Test set accuracy scaled data: 0.96
|
||||
Test set accuracy scaled data: 0.96
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
|
||||
@@ -953,10 +971,10 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07206099142503472
|
||||
4.125937788943556
|
||||
[[ 1.03889255 3.1254397 ]
|
||||
[ 3.1254397 10.77395298]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.11455861677725063
|
||||
4.414311328416008
|
||||
[[1.04460586 2.98508511]
|
||||
[2.98508511 9.60711152]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -996,10 +1014,10 @@ a more brute force way. Here we scale the mean values for each column of the des
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07865526085644693
|
||||
1.4346285243728323
|
||||
[[1. 0.63566281]
|
||||
[0.63566281 1. ]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.07650519724946082
|
||||
1.5077307406836722
|
||||
[[1. 0.63582252]
|
||||
[0.63582252 1. ]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1031,30 +1049,30 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[ 0.53680052 2.50858953]
|
||||
[-0.62753667 -1.74895806]
|
||||
[ 0.16570918 0.64706639]
|
||||
[-0.27228985 -0.08345099]
|
||||
[ 0.85643181 1.55737697]
|
||||
[-0.329196 -2.18391809]
|
||||
[ 0.92272798 2.91370357]
|
||||
[-0.37864859 -0.25719203]
|
||||
[-0.55226917 -1.4496257 ]
|
||||
[-0.32172921 -1.9035916 ]]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[[-0.21054401 -1.67179046]
|
||||
[-0.27924545 -1.98111498]
|
||||
[-1.24089109 -3.51965915]
|
||||
[ 2.17042981 6.77708985]
|
||||
[-0.36489938 -0.63821317]
|
||||
[-1.61222872 -5.67668711]
|
||||
[-0.67184873 -0.62798419]
|
||||
[ 1.21860807 4.36876959]
|
||||
[ 0.0351748 -0.93789716]
|
||||
[ 0.95544469 3.90748678]]
|
||||
0 1
|
||||
0 0.536801 2.508590
|
||||
1 -0.627537 -1.748958
|
||||
2 0.165709 0.647066
|
||||
3 -0.272290 -0.083451
|
||||
4 0.856432 1.557377
|
||||
5 -0.329196 -2.183918
|
||||
6 0.922728 2.913704
|
||||
7 -0.378649 -0.257192
|
||||
8 -0.552269 -1.449626
|
||||
9 -0.321729 -1.903592
|
||||
0 -0.210544 -1.671790
|
||||
1 -0.279245 -1.981115
|
||||
2 -1.240891 -3.519659
|
||||
3 2.170430 6.777090
|
||||
4 -0.364899 -0.638213
|
||||
5 -1.612229 -5.676687
|
||||
6 -0.671849 -0.627984
|
||||
7 1.218608 4.368770
|
||||
8 0.035175 -0.937897
|
||||
9 0.955445 3.907487
|
||||
0 1
|
||||
0 1.000000 0.907143
|
||||
1 0.907143 1.000000
|
||||
0 1.000000 0.972249
|
||||
1 0.972249 1.000000
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1114,37 +1132,37 @@ this matrix we easily see that it is a positive definite matrix.</p>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1 2 3 4 5 6 7 \
|
||||
0 0.0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.0 0.076168 0.080935 0.078268 0.081921 0.085682 0.070727 0.073798
|
||||
2 0.0 0.080935 0.086925 0.081766 0.086005 0.090410 0.072900 0.076302
|
||||
3 0.0 0.078268 0.081766 0.085075 0.088305 0.091557 0.079623 0.082640
|
||||
4 0.0 0.081921 0.086005 0.088305 0.091884 0.095514 0.082116 0.085368
|
||||
5 0.0 0.085682 0.090410 0.091557 0.095514 0.099557 0.084565 0.088068
|
||||
6 0.0 0.070727 0.072900 0.079623 0.082116 0.084565 0.076371 0.078937
|
||||
7 0.0 0.073798 0.076302 0.082640 0.085368 0.088068 0.078937 0.081684
|
||||
8 0.0 0.077019 0.079892 0.085767 0.088751 0.091725 0.081567 0.084509
|
||||
9 0.0 0.080401 0.083686 0.089005 0.092271 0.095547 0.084258 0.087409
|
||||
10 0.0 0.062426 0.063699 0.072024 0.073913 0.075721 0.070361 0.072486
|
||||
11 0.0 0.065031 0.066507 0.074745 0.076802 0.078786 0.072797 0.075066
|
||||
12 0.0 0.067773 0.069474 0.077585 0.079825 0.082002 0.075324 0.077748
|
||||
13 0.0 0.070658 0.072612 0.080551 0.082990 0.085376 0.077944 0.080532
|
||||
14 0.0 0.073697 0.075932 0.083645 0.086302 0.088919 0.080656 0.083422
|
||||
1 0.0 0.062851 0.073533 0.065769 0.068538 0.069947 0.059538 0.060320
|
||||
2 0.0 0.073533 0.087625 0.078561 0.082559 0.084744 0.071705 0.072983
|
||||
3 0.0 0.065769 0.078561 0.072769 0.076696 0.079022 0.068407 0.069803
|
||||
4 0.0 0.068538 0.082559 0.076696 0.081211 0.083977 0.072511 0.074206
|
||||
5 0.0 0.069947 0.084744 0.079022 0.083977 0.087102 0.075127 0.077070
|
||||
6 0.0 0.059538 0.071705 0.068407 0.072511 0.075127 0.066156 0.067782
|
||||
7 0.0 0.060320 0.072983 0.069803 0.074206 0.077070 0.067782 0.069583
|
||||
8 0.0 0.060735 0.073737 0.070715 0.075353 0.078424 0.068928 0.070878
|
||||
9 0.0 0.060967 0.074217 0.071359 0.076188 0.079434 0.069792 0.071870
|
||||
10 0.0 0.052371 0.063245 0.061909 0.065807 0.068404 0.061247 0.062897
|
||||
11 0.0 0.052613 0.063714 0.062491 0.066554 0.069297 0.062003 0.063759
|
||||
12 0.0 0.052756 0.064027 0.062920 0.067118 0.069987 0.062592 0.064441
|
||||
13 0.0 0.052853 0.064260 0.063261 0.067574 0.070552 0.063079 0.065009
|
||||
14 0.0 0.052933 0.064453 0.063553 0.067966 0.071040 0.063502 0.065504
|
||||
|
||||
8 9 10 11 12 13 14
|
||||
0 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
|
||||
1 0.077019 0.080401 0.062426 0.065031 0.067773 0.070658 0.073697
|
||||
2 0.079892 0.083686 0.063699 0.066507 0.069474 0.072612 0.075932
|
||||
3 0.085767 0.089005 0.072024 0.074745 0.077585 0.080551 0.083645
|
||||
4 0.088751 0.092271 0.073913 0.076802 0.079825 0.082990 0.086302
|
||||
5 0.091725 0.095547 0.075721 0.078786 0.082002 0.085376 0.088919
|
||||
6 0.081567 0.084258 0.070361 0.072797 0.075324 0.077944 0.080656
|
||||
7 0.084509 0.087409 0.072486 0.075066 0.077748 0.080532 0.083422
|
||||
8 0.087542 0.090669 0.074644 0.077376 0.080221 0.083181 0.086261
|
||||
9 0.090669 0.094040 0.076827 0.079721 0.082739 0.085887 0.089170
|
||||
10 0.074644 0.076827 0.065759 0.067867 0.070042 0.072284 0.074590
|
||||
11 0.077376 0.079721 0.067867 0.070098 0.072403 0.074782 0.077234
|
||||
12 0.080221 0.082739 0.070042 0.072403 0.074845 0.077371 0.079979
|
||||
13 0.083181 0.085887 0.072284 0.074782 0.077371 0.080053 0.082828
|
||||
14 0.086261 0.089170 0.074590 0.077234 0.079979 0.082828 0.085781
|
||||
1 0.060735 0.060967 0.052371 0.052613 0.052756 0.052853 0.052933
|
||||
2 0.073737 0.074217 0.063245 0.063714 0.064027 0.064260 0.064453
|
||||
3 0.070715 0.071359 0.061909 0.062491 0.062920 0.063261 0.063553
|
||||
4 0.075353 0.076188 0.065807 0.066554 0.067118 0.067574 0.067966
|
||||
5 0.078424 0.079434 0.068404 0.069297 0.069987 0.070552 0.071040
|
||||
6 0.068928 0.069792 0.061247 0.062003 0.062592 0.063079 0.063502
|
||||
7 0.070878 0.071870 0.062897 0.063759 0.064441 0.065009 0.065504
|
||||
8 0.072303 0.073408 0.064109 0.065065 0.065830 0.066473 0.067035
|
||||
9 0.073408 0.074615 0.065054 0.066094 0.066934 0.067645 0.068268
|
||||
10 0.064109 0.065054 0.057780 0.058594 0.059248 0.059799 0.060281
|
||||
11 0.065065 0.066094 0.058594 0.059478 0.060193 0.060800 0.061333
|
||||
12 0.065830 0.066934 0.059248 0.060193 0.060965 0.061622 0.062202
|
||||
13 0.066473 0.067645 0.059799 0.060800 0.061622 0.062326 0.062949
|
||||
14 0.067035 0.068268 0.060281 0.061333 0.062202 0.062949 0.063612
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1341,10 +1359,10 @@ We can write our own code or simply use either the functionaly of <strong>numpy<
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0 1
|
||||
0 3.884269 1.937050
|
||||
1 1.937050 1.951359
|
||||
[[3.88426936 1.93705016]
|
||||
[1.93705016 1.95135877]]
|
||||
0 3.963873 1.979586
|
||||
1 1.979586 1.976796
|
||||
[[3.96387325 1.97958566]
|
||||
[1.97958566 1.97679551]]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1371,8 +1389,8 @@ Our own code here is not very elegant and asks for obvious improvements. It is t
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Centered covariance using own code
|
||||
[[3.88426936 1.93705016]
|
||||
[1.93705016 1.95135877]]
|
||||
[[3.96387325 1.97958566]
|
||||
[1.97958566 1.97679551]]
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter8_77_1.png" src="_images/chapter8_77_1.png" />
|
||||
@@ -1432,16 +1450,14 @@ questions.</p>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvalues of Covariance matrix
|
||||
5.082577137078103
|
||||
0.7530509884519185
|
||||
5.185256249406441
|
||||
0.7554125101643117
|
||||
First eigenvector
|
||||
[0.85042593 0.5260948 ]
|
||||
[0.85104821 0.52508756]
|
||||
Second eigenvector
|
||||
[-0.5260948 0.85042593]
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Eigenvector of largest eigenvalue
|
||||
[-0.85042593 -0.5260948 ]
|
||||
[-0.52508756 0.85104821]
|
||||
Eigenvector of largest eigenvalue
|
||||
[-0.85104821 -0.52508756]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -38,6 +38,7 @@
|
||||
<script async="async" src="_static/sphinx-thebe.js"></script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="9. Recurrent Neural Networks" href="chapter10.html" />
|
||||
<link rel="prev" title="7. Dimensionality Reduction" href="chapter8.html" />
|
||||
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
@@ -124,6 +125,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -731,88 +752,88 @@ labels = (n_inputs) = (1797,)
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 2.5817 - accuracy: 0.2500
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 3.0150 - accuracy: 0.1250
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 917us/step - loss: 2.8078 - accuracy: 0.1139
|
||||
12/12 [==============================] - 0s 958us/step - loss: 2.9441 - accuracy: 0.0722
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 1e-05
|
||||
Test accuracy: 0.114
|
||||
Test accuracy: 0.072
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 3.0782 - accuracy: 0.1875
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 3.3024 - accuracy: 0.0312
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 932us/step - loss: 3.3640 - accuracy: 0.1500
|
||||
12/12 [==============================] - 0s 957us/step - loss: 3.0967 - accuracy: 0.1389
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.0001
|
||||
Test accuracy: 0.150
|
||||
Test accuracy: 0.139
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 2.7005 - accuracy: 0.0312
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 2.9008 - accuracy: 0.2188
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 933us/step - loss: 2.7588 - accuracy: 0.0944
|
||||
12/12 [==============================] - 0s 1ms/step - loss: 3.4843 - accuracy: 0.1028
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.001
|
||||
Test accuracy: 0.094
|
||||
Test accuracy: 0.103
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 3.9994 - accuracy: 0.1250
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 3.7695 - accuracy: 0.1250
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 1ms/step - loss: 4.2041 - accuracy: 0.1028
|
||||
12/12 [==============================] - 0s 968us/step - loss: 3.6647 - accuracy: 0.1417
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.01
|
||||
Test accuracy: 0.103
|
||||
Test accuracy: 0.142
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 12.9177 - accuracy: 0.1250
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 12.4141 - accuracy: 0.0938
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 939us/step - loss: 12.8693 - accuracy: 0.1056
|
||||
12/12 [==============================] - 0s 960us/step - loss: 12.3084 - accuracy: 0.1028
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 0.1
|
||||
Test accuracy: 0.106
|
||||
Test accuracy: 0.103
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 92.4325 - accuracy: 0.1250
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 92.7441 - accuracy: 0.0938
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 2ms/step - loss: 92.4491 - accuracy: 0.1111
|
||||
12/12 [==============================] - 0s 959us/step - loss: 91.7224 - accuracy: 0.1306
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 1.0
|
||||
Test accuracy: 0.111
|
||||
Test accuracy: 0.131
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 514.5186 - accuracy: 0.0312
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 1/12 [=>............................] - ETA: 0s - loss: 514.1083 - accuracy: 0.0625
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
|
||||
12/12 [==============================] - 0s 939us/step - loss: 514.2589 - accuracy: 0.1028
|
||||
12/12 [==============================] - 0s 958us/step - loss: 513.9800 - accuracy: 0.1111
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Learning rate = 1e-05
|
||||
Lambda = 10.0
|
||||
Test accuracy: 0.103
|
||||
Test accuracy: 0.111
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -1679,6 +1700,7 @@ of recurrent neural networks (see the link above and also
|
||||
<div class='prev-next-bottom'>
|
||||
|
||||
<a class='left-prev' id="prev-link" href="chapter8.html" title="previous page"><span class="section-number">7. </span>Dimensionality Reduction</a>
|
||||
<a class='right-next' id="next-link" href="chapter10.html" title="next page"><span class="section-number">9. </span>Recurrent Neural Networks</a>
|
||||
|
||||
</div>
|
||||
<footer class="footer mt-5 mt-md-0">
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,39 @@
|
||||
Traceback (most recent call last):
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 604, in _async_poll_for_reply
|
||||
msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 85, in ensure_async
|
||||
result = await obj
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_client/asynchronous/channels.py", line 48, in get_msg
|
||||
raise Empty
|
||||
_queue.Empty
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution
|
||||
executenb(
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 1082, in execute
|
||||
return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
|
||||
return just_run(coro(*args, **kwargs))
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
|
||||
return loop.run_until_complete(coro)
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
|
||||
return future.result()
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 535, in async_execute
|
||||
await self.async_execute_cell(
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 811, in async_execute_cell
|
||||
exec_reply = await self.task_poll_for_reply
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 628, in _async_poll_for_reply
|
||||
await self._async_handle_timeout(timeout, cell)
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 678, in _async_handle_timeout
|
||||
raise CellTimeoutError.error_from_timeout_and_cell(
|
||||
nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 30 seconds.
|
||||
The message was: Cell execution timed out.
|
||||
Here is a preview of the cell contents:
|
||||
-------------------
|
||||
['import autograd.numpy as np', 'from autograd import grad, elementwise_grad', 'import autograd.numpy.random as npr', 'from matplotlib import pyplot as plt', '']
|
||||
...
|
||||
[' plt.plot(x, res[0,:])', " plt.legend(['analytical','nn'])", " plt.xlabel('x')", " plt.ylabel('g(x)')", ' plt.show()']
|
||||
-------------------
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
Traceback (most recent call last):
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution
|
||||
executenb(
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 1082, in execute
|
||||
return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 74, in wrapped
|
||||
return just_run(coro(*args, **kwargs))
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 53, in just_run
|
||||
return loop.run_until_complete(coro)
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
|
||||
return future.result()
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 535, in async_execute
|
||||
await self.async_execute_cell(
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 827, in async_execute_cell
|
||||
self._check_raise_for_error(cell, exec_reply)
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 735, in _check_raise_for_error
|
||||
raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
|
||||
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
|
||||
------------------
|
||||
np.random.seed(999) # for reproducibility
|
||||
a=0.6 # biased coin
|
||||
flips=np.random.rand(2**12) # simulates 4096 coin flips
|
||||
heads=flips<a # boolean array, heads[i]=True if flip i is heads
|
||||
------------------
|
||||
|
||||
[0;31m---------------------------------------------------------------------------[0m
|
||||
[0;31mNameError[0m Traceback (most recent call last)
|
||||
[0;32m<ipython-input-1-3f1158901148>[0m in [0;36m<module>[0;34m[0m
|
||||
[0;32m----> 1[0;31m [0mnp[0m[0;34m.[0m[0mrandom[0m[0;34m.[0m[0mseed[0m[0;34m([0m[0;36m999[0m[0;34m)[0m [0;31m# for reproducibility[0m[0;34m[0m[0;34m[0m[0m
|
||||
[0m[1;32m 2[0m [0ma[0m[0;34m=[0m[0;36m0.6[0m [0;31m# biased coin[0m[0;34m[0m[0;34m[0m[0m
|
||||
[1;32m 3[0m [0mflips[0m[0;34m=[0m[0mnp[0m[0;34m.[0m[0mrandom[0m[0;34m.[0m[0mrand[0m[0;34m([0m[0;36m2[0m[0;34m**[0m[0;36m12[0m[0;34m)[0m [0;31m# simulates 4096 coin flips[0m[0;34m[0m[0;34m[0m[0m
|
||||
[1;32m 4[0m [0mheads[0m[0;34m=[0m[0mflips[0m[0;34m<[0m[0ma[0m [0;31m# boolean array, heads[i]=True if flip i is heads[0m[0;34m[0m[0;34m[0m[0m
|
||||
|
||||
[0;31mNameError[0m: name 'np' is not defined
|
||||
NameError: name 'np' is not defined
|
||||
|
||||
@@ -1,14 +1,3 @@
|
||||
Traceback (most recent call last):
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 604, in _async_poll_for_reply
|
||||
msg = await ensure_async(self.kc.shell_channel.get_msg(timeout=new_timeout))
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/util.py", line 85, in ensure_async
|
||||
result = await obj
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_client/asynchronous/channels.py", line 48, in get_msg
|
||||
raise Empty
|
||||
_queue.Empty
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/jupyter_cache/executors/utils.py", line 51, in single_nb_execution
|
||||
executenb(
|
||||
@@ -22,18 +11,19 @@ Traceback (most recent call last):
|
||||
return future.result()
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 535, in async_execute
|
||||
await self.async_execute_cell(
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 811, in async_execute_cell
|
||||
exec_reply = await self.task_poll_for_reply
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 628, in _async_poll_for_reply
|
||||
await self._async_handle_timeout(timeout, cell)
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 678, in _async_handle_timeout
|
||||
raise CellTimeoutError.error_from_timeout_and_cell(
|
||||
nbclient.exceptions.CellTimeoutError: A cell timed out while it was being executed, after 30 seconds.
|
||||
The message was: Cell execution timed out.
|
||||
Here is a preview of the cell contents:
|
||||
-------------------
|
||||
['eta_vals = np.logspace(-5, 1, 7)', 'lmbd_vals = np.logspace(-5, 1, 7)', '# store the models for later use', 'DNN_numpy = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)', '']
|
||||
...
|
||||
[' ', ' print("Learning rate = ", eta)', ' print("Lambda = ", lmbd)', ' print("Accuracy score on test set: ", accuracy_score(Y_test, test_predict))', ' print()']
|
||||
-------------------
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 827, in async_execute_cell
|
||||
self._check_raise_for_error(cell, exec_reply)
|
||||
File "/Users/hjensen/opt/anaconda3/lib/python3.8/site-packages/nbclient/client.py", line 735, in _check_raise_for_error
|
||||
raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
|
||||
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
|
||||
------------------
|
||||
pip3 install tensorflow
|
||||
------------------
|
||||
|
||||
[0;36m File [0;32m"<ipython-input-13-6ea927cc6e88>"[0;36m, line [0;32m1[0m
|
||||
[0;31m pip3 install tensorflow[0m
|
||||
[0m ^[0m
|
||||
[0;31mSyntaxError[0m[0;31m:[0m invalid syntax
|
||||
|
||||
SyntaxError: invalid syntax (<ipython-input-13-6ea927cc6e88>, line 1)
|
||||
|
||||
|
||||
@@ -128,6 +128,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -1553,8 +1573,8 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-1.17029256 1.16440383 -0.52324284 0.91883189 -0.05688344 -0.80125589
|
||||
1.85759035 -1.11063616 0.67761635 0.09188807]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[ 1.8413241 -1.44931127 0.91189378 1.82553587 0.69940044 -0.65726179
|
||||
1.72526513 1.82825747 -1.34003257 -0.5817835 ]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
+20
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
@@ -1553,8 +1573,8 @@ developed in the 1970s, namely EISPACK and LINPACK. We describe them shortly he
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-0.15903306 -1.03373868 0.31582494 -0.65395262 0.96308377 -0.89760255
|
||||
1.62466988 1.27946914 -0.71331923 -0.2179017 ]
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>[-1.04569783 1.68833083 1.59472197 -0.22027464 -1.12477388 0.78480985
|
||||
0.8799874 -0.43301111 -0.53140431 0.2976373 ]
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../../../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -123,6 +123,26 @@
|
||||
8. Convolutional Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html">
|
||||
9. Recurrent Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter10.html#solving-odes-with-deep-learning">
|
||||
10. Solving ODEs with Deep Learning
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html">
|
||||
11. Data Analysis and Machine Learning:
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="../chapter11.html#elements-of-bayesian-theory-and-bayesian-neural-networks">
|
||||
12. Elements of Bayesian theory and Bayesian Neural Networks
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</nav>
|
||||
|
||||
@@ -416,6 +416,8 @@
|
||||
"chapter7.ipynb\n",
|
||||
"chapter8.ipynb\n",
|
||||
"chapter9.ipynb\n",
|
||||
"chapter10.ipynb\n",
|
||||
"chapter11.ipynb\n",
|
||||
"```\n"
|
||||
]
|
||||
}
|
||||
|
||||
@@ -405,4 +405,6 @@ chapter6.ipynb
|
||||
chapter7.ipynb
|
||||
chapter8.ipynb
|
||||
chapter9.ipynb
|
||||
chapter10.ipynb
|
||||
chapter11.ipynb
|
||||
```
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 19 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 31 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 13 KiB |
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,712 @@
|
||||
<!-- dom:TITLE: Data Analysis and Machine Learning: -->
|
||||
# Data Analysis and Machine Learning:
|
||||
<!-- dom:AUTHOR: Christian Forssén at Department of Physics, Chalmers University of Technology, Sweden -->
|
||||
<!-- Author: -->
|
||||
**Christian Forssén**, Department of Physics, Chalmers University of Technology, Sweden
|
||||
<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->
|
||||
<!-- Author: --> **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
|
||||
|
||||
Date: **Dec 23, 2020**
|
||||
|
||||
Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
|
||||
|
||||
|
||||
|
||||
|
||||
# Elements of Bayesian theory and Bayesian Neural Networks
|
||||
|
||||
|
||||
## Why Bayesian Statistics?
|
||||
|
||||
We have already made ourselves familiar with elements of a statistical
|
||||
data analysis via quantities like the bias-variance tradeoff as well
|
||||
as some central distribution functions such as the Normal
|
||||
distribution, the binomial distribution and other probability
|
||||
distribution functions.
|
||||
|
||||
In essentially all the Machine Learning algorithms we have studied,
|
||||
our focus has been on a so-called **frequentist approach**, where
|
||||
knowledge of an underlying likelihood function has not been
|
||||
emphasized. Our data, whether we had a classification or a regression
|
||||
problem, have been our central points of departure.
|
||||
|
||||
Here we wish to merge this approach with the derivation of a
|
||||
likelihood function which can be used to make prediction on how our
|
||||
system under study evolves. We will venture into the realm of what is
|
||||
called Bayesian Neural Networks. To get an overarching view on what
|
||||
this entails, the following figure conveys the essential differences
|
||||
between a standard Neural network that we have met earlier and a
|
||||
Bayesian Neural Network. In order to get there, we need to present
|
||||
some of the basic elements of Bayesian statistics, starting with the
|
||||
product rule and Bayes' theorem.
|
||||
|
||||
|
||||
|
||||
|
||||
## Inference
|
||||
Inference:
|
||||
:
|
||||
"the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former" (Webster)
|
||||
Do premises $A, B, \ldots \to$ hypothesis, $H$?
|
||||
|
||||
Deductive inference:
|
||||
:
|
||||
Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
|
||||
$B(H|A,B,...) = 0$ or $1$
|
||||
|
||||
Inductive inference:
|
||||
:
|
||||
Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
|
||||
$A, B, C, D$ share properties $x, y, z$; $E$ has properties $x, y$
|
||||
$\to$ $E$ probably has property $z$.
|
||||
|
||||
|
||||
|
||||
|
||||
## Statistical Inference
|
||||
* Quantify the strength of inductive inferences from facts, in the form of data ($D$), and other premises, e.g. models, to hypotheses about the phenomena producing the data.
|
||||
|
||||
* Quantify via probabilities, or averages calculated using probabilities. Frequentists ($\mathcal{F}$) and Bayesians ($\mathcal{B}$) use probabilities very differently for this.
|
||||
|
||||
* To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a *degree-of-belief* or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.
|
||||
|
||||
* To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the *long run relative frequency* with which an event occurred, given (infinitely) many repeated (experimental) trials.
|
||||
|
||||
|
||||
|
||||
|
||||
## Some history
|
||||
Adapted from D.S. Sivia[^Sivia]:
|
||||
|
||||
[^Sivia]: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006
|
||||
|
||||
> Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2].
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- dom:FIGURE: [fig/sivia_fig_1_2.png, width=700 frac=0.9] -->
|
||||
<!-- begin figure -->
|
||||
|
||||
<p></p>
|
||||
<img src="fig/sivia_fig_1_2.png" width=700>
|
||||
|
||||
<!-- end figure -->
|
||||
|
||||
|
||||
|
||||
> To Laplace, the (shaded) area under the posterior pdf curve between $m_1$ and $m_2$ was a measure of how much he believed that the mass of Saturn lay in the range $m_1 \le M \le m_2$. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%!
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.
|
||||
>
|
||||
> If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.
|
||||
> For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
> Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## The Bayesian recipe
|
||||
Assess hypotheses by calculating their probabilities $p(H_i | \ldots)$ conditional on known and/or presumed information using the rules of probability theory.
|
||||
|
||||
|
||||
Probability Theory Axioms:
|
||||
Product (AND) rule :
|
||||
:
|
||||
$p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)$
|
||||
Should read $p(A,B|I)$ as the probability for propositions $A$ AND $B$ being true given that $I$ is true.
|
||||
|
||||
Sum (OR) rule:
|
||||
:
|
||||
$p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)$
|
||||
$p(A+B|I)$ is the probability that proposition $A$ OR $B$ is true given that $I$ is true.
|
||||
|
||||
Normalization:
|
||||
:
|
||||
$p(A|I) + p(\bar{A}|I) = 1$
|
||||
$\bar{A}$ denotes the proposition that $A$ is false.
|
||||
|
||||
|
||||
|
||||
|
||||
## Bayes' theorem
|
||||
Bayes' theorem follows directly from the product rule
|
||||
|
||||
$$
|
||||
$$
|
||||
p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}.
|
||||
$$
|
||||
$$
|
||||
|
||||
The importance of this property to data analysis becomes apparent if we replace $A$ and $B$ by hypothesis($H$) and data($D$):
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="eq:bayes"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}.
|
||||
\label{eq:bayes} \tag{1}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.
|
||||
|
||||
|
||||
|
||||
|
||||
The various terms in Bayes’ theorem have formal names.
|
||||
* The quantity on the far right, $p(H|I)$, is called the *prior* probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data.
|
||||
|
||||
* This is modified by the experimental measurements through $p(D|H,I)$, the *likelihood* function,
|
||||
|
||||
* The denominator $p(D|I)$ is called the *evidence*. It does not depend on the hypothesis and can be regarded as a normalization constant.
|
||||
|
||||
* Together, these yield the *posterior* probability, $p(H|D, I )$, representing our state of knowledge about the truth of the hypothesis in the light of the data.
|
||||
|
||||
In a sense, Bayes’ theorem encapsulates the process of learning.
|
||||
|
||||
|
||||
|
||||
|
||||
## The friends of Bayes' theorem
|
||||
Normalization:
|
||||
:
|
||||
$\sum_i p(H_i|\ldots) = 1$.
|
||||
|
||||
Marginalization:
|
||||
:
|
||||
$\sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I)$.
|
||||
|
||||
Marginalization (continuum limit):
|
||||
:
|
||||
$\int dx p(A,H(x)|I) = p(A|I)$.
|
||||
|
||||
In the above, $H_i$ is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then $H_1$ could be the proposition that the first candidate will win, and so on. The probability that $A$ is true, for example that unemployment will be lower in a year’s time (given all relevant information $I$, but irrespective of whoever becomes president) is then given by $\sum_i p(A,H_i|I)$.
|
||||
|
||||
In the continuum limit of propositions we must understand $p(\ldots)$ as a pdf (probability density function).
|
||||
|
||||
Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.
|
||||
|
||||
|
||||
|
||||
|
||||
## Inference With Parametric Models
|
||||
Inductive inference with parametric models is a very important tool in the natural sciences.
|
||||
* Consider $N$ different models $M_i$ ($i = 1, \ldots, N$), each with parameters $\boldsymbol{\alpha}_i$. Each of them implies a sampling distribution (conditional predictive distribution for possible data)
|
||||
|
||||
$$
|
||||
$$
|
||||
p(D|\boldsymbol{\alpha}_i, M_i)
|
||||
$$
|
||||
$$
|
||||
|
||||
* The $\boldsymbol{\alpha}_i$ dependence when we fix attention on the actual, observed data ($D_\mathrm{obs}$) is the likelihood function:
|
||||
|
||||
$$
|
||||
$$
|
||||
\mathcal{L}_i (\boldsymbol{\alpha}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\alpha}_i, M_i)
|
||||
$$
|
||||
$$
|
||||
|
||||
* We may be uncertain about $i$ (model uncertainty),
|
||||
|
||||
* or uncertain about $\boldsymbol{\alpha}_i$ (parameter uncertainty).
|
||||
|
||||
|
||||
|
||||
|
||||
Parameter Estimation:
|
||||
:
|
||||
Premise = choice of model (pick specific $i$)
|
||||
$\Rightarrow$ What can we say about $\boldsymbol{\alpha}_i$?
|
||||
|
||||
Model comparison:
|
||||
:
|
||||
Premise = $\{M_i\}$
|
||||
$\Rightarrow$ What can we say about $i$?
|
||||
|
||||
Model adequacy:
|
||||
:
|
||||
Premise = $M_1$
|
||||
$\Rightarrow$ Is $M_1$ adequate?
|
||||
|
||||
Hybrid Uncertainty:
|
||||
:
|
||||
Models share some common params: $\boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\}$
|
||||
$\Rightarrow$ What can we say about $\boldsymbol{\varphi}$? (Systematic error is an example)
|
||||
|
||||
|
||||
|
||||
|
||||
## Illustrative examples with python code
|
||||
* Is this a fair coin? (analytical)
|
||||
|
||||
* Flux from a star (single parameter, MCMC)
|
||||
|
||||
* The lighthouse problem (two parameters, MCMC)
|
||||
|
||||
* Linear fit with outliers (nuisance parameters)
|
||||
|
||||
* ...
|
||||
|
||||
|
||||
|
||||
|
||||
## Example: Is this a fair coin?
|
||||
Let us begin with the analysis of data from a simple coin-tossing experiment.
|
||||
Given that we had observed 6 heads in 8 flips, would you think it was a fair coin? By fair, we mean that we would be prepared to lay an even 1 : 1 bet on the outcome of a flip being a head or a tail. If we decide that the coin was fair, the question which follows naturally is how sure are we that this was so; if it was not fair, how unfair do we think it was? Furthermore, if we were to continue collecting data for this particular coin, observing the outcomes of additional flips, how would we update our belief on the fairness of the coin?
|
||||
|
||||
A sensible way of formulating this problem is to consider a large number of hypotheses about the range in which the bias-weighting of the coin might lie. If we denote the bias-weighting by $H$, then $H = 0$ and $H = 1$ can represent a coin which produces a tail or a head on every flip, respectively. There is a continuum of possibilities for the value of H between these limits, with $H = 0.5$ indicating a fair coin. Our state of knowledge about the fairness, or the degree of unfairness, of the coin is then completely summarized by specifying how much we believe these various propositions to be true.
|
||||
|
||||
Let us perform a computer simulation of a coin-tossing experiment. This provides the data that we will be analysing.
|
||||
|
||||
0
|
||||
|
||||
<
|
||||
<
|
||||
<
|
||||
!
|
||||
!
|
||||
C
|
||||
O
|
||||
D
|
||||
E
|
||||
_
|
||||
B
|
||||
L
|
||||
O
|
||||
C
|
||||
K
|
||||
|
||||
|
||||
p
|
||||
y
|
||||
c
|
||||
o
|
||||
d
|
||||
|
||||
np.random.seed(999) # for reproducibility
|
||||
a=0.6 # biased coin
|
||||
flips=np.random.rand(2**12) # simulates 4096 coin flips
|
||||
heads=flips<a # boolean array, heads[i]=True if flip i is heads
|
||||
|
||||
In the light of this data, our inference about the fairness of this coin is summarized by the conditional pdf: $p(H|D,I)$. This is, of course, shorthand for the limiting case of a continuum of propositions for the value of $H$; that is to say, the probability that $H$ lies in an infinitesimally narrow range is given by $p(H|D,I) dH$.
|
||||
|
||||
To estimate this posterior pdf, we need to use Bayes’ theorem ([1](#eq:bayes)). We will ignore the denominator $p(D|I)$ as it does not involve bias-weighting explicitly, and it will therefore not affect the shape of the desired pdf. At the end we can evaluate the missing constant subsequently from the normalization condition
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="eq:coin_posterior_norm"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
\int_0^1 p(H|D,I) dH = 1.
|
||||
\label{eq:coin_posterior_norm} \tag{2}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
The prior pdf, $p(H|I)$, represents what we know about the coin given only the information $I$ that we are dealing with a ‘strange coin’. We could keep a very open mind about the nature of the coin; a simple probability assignment which reflects this is a uniform, or flat, prior
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="eq:coin_prior_uniform"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
p(H|I) = \left\{ \begin{array}{ll}
|
||||
1 & 0 \le H \le 1, \\
|
||||
0 & \mathrm{otherwise}.
|
||||
\end{array} \right.
|
||||
\label{eq:coin_prior_uniform} \tag{3}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
We will get back later to the choice of prior and its effect on the analysis.
|
||||
|
||||
This prior state of knowledge, or ignorance, is modified by the data through the likelihood function $p(D|H,I)$. It is a measure of the chance that we would have obtained the data that we actually observed, if the value of the bias-weighting was given (as known). If, in the conditioning information $I$, we assume that the flips of the coin were independent events, so that the outcome of one did not influence that of another, then the probability of obtaining the data `R heads in N tosses' is given by the binomial distribution (we leave a formal definition of this to a statistics textbook)
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto1"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
p(D|H,I) \propto H^R (1-H)^{N-R}.
|
||||
\label{_auto1} \tag{4}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
It seems reasonable because $H$ is the chance of obtaining a head on any flip, and there were $R$ of them, and $1-H$ is the corresponding probability for a tail, of which there were $N-R$. We note that this binomial distribution also contains a normalization factor, but we will ignore it since it does not depend explicitly on $H$, the quantity of interest. It will be absorbed by the normalization condition ([2](#eq:coin_posterior_norm)).
|
||||
|
||||
We perform the setup of this Bayesian framework on the computer.
|
||||
|
||||
def prior(H):
|
||||
p=np.zeros_like(H)
|
||||
p[(0<=x)&(x<=1)]=1 # allowed range: 0<=H<=1
|
||||
return p # uniform prior
|
||||
def likelihood(H,data):
|
||||
N = len(data)
|
||||
no_of_heads = sum(data)
|
||||
no_of_tails = N - no_of_heads
|
||||
return H**no_of_heads * (1-H)**no_of_tails
|
||||
def posterior(H,data):
|
||||
p=prior(H)*likelihood(H,data)
|
||||
norm=np.trapz(p,H)
|
||||
return p/norm
|
||||
|
||||
The next step is to confront this setup with the simulated data. To get a feel for the result, it is instructive to see how the posterior pdf evolves as we obtain more and more data pertaining to the coin. The results of such an analyses is shown in Fig. [fig:coinflipping](#fig:coinflipping).
|
||||
|
||||
x=np.linspace(0,1,100)
|
||||
fig, axs = plt.subplots(nrows=4,ncols=3,sharex=True,sharey='row')
|
||||
axs_vec=np.reshape(axs,-1)
|
||||
axs_vec[0].plot(x,prior(x))
|
||||
for ndouble in range(11):
|
||||
ax=axs_vec[1+ndouble]
|
||||
ax.plot(x,posterior(x,heads[:2**ndouble]))
|
||||
ax.text(0.1, 0.8, '$N={0}$'.format(2**ndouble), transform=ax.transAxes)
|
||||
for row in range(4): axs[row,0].set_ylabel('$p(H|D_\mathrm{obs},I)$')
|
||||
for col in range(3): axs[-1,col].set_xlabel('$H$')
|
||||
|
||||
<!-- dom:FIGURE:[fig/coinflipping_fig_1.png, width=500 frac=0.95] The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis. <div id="fig:coinflipping"></div> -->
|
||||
<!-- begin figure -->
|
||||
<div id="fig:coinflipping"></div>
|
||||
|
||||
<p>The evolution of the posterior pdf for the bias-weighting of a coin, as the number of data available increases. The figure on the top left-hand corner of each panel shows the number of data included in the analysis.</p>
|
||||
<img src="fig/coinflipping_fig_1.png" width=500>
|
||||
|
||||
<!-- end figure -->
|
||||
|
||||
|
||||
The panel in the top left-hand corner shows the posterior pdf for $H$ given no data, i.e., it is the same as the prior pdf of Eq. ([3](#eq:coin_prior_uniform)). It indicates that we have no more reason to believe that the coin is fair than we have to think that it is double-headed, double-tailed, or of any other intermediate bias-weighting.
|
||||
|
||||
The first flip is obviously tails. At this point we have no evidence that the coin has a side with heads, as indicated by the pdf going to zero as $H \to 1$. The second flip is obviously heads and we have now excluded both extreme options $H=0$ (double-tailed) and $H=1$ (double-headed). We can note that the posterior at this point has the simple form $p(H|D,I) = H(1-H)$ for $0 \le H \le 1$.
|
||||
|
||||
The remainder of Fig. [fig:coinflipping](#fig:coinflipping) shows how the posterior pdf evolves as the number of data analysed becomes larger and larger. We see that the position of the maximum moves around, but that the amount by which it does so decreases with the increasing number of observations. The width of the posterior pdf also becomes narrower with more data, indicating that we are becoming increasingly confident in our estimate of the bias-weighting. For the coin in this example, the best estimate of $H$ eventually converges to 0.6, which, of course, was the value chosen to simulate the flips.
|
||||
|
||||
|
||||
## A few words on different priors
|
||||
* uniform
|
||||
|
||||
* Gaussian
|
||||
|
||||
* Jeffrey's prior
|
||||
|
||||
Repeat the coin flipping experiment with other priors.
|
||||
|
||||
|
||||
## Bayesian parameter estimation (single parameter)
|
||||
We will now consider the very important task of model parameter estimation using statistical inference.
|
||||
[CF 1: maybe stress that model parameters are not random variables, and the meaning of parameter estimation is therefore very different between frequentist and bayesian approaches.]
|
||||
|
||||
Throughout this section we will consider a specific example that involves a model with a single parameter: "Measured flux from a star".
|
||||
|
||||
|
||||
|
||||
|
||||
### Example: Measured flux from a star
|
||||
|
||||
Adapted from the blog [Pythonic Perambulations](http://jakevdp.github.io) by Jake VanderPlas.
|
||||
|
||||
Imagine that we point our telescope to the sky, and observe the light coming from a single star. For the time being, we'll assume that the star's true flux is constant with time, i.e. that is it has a fixed value $F_\mathrm{true}$ (we'll also ignore effects like sky noise and other sources of systematic error). We'll assume that we perform a series of $N$ measurements with our telescope, where the ith measurement reports the observed photon flux $F_i$ and error $e_i$[^errors].
|
||||
The question is, given this set of measurements $D = \{F_i, e_i\}$, what is our best estimate of the true flux $F_\mathrm{true}$?
|
||||
|
||||
[^errors]: We'll make the reasonable assumption that errors are Gaussian. In a Frequentist perspective, $e_i$ is the standard deviation of the results of a single measurement event in the limit of repetitions of *that event*. In the Bayesian perspective, $e_i$ is the standard deviation of the (Gaussian) probability distribution describing our knowledge of that particular measurement given its observed value.
|
||||
|
||||
Because the measurements are number counts, a Poisson distribution is a good approximation to the measurement process:
|
||||
|
||||
np.random.seed(1) # for repeatability
|
||||
F_true = 1000 # true flux, say number of photons measured in 1 second
|
||||
N = 50 # number of measurements
|
||||
F = stats.poisson(F_true).rvs(N)
|
||||
# N measurements of the flux
|
||||
e = np.sqrt(F) # errors on Poisson counts estimated via square root
|
||||
|
||||
Now let's make a simple visualization of the "observed" data, see Fig. [fig:flux](#fig:flux).
|
||||
|
||||
fig, ax = plt.subplots()
|
||||
ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
|
||||
ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
|
||||
ax.set_xlabel("Flux");ax.set_ylabel("measurement number");
|
||||
|
||||
<!-- dom:FIGURE:[fig/singlephotoncount_fig_1.png, width=400 frac=0.8] Single photon counts (flux measurements). <div id="fig:flux"></div> -->
|
||||
<!-- begin figure -->
|
||||
<div id="fig:flux"></div>
|
||||
|
||||
<p>Single photon counts (flux measurements).</p>
|
||||
<img src="fig/singlephotoncount_fig_1.png" width=400>
|
||||
|
||||
<!-- end figure -->
|
||||
|
||||
|
||||
These measurements each have a different error $e_i$ which is estimated from Poisson statistics using the standard square-root rule. In this toy example we already know the true flux $F_\mathrm{true}$, but the question is this: given our measurements and errors, what is our best estimate of the true flux?
|
||||
|
||||
Let's take a look at the frequentist and Bayesian approaches to solving this.
|
||||
|
||||
### Simple Photon Counts: Frequentist Approach
|
||||
|
||||
We'll start with the classical frequentist maximum likelihood approach. Given a single observation $D_i = (F_i, e_i)$, we can compute the probability distribution of the measurement given the true flux Ftrue given our assumption of Gaussian errors
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto2"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
p(D_i | F_\mathrm{true}, I) = \frac{1}{\sqrt{2\pi e_i^2}} \exp \left( \frac{-(F_i-F_\mathrm{true})^2}{2e_i^2} \right).
|
||||
\label{_auto2} \tag{5}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
This should be read "the probability of $D_i$ given $F_\mathrm{true}$
|
||||
equals ...". You should recognize this as a normal distribution with mean $F_\mathrm{true}$ and standard deviation $e_i$.
|
||||
|
||||
We construct the *likelihood function* by computing the product of the probabilities for each data point
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto3"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
\mathcal{L}(D | F_\mathrm{true}, I) = \prod_{i=1}^N p(D_i | F_\mathrm{true}, I),
|
||||
\label{_auto3} \tag{6}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
here $D = \{D_i\}$ represents the entire set of measurements. Because the value of the likelihood can become very small, it is often more convenient to instead compute the log-likelihood. Combining the previous two equations and computing the log, we have
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto4"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
\log\mathcal{L} = -\frac{1}{2} \sum_{i=1}^N \left[ \log(2\pi e_i^2) + \frac{(F_i-F_\mathrm{true})^2}{e_i^2} \right].
|
||||
\label{_auto4} \tag{7}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
What we'd like to do is determine $F_\mathrm{true}$ such that the likelihood is maximized. For this simple problem, the maximization can be computed analytically (i.e. by setting $d\log\mathcal{L}/d F_\mathrm{true} = 0$). This results in the following observed estimate of $F_\mathrm{true}$
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto5"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
F_\mathrm{est} = \sum_{i=1}^N w_i F_i; \quad w_i = 1/e_i^2.
|
||||
\label{_auto5} \tag{8}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
Notice that in the special case of all errors $e_i$ being equal, this reduces to
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto6"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
F_\mathrm{est} = \frac{1}{N} \sum_{i=1} F_i.
|
||||
\label{_auto6} \tag{9}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
That is, in agreement with intuition, $F_\mathrm{est}$ is simply the mean of the observed data when errors are equal.
|
||||
|
||||
We can go further and ask what the error of our estimate is. In the frequentist approach, this can be accomplished by fitting a Gaussian approximation to the likelihood curve at maximum; in this simple case this can also be solved analytically (the sum of Gaussians is also a Gaussian). It can be shown that the standard deviation of this Gaussian approximation is
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto7"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
\sigma_\mathrm{est} = \sum_{i=1}^N w_i.
|
||||
\label{_auto7} \tag{10}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
These results are fairly simple calculations; let's evaluate them for our toy dataset:
|
||||
|
||||
w=1./e**2
|
||||
print("""
|
||||
F_true = {0}
|
||||
F_est = {1:.0f} +/- {2:.0f} (based on {3} measurements) """\
|
||||
.format(F_true, (w * F).sum() / w.sum(), w.sum() ** -0.5, N))
|
||||
|
||||
`F_true = 1000`
|
||||
`F_est = 998 +/- 4 (based on 50 measurements)`
|
||||
|
||||
We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.
|
||||
|
||||
|
||||
### Simple Photon Counts: Bayesian Approach
|
||||
|
||||
The Bayesian approach, as you might expect, begins and ends with probabilities. Our hypothesis is that the star has a constant flux $F_\mathrm{true}$. It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question given the data and other information (such as our knowledge of uncertainties for the observed values), i.e. in this case, $p(F_\mathrm{true} | D,I)$.
|
||||
Note that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like $F_\mathrm{true}$. Nevertheless, within the Bayesian philosophy this is perfectly acceptable.
|
||||
|
||||
To compute this result, Bayesians next apply Bayes' Theorem ([1](#eq:bayes)).
|
||||
If we set the prior $p(F_\mathrm{true}|I) \propto 1$ (a flat prior), we find
|
||||
$p(F_\mathrm{true}|D,I) \propto p(D | F_\mathrm{true},I) \equiv \mathcal{L}(D | F_\mathrm{true},I)$
|
||||
and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.
|
||||
|
||||
### A note about priors
|
||||
|
||||
The prior allows inclusion of other information into the computation, which becomes very useful in cases where multiple measurement strategies are being combined to constrain a single model. The necessity to specify a prior, however, is one of the more controversial pieces of Bayesian analysis.
|
||||
A frequentist will point out that the prior is problematic when no true prior information is available. Though it might seem straightforward to use a noninformative prior like the flat prior mentioned above, there are some [surprisingly subtleties](http://normaldeviate.wordpress.com/2013/07/13/lost-causes-in-statistics-ii-noninformative- priors/comment-page-1/) involved. It turns out that in many situations, a truly noninformative prior does not exist! Frequentists point out that the subjective choice of a prior which necessarily biases your result has no place in statistical data analysis.
|
||||
A Bayesian would counter that frequentism doesn't solve this problem, but simply skirts the question. Frequentism can often be viewed as simply a special case of the Bayesian approach for some (implicit) choice of the prior: a Bayesian would say that it's better to make this implicit choice explicit, even if the choice might include some subjectivity.
|
||||
|
||||
### Simple Photon Counts: Bayesian approach in practice
|
||||
|
||||
Leaving these philosophical debates aside for the time being, let's address how Bayesian results are generally computed in practice. For a one parameter problem like the one considered here, it's as simple as computing the posterior probability $p(F_\mathrm{true} | D,I)$ as a function of $F_\mathrm{true}$: this is the distribution reflecting our knowledge of the parameter $F_\mathrm{true}$.
|
||||
But as the dimension of the model grows, this direct approach becomes increasingly intractable. For this reason, Bayesian calculations often depend on sampling methods such as Markov Chain Monte Carlo (MCMC). For this practical example, let us apply an MCMC approach using Dan Foreman-Mackey's [emcee](http://dan.iel.fm/emcee/current/) package. Keep in mind here that the goal is to generate a set of points drawn from the posterior probability distribution, and to use those points to determine the answer we seek.
|
||||
To perform this MCMC, we start by defining Python functions for the prior $p(F_\mathrm{true} | I)$, the likelihood $p(D | F_\mathrm{true},I)$, and the posterior $p(F_\mathrm{true} | D,I)$, noting that none of these need be properly normalized. Our model here is one-dimensional, but to handle multi-dimensional models we'll define the model in terms of an array of parameters $\boldsymbol{\alpha}$, which in this case is $\boldsymbol{\alpha} = [F_\mathrm{true}]$
|
||||
|
||||
def log_prior(alpha):
|
||||
return 0 # flat prior
|
||||
|
||||
def log_likelihood(alpha, F, e):
|
||||
return -0.5 * np.sum(np.log(2 * np.pi * e ** 2) \
|
||||
+ (F - alpha[0]) ** 2 / e ** 2)
|
||||
|
||||
def log_posterior(alpha, F, e):
|
||||
return log_prior(alpha) + log_likelihood(alpha, F, e)
|
||||
|
||||
Now we set up the problem, including generating some random starting guesses for the multiple chains of points.
|
||||
|
||||
ndim = 1 # number of parameters in the model
|
||||
nwalkers = 50 # number of MCMC walkers
|
||||
nburn = 1000 # "burn-in" period to let chains stabilize
|
||||
nsteps = 2000 # number of MCMC steps to take
|
||||
# we'll start at random locations between 0 and 2000
|
||||
starting_guesses = 2000 * np.random.rand(nwalkers, ndim)
|
||||
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F,e])
|
||||
sampler.run_mcmc(starting_guesses, nsteps)
|
||||
# Shape of sampler.chain = (nwalkers, nsteps, ndim)
|
||||
# Flatten the sampler chain and discard burn-in points:
|
||||
samples = sampler.chain[:, nburn:, :].reshape((-1, ndim))
|
||||
|
||||
If this all worked correctly, the array sample should contain a series of 50,000 points drawn from the posterior. Let's plot them and check. See results in Fig. [fig:flux-bayesian](#fig:flux-bayesian).
|
||||
|
||||
fig, ax = plt.subplots()
|
||||
ax.hist(samples, bins=50, histtype="stepfilled", alpha=0.3, normed=True)
|
||||
ax.set_xlabel(r'$F_\mathrm{est}$')
|
||||
ax.set_ylabel(r'$p(F_\mathrm{est}|D,I)$')
|
||||
|
||||
<!-- dom:FIGURE:[fig/singlephotoncount_fig_2.png, width=400 frac=0.8] Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements. <div id="fig:flux-bayesian"></div> -->
|
||||
<!-- begin figure -->
|
||||
<div id="fig:flux-bayesian"></div>
|
||||
|
||||
<p>Bayesian posterior pdf (represented by a histogram of MCMC samples) from flux measurements.</p>
|
||||
<img src="fig/singlephotoncount_fig_2.png" width=400>
|
||||
|
||||
<!-- end figure -->
|
||||
|
||||
|
||||
### Best estimates and confidence intervals
|
||||
|
||||
The posterior distribution from our Bayesian data analysis is the key quantity that encodes our inference about the values of the model parameters, given the data and the relevant background information. Often, however, we wish to summarize this result with just a few numbers: the best estimate and a measure of its reliability.
|
||||
|
||||
There are a few different options for this. The choice of the most appropriate one depends mainly on the shape of the posterior distribution:
|
||||
|
||||
*Symmetric posterior pdfs*: Since the probability (density) associated with any particular value of the parameter is a measure of how much we believe that it lies in the neighbourhood of that point, our best estimate is given by the maximum of the posterior pdf. If we denote the quantity of interest by $X$, with a posterior pdf $P =p(X|D,I)$, then the best estimate of its value $X_0$ is given by the condition $dP/dX|_{X=X_0}=0$. Strictly speaking, we should also check the sign of the second derivative to ensure that $X_0$ represents a maximum.
|
||||
|
||||
To obtain a measure of the reliability of this best estimate, we need to look at the width or spread of the posterior pdf about $X_0$. When considering the behaviour of any function in the neighbourhood of a particular point, it is often helpful to carry out a Taylor series expansion; this is simply a standard tool for (locally) approximating a complicated function by a low-order polynomial. The linear term is zero at the maximum and the quadratic term is often the dominating one determining the width of the posterior pdf. Ignoring all the higher-order terms we arrive at the Gaussian approximation
|
||||
|
||||
<!-- Equation labels as ordinary links -->
|
||||
<div id="_auto8"></div>
|
||||
|
||||
$$
|
||||
\begin{equation}
|
||||
p(X|D,I) \approx \frac{1}{\sigma\sqrt{2\pi}} \exp \left[ -\frac{(x-\mu)^2}{2\sigma^2} \right],
|
||||
\label{_auto8} \tag{11}
|
||||
\end{equation}
|
||||
$$
|
||||
|
||||
where the mean $\mu = X_0$ and the variance $\sigma = \left( - \left. \frac{d^2L}{dX^2} \right|_{X_0} \right)^{-1/2}$, where $L$ is the logarithm of the posterior $P$. Our inference about the quantity of interest is conveyed very concisely, therefore, by the statement $X = X_0 \pm \sigma$, and
|
||||
|
||||
$$
|
||||
$$
|
||||
p(X_0-\sigma < X < X_0+\sigma | D,I) = \int_{X_0-\sigma}^{X_0+\sigma} p(X|D,I) dX \approx 0.67.
|
||||
$$
|
||||
$$
|
||||
|
||||
*Asymmetric posterior pdfs*: While the maximum of the posterior ($X_0$) can still be regarded as giving the best estimate, the true value is now more likely to be on one side of this rather than the other. Alternatively one can compute the mean value, $\langle X \rangle = \int X p(X|D,I) dX$, although this tends to overemphasise very long tails. The best option is probably a compromise that can be employed when having access to a large sample from the posterior (as provided by an MCMC), namely to give the median of this ensamble.
|
||||
|
||||
Furthermore, the concept of an error-bar does not seem appropriate in this case, as it implicitly entails the idea of symmetry. A good way of expressing the reliability with which a parameter can be inferred, for an asymmetric posterior pdf, is rather through a *confidence interval*. Since the area under the posterior pdf between $X_1$ and $X_2$ is proportional to how much we believe that $X$ lies in that range, the shortest interval that encloses 67% of the area represents a sensible measure of the uncertainty of the estimate. Obviously we can choose to provide some other degree-of-belief that we think is relevant for the case at hand. Assuming that the posterior pdf has been normalized, to have unit area, we need to find $X_1$ and $X_2$ such that:
|
||||
|
||||
$$
|
||||
$$
|
||||
p(X_1 < X < X_2 | D,I) = \int_{X_1}^{X_2} p(X|D,I) dX \approx 0.67,
|
||||
$$
|
||||
$$
|
||||
|
||||
where the difference $X_2 - X_1$ is as small as possible. The region $X_1 < X < X_2$ is then called the shortest 67% confidence interval.
|
||||
|
||||
*Multimodal posterior pdfs*: We can sometimes obtain posteriors which are multimodal; i.e. contains several disconnected regions with large probabilities. There is no difficulty when one of the maxima is very much larger than the others: we can simply ignore the subsidiary solutions, to a good approximation, and concentrate on the global maximum. The problem arises when there are several maxima of comparable magnitude. What do we now mean by a best estimate, and how should we quantify its reliability? The idea of a best estimate and an error-bar, or even a confidence interval, is merely an attempt to summarize the posterior with just two or three numbers; sometimes this just can’t be done, and so these concepts are not valid. For the bimodal case we might be able to characterize the posterior in terms of a few numbers: two best estimates and their associated error-bars, or disjoint confidence intervals. For a general multimodal pdf, the most honest thing we can do is just display the posterior itself.
|
||||
|
||||
### Simple Photon Counts: Best estimates and confidence intervals
|
||||
|
||||
To compute these numbers for our example, you would run:
|
||||
|
||||
sampper=np.percentile(samples, [2.5, 16.5, 50, 83.5, 97.5],axis=0).flatten()
|
||||
print("""
|
||||
F_true = {0}
|
||||
Based on {1} measurements the posterior point estimates are:
|
||||
...F_est = {2:.0f} +/- {3:.0f}
|
||||
or using credible intervals:
|
||||
...F_est = {4:.0f} (posterior median)
|
||||
...F_est in [{5:.0f}, {6:.0f}] (67% credible interval)
|
||||
...F_est in [{7:.0f}, {8:.0f}] (95% credible interval) """\
|
||||
.format(F_true, N, np.mean(samples), np.std(samples), \
|
||||
sampper[2], sampper[1], sampper[3], sampper[0], sampper[4]))
|
||||
|
||||
`F_true = 1000`
|
||||
`Based on 50 measurements the posterior point estimates are:`
|
||||
`...F_est = 998 +/- 4`
|
||||
`or using credible intervals:`
|
||||
`...F_est = 998 (posterior median)`
|
||||
`...F_est in [993, 1002] (67% credible interval)`
|
||||
`...F_est in [989, 1006] (95% credible interval)`
|
||||
|
||||
In this particular example, the posterior pdf is actually a Gaussian (since it is constructed as a product of Gaussians), and the mean and variance from the quadratic approximation will agree exactly with the frequentist approach.
|
||||
|
||||
From this final result you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an MCMC sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball, but we did this here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible.
|
||||
|
||||
Furthermore, as data and models grow in complexity, the two approaches can diverge greatly.
|
||||
|
||||
|
||||
## Bayesian parameter estimation (multiple parameters, covariance)
|
||||
* multidimensional posterior pdf:s
|
||||
|
||||
* nuisance parameters (e.g. background subtraction?)
|
||||
|
||||
* corner plots, covariance, correlations
|
||||
|
||||
* best example?
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Bayesian model selection
|
||||
* Bayesian evidence
|
||||
|
||||
* Occam's razor
|
||||
|
||||
* Best example? How many spectral lines are there?
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
|
Before Width: | Height: | Size: 5.0 KiB After Width: | Height: | Size: 5.0 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 10 KiB After Width: | Height: | Size: 9.6 KiB |
@@ -440,8 +440,8 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"[-0.36370241 0.57281693 0.31446735 -1.74797657 -1.47723222 -1.24113195\n",
|
||||
" -0.714415 -1.26679543 -0.27301697 0.6793838 ]\n"
|
||||
"[-2.57325284 -1.11383985 0.52678895 -0.47530983 -0.1807752 0.52810513\n",
|
||||
" -1.50448048 0.33629261 -1.41402417 -0.39264297]\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user