Data Analysis and Machine Learning: Elements of Bayesian theory
- +
-
-
Dec 13, 2017
Mar 10, 2018
@@ -112,6 +149,16 @@ end of tocinfo -->
@@ -129,7 +176,7 @@ end of tocinfo -->
@@ -63,6 +87,14 @@ end of tocinfo -->
Contents
@@ -82,7 +114,7 @@ end of tocinfo -->
+
-Morten Hjorth-Jensen [1, 2]
+Christian Forssén [1]
+
+
+
+Morten Hjorth-Jensen [2, 3]
[1] Department of Physics, University of Oslo
-[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+[1] Department of Physics, Chalmers University of Technology, Sweden
+[2] Department of Physics, University of Oslo
+[3] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+
+ +
- © 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
+ © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
diff --git a/doc/pub/Bayesian/html/Bayesian.html b/doc/pub/Bayesian/html/Bayesian.html
index 1dd56fc73..11e464069 100644
--- a/doc/pub/Bayesian/html/Bayesian.html
+++ b/doc/pub/Bayesian/html/Bayesian.html
@@ -65,11 +65,35 @@ div { text-align: justify; text-justify: inter-word; }
+
+
+
+
+
+
+
@@ -78,20 +102,25 @@ end of tocinfo -->
-Morten Hjorth-Jensen [1, 2]
+Christian Forssén [1]
+
+
+
+Morten Hjorth-Jensen [2, 3]
[1] Department of Physics, University of Oslo
-[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+[1] Department of Physics, Chalmers University of Technology, Sweden
+[2] Department of Physics, University of Oslo
+[3] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+
+ +
- © 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
+ © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
diff --git a/doc/pub/Bayesian/ipynb/Bayesian.ipynb b/doc/pub/Bayesian/ipynb/Bayesian.ipynb
index d969f8c75..bf2f80296 100644
--- a/doc/pub/Bayesian/ipynb/Bayesian.ipynb
+++ b/doc/pub/Bayesian/ipynb/Bayesian.ipynb
@@ -6,19 +6,21 @@
"source": [
"\n",
"# Data Analysis and Machine Learning: Elements of Bayesian theory\n",
- "\n",
+ "\n",
" \n",
- "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
+ "**Christian Forssén**, Department of Physics, Chalmers University of Technology, Sweden \n",
+ "\n",
+ " **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Dec 13, 2017**\n",
+ "Date: **Mar 10, 2018**\n",
"\n",
- "Copyright 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
+ "Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"\n",
"## What is Bayesian Statistics\n",
- "Say something more general here. Reminder about probabilities from the statistics section\n",
+ "Morten's original plan: Reminder about probabilities from the statistics section\n",
"1. Product rule\n",
"\n",
"2. Binomial distribution\n",
@@ -27,7 +29,260 @@
"\n",
"4. other PDFs\n",
"\n",
- "5. Bayesian regression analysis"
+ "5. Bayesian regression analysis\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Inference\n",
+ "Inference:\n",
+ " : \n",
+ " \"the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former\" (Webster) \n",
+ " Do premises $A, B, \\ldots \\to$ hypothesis, $H$? \n",
+ "\n",
+ "Deductive inference:\n",
+ " : \n",
+ " Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra) \n",
+ " $B(H|A,B,...) = 0$ or $1$\n",
+ "\n",
+ "Inductive inference:\n",
+ " : \n",
+ " Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)\n",
+ " $A, B, C, D$ share properties $x, y, z$; $E$ has properties $x, y$\n",
+ " $\\to$ $E$ probably has property $z$.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Statistical Inference\n",
+ "* Quantify the strength of inductive inferences from facts, in the form of data ($D$), and other premises, e.g. models, to hypotheses about the phenomena producing the data.\n",
+ "\n",
+ "* Quantify via probabilities, or averages calculated using probabilities. Frequentists ($\\mathcal{F}$) and Bayesians ($\\mathcal{B}$) use probabilities very differently for this.\n",
+ "\n",
+ "* To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a *degree-of-belief* or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.\n",
+ "\n",
+ "* To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the *long run relative frequency* with which an event occurred, given (infinitely) many repeated (experimental) trials.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Some history\n",
+ "Adapted from D.S. Sivia[^Sivia]:\n",
+ "\n",
+ "[^Sivia]: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006\n",
+ "\n",
+ "> Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2].\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "![](\"fig/sivia_fig_1_2.png\")
\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> To Laplace, the (shaded) area under the posterior pdf curve between $m_1$ and $m_2$ was a measure of how much he believed that the mass of Saturn lay in the range $m_1 \\le M \\le m_2$. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%!\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.\n",
+ "> \n",
+ "> If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.\n",
+ "> For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## The Bayesian recipe\n",
+ "Assess hypotheses by calculating their probabilities $p(H_i | \\ldots)$ conditional on known and/or presumed information using the rules of probability theory.\n",
+ "\n",
+ "\n",
+ "Probability Theory Axioms:\n",
+ "Product (AND) rule :\n",
+ " : \n",
+ " $p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)$\n",
+ " Should read $p(A,B|I)$ as the probability for propositions $A$ AND $B$ being true given that $I$ is true.\n",
+ "\n",
+ "Sum (OR) rule:\n",
+ " : \n",
+ " $p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)$\n",
+ " $p(A+B|I)$ is the probability that proposition $A$ OR $B$ is true given that $I$ is true.\n",
+ "\n",
+ "Normalization:\n",
+ " : \n",
+ " $p(A|I) + p(\\bar{A}|I) = 1$\n",
+ " $\\bar{A}$ denotes the proposition that $A$ is false.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Bayes' theorem\n",
+ "Bayes' theorem follows directly from the product rule"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "p(A|B,I) = \\frac{p(B|A,I) p(A|I)}{p(B|I)}.\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The importance of this property to data analysis becomes apparent if we replace $A$ and $B$ by hypothesis($H$) and data($D$):"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "p(H|D,I) = \\frac{p(D|H,I) p(H|I)}{p(D|I)}.\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.\n",
+ "\n",
+ "\n",
+ "\n",
+ "The various terms in Bayes’ theorem have formal names. \n",
+ "* The quantity on the far right, $p(H|I)$, is called the *prior* probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data. \n",
+ "\n",
+ "* This is modified by the experimental measurements through $p(D|H,I)$, the *likelihood* function, \n",
+ "\n",
+ "* The denominator $p(D|I)$ is called the *evidence*. It does not depend on the hypothesis and can be regarded as a normalization constant.\n",
+ "\n",
+ "* Together, these yield the *posterior* probability, $p(H|D, I )$, representing our state of knowledge about the truth of the hypothesis in the light of the data. \n",
+ "\n",
+ "In a sense, Bayes’ theorem encapsulates the process of learning.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## The friends of Bayes' theorem\n",
+ "Normalization:\n",
+ " : \n",
+ " $\\sum_i p(H_i|\\ldots) = 1$.\n",
+ "\n",
+ "Marginalization:\n",
+ " : \n",
+ " $\\sum_i p(A,H_i|I) = \\sum_i p(H_i|A,I) p(A|I) = p(A|I)$.\n",
+ "\n",
+ "Marginalization (continuum limit):\n",
+ " : \n",
+ " $\\int dx p(A,H(x)|I) = p(A|I)$.\n",
+ "\n",
+ "In the above, $H_i$ is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then $H_1$ could be the proposition that the first candidate will win, and so on. The probability that $A$ is true, for example that unemployment will be lower in a year’s time (given all relevant information $I$, but irrespective of whoever becomes president) is then given by $\\sum_i p(A,H_i|I)$.\n",
+ "\n",
+ "In the continuum limit of propositions we must understand $p(\\ldots)$ as a pdf (probability density function).\n",
+ "\n",
+ "Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Inference With Parametric Models\n",
+ "Inductive inference with parametric models is a very important tool in the natural sciences.\n",
+ "* Consider $N$ different models $M_i$ ($i = 1, \\ldots, N$), each with parameters $\\boldsymbol{\\alpha}_i$. Each of them implies a sampling distribution (conditional predictive distribution for possible data)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "p(D|\\boldsymbol{\\alpha}_i, M_i)\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* The $\\boldsymbol{\\alpha}_i$ dependence when we fix attention on the actual, observed data ($D_\\mathrm{obs}$) is the likelihood function:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "\\mathcal{L}_i (\\boldsymbol{\\alpha}_i) \\equiv p(D_\\mathrm{obs}|\\boldsymbol{\\alpha}_i, M_i)\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* We may be uncertain about $i$ (model uncertainty),\n",
+ "\n",
+ "* or uncertain about $\\boldsymbol{\\alpha}_i$ (parameter uncertainty).\n",
+ "\n",
+ "\n",
+ "\n",
+ "Parameter Estimation:\n",
+ " : \n",
+ " Premise = choice of model (pick specific $i$)\n",
+ " $\\Rightarrow$ What can we say about $\\boldsymbol{\\alpha}_i$?\n",
+ "\n",
+ "Model comparison:\n",
+ " : \n",
+ " Premise = $\\{M_i\\}$\n",
+ " $\\Rightarrow$ What can we say about $i$?\n",
+ "\n",
+ "Model adequacy:\n",
+ " : \n",
+ " Premise = $M_1$\n",
+ " $\\Rightarrow$ Is $M_1$ adequate?\n",
+ "\n",
+ "Hybrid Uncertainty:\n",
+ " : \n",
+ " Models share some common params: $\\boldsymbol{\\alpha}_1 = \\{ \\boldsymbol{\\varphi}, \\boldsymbol{\\eta}_i\\}$\n",
+ " $\\Rightarrow$ What can we say about $\\boldsymbol{\\varphi}$? (Systematic error is an example)\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Illustrative examples with python code\n",
+ "* Is this a fair coin? (analytical)\n",
+ "\n",
+ "* Flux from a star (single parameter, MCMC)\n",
+ "\n",
+ "* The lighthouse problem (two parameters, MCMC)\n",
+ "\n",
+ "* Linear fit with outliers (nuisance parameters)\n",
+ "\n",
+ "* ..."
]
}
],
diff --git a/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz b/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz
index afdbddbca..aac2f9fee 100644
Binary files a/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz and b/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz differ
diff --git a/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf b/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf
index 3f095066b..ebd8a2824 100644
Binary files a/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf and b/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf b/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf
index 246f3cb3a..5247e17bf 100644
Binary files a/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf and b/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf differ
diff --git a/doc/pub/Bayesian/pdf/Bayesian-minted.pdf b/doc/pub/Bayesian/pdf/Bayesian-minted.pdf
index 0281d83a8..223188b24 100644
Binary files a/doc/pub/Bayesian/pdf/Bayesian-minted.pdf and b/doc/pub/Bayesian/pdf/Bayesian-minted.pdf differ
diff --git a/doc/src/Bayesian/Bayesian.do.txt b/doc/src/Bayesian/Bayesian.do.txt
index bb849e14f..c5ed2f130 100644
--- a/doc/src/Bayesian/Bayesian.do.txt
+++ b/doc/src/Bayesian/Bayesian.do.txt
@@ -1,4 +1,5 @@
TITLE: Data Analysis and Machine Learning: Elements of Bayesian theory
+AUTHOR: Christian Forssén at Department of Physics, Chalmers University of Technology, Sweden
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -6,7 +7,7 @@ DATE: today
!split
===== What is Bayesian Statistics =====
!bblock
-Say something more general here. Reminder about probabilities from the statistics section
+Morten's original plan: Reminder about probabilities from the statistics section
o Product rule
o Binomial distribution
o Gaussian PDF
@@ -14,3 +15,162 @@ o other PDFs
o Bayesian regression analysis
!eblock
+!split
+===== Inference =====
+!bblock
+- Inference: ``the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former'' (Webster)
+ Do premises $A, B, \ldots \to$ hypothesis, $H$?
+
+- Deductive inference: Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
+ $B(H|A,B,...) = 0$ or $1$
+
+- Inductive inference: Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
+ $A, B, C, D$ share properties $x, y, z$; $E$ has properties $x, y$
+ $\to$ $E$ probably has property $z$.
+!eblock
+
+!split
+===== Statistical Inference =====
+!bblock
+* Quantify the strength of inductive inferences from facts, in the form of data ($D$), and other premises, e.g. models, to hypotheses about the phenomena producing the data.
+* Quantify via probabilities, or averages calculated using probabilities. Frequentists ($\mathcal{F}$) and Bayesians ($\mathcal{B}$) use probabilities very differently for this.
+* To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a *degree-of-belief* or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach.
+* To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the *long run relative frequency* with which an event occurred, given (infinitely) many repeated (experimental) trials.
+!eblock
+
+!split
+===== Some history =====
+Adapted from D.S. Sivia[^Sivia]:
+
+[^Sivia]: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006
+
+!bquote
+Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2].
+!equote
+
+!split
+FIGURE: [fig/sivia_fig_1_2.png, width=700 frac=0.9]
+
+!split
+!bquote
+To Laplace, the (shaded) area under the posterior pdf curve between $m_1$ and $m_2$ was a measure of how much he believed that the mass of Saturn lay in the range $m_1 \le M \le m_2$. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%!
+!equote
+
+!split
+!bquote
+According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.
+
+If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn.
+!equote
+
+!split
+!bquote
+As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.
+For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.
+!equote
+
+!split
+!bquote
+Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.
+!equote
+
+!split
+===== The Bayesian recipe =====
+!bblock
+Assess hypotheses by calculating their probabilities $p(H_i | \ldots)$ conditional on known and/or presumed information using the rules of probability theory.
+!eblock
+!bblock
+Probability Theory Axioms:
+- Product (AND) rule : $p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)$
+ Should read $p(A,B|I)$ as the probability for propositions $A$ AND $B$ being true given that $I$ is true.
+- Sum (OR) rule: $p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)$
+ $p(A+B|I)$ is the probability that proposition $A$ OR $B$ is true given that $I$ is true.
+- Normalization: $p(A|I) + p(\bar{A}|I) = 1$
+ $\bar{A}$ denotes the proposition that $A$ is false.
+
+!eblock
+
+!split
+===== Bayes' theorem =====
+!bblock
+Bayes' theorem follows directly from the product rule
+!bt
+$$
+p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}.
+$$
+!et
+The importance of this property to data analysis becomes apparent if we replace $A$ and $B$ by hypothesis($H$) and data($D$):
+!bt
+$$
+p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}.
+$$
+!et
+The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.
+!eblock
+
+!split
+!bblock
+The various terms in Bayes’ theorem have formal names.
+* The quantity on the far right, $p(H|I)$, is called the *prior* probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data.
+* This is modified by the experimental measurements through $p(D|H,I)$, the *likelihood* function,
+* The denominator $p(D|I)$ is called the *evidence*. It does not depend on the hypothesis and can be regarded as a normalization constant.
+* Together, these yield the *posterior* probability, $p(H|D, I )$, representing our state of knowledge about the truth of the hypothesis in the light of the data.
+
+In a sense, Bayes’ theorem encapsulates the process of learning.
+!eblock
+
+!split
+===== The friends of Bayes' theorem =====
+!bblock
+- Normalization: $\sum_i p(H_i|\ldots) = 1$.
+- Marginalization: $\sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I)$.
+- Marginalization (continuum limit): $\int dx p(A,H(x)|I) = p(A|I)$.
+
+In the above, $H_i$ is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then $H_1$ could be the proposition that the first candidate will win, and so on. The probability that $A$ is true, for example that unemployment will be lower in a year’s time (given all relevant information $I$, but irrespective of whoever becomes president) is then given by $\sum_i p(A,H_i|I)$.
+
+In the continuum limit of propositions we must understand $p(\ldots)$ as a pdf (probability density function).
+
+Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.
+!eblock
+
+!split
+===== Inference With Parametric Models =====
+!bblock
+Inductive inference with parametric models is a very important tool in the natural sciences.
+* Consider $N$ different models $M_i$ ($i = 1, \ldots, N$), each with parameters $\boldsymbol{\alpha}_i$. Each of them implies a sampling distribution (conditional predictive distribution for possible data)
+!bt
+$$
+p(D|\boldsymbol{\alpha}_i, M_i)
+$$
+!et
+* The $\boldsymbol{\alpha}_i$ dependence when we fix attention on the actual, observed data ($D_\mathrm{obs}$) is the likelihood function:
+!bt
+$$
+\mathcal{L}_i (\boldsymbol{\alpha}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\alpha}_i, M_i)
+$$
+!et
+* We may be uncertain about $i$ (model uncertainty),
+* or uncertain about $\boldsymbol{\alpha}_i$ (parameter uncertainty).
+!eblock
+
+!split
+!bblock
+- Parameter Estimation: Premise = choice of model (pick specific $i$)
+ $\Rightarrow$ What can we say about $\boldsymbol{\alpha}_i$?
+- Model comparison: Premise = $\{M_i\}$
+ $\Rightarrow$ What can we say about $i$?
+- Model adequacy: Premise = $M_1$
+ $\Rightarrow$ Is $M_1$ adequate?
+- Hybrid Uncertainty: Models share some common params: $\boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\}$
+ $\Rightarrow$ What can we say about $\boldsymbol{\varphi}$? (Systematic error is an example)
+!eblock
+
+!split
+===== Illustrative examples with python code =====
+!bblock
+* Is this a fair coin? (analytical)
+* Flux from a star (single parameter, MCMC)
+* The lighthouse problem (two parameters, MCMC)
+* Linear fit with outliers (nuisance parameters)
+* ...
+!eblock
\ No newline at end of file
-Say something more general here. Reminder about probabilities from the statistics section +Morten's original plan: Reminder about probabilities from the statistics section
- Product rule @@ -95,12 +127,25 @@ Say something more general here. Reminder about probabilities from the statistic
diff --git a/doc/pub/Bayesian/html/Bayesian-bs.html b/doc/pub/Bayesian/html/Bayesian-bs.html index e8eb2961e..d94e0043e 100644 --- a/doc/pub/Bayesian/html/Bayesian-bs.html +++ b/doc/pub/Bayesian/html/Bayesian-bs.html @@ -40,11 +40,35 @@ Automatically generated HTML file from DocOnce source + + + + + + +
@@ -63,6 +87,14 @@ end of tocinfo -->
Contents
@@ -84,20 +116,25 @@ end of tocinfo -->
-Morten Hjorth-Jensen [1, 2]
+Christian Forssén [1]
+
+
+
+Morten Hjorth-Jensen [2, 3]
[1] Department of Physics, University of Oslo
-[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+[1] Department of Physics, Chalmers University of Technology, Sweden
+[2] Department of Physics, University of Oslo
+[3] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
- © 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
+ © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
diff --git a/doc/pub/Bayesian/html/Bayesian-reveal.html b/doc/pub/Bayesian/html/Bayesian-reveal.html
index 27b99f8d4..8e8487d10 100644
--- a/doc/pub/Bayesian/html/Bayesian-reveal.html
+++ b/doc/pub/Bayesian/html/Bayesian-reveal.html
@@ -107,6 +107,22 @@ td.padding {
+
+
+
+
+
+
+
@@ -119,25 +135,30 @@ td.padding {
-Morten Hjorth-Jensen [1, 2]
+Christian Forssén [1]
+
+
+
+Morten Hjorth-Jensen [2, 3]
[1] Department of Physics, University of Oslo
-[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
+[1] Department of Physics, Chalmers University of Technology, Sweden
+[2] Department of Physics, University of Oslo
+[3] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
- © 1999-2017, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
+ © 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
@@ -147,7 +168,7 @@ td.padding {
diff --git a/doc/pub/Bayesian/html/Bayesian-solarized.html b/doc/pub/Bayesian/html/Bayesian-solarized.html
index 8b51f4470..ae99ba2b5 100644
--- a/doc/pub/Bayesian/html/Bayesian-solarized.html
+++ b/doc/pub/Bayesian/html/Bayesian-solarized.html
@@ -60,11 +60,35 @@ div { text-align: justify; text-justify: inter-word; }
+
+
+
+
+
+
+
@@ -73,20 +97,25 @@ end of tocinfo -->
Data Analysis and Machine Learning: Elements of Bayesian theory
- +
-
-
Dec 13, 2017
Mar 10, 2018
@@ -112,6 +149,16 @@ end of tocinfo -->
@@ -129,7 +176,7 @@ end of tocinfo -->Data Analysis and Machine Learning: Elements of Bayesian theory
- +
-
-
Dec 13, 2017
Mar 10, 2018
-Say something more general here. Reminder about probabilities from the statistics section +Morten's original plan: Reminder about probabilities from the statistics section
- Product rule @@ -160,6 +181,278 @@ Say something more general here. Reminder about probabilities from the statistic +
- Inference:
-
+ "the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former" (Webster)
+ Do premises \( A, B, \ldots \to \) hypothesis, \( H \)? + - Deductive inference:
-
+ Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
+ \( B(H|A,B,...) = 0 \) or \( 1 \) + - Inductive inference:
-
+ Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
+ \( A, B, C, D \) share properties \( x, y, z \); \( E \) has properties \( x, y \)
+ \( \to \) $E$ probably has property \( z \). + - Quantify the strength of inductive inferences from facts, in the form of data (\( D \)), and other premises, e.g. models, to hypotheses about the phenomena producing the data. +
- Quantify via probabilities, or averages calculated using probabilities. Frequentists (\( \mathcal{F} \)) and Bayesians (\( \mathcal{B} \)) use probabilities very differently for this. +
- To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a degree-of-belief or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach. +
- To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the long run relative frequency with which an event occurred, given (infinitely) many repeated (experimental) trials. +
- Product (AND) rule :
-
+ \( p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I) \)
+ Should read \( p(A,B|I) \) as the probability for propositions \( A \) AND \( B \) being true given that \( I \) is true. + - Sum (OR) rule:
-
+ \( p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I) \)
+ \( p(A+B|I) \) is the probability that proposition \( A \) OR \( B \) is true given that \( I \) is true. + - Normalization:
-
+ \( p(A|I) + p(\bar{A}|I) = 1 \)
+ \( \bar{A} \) denotes the proposition that \( A \) is false. + - The quantity on the far right, \( p(H|I) \), is called the prior probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data. +
- This is modified by the experimental measurements through \( p(D|H,I) \), the likelihood function, +
- The denominator \( p(D|I) \) is called the evidence. It does not depend on the hypothesis and can be regarded as a normalization constant. +
- Together, these yield the posterior probability, \( p(H|D, I ) \), representing our state of knowledge about the truth of the hypothesis in the light of the data. +
- Normalization:
- + \( \sum_i p(H_i|\ldots) = 1 \). +
- Marginalization:
- + \( \sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I) \). +
- Marginalization (continuum limit):
- + \( \int dx p(A,H(x)|I) = p(A|I) \). +
- Consider \( N \) different models \( M_i \) (\( i = 1, \ldots, N \)), each with parameters \( \boldsymbol{\alpha}_i \). Each of them implies a sampling distribution (conditional predictive distribution for possible data) +
- The \( \boldsymbol{\alpha}_i \) dependence when we fix attention on the actual, observed data (\( D_\mathrm{obs} \)) is the likelihood function: +
- We may be uncertain about \( i \) (model uncertainty), +
- or uncertain about \( \boldsymbol{\alpha}_i \) (parameter uncertainty). +
- Parameter Estimation:
-
+ Premise = choice of model (pick specific \( i \))
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\alpha}_i \)? + - Model comparison:
-
+ Premise = \( \{M_i\} \)
+ \( \Rightarrow \) What can we say about \( i \)? + - Model adequacy:
-
+ Premise = \( M_1 \)
+ \( \Rightarrow \) Is \( M_1 \) adequate? + - Hybrid Uncertainty:
-
+ Models share some common params: \( \boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\} \)
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\varphi} \)? (Systematic error is an example) + - Is this a fair coin? (analytical) +
- Flux from a star (single parameter, MCMC) +
- The lighthouse problem (two parameters, MCMC) +
- Linear fit with outliers (nuisance parameters) +
- ... +
Inference
+
+
+
++ +
-
+
Statistical Inference
+
+
+
+-
+
Some history
+Adapted from D.S. Sivia [1]: + +1: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006
+ ++
+ Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2]. ++
+
+ To Laplace, the (shaded) area under the posterior pdf curve between \( m_1 \) and \( m_2 \) was a measure of how much he believed that the mass of Saturn lay in the range \( m_1 \le M \le m_2 \). As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%! ++
+ According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used. + +++ If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn. +
+ As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse. + For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability. ++
+ Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today. ++
The Bayesian recipe
+
+
+
+
++Assess hypotheses by calculating their probabilities \( p(H_i | \ldots) \) conditional on known and/or presumed information using the rules of probability theory. +
+
+
++Probability Theory Axioms: + +
-
+
Bayes' theorem
+
+
+
++Bayes' theorem follows directly from the product rule +
+$$
+$$
+
+p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}.
+
+$$
+$$
+
+
+The importance of this property to data analysis becomes apparent if we replace \( A \) and \( B \) by hypothesis(\( H \)) and data(\( D \)):
+
+$$
+$$
+
+p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}.
+
+$$
+$$
+
+
+The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.
+
+
+
++The various terms in Bayes’ theorem have formal names. + +
-
+
+ +In a sense, Bayes’ theorem encapsulates the process of learning. +
The friends of Bayes' theorem
+
+
+
++ +
-
+
+In the continuum limit of propositions we must understand \( p(\ldots) \) as a pdf (probability density function). + +
+Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters. +
Inference With Parametric Models
+
+
+
++Inductive inference with parametric models is a very important tool in the natural sciences. + +
-
+
+$$
+$$
+
+p(D|\boldsymbol{\alpha}_i, M_i)
+
+$$
+$$
+
+
+
+
-
+
+$$
+$$
+
+\mathcal{L}_i (\boldsymbol{\alpha}_i) \equiv p(D_\mathrm{obs}|\boldsymbol{\alpha}_i, M_i)
+
+$$
+$$
+
+
+
+
-
+
+
+
++ +
-
+
Illustrative examples with python code
+
+
+
+-
+
Data Analysis and Machine Learning: Elements of Bayesian theory
- +
-
-
Dec 13, 2017
Mar 10, 2018
@@ -95,7 +124,7 @@ end of tocinfo -->
-Say something more general here. Reminder about probabilities from the statistics section +Morten's original plan: Reminder about probabilities from the statistics section
- Product rule @@ -107,11 +136,279 @@ Say something more general here. Reminder about probabilities from the statistic
+
+
+
Inference
+
+
+
+
+
++ +
-
+
- Inference:
-
+ "the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former" (Webster)
+ Do premises \( A, B, \ldots \to \) hypothesis, \( H \)? + - Deductive inference:
-
+ Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
+ \( B(H|A,B,...) = 0 \) or \( 1 \) + - Inductive inference:
-
+ Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
+ \( A, B, C, D \) share properties \( x, y, z \); \( E \) has properties \( x, y \)
+ \( \to \) $E$ probably has property \( z \). +
+
+
+
Statistical Inference
+
+
+
+
+
++ +
-
+
- Quantify the strength of inductive inferences from facts, in the form of data (\( D \)), and other premises, e.g. models, to hypotheses about the phenomena producing the data. +
- Quantify via probabilities, or averages calculated using probabilities. Frequentists (\( \mathcal{F} \)) and Bayesians (\( \mathcal{B} \)) use probabilities very differently for this. +
- To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a degree-of-belief or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach. +
- To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the long run relative frequency with which an event occurred, given (infinitely) many repeated (experimental) trials. +
+
+
+
Some history
+Adapted from D.S. Sivia [1]: + +1: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006
+ ++
+ Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2]. ++ + +
+
+
+ +
+
+
+ To Laplace, the (shaded) area under the posterior pdf curve between \( m_1 \) and \( m_2 \) was a measure of how much he believed that the mass of Saturn lay in the range \( m_1 \le M \le m_2 \). As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%! ++ + +
+
+
+ According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used. + ++ + ++ If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn. +
+
+
+ As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse. + For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability. ++ + +
+
+
+ Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today. ++ + +
+
+
+
The Bayesian recipe
+
+
+
+
++Assess hypotheses by calculating their probabilities \( p(H_i | \ldots) \) conditional on known and/or presumed information using the rules of probability theory. +
+
+
+
+
++Probability Theory Axioms: + +
-
+
- Product (AND) rule :
-
+ \( p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I) \)
+ Should read \( p(A,B|I) \) as the probability for propositions \( A \) AND \( B \) being true given that \( I \) is true. + - Sum (OR) rule:
-
+ \( p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I) \)
+ \( p(A+B|I) \) is the probability that proposition \( A \) OR \( B \) is true given that \( I \) is true. + - Normalization:
-
+ \( p(A|I) + p(\bar{A}|I) = 1 \)
+ \( \bar{A} \) denotes the proposition that \( A \) is false. +
+
+
+
Bayes' theorem
+
+
+
+
+
++Bayes' theorem follows directly from the product rule +$$ +$$ +p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}. +$$ +$$ + +The importance of this property to data analysis becomes apparent if we replace \( A \) and \( B \) by hypothesis(\( H \)) and data(\( D \)): +$$ +$$ +p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}. +$$ +$$ + +The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true. +
+
+
+
+
+
+
++The various terms in Bayes’ theorem have formal names. + +
-
+
- The quantity on the far right, \( p(H|I) \), is called the prior probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data. +
- This is modified by the experimental measurements through \( p(D|H,I) \), the likelihood function, +
- The denominator \( p(D|I) \) is called the evidence. It does not depend on the hypothesis and can be regarded as a normalization constant. +
- Together, these yield the posterior probability, \( p(H|D, I ) \), representing our state of knowledge about the truth of the hypothesis in the light of the data. +
+
+
+
The friends of Bayes' theorem
+
+
+
+
+
++ +
-
+
- Normalization:
- + \( \sum_i p(H_i|\ldots) = 1 \). +
- Marginalization:
- + \( \sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I) \). +
- Marginalization (continuum limit):
- + \( \int dx p(A,H(x)|I) = p(A|I) \). +
+In the continuum limit of propositions we must understand \( p(\ldots) \) as a pdf (probability density function). + +
+Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters. +
+
+
+
Inference With Parametric Models
+
+
+
+
+
++Inductive inference with parametric models is a very important tool in the natural sciences. + +
-
+
- Consider \( N \) different models \( M_i \) (\( i = 1, \ldots, N \)), each with parameters \( \boldsymbol{\alpha}_i \). Each of them implies a sampling distribution (conditional predictive distribution for possible data) +
-
+
- The \( \boldsymbol{\alpha}_i \) dependence when we fix attention on the actual, observed data (\( D_\mathrm{obs} \)) is the likelihood function: +
-
+
- We may be uncertain about \( i \) (model uncertainty), +
- or uncertain about \( \boldsymbol{\alpha}_i \) (parameter uncertainty). +
+
+
+
+
+
+
++ +
-
+
- Parameter Estimation:
-
+ Premise = choice of model (pick specific \( i \))
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\alpha}_i \)? + - Model comparison:
-
+ Premise = \( \{M_i\} \)
+ \( \Rightarrow \) What can we say about \( i \)? + - Model adequacy:
-
+ Premise = \( M_1 \)
+ \( \Rightarrow \) Is \( M_1 \) adequate? + - Hybrid Uncertainty:
-
+ Models share some common params: \( \boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\} \)
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\varphi} \)? (Systematic error is an example) +
+
+
+
Illustrative examples with python code
+
+
+
+
+
+ +
-
+
- Is this a fair coin? (analytical) +
- Flux from a star (single parameter, MCMC) +
- The lighthouse problem (two parameters, MCMC) +
- Linear fit with outliers (nuisance parameters) +
- ... +
Data Analysis and Machine Learning: Elements of Bayesian theory
- +
-
-
Dec 13, 2017
Mar 10, 2018
@@ -100,7 +129,7 @@ end of tocinfo -->
-Say something more general here. Reminder about probabilities from the statistics section +Morten's original plan: Reminder about probabilities from the statistics section
- Product rule @@ -112,11 +141,279 @@ Say something more general here. Reminder about probabilities from the statistic
+
+
+
Inference
+
+
+
+
+
++ +
-
+
- Inference:
-
+ "the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former" (Webster)
+ Do premises \( A, B, \ldots \to \) hypothesis, \( H \)? + - Deductive inference:
-
+ Premises allow definite determination of truth/falsity of H (syllogisms, symbolic logic, Boolean algebra)
+ \( B(H|A,B,...) = 0 \) or \( 1 \) + - Inductive inference:
-
+ Premises bear on truth/falsity of H, but don’t allow its definite determination (weak syllogisms, analogies)
+ \( A, B, C, D \) share properties \( x, y, z \); \( E \) has properties \( x, y \)
+ \( \to \) $E$ probably has property \( z \). +
+
+
+
Statistical Inference
+
+
+
+
+
++ +
-
+
- Quantify the strength of inductive inferences from facts, in the form of data (\( D \)), and other premises, e.g. models, to hypotheses about the phenomena producing the data. +
- Quantify via probabilities, or averages calculated using probabilities. Frequentists (\( \mathcal{F} \)) and Bayesians (\( \mathcal{B} \)) use probabilities very differently for this. +
- To the pioneers such as Bernoulli, Bayes and Laplace, a probability represented a degree-of-belief or plausability: how much they thought that something as true based on the evidence at hand. This is the Bayesian approach. +
- To the 19th century scholars, this seemed too vague and subjective. They redefined probability as the long run relative frequency with which an event occurred, given (infinitely) many repeated (experimental) trials. +
+
+
+
Some history
+Adapted from D.S. Sivia [1]: + +1: Sivia, Devinderjit, and John Skilling. Data Analysis : A Bayesian Tutorial, OUP Oxford, 2006
+ ++
+ Although the frequency definition appears to be more objective, its range of validity is also far more limited. For example, Laplace used (his) probability theory to estimate the mass of Saturn, given orbital data that were available to him from various astronomical observatories. In essence, he computed the posterior pdf for the mass M , given the data and all the relevant background information I (such as a knowledge of the laws of classical mechanics): prob(M|{data},I); this is shown schematically in the figure [Fig. 1.2]. ++ + +
+
+
+ +
+
+
+ To Laplace, the (shaded) area under the posterior pdf curve between \( m_1 \) and \( m_2 \) was a measure of how much he believed that the mass of Saturn lay in the range \( m_1 \le M \le m_2 \). As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%! ++ + +
+
+
+ According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used. + ++ + ++ If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn. +
+
+
+ As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse. + For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability. ++ + +
+
+
+ Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today. ++ + +
+
+
+
The Bayesian recipe
+
+
+
+
++Assess hypotheses by calculating their probabilities \( p(H_i | \ldots) \) conditional on known and/or presumed information using the rules of probability theory. +
+
+
+
+
++Probability Theory Axioms: + +
-
+
- Product (AND) rule :
-
+ \( p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I) \)
+ Should read \( p(A,B|I) \) as the probability for propositions \( A \) AND \( B \) being true given that \( I \) is true. + - Sum (OR) rule:
-
+ \( p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I) \)
+ \( p(A+B|I) \) is the probability that proposition \( A \) OR \( B \) is true given that \( I \) is true. + - Normalization:
-
+ \( p(A|I) + p(\bar{A}|I) = 1 \)
+ \( \bar{A} \) denotes the proposition that \( A \) is false. +
+
+
+
Bayes' theorem
+
+
+
+
+
++Bayes' theorem follows directly from the product rule +$$ +$$ +p(A|B,I) = \frac{p(B|A,I) p(A|I)}{p(B|I)}. +$$ +$$ + +The importance of this property to data analysis becomes apparent if we replace \( A \) and \( B \) by hypothesis(\( H \)) and data(\( D \)): +$$ +$$ +p(H|D,I) = \frac{p(D|H,I) p(H|I)}{p(D|I)}. +$$ +$$ + +The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true. +
+
+
+
+
+
+
++The various terms in Bayes’ theorem have formal names. + +
-
+
- The quantity on the far right, \( p(H|I) \), is called the prior probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data. +
- This is modified by the experimental measurements through \( p(D|H,I) \), the likelihood function, +
- The denominator \( p(D|I) \) is called the evidence. It does not depend on the hypothesis and can be regarded as a normalization constant. +
- Together, these yield the posterior probability, \( p(H|D, I ) \), representing our state of knowledge about the truth of the hypothesis in the light of the data. +
+
+
+
The friends of Bayes' theorem
+
+
+
+
+
++ +
-
+
- Normalization:
- + \( \sum_i p(H_i|\ldots) = 1 \). +
- Marginalization:
- + \( \sum_i p(A,H_i|I) = \sum_i p(H_i|A,I) p(A|I) = p(A|I) \). +
- Marginalization (continuum limit):
- + \( \int dx p(A,H(x)|I) = p(A|I) \). +
+In the continuum limit of propositions we must understand \( p(\ldots) \) as a pdf (probability density function). + +
+Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters. +
+
+
+
Inference With Parametric Models
+
+
+
+
+
++Inductive inference with parametric models is a very important tool in the natural sciences. + +
-
+
- Consider \( N \) different models \( M_i \) (\( i = 1, \ldots, N \)), each with parameters \( \boldsymbol{\alpha}_i \). Each of them implies a sampling distribution (conditional predictive distribution for possible data) +
-
+
- The \( \boldsymbol{\alpha}_i \) dependence when we fix attention on the actual, observed data (\( D_\mathrm{obs} \)) is the likelihood function: +
-
+
- We may be uncertain about \( i \) (model uncertainty), +
- or uncertain about \( \boldsymbol{\alpha}_i \) (parameter uncertainty). +
+
+
+
+
+
+
++ +
-
+
- Parameter Estimation:
-
+ Premise = choice of model (pick specific \( i \))
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\alpha}_i \)? + - Model comparison:
-
+ Premise = \( \{M_i\} \)
+ \( \Rightarrow \) What can we say about \( i \)? + - Model adequacy:
-
+ Premise = \( M_1 \)
+ \( \Rightarrow \) Is \( M_1 \) adequate? + - Hybrid Uncertainty:
-
+ Models share some common params: \( \boldsymbol{\alpha}_1 = \{ \boldsymbol{\varphi}, \boldsymbol{\eta}_i\} \)
+ \( \Rightarrow \) What can we say about \( \boldsymbol{\varphi} \)? (Systematic error is an example) +
+
+
+
Illustrative examples with python code
+
+
+
+
+
+ +
-
+
- Is this a fair coin? (analytical) +
- Flux from a star (single parameter, MCMC) +
- The lighthouse problem (two parameters, MCMC) +
- Linear fit with outliers (nuisance parameters) +
- ... +
\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> To Laplace, the (shaded) area under the posterior pdf curve between $m_1$ and $m_2$ was a measure of how much he believed that the mass of Saturn lay in the range $m_1 \\le M \\le m_2$. As such, the position of the maximum of the posterior pdf represents a best estimate of the mass; its width, or spread, about this optimal value gives an indication of the uncertainty in the estimate. Laplace stated that: ‘ . . . it is a bet of 11,000 to 1 that the error of this result is not 1/100th of its value.’ He would have won the bet, as another 150 years’ accumulation of data has changed the estimate by only 0.63%!\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> According to the frequency definition, however, we are not permitted to use probability theory to tackle this problem. This is because the mass of Saturn is a constant and not a random variable; therefore, it has no frequency distribution and so probability theory cannot be used.\n",
+ "> \n",
+ "> If the pdf [of Fig. 1.2] had to be interpreted in terms of the frequency definition, we would have to imagine a large ensemble of universes in which everything remains constant apart from the mass of Saturn.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> As this scenario appears quite far-fetched, we might be inclined to think of [Fig. 1.2] in terms of the distribution of the measurements of the mass in many repetitions of the experiment. Although we are at liberty to think about a problem in any way that facilitates its solution, or our understanding of it, having to seek a frequency interpretation for every data analysis problem seems rather perverse.\n",
+ "> For example, what do we mean by the ‘measurement of the mass’ when the data consist of orbital periods? Besides, why should we have to think about many repetitions of an experiment that never happened? What we really want to do is to make the best inference of the mass given the (few) data that we actually have; this is precisely the Bayes and Laplace view of probability.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "> Faced with the realization that the frequency definition of probability theory did not permit most real-life scientific problems to be addressed, a new subject was invented — statistics! To estimate the mass of Saturn, for example, one has to relate the mass to the data through some function called the statistic; since the data are subject to ‘random’ noise, the statistic becomes the random variable to which the rules of probability the- ory can be applied. But now the question arises: How should we choose the statistic? The frequentist approach does not yield a natural way of doing this and has, therefore, led to the development of several alternative schools of orthodox or conventional statis- tics. The masters, such as Fisher, Neyman and Pearson, provided a variety of different principles, which has merely resulted in a plethora of tests and procedures without any clear underlying rationale. This lack of unifying principles is, perhaps, at the heart of the shortcomings of the cook-book approach to statistics that students are often taught even today.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## The Bayesian recipe\n",
+ "Assess hypotheses by calculating their probabilities $p(H_i | \\ldots)$ conditional on known and/or presumed information using the rules of probability theory.\n",
+ "\n",
+ "\n",
+ "Probability Theory Axioms:\n",
+ "Product (AND) rule :\n",
+ " : \n",
+ " $p(A, B | I) = p(A|I) p(B|A, I) = p(B|I)p(A|B,I)$\n",
+ " Should read $p(A,B|I)$ as the probability for propositions $A$ AND $B$ being true given that $I$ is true.\n",
+ "\n",
+ "Sum (OR) rule:\n",
+ " : \n",
+ " $p(A + B | I) = p(A | I) + p(B | I) - p(A, B | I)$\n",
+ " $p(A+B|I)$ is the probability that proposition $A$ OR $B$ is true given that $I$ is true.\n",
+ "\n",
+ "Normalization:\n",
+ " : \n",
+ " $p(A|I) + p(\\bar{A}|I) = 1$\n",
+ " $\\bar{A}$ denotes the proposition that $A$ is false.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Bayes' theorem\n",
+ "Bayes' theorem follows directly from the product rule"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "p(A|B,I) = \\frac{p(B|A,I) p(A|I)}{p(B|I)}.\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The importance of this property to data analysis becomes apparent if we replace $A$ and $B$ by hypothesis($H$) and data($D$):"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "p(H|D,I) = \\frac{p(D|H,I) p(H|I)}{p(D|I)}.\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The power of Bayes’ theorem lies in the fact that it relates the quantity of interest, the probability that the hypothesis is true given the data, to the term we have a better chance of being able to assign, the probability that we would have observed the measured data if the hypothesis was true.\n",
+ "\n",
+ "\n",
+ "\n",
+ "The various terms in Bayes’ theorem have formal names. \n",
+ "* The quantity on the far right, $p(H|I)$, is called the *prior* probability; it represents our state of knowledge (or ignorance) about the truth of the hypothesis before we have analysed the current data. \n",
+ "\n",
+ "* This is modified by the experimental measurements through $p(D|H,I)$, the *likelihood* function, \n",
+ "\n",
+ "* The denominator $p(D|I)$ is called the *evidence*. It does not depend on the hypothesis and can be regarded as a normalization constant.\n",
+ "\n",
+ "* Together, these yield the *posterior* probability, $p(H|D, I )$, representing our state of knowledge about the truth of the hypothesis in the light of the data. \n",
+ "\n",
+ "In a sense, Bayes’ theorem encapsulates the process of learning.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## The friends of Bayes' theorem\n",
+ "Normalization:\n",
+ " : \n",
+ " $\\sum_i p(H_i|\\ldots) = 1$.\n",
+ "\n",
+ "Marginalization:\n",
+ " : \n",
+ " $\\sum_i p(A,H_i|I) = \\sum_i p(H_i|A,I) p(A|I) = p(A|I)$.\n",
+ "\n",
+ "Marginalization (continuum limit):\n",
+ " : \n",
+ " $\\int dx p(A,H(x)|I) = p(A|I)$.\n",
+ "\n",
+ "In the above, $H_i$ is an exclusive and exhaustive list of hypotheses. For example,let’s imagine that there are five candidates in a presidential election; then $H_1$ could be the proposition that the first candidate will win, and so on. The probability that $A$ is true, for example that unemployment will be lower in a year’s time (given all relevant information $I$, but irrespective of whoever becomes president) is then given by $\\sum_i p(A,H_i|I)$.\n",
+ "\n",
+ "In the continuum limit of propositions we must understand $p(\\ldots)$ as a pdf (probability density function).\n",
+ "\n",
+ "Marginalization is a very powerful device in data analysis because it enables us to deal with nuisance parameters; that is, quantities which necessarily enter the analysis but are of no intrinsic interest. The unwanted background signal present in many experimental measurements are examples of nuisance parameters.\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Inference With Parametric Models\n",
+ "Inductive inference with parametric models is a very important tool in the natural sciences.\n",
+ "* Consider $N$ different models $M_i$ ($i = 1, \\ldots, N$), each with parameters $\\boldsymbol{\\alpha}_i$. Each of them implies a sampling distribution (conditional predictive distribution for possible data)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "p(D|\\boldsymbol{\\alpha}_i, M_i)\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* The $\\boldsymbol{\\alpha}_i$ dependence when we fix attention on the actual, observed data ($D_\\mathrm{obs}$) is the likelihood function:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "$$\n",
+ "\\mathcal{L}_i (\\boldsymbol{\\alpha}_i) \\equiv p(D_\\mathrm{obs}|\\boldsymbol{\\alpha}_i, M_i)\n",
+ "$$\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* We may be uncertain about $i$ (model uncertainty),\n",
+ "\n",
+ "* or uncertain about $\\boldsymbol{\\alpha}_i$ (parameter uncertainty).\n",
+ "\n",
+ "\n",
+ "\n",
+ "Parameter Estimation:\n",
+ " : \n",
+ " Premise = choice of model (pick specific $i$)\n",
+ " $\\Rightarrow$ What can we say about $\\boldsymbol{\\alpha}_i$?\n",
+ "\n",
+ "Model comparison:\n",
+ " : \n",
+ " Premise = $\\{M_i\\}$\n",
+ " $\\Rightarrow$ What can we say about $i$?\n",
+ "\n",
+ "Model adequacy:\n",
+ " : \n",
+ " Premise = $M_1$\n",
+ " $\\Rightarrow$ Is $M_1$ adequate?\n",
+ "\n",
+ "Hybrid Uncertainty:\n",
+ " : \n",
+ " Models share some common params: $\\boldsymbol{\\alpha}_1 = \\{ \\boldsymbol{\\varphi}, \\boldsymbol{\\eta}_i\\}$\n",
+ " $\\Rightarrow$ What can we say about $\\boldsymbol{\\varphi}$? (Systematic error is an example)\n",
+ "\n",
+ "\n",
+ "\n",
+ "## Illustrative examples with python code\n",
+ "* Is this a fair coin? (analytical)\n",
+ "\n",
+ "* Flux from a star (single parameter, MCMC)\n",
+ "\n",
+ "* The lighthouse problem (two parameters, MCMC)\n",
+ "\n",
+ "* Linear fit with outliers (nuisance parameters)\n",
+ "\n",
+ "* ..."
]
}
],
diff --git a/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz b/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz
index afdbddbca..aac2f9fee 100644
Binary files a/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz and b/doc/pub/Bayesian/ipynb/ipynb-Bayesian-src.tar.gz differ
diff --git a/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf b/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf
index 3f095066b..ebd8a2824 100644
Binary files a/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf and b/doc/pub/Bayesian/pdf/Bayesian-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf b/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf
index 246f3cb3a..5247e17bf 100644
Binary files a/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf and b/doc/pub/Bayesian/pdf/Bayesian-beamer.pdf differ
diff --git a/doc/pub/Bayesian/pdf/Bayesian-minted.pdf b/doc/pub/Bayesian/pdf/Bayesian-minted.pdf
index 0281d83a8..223188b24 100644
Binary files a/doc/pub/Bayesian/pdf/Bayesian-minted.pdf and b/doc/pub/Bayesian/pdf/Bayesian-minted.pdf differ
diff --git a/doc/src/Bayesian/Bayesian.do.txt b/doc/src/Bayesian/Bayesian.do.txt
index bb849e14f..c5ed2f130 100644
--- a/doc/src/Bayesian/Bayesian.do.txt
+++ b/doc/src/Bayesian/Bayesian.do.txt
@@ -1,4 +1,5 @@
TITLE: Data Analysis and Machine Learning: Elements of Bayesian theory
+AUTHOR: Christian Forssén at Department of Physics, Chalmers University of Technology, Sweden
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -6,7 +7,7 @@ DATE: today
!split
===== What is Bayesian Statistics =====
!bblock
-Say something more general here. Reminder about probabilities from the statistics section
+Morten's original plan: Reminder about probabilities from the statistics section
o Product rule
o Binomial distribution
o Gaussian PDF
@@ -14,3 +15,162 @@ o other PDFs
o Bayesian regression analysis
!eblock
+!split
+===== Inference =====
+!bblock
+- Inference: ``the act of passing from one proposition, statement or judgment considered as true to another whose truth is believed to follow from that of the former'' (Webster)