lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added files

Browse Source
This commit is contained in:
mhjensen
2020-12-18 06:53:59 +01:00
parent cd0f926a6a
commit 1825fb4b80
8 changed files with 15112 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/Statistics/Statistics.do.txt
+75 -79
View File
@@ -1,17 +1,13 @@
TITLE: Data Analysis and Machine Learning: Elements of Probability Theory and Statistical Data Analysis
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
======= Elements of Probability Theory and Statistical Data Analysis =======
!split
===== To do list =====
* add math about MVN and define MLE and other quantities
* rewrite about covariance matrix
* add KL theorem
!split
===== Domains and probabilities =====
!bblock
Consider the following simple example, namely the tossing of two dice, resulting in the following possible values
@@ -29,7 +25,7 @@ To this domain we have the corresponding *probabilities*
!et
!eblock
!split
===== Tossing the dice =====
!bblock
The numbers in the domain are the outcomes of the physical process of tossing say two dice.
@@ -55,14 +51,14 @@ appear in a random order. After 11 throws the results may look like
!et
!eblock
!split
===== Stochastic variables =====
!bblock
_Random variables are characterized by a domain which contains all possible values that the random value may take. This domain has a corresponding probability distribution function(PDF)_.
!eblock
!split
===== Stochastic variables and the main concepts, the discrete case =====
!bblock
There are two main concepts associated with a stochastic variable. The
@@ -83,7 +79,7 @@ p(x) = \mathrm{Prob}(X=x).
!eblock
!split
===== Stochastic variables and the main concepts, the continuous case =====
!bblock
In the continuous case, the PDF does not directly depict the
@@ -104,7 +100,7 @@ numbers chosen as if by chance from some specified PDF so that the
selection of a large set of these numbers reproduces this PDF.
!eblock
!split
===== The cumulative probability =====
!bblock
Of interest to us is the *cumulative probability
@@ -125,7 +121,7 @@ p(x) = \frac{d}{dx}P(x).
!et
!eblock
!split
===== Properties of PDFs =====
!bblock
@@ -150,7 +146,7 @@ both discrete and continuous PDFs, this condition is
!et
!eblock
!split
===== Important distributions, the uniform distribution =====
!bblock
The first one
@@ -172,7 +168,7 @@ p(x)dx = dx & \in [0,1].
The latter distribution is used to generate random numbers. For other PDFs, one needs normally a mapping from this distribution to say for example the exponential distribution.
!eblock
!split
===== Gaussian distribution =====
!bblock
The second one is the Gaussian Distribution
@@ -231,7 +227,7 @@ plt.show()
!eblock
!split
===== Exponential distribution =====
!bblock
Another important distribution in science is the exponential distribution
@@ -242,7 +238,7 @@ p(x) = \alpha\exp{-(\alpha x)}.
!et
!eblock
!split
===== Expectation values =====
!bblock
Let $h(x)$ be an arbitrary continuous function on the domain of the stochastic
@@ -267,7 +263,7 @@ follows
!et
!eblock
!split
===== Stochastic variables and the main concepts, mean values =====
!bblock
The zero-th moment $\langle 1\rangle$ is just the normalization condition of
@@ -289,7 +285,7 @@ Qualitatively it represents the centroid or the average value of the
PDF and is therefore simply called the expectation value of $p(x)$.
!eblock
!split
===== Stochastic variables and the main concepts, central moments, the variance =====
!bblock
@@ -320,7 +316,7 @@ qualitatively as the ``spread'' of $p$ around its mean.
!split
===== Probability Distribution Functions =====
!bblock
@@ -344,7 +340,7 @@ while $P(x)$ is the cumulative probability.
!eblock
!split
===== Probability Distribution Functions =====
!bblock
With a PDF we can compute expectation values of selected quantities such as
@@ -365,7 +361,7 @@ in the case of a continuous PDF. We have already defined the mean value $\mu$
and the variance $\sigma^2$.
!eblock
!split
===== The three famous Probability Distribution Functions =====
!bblock
@@ -399,7 +395,7 @@ with variance
\end{equation*}
!et
!split
===== Probability Distribution Functions, the normal distribution =====
!bblock
Finally, we have the so-called univariate normal distribution, or just the _normal distribution_
@@ -425,7 +421,7 @@ which becomes with a suitable change of variables
!et
!eblock
!split
===== Probability Distribution Functions, the normal distribution =====
!bblock
Similarly, the variance becomes
@@ -453,7 +449,7 @@ centered around zero and with a variance $\sigma^2=1$, leading to
!et
!eblock
!split
===== Probability Distribution Functions, the cumulative distribution =====
!bblock
@@ -470,7 +466,7 @@ which is difficult to evaluate in a quick way.
!eblock
!split
===== Probability Distribution Functions, other important distribution =====
!bblock
@@ -492,7 +488,7 @@ The sequence of binomial trials is characterized by the following definitions
* The probability for every outcome in a single trial has the same value, for example the outcome of tossing (either heads or tails) a coin is always $1/2$.
!eblock
!split
===== Probability Distribution Functions, the binomial distribution =====
!bblock
@@ -535,7 +531,7 @@ which we rewrite as
!eblock
The variance is slightly trickier to get. It reads $\sigma^2=ny(1-y)$.
!split
===== Probability Distribution Functions, Poisson's distribution =====
!bblock
@@ -559,7 +555,7 @@ and the variance is $\sigma^2=\lambda$.
!split
===== Probability Distribution Functions, Poisson's distribution =====
!bblock
An example of applications of the Poisson distribution could be the counting
@@ -576,7 +572,7 @@ the binomial distribution we can show that
!eblock
!split
===== Meet the covariance! =====
!bblock
An important quantity in a statistical analysis is the so-called covariance.
@@ -604,7 +600,7 @@ with
!split
===== Meet the covariance in matrix disguise =====
!bblock
If we consider the above covariance as a matrix
@@ -619,7 +615,7 @@ all the off-diagonal elements are zero if the stochastic variables are
uncorrelated.
!eblock
!split
===== Covariance =====
!bc pycod
# Importing various packages
@@ -651,7 +647,7 @@ print(c)
!split
===== Meet the covariance, uncorrelated events =====
!bblock
@@ -683,7 +679,7 @@ Cov(X_i, X_j) = 0 \hspace{0.1cm} (i\neq j).
!split
===== Numerical experiments and the covariance =====
!bblock
@@ -704,7 +700,7 @@ chain of values
!eblock
!split
===== Numerical experiments and the covariance =====
!bblock
We will call these
@@ -722,7 +718,7 @@ $\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$.
!eblock
!split
===== Numerical experiments and the covariance, actual situations =====
!bblock
In practical situations however, a sample is always of finite size. Let that
@@ -742,7 +738,7 @@ with its square root being the *standard deviation of the sample*.
!eblock
!split
===== Numerical experiments and the covariance, our observables =====
!bblock
You can think of the above observables as a set of quantities which define
@@ -770,7 +766,7 @@ label{eq:exptvariance}
!et
!eblock
!split
===== Numerical experiments and the covariance, the sample variance =====
!bblock
@@ -791,7 +787,7 @@ exact values for the mean value $\mu_X$, the variance $\mathrm{Var}(X)$
and the covariance $\mathrm{Cov}(X,Y)$.
!eblock
!split
===== Numerical experiments and the covariance, central limit theorem =====
!bblock
@@ -813,7 +809,7 @@ standard deviation, given by
In many cases the above estimate for the standard deviation, in particular if correlations are strong, may be too simplistic. We need therefore a more precise defintion of the error and the variance in our results.
!eblock
!split
===== Definition of Correlation Functions and Standard Deviation =====
!bblock
Our estimate of the true average $\mu_{X}$ is the sample mean $\langle X_m \rangle$
@@ -845,7 +841,7 @@ and the last term is nothing but the covariance which arises when $k\ne l$.
!split
===== Definition of Correlation Functions and Standard Deviation =====
!bblock
Our estimate of the true average $\mu_{X}$ is the sample mean $\langle X_m \rangle$
@@ -863,7 +859,7 @@ measurements to calculate the cross terms. Therefore, all measurements
have to be stored throughout the experiment.
!eblock
!split
===== Definition of Correlation Functions and Standard Deviation =====
!bblock
@@ -887,7 +883,7 @@ $f_d$
!eblock
!split
===== Definition of Correlation Functions and Standard Deviation =====
!bblock
The value of $f_d$ reflects the correlation between measurements
@@ -905,7 +901,7 @@ which gives us a useful measure of the correlation pair correlation
starting always at $1$ for $d=0$.
!eblock
!split
===== Definition of Correlation Functions and Standard Deviation, sample variance =====
!bblock
@@ -938,7 +934,7 @@ equals 1.
!split
===== Definition of Correlation Functions and Standard Deviation =====
!bblock
From the point of view of
@@ -961,7 +957,7 @@ more practically oriented methods like the blocking technique.
!eblock
!split
===== Code to compute the Covariance matrix and the Covariance =====
!bc pycod
# Importing various packages
@@ -1003,7 +999,7 @@ Eigvals, Eigvecs = np.linalg.eig(c)
print(Eigvals)
!ec
!split
======= Random Numbers =======
!bblock
@@ -1020,7 +1016,7 @@ or Monte Carlo computer work.
!eblock
!split
======= Random Numbers, better name: pseudo random numbers =======
!bblock
@@ -1040,7 +1036,7 @@ numbers, hopefully abiding to the following criteria:
!eblock
!split
======= Random number generator RNG =======
!bblock
The most common random number generators are based on so-called
@@ -1067,7 +1063,7 @@ that is if we were to evaluate $(13)\mathrm{MOD}(9)$, the outcome is the remaind
of the division $13/9$, namely $4$.
!eblock
!split
======= Random number generator RNG and periodic outputs =======
!bblock
@@ -1097,7 +1093,7 @@ which still, with $N_0=2$, results in $11,38,11,38,11,38,\dots$, a period of
just $2$.
!eblock
!split
======= Random number generator RNG and its period =======
!bblock
Typical periods for the random generators provided in the program library
@@ -1115,7 +1111,7 @@ number is the sum of the $l-i$th and $l-j$th values with modulo $M$,
!et
!eblock
!split
======= Random number generator RNG, other examples =======
!bblock
Such a generator again produces a sequence of pseudorandom numbers
@@ -1141,7 +1137,7 @@ followed by
which according to the authors has a period larger than $2^{94}$.
!eblock
!split
======= Random number generator RNG, other examples =======
!bblock
Instead of using modular addition, we could use the bitwise
@@ -1165,7 +1161,7 @@ it is given by $m\wedge n$.
!eblock
!split
======= Random number generator RNG, RAN0 =======
!bblock
@@ -1203,7 +1199,7 @@ $q$ and $r$ are chosen so that $r < q$.
!split
======= Random number generator RNG, RAN0 =======
!bblock
@@ -1220,7 +1216,7 @@ $[N_{i-1}/q]$ just yields a constant which is multiplied with $M$.
!eblock
!split
======= Random number generator RNG, RAN0 =======
!bblock
We can now rewrite Eq. (ref{eq:rntrick1}) as
@@ -1250,7 +1246,7 @@ label{eq:rntrick4}
!eblock
!split
======= Random number generator RNG, RAN0 =======
!bblock
The term $[N_{i-1}/q]r$ is always smaller or equal $N_{i-1}(r/q)$ and with $r < q$ we obtain always a
@@ -1268,7 +1264,7 @@ set to $0$.
!eblock
!split
======= Random number generator RNG, RAN0 code =======
!bblock
@@ -1302,7 +1298,7 @@ double ran0(long &idum)
!ec
!eblock
!split
===== Properties of Selected Random Number Generators =====
!bblock
@@ -1321,7 +1317,7 @@ Two additional measures are the standard deviation $\sigma$ and the mean
$\mu=\langle x\rangle$.
!eblock
!split
===== Properties of Selected Random Number Generators =====
!bblock
For the uniform distribution, the mean value $\mu$ is then
@@ -1340,7 +1336,7 @@ while the standard deviation is
!et
!eblock
!split
===== Properties of Selected Random Number Generators =====
!bblock
The various random number generators produce results which agree rather well with
@@ -1365,7 +1361,7 @@ these limiting values.
!eblock
!split
===== Simple demonstration of RNGs using python =====
!bblock
The following simple Python code plots the distribution of the produced random numbers using the linear congruential RNG employed by Python. The trend displayed in the previous table is seen rather clearly.
@@ -1396,7 +1392,7 @@ plt.show()
!eblock
!split
===== Properties of Selected Random Number Generators =====
!bblock
Since our random numbers, which are typically generated via a linear congruential algorithm,
@@ -1427,7 +1423,7 @@ assumption for approximating $\sigma_N$ is no longer valid.
!eblock
!split
===== Autocorrelation function =====
This program computes the autocorrelation function as discussed in the equation on the previous slide for random numbers generated with the normal distribution $N(0,1)$.
!bc pycod
@@ -1479,7 +1475,7 @@ For the remaining values we notice that there are still non-zero values for the
!split
===== Correlation function and which random number generators should I use =====
!bblock
The program here computes the correlation function for one of the standard functions included with the c++ compiler.
@@ -1555,7 +1551,7 @@ int main(int argc, char* argv[])
!split
======= Which RNG should I use? =======
!bblock
* C++ has a class called _random_. The "random class":"http://www.cplusplus.com/reference/random/" contains a large selection of RNGs and is highly recommended. Some of these RNGs have very large periods making it thereby very safe to use these RNGs in case one is performing large calculations. In particular, the "Mersenne twister random number engine":"http://www.cplusplus.com/reference/random/mersenne_twister_engine/" has a period of $2^{19937}$.
@@ -1564,7 +1560,7 @@ int main(int argc, char* argv[])
!eblock
!split
===== How to use the Mersenne generator =====
!bblock
The following part of a c++ code (from project 4) sets up the uniform distribution for $x\in [0,1]$.
@@ -1588,7 +1584,7 @@ int ix = (int) (RandomNumberGenerator(gen)*NSpins);
!split
===== Why blocking? =====
!bblock Statistical analysis
* Monte Carlo simulations can be treated as *computer experiments*
@@ -1601,7 +1597,7 @@ A very good article which explains blocking is H. Flyvbjerg and H. G. Petersen,
!split
===== Why blocking? =====
!bblock Statistical analysis
* As in other experiments, Monte Carlo experiments have two classes of errors:
@@ -1612,7 +1608,7 @@ A very good article which explains blocking is H. Flyvbjerg and H. G. Petersen,
!eblock
!split
===== Code to demonstrate the calculation of the autocorrelation function =====
The following code computes the autocorrelation function, the covariance and the standard deviation
for standard RNG.
@@ -1700,7 +1696,7 @@ int main(int argc, char* argv[])
!ec
!split
===== What is blocking? =====
!bblock Blocking
* Say that we have a set of samples from a Monte Carlo experiment
@@ -1719,7 +1715,7 @@ int main(int argc, char* argv[])
where $\tau$ is the correlation time (the time between a sample and the next uncorrelated sample) and $\Delta t$ is time between each sample
!eblock
!split
===== What is blocking? =====
!bblock Blocking
* If $\Delta t\gg\tau$ our first estimate of $\sigma$ still holds
@@ -1730,7 +1726,7 @@ int main(int argc, char* argv[])
* The correlation time $\tau$ would be a good choice
!eblock
!split
===== What is blocking? =====
!bblock Blocking
* Problem: We don't know $\tau$ or it is too expensive to compute
@@ -1739,7 +1735,7 @@ int main(int argc, char* argv[])
* When the std. dev. stops increasing the blocks are uncorrelated
!eblock
!split
===== Implementation =====
!bblock
* Do a Monte Carlo simulation, storing all samples to file
@@ -1756,7 +1752,7 @@ int main(int argc, char* argv[])
!split
===== Actual implementation with code, main function =====
When the file gets large, it can be useful to write your data in binary mode instead of ascii characters.
The "following python file":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py" reads data from file with the output from every Monte Carlo cycle.
@@ -1797,7 +1793,7 @@ The "following python file":"https://github.com/CompPhysics/MachineLearning/blob
!split
===== The Bootstrap method =====
The Bootstrap resampling method is also very popular. It is very simple:
@@ -1811,7 +1807,7 @@ that it produces the correct standard deviation.
This method is very useful for small ensembles of data points.
!split
===== Bootstrapping =====
Given a set of $N$ data, assume that we are interested in some
observable $\theta$ which may be estimated from that set. This observable can also be for example the result of a fit based on all $N$ raw data.
@@ -1820,7 +1816,7 @@ data set $\hat{\theta}$. One recreates from the sample repeatedly
other samples by choosing randomly $N$ data out of the original set.
This costs essentially nothing, since we just recycle the original data set for the building of new sets.
!split
===== Bootstrapping, recipe =====
Let us assume we have done this $K$ times and thus have $K$ sets of $N$
data values each.
@@ -1844,7 +1840,7 @@ The difference is called bias and gives an idea on how far away the result may b
the true $\angle\theta\rangle$. As final result for the observable one quotes $\angle\theta\rangle = \tilde{\theta} \pm \sigma_{\tilde{\theta}}$ .
!split
===== Bootstrapping, "code":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py" =====
!bc
# Bootstrap
@@ -1858,7 +1854,7 @@ the true $\angle\theta\rangle$. As final result for the observable one quotes $\
self.bootStd = np.std(bootVec)
!ec
!split
===== Jackknife, "code":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/Programs/Sampling/analysis.py" =====
!bc
# Jackknife