diff --git a/doc/src/Regression/Regression.do.txt b/doc/src/Regression/Regression.do.txt index c59026388..49576bf37 100644 --- a/doc/src/Regression/Regression.do.txt +++ b/doc/src/Regression/Regression.do.txt @@ -2251,664 +2251,11 @@ bootstrap is widely used. -!split -===== Statistics ===== -!bblock -The *probability distribution function (PDF)* is a function -$p(x)$ on the domain which, in the discrete case, gives us the -probability or relative frequency with which these values of $X$ occur: -!bt -\[ -p(x) = \mathrm{prob}(X=x) -\] -!et -In the continuous case, the PDF does not directly depict the -actual probability. Instead we define the probability for the -stochastic variable to assume any value on an infinitesimal interval -around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us -the *density* of the probability rather than the probability -itself. The probability for a stochastic variable to assume any value -on a non-infinitesimal interval $[a,\,b]$ is then just the integral: -!bt -\[ -\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx -\] -!et -Qualitatively speaking, a stochastic variable represents the values of -numbers chosen as if by chance from some specified PDF so that the -selection of a large set of these numbers reproduces this PDF. -!eblock - - -!split -===== Statistics, moments ===== -!bblock -A particularly useful class of special expectation values are the -*moments*. The $n$-th moment of the PDF $p$ is defined as -follows: -!bt -\[ -\langle x^n\rangle \equiv \int\! x^n p(x)\,dx -\] -!et -The zero-th moment $\langle 1\rangle$ is just the normalization condition of -$p$. The first moment, $\langle x\rangle$, is called the *mean* of $p$ -and often denoted by the letter $\mu$: -!bt -\[ -\langle x\rangle = \mu \equiv \int\! x p(x)\,dx -\] -!et -!eblock - -!split -===== Statistics, central moments ===== -!bblock -A special version of the moments is the set of *central moments*, -the n-th central moment defined as: -!bt -\[ -\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx -\] -!et -The zero-th and first central moments are both trivial, equal $1$ and -$0$, respectively. But the second central moment, known as the -*variance* of $p$, is of particular interest. For the stochastic -variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$: -!bt -\begin{align} -\sigma^2_X\ \ =\ \ \mathrm{var}(X) & = \langle (x-\langle x\rangle)^2\rangle = -\int\! (x-\langle x\rangle)^2 p(x)\,dx\\ -& = \int\! \left(x^2 - 2 x \langle x\rangle^{2} + - \langle x\rangle^2\right)p(x)\,dx\\ -& = \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2\\ -& = \langle x^2\rangle - \langle x\rangle^2 -\end{align} -!et -The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the *standard deviation* of $p$. It is clearly just the RMS (root-mean-square) -value of the deviation of the PDF from its mean value, interpreted -qualitatively as the *spread* of $p$ around its mean. -!eblock - -!split -===== Statistics, covariance ===== -!bblock -Another important quantity is the so called covariance, a variant of -the above defined variance. Consider again the set $\{X_i\}$ of $n$ -stochastic variables (not necessarily uncorrelated) with the -multivariate PDF $P(x_1,\dots,x_n)$. The *covariance* of two -of the stochastic variables, $X_i$ and $X_j$, is defined as follows: -!bt -\begin{align} -\mathrm{cov}(X_i,\,X_j) &\equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle -\nonumber\\ -&= -\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\, -P(x_1,\dots,x_n)\,dx_1\dots dx_n -label{eq:def_covariance} -\end{align} -!et -with -!bt -\[ -\langle x_i\rangle = -\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n -\] -!et -!eblock - -!split -===== Statistics, more covariance ===== -!bblock -If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar -variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that -all the off-diagonal elements are zero if the stochastic variables are -uncorrelated. This is easy to show, keeping in mind the linearity of -the expectation value. Consider the stochastic variables $X_i$ and -$X_j$, ($i\neq j$): -!bt -\begin{align} -\mathrm{cov}(X_i,\,X_j) &= \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle\\ -&=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle \\ -&=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle + -\langle \langle x_i\rangle\langle x_j\rangle\rangle\\ -&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle + -\langle x_i\rangle\langle x_j\rangle\\ -&=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle -\end{align} -!et -!eblock - -!split -===== Covariance example ===== - -Suppose we have defined three vectors $\bm{x}, \bm{y}, \bm{z}$ with -$n$ elements each. The covariance matrix is defined as - - -!bt -\[ -\bm{\Sigma} = \begin{bmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xz} \\ - \sigma_{yx} & \sigma_{yy} & \sigma_{yz} \\ - \sigma_{zx} & \sigma_{zy} & \sigma_{zz} - \end{bmatrix}, -\] -!et -where for example -!bt -\[ -\sigma_{xy} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). -\] -!et - -The Numpy function _np.cov_ calculates the covariance elements using -the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have -the exact mean valu\ es. - -The following simple function uses the _np.vstack_ function which -takes each vector of dimension $1\times n$ and produces a $3\times n$ -matrix $\bm{W}$ - -!bt -\[ -\bm{W} = \begin{bmatrix} x_0 & y_0 & z_0 \\ - x_1 & y_1 & z_1 \\ - x_2 & y_2 & z_2 \\ - \dots & \dots & \dots \\ - x_{n-2} & y_{n-2} & z_{n-2} \\ - x_{n-1} & y_{n-1} & z_{n-1} - \end{bmatrix}, -\] -!et - -which in turn is converted into into the $3\times 3$ covariance matrix -$\bm{\Sigma}$ via the Numpy function _np.cov()_. We note that we can -also calculate the mean value of each set of samples $\bm{x}$ etc -using the Numpy function _np.mean(x)_. We can also extract the -eigenvalues of the covariance matrix through the _np.linalg.eig()_ -function. - - -!split -===== Covariance in numpy ===== - -!bc pycod -# Importing various packages -import numpy as np - -n = 100 -x = np.random.normal(size=n) -print(np.mean(x)) -y = 4+3*x+np.random.normal(size=n) -print(np.mean(y)) -z = x**3+np.random.normal(size=n) -print(np.mean(z)) -W = np.vstack((x, y, z)) -Sigma = np.cov(W) -print(Sigma) -Eigvals, Eigvecs = np.linalg.eig(Sigma) -print(Eigvals) -!ec - -!bc pycod -import numpy as np -import matplotlib.pyplot as plt -from scipy import sparse -eye = np.eye(4) -print(eye) -sparse_mtx = sparse.csr_matrix(eye) -print(sparse_mtx) -x = np.linspace(-10,10,100) -y = np.sin(x) -plt.plot(x,y,marker='x') -plt.show() -!ec - - -!split -===== Statistics, independent variables ===== -!bblock -If $X_i$ and $X_j$ are independent, we get -$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$. - -Also useful for us is the covariance of linear combinations of -stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of -stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of -scalars. Consider the linear combination: -!bt -\[ -U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j -\] -!et -By the linearity of the expectation value -!bt -\[ -\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j) -\] -!et -!eblock - -!split -===== Statistics, more variance ===== -!bblock -Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get -the variance of the linear combination $U = \sum_i a_i X_i$: -!bt -\begin{equation} -\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j) -label{eq:variance_linear_combination} -\end{equation} -!et -And in the special case when the stochastic variables are -uncorrelated, the off-diagonal elements of the covariance are as we -know zero, resulting in: -!bt -\[ -\mathrm{var}(U) = \sum_i a_i^2 \mathrm{cov}(X_i, X_i) = \sum_i a_i^2 \mathrm{var}(X_i) -\] -!et -!bt -\[ -\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i) -\] -!et -which will become very useful in our study of the error in the mean -value of a set of measurements. -!eblock - -!split -===== Statistics and stochastic processes ===== -!bblock -A *stochastic process* is a process that produces sequentially a -chain of values: -!bt -\[ -\{x_1, x_2,\dots\,x_k,\dots\}. -\] -!et -We will call these -values our *measurements* and the entire set as our measured -*sample*. The action of measuring all the elements of a sample -we will call a stochastic *experiment* since, operationally, -they are often associated with results of empirical observation of -some physical or mathematical phenomena; precisely an experiment. We -assume that these values are distributed according to some -PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the -stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of -trying to determine the full distribution $p$ we are often only -interested in finding the few lowest moments, like the mean -$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$. -!eblock - - -!split -===== Statistics and sample variables ===== -!bblock -In practical situations a sample is always of finite size. Let that -size be $n$. The expectation value of a sample, the *sample mean*, is then defined as follows: -!bt -\[ -\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k -\] -!et -The *sample variance* is: -!bt -\[ -\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2 -\] -!et -its square root being the *standard deviation of the sample*. The -*sample covariance* is: -!bt -\[ -\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n) -\] -!et -!eblock - -!split -===== Statistics, sample variance and covariance ===== -!bblock -Note that the sample variance is the sample covariance without the -cross terms. In a similar manner as the covariance in Eq.~(ref{eq:def_covariance}) is a measure of the correlation between -two stochastic variables, the above defined sample covariance is a -measure of the sequential correlation between succeeding measurements -of a sample. - -These quantities, being known experimental values, differ -significantly from and must not be confused with the similarly named -quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$ -and covariance $\mathrm{cov}(X,Y)$. -!eblock - -!split -===== Statistics, law of large numbers ===== -!bblock -The law of large numbers -states that as the size of our sample grows to infinity, the sample -mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF: -!bt -\[ -\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X} -\] -!et -The sample mean $\bar{x}_n$ works therefore as an estimate of the true -mean $\mu_X^{\phantom X}$. - -What we need to find out is how good an approximation $\bar{x}_n$ is to -$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated -mean is of no use to us without a measure of its error. A quantity -that tells us how well we can reproduce it in another experiment. We -are therefore interested in the PDF of the sample mean itself. Its -standard deviation will be a measure of the spread of sample means, -and we will simply call it the *error* of the sample mean, or -just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In -practice, we will only be able to produce an *estimate* of the -sample error since the exact value would require the knowledge of the -true PDFs behind, which we usually do not have. -!eblock - - -!split -===== Statistics, more on sample error ===== -!bblock -Let us first take a look at what happens to the sample error as the -size of the sample grows. In a sample, each of the measurements $x_i$ -can be associated with its own stochastic variable $X_i$. The -stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is -then just a linear combination, already familiar to us: -!bt -\[ -\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i -\] -!et -All the coefficients are just equal $1/n$. The PDF of $\overline X_n$, -denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample -means. -!eblock - -!split -===== Statistics ===== -!bblock -The probability density of obtaining a sample mean $\bar x_n$ -is the product of probabilities of obtaining arbitrary values $x_1, -x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$ -is $\bar x_n$: -!bt -\[ -p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots -\int p_X^{\phantom X}(x_n)\ -\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1 -\] -!et -And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$. -!eblock - - - -!split -===== Statistics, central limit theorem ===== -!bblock -It is generally not possible to express $p_{\overline X_n}(x)$ in a -closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number -$n$. But for the limit $n\to\infty$ it is possible to make an -approximation. The very important result is called *the central limit theorem*. It tells us that as $n$ goes to infinity, -$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean -and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$ -and $\sigma_{X}^{2}$, respectively: -!bt -\begin{equation} -\lim_{n\to\infty} p_{\overline X_n}(x) = -\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2} -e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}} -label{eq:central_limit_gaussian} -\end{equation} -!et -!eblock - -!split -===== Statistics, more technicalities ===== -!bblock -The desired variance -$\mathrm{var}(\overline X_n)$, i.e. the sample error squared -$\mathrm{err}_X^2$, is given by: -!bt -\begin{equation} -\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2} -\sum_{ij} \mathrm{cov}(X_i, X_j) -label{eq:error_exact} -\end{equation} -!et -We see now that in order to calculate the exact error of the sample -with the above expression, we would need the true means -$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To -calculate these requires that we know the true multivariate PDF of all -the $X_i$. But this PDF is unknown to us, we have only got the measurements of -one sample. The best we can do is to let the sample itself be an -estimate of the PDF of each of the $X_i$, estimating all properties of -$X_i$ through the measurements of the sample. -!eblock - - -!split -===== Statistics ===== -!bblock -Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$ -itself, in accordance with the the central limit theorem: -!bt -\[ -\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x -\] -!et -Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an -*estimate* of the covariance in Eq.~(ref{eq:error_exact}) -!bt -\[ -\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle -\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle, -\] -!et -resulting in -!bt -\[ -\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x) -\] -!et -!eblock - -!split -===== Statistics and sample variance ===== -!bblock -By the same procedure we can use the sample variance as an -estimate of the variance of any of the stochastic variables $X_i$ -!bt -\[ -\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber, -\] -!et -which is approximated as -!bt -\begin{equation} -\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x) -label{eq:var_estimate_i_think} -\end{equation} -!et - -Now we can calculate an estimate of the error -$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$: -!bt -\begin{align} -\mathrm{err}_X^2 -&=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber \\ -&\approx&\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber\\ -&=\frac{1}{n}\mathrm{cov}(x) -label{eq:error_estimate} -\end{align} -!et -which is nothing but the sample covariance divided by the number of -measurements in the sample. -!eblock - -!split -===== Statistics, uncorrelated results ===== -!bblock - -In the special case that the measurements of the sample are -uncorrelated (equivalently the stochastic variables $X_i$ are -uncorrelated) we have that the off-diagonal elements of the covariance -are zero. This gives the following estimate of the sample error: -!bt -\[ -\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) = -\frac{1}{n^2} \sum_i \mathrm{var}(X_i), -\] -!et -resulting in -!bt -\begin{equation} -\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x) -label{eq:error_estimate_uncorrel} -\end{equation} -!et -where in the second step we have used Eq.~(ref{eq:var_estimate_i_think}). -The error of the sample is then just its standard deviation divided by -the square root of the number of measurements the sample contains. -This is a very useful formula which is easy to compute. It acts as a -first approximation to the error, but in numerical experiments, we -cannot overlook the always present correlations. -!eblock - -!split -===== Statistics, computations ===== -!bblock -For computational purposes one usually splits up the estimate of -$\mathrm{err}_X^2$, given by Eq.~(ref{eq:error_estimate}), into two -parts -!bt -\[ -\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)), -\] -!et -which equals -!bt -\begin{equation} -\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k