-
@@ -180,7 +192,7 @@ MathJax.Hub.Config({
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring $$ -\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0, +\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, $$ which results in $$ -\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0, +\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ or in a matrix-vector form as @@ -192,6 +204,12 @@ $$
-How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then + +
+Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable. + +
+Introducing the standard deviation \( \sigma_i \) for each measurement \( y_i \), we define now the \( \chi^2 \) function as $$ -\hat{A} = \hat{U}\hat{\Sigma}\hat{V} +\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right), $$ + +where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements. + +
-
diff --git a/doc/pub/Regression/html/._Regression-bs013.html b/doc/pub/Regression/html/._Regression-bs013.html new file mode 100644 index 000000000..7d35cf836 --- /dev/null +++ b/doc/pub/Regression/html/._Regression-bs013.html @@ -0,0 +1,223 @@ + + + + + + + +
+ + + + +
+ +
+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\hat{\beta}) \) by requiring +$$ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0, +$$ + +which results in +$$ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0, +$$ + +or in a matrix-vector form as +$$ +\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right). +$$ + +where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \hat{b} \) with elements \( b_i = y_i/\sigma_i \). +
+
+ +
+ + +
+ + + + +
+ +
+We can rewrite +$$ +\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right), +$$ + +as +$$ +\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta}, +$$ + +and if the matrix \( \hat{A}^T\hat{A} \) is invertible we have the solution +$$ +\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}. +$$ +
+
+ +
+ + +
+ + + + +
+ +
+If we then introduce the matrix +$$ +\hat{H} = \hat{A}^T\hat{A}, +$$ + +we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \hat{H} \) are \( h_{ij} \)) +$$ +\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik} +$$ + +We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as +$$ +\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2, +$$ + +resulting in +$$ +\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}! +$$ +
+
+ +
+ + +
+ + + + +
+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write +$$ +y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i. +$$ + +By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by +$$ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0, +$$ + +and +$$ +\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0. +$$ +
+
+ +
+ + +
+ + + + +
+ +
+We define then +$$ +\gamma = \sum_{i=0}^{1}\frac{1}{\sigma_i^2}, +$$ + + +$$ +\gamma_x = \sum_{i=0}^{1}\frac{x_{i}}{\sigma_i^2}, +$$ + +$$ +\gamma_y = \sum_{i=0}^{1}\left(\frac{y_i}{\sigma_i^2}\right), +$$ + +$$ +\gamma_{xx} = \sum_{i=0}^{1}\frac{x_ix_{i}}{\sigma_i^2}, +$$ + +$$ +\gamma_{xy} = \sum_{i=0}^{1}\frac{y_ix_{i}}{\sigma_i^2}, +$$ + +and show that +$$ +\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}, +$$ + +$$ +\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}. +$$ + +
+The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients \( \beta_i \). A better approach is to use the Singular Value Decomposition (SVD) method discussed below. +
+
+ +
+ + +
+ + + + +
+How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then +$$ +\hat{A} = \hat{U}\hat{\Sigma}\hat{V} +$$ +
+ +
+ +
+ + +-
@@ -180,7 +192,7 @@ MathJax.Hub.Config({
-
@@ -404,14 +403,14 @@ where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till n In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
+Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
+
+
+Introducing the standard deviation \( \sigma_i \) for each measurement \( y_i \), we define now the \( \chi^2 \) function as
+
+
+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\hat{\beta}) \) by requiring
+
+We can rewrite
+
+If we then introduce the matrix
+
+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write
+
+We define then
+
+The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients \( \beta_i \). A better approach is to use the Singular Value Decomposition (SVD) method discussed below.
+
diff --git a/doc/pub/Regression/html/Regression-solarized.html b/doc/pub/Regression/html/Regression-solarized.html
index f72b8d2f1..1c426581d 100644
--- a/doc/pub/Regression/html/Regression-solarized.html
+++ b/doc/pub/Regression/html/Regression-solarized.html
@@ -92,7 +92,13 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec10'),
- ('The singular value decompostion', 2, None, '___sec11')]}
+ ('The $\\chi^2$ function', 2, None, '___sec11'),
+ ('The $\\chi^2$ function', 2, None, '___sec12'),
+ ('The $\\chi^2$ function', 2, None, '___sec13'),
+ ('The $\\chi^2$ function', 2, None, '___sec14'),
+ ('The $\\chi^2$ function', 2, None, '___sec15'),
+ ('The $\\chi^2$ function', 2, None, '___sec16'),
+ ('The singular value decompostion', 2, None, '___sec17')]}
end of tocinfo -->
-
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as
@@ -434,7 +440,184 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
+
+
+Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
+
+
+Introducing the standard deviation \( \sigma_i \) for each measurement \( y_i \), we define now the \( \chi^2 \) function as
+$$
+\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right),
+$$
+
+where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements.
+
+
+
+
+
+
+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\hat{\beta}) \) by requiring
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
+$$
+
+which results in
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
+$$
+
+or in a matrix-vector form as
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right).
+$$
+
+where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \hat{b} \) with elements \( b_i = y_i/\sigma_i \).
+
+
+
+
+We can rewrite
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right),
+$$
+
+as
+$$
+\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta},
+$$
+
+and if the matrix \( \hat{A}^T\hat{A} \) is invertible we have the solution
+$$
+\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}.
+$$
+
+
+
+
+If we then introduce the matrix
+$$
+\hat{H} = \hat{A}^T\hat{A},
+$$
+
+we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \hat{H} \) are \( h_{ij} \))
+$$
+\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
+$$
+
+We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as
+$$
+\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
+$$
+
+resulting in
+$$
+\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
+$$
+
+
+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write
+$$
+y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
+$$
+
+By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
+$$
+
+and
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
+$$
+
+
+
+
+We define then
+$$
+\gamma = \sum_{i=0}^{1}\frac{1}{\sigma_i^2},
+$$
+
+
+$$
+\gamma_x = \sum_{i=0}^{1}\frac{x_{i}}{\sigma_i^2},
+$$
+
+$$
+\gamma_y = \sum_{i=0}^{1}\left(\frac{y_i}{\sigma_i^2}\right),
+$$
+
+$$
+\gamma_{xx} = \sum_{i=0}^{1}\frac{x_ix_{i}}{\sigma_i^2},
+$$
+
+$$
+\gamma_{xy} = \sum_{i=0}^{1}\frac{y_ix_{i}}{\sigma_i^2},
+$$
+
+and show that
+$$
+\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
+$$
+
+$$
+\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
+$$
+
+
+The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients \( \beta_i \). A better approach is to use the Singular Value Decomposition (SVD) method discussed below.
+
+
diff --git a/doc/pub/Regression/html/Regression.html b/doc/pub/Regression/html/Regression.html
index 80a3c96fb..1dac246a3 100644
--- a/doc/pub/Regression/html/Regression.html
+++ b/doc/pub/Regression/html/Regression.html
@@ -97,7 +97,13 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec10'),
- ('The singular value decompostion', 2, None, '___sec11')]}
+ ('The $\\chi^2$ function', 2, None, '___sec11'),
+ ('The $\\chi^2$ function', 2, None, '___sec12'),
+ ('The $\\chi^2$ function', 2, None, '___sec13'),
+ ('The $\\chi^2$ function', 2, None, '___sec14'),
+ ('The $\\chi^2$ function', 2, None, '___sec15'),
+ ('The $\\chi^2$ function', 2, None, '___sec16'),
+ ('The singular value decompostion', 2, None, '___sec17')]}
end of tocinfo -->
-
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as
@@ -439,7 +445,184 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
+
+
+Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
+
+
+Introducing the standard deviation \( \sigma_i \) for each measurement \( y_i \), we define now the \( \chi^2 \) function as
+$$
+\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right),
+$$
+
+where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements.
+
+
+
+
+
+
+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\hat{\beta}) \) by requiring
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
+$$
+
+which results in
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
+$$
+
+or in a matrix-vector form as
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right).
+$$
+
+where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \hat{b} \) with elements \( b_i = y_i/\sigma_i \).
+
+
+
+
+We can rewrite
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right),
+$$
+
+as
+$$
+\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta},
+$$
+
+and if the matrix \( \hat{A}^T\hat{A} \) is invertible we have the solution
+$$
+\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}.
+$$
+
+
+
+
+If we then introduce the matrix
+$$
+\hat{H} = \hat{A}^T\hat{A},
+$$
+
+we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \hat{H} \) are \( h_{ij} \))
+$$
+\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
+$$
+
+We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as
+$$
+\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
+$$
+
+resulting in
+$$
+\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
+$$
+
+
+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write
+$$
+y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
+$$
+
+By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
+$$
+
+and
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
+$$
+
+
+
+
+We define then
+$$
+\gamma = \sum_{i=0}^{1}\frac{1}{\sigma_i^2},
+$$
+
+
+$$
+\gamma_x = \sum_{i=0}^{1}\frac{x_{i}}{\sigma_i^2},
+$$
+
+$$
+\gamma_y = \sum_{i=0}^{1}\left(\frac{y_i}{\sigma_i^2}\right),
+$$
+
+$$
+\gamma_{xx} = \sum_{i=0}^{1}\frac{x_ix_{i}}{\sigma_i^2},
+$$
+
+$$
+\gamma_{xy} = \sum_{i=0}^{1}\frac{y_ix_{i}}{\sigma_i^2},
+$$
+
+and show that
+$$
+\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
+$$
+
+$$
+\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
+$$
+
+
+The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients \( \beta_i \). A better approach is to use the Singular Value Decomposition (SVD) method discussed below.
+
+
diff --git a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz
index 633f50773..bae73d029 100644
Binary files a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz and b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf
index d166a4556..aff0899eb 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf and b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer.pdf b/doc/pub/Regression/pdf/Regression-beamer.pdf
index 012624e18..ca6edc2f4 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer.pdf and b/doc/pub/Regression/pdf/Regression-beamer.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-minted.pdf b/doc/pub/Regression/pdf/Regression-minted.pdf
index 867b17236..e2b7d2b40 100644
Binary files a/doc/pub/Regression/pdf/Regression-minted.pdf and b/doc/pub/Regression/pdf/Regression-minted.pdf differ
diff --git a/doc/src/Regression/Regression.do.txt b/doc/src/Regression/Regression.do.txt
index 9c408a463..4d456bf1b 100644
--- a/doc/src/Regression/Regression.do.txt
+++ b/doc/src/Regression/Regression.do.txt
@@ -198,13 +198,13 @@ where $\langle y_i \rangle$ is the mean value. Keep in mind also that till now
In order to find the parameters $\beta_i$ we will then minimize the spread of $Q(\hat{\beta})$ by requiring
!bt
\[
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
\]
!et
which results in
!bt
\[
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
\]
!et
or in a matrix-vector form as
@@ -267,8 +267,169 @@ meaning that the solution for $\hat{\beta}$ is the one which minimizes the resid
!eblock
+!split
+===== The $\chi^2$ function =====
+!bblock
+Normally, the response (dependent or outcome) variable $y_i$ the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat $y_i$ as our exact value for the response variable.
+Introducing the standard deviation $\sigma_i$ for each measurement $y_i$, we define now the $\chi^2$ function as
+!bt
+\[
+\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right),
+\]
+!et
+where the matrix $\hat{\Sigma}$ is a diagonal matrix with $\sigma_i$ as matrix elements.
+
+!eblock
+
+!split
+===== The $\chi^2$ function =====
+!bblock
+
+In order to find the parameters $\beta_i$ we will then minimize the spread of $\chi^2(\hat{\beta})$ by requiring
+!bt
+\[
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
+\]
+!et
+which results in
+!bt
+\[
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
+\]
+!et
+or in a matrix-vector form as
+!bt
+\[
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right).
+\]
+!et
+where we have defined the matrix $\hat{A} =\hat{X}/\hat{\Sigma}$ with matrix elements $a_{ij} = x_{ij}/\sigma_i$ and the vector $\hat{b}$ with elements $b_i = y_i/\sigma_i$.
+!eblock
+
+!split
+===== The $\chi^2$ function =====
+!bblock
+
+We can rewrite
+!bt
+\[
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right),
+\]
+!et
+as
+!bt
+\[
+\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta},
+\]
+!et
+and if the matrix $\hat{A}^T\hat{A}$ is invertible we have the solution
+!bt
+\[
+\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}.
+\]
+!et
+!eblock
+
+!split
+===== The $\chi^2$ function =====
+!bblock
+
+If we then introduce the matrix
+!bt
+\[
+\hat{H} = \hat{A}^T\hat{A},
+\]
+!et
+we have then the following expression for the parameters $\beta_j$ (the matrix elements of $\hat{H}$ are $h_{ij}$)
+!bt
+\[
+\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
+\]
+!et
+We state without proof the expression for the uncertainty in the parameters $\beta_j$ as
+!bt
+\[
+\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
+\]
+!et
+resulting in
+!bt
+\[
+\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
+\]
+!et
+!eblock
+
+!split
+===== The $\chi^2$ function =====
+!bblock
+The first step here is to approximate the function $y$ with a first-order polynomial, that is we write
+!bt
+\[
+y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
+\]
+!et
+By computing the derivatives of $\chi^2$ with respect to $\beta_0$ and $\beta_1$ show that these are given by
+!bt
+\[
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
+\]
+!et
+and
+!bt
+\[
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
+\]
+!et
+!eblock
+
+!split
+===== The $\chi^2$ function =====
+!bblock
+
+We define then
+!bt
+\[
+\gamma = \sum_{i=0}^{1}\frac{1}{\sigma_i^2},
+\]
+!et
+
+!bt
+\[
+\gamma_x = \sum_{i=0}^{1}\frac{x_{i}}{\sigma_i^2},
+\]
+!et
+!bt
+\[
+\gamma_y = \sum_{i=0}^{1}\left(\frac{y_i}{\sigma_i^2}\right),
+\]
+!et
+!bt
+\[
+\gamma_{xx} = \sum_{i=0}^{1}\frac{x_ix_{i}}{\sigma_i^2},
+\]
+!et
+!bt
+\[
+\gamma_{xy} = \sum_{i=0}^{1}\frac{y_ix_{i}}{\sigma_i^2},
+\]
+!et
+and show that
+!bt
+\[
+\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
+\]
+!et
+!bt
+\[
+\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
+\]
+!et
+
+The LSM suffers often from both being underdetermined and overdetermined in the unknown coefficients $\beta_i$. A better approach is to use the Singular Value Decomposition (SVD) method discussed below.
+!eblock
@@ -287,13 +448,7 @@ How can we use the singular value decomposition to find the parameters $\beta_j$
-Suppose M is a m × n matrix whose entries come from the field K, which is either the field of real numbers or the field of complex numbers. Then there exists a factorization, called a singular value decomposition of M, of the form
-{\displaystyle \mathbf {M} =\mathbf {U} {\boldsymbol {\Sigma }}\mathbf {V} ^{*}} \mathbf {M} =\mathbf {U} {\boldsymbol {\Sigma }}\mathbf {V} ^{*}
-where
-U is an m × m unitary matrix (if K = {\displaystyle \mathbb {R} } \mathbb {R} , unitary matrices are orthogonal matrices),
-Σ is a diagonal m × n matrix with non-negative real numbers on the diagonal,
-V is an n × n unitary matrix over K, and
-V∗ is the conjugate transpose of V.
-The diagonal entries σi of Σ are known as the singular values of M. A common convention is to list the singular values in descending order. In this case, the diagonal matrix, Σ, is uniquely determined by M (though not the matrices U and V, see below).
+
+
@@ -492,7 +491,210 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
The singular value decompostion
+The \( \chi^2 \) function
+
+$$
+\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right),
+$$
+
+
+where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements.
+
+
+The \( \chi^2 \) function
+
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
+$$
+
+
+which results in
+
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
+$$
+
+
+or in a matrix-vector form as
+
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right).
+$$
+
+
+where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \hat{b} \) with elements \( b_i = y_i/\sigma_i \).
+The \( \chi^2 \) function
+
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right),
+$$
+
+
+as
+
+$$
+\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta},
+$$
+
+
+and if the matrix \( \hat{A}^T\hat{A} \) is invertible we have the solution
+
+$$
+\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}.
+$$
+
+The \( \chi^2 \) function
+
+$$
+\hat{H} = \hat{A}^T\hat{A},
+$$
+
+
+we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \hat{H} \) are \( h_{ij} \))
+
+$$
+\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
+$$
+
+
+We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as
+
+$$
+\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
+$$
+
+
+resulting in
+
+$$
+\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
+$$
+
+The \( \chi^2 \) function
+
+$$
+y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
+$$
+
+
+By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by
+
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
+$$
+
+
+and
+
+$$
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
+$$
+
+The \( \chi^2 \) function
+
+$$
+\gamma = \sum_{i=0}^{1}\frac{1}{\sigma_i^2},
+$$
+
+
+
+$$
+\gamma_x = \sum_{i=0}^{1}\frac{x_{i}}{\sigma_i^2},
+$$
+
+
+
+$$
+\gamma_y = \sum_{i=0}^{1}\left(\frac{y_i}{\sigma_i^2}\right),
+$$
+
+
+
+$$
+\gamma_{xx} = \sum_{i=0}^{1}\frac{x_ix_{i}}{\sigma_i^2},
+$$
+
+
+
+$$
+\gamma_{xy} = \sum_{i=0}^{1}\frac{y_ix_{i}}{\sigma_i^2},
+$$
+
+
+and show that
+
+$$
+\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
+$$
+
+
+
+$$
+\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
+$$
+
+
+The singular value decompostion
Oct 18, 2017
Oct 24, 2017
@@ -360,12 +366,12 @@ where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till n
-The singular value decompostion
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The singular value decompostion
Oct 18, 2017
Oct 24, 2017
@@ -365,12 +371,12 @@ where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till n
-The singular value decompostion
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The \( \chi^2 \) function
+
+
+The singular value decompostion