Updated regression analysis, sad to come

doc/src/Regression/Regression.do.txt

@@ -37,13 +37,13 @@ where $\epsilon_i$ is the error in our approximation.
 !bblock
 For every set of values $y_i,x_i$ we have thus the corresponding set of equations
 !bt
-\begin{align}
+\begin{align*}
 y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
 y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
 y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
 \dots & \dots \\
 y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
-\end{align}
+\end{align*}
 !et
 Defining the vectors
 !bt
@@ -89,7 +89,7 @@ we can rewrite our equations as
 We are obviously not limited to the above polynomial. We could replace the various powers of $x$ with elements of Fourier series, that is, instead of $x_i^j$ we could have $\cos{(j x_i)}$ or $\sin{(j x_i)}$, or time series or other orthogonal functions.
 For every set of values $y_i,x_i$ we can then generalize the equations to 
 !bt
-\begin{align}
+\begin{align*}
 y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
 y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
 y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
@@ -97,7 +97,7 @@ y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilo
 y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
 \dots & \dots \\
 y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1}^{n-1,n-1}+\epsilon_{n-1}.\\
-\end{align}
+\end{align*}
 !et
 We redefine in turn the matrix $\hat{X}$ as
 !bt
@@ -127,7 +127,7 @@ The left-hand side of this equation forms know. Our error vector $\hat{\epsilon}
 !bblock
 We have defined the matrix $\hat{X}$
 !bt
-\begin{align}
+\begin{align*}
 y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
 y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
 y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
@@ -135,7 +135,7 @@ y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilo
 y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
 \dots & \dots \\
 y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
-\end{align}
+\end{align*}
 !et
 We well use this matrix to define the approximation $\hat{\tilde{y}}$ via the unknown quantity $\hat{\beta}$ as
 !bt
@@ -146,13 +146,13 @@ We well use this matrix to define the approximation $\hat{\tilde{y}}$ via the un
 and in order to find the optimal parameters $\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parametrized values $\tilde{y}_i$, namely
 !bt
 \[
-Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left \hat{y}-\hat{\tilde{y}}\right)^T\left \hat{y}-\hat{\tilde{y}}\right),
+Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
 \]
 !et
 or using the matrix $\hat{X}$ as
 !bt
 \[
-Q(\hat{\beta})=\left \hat{y}-\hat{X}\hat{\beta}\right)^T\left \hat{y}-\hat{X}\hat{\beta}\right).
+Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
 \]
 !et
 !eblock
@@ -164,7 +164,7 @@ Q(\hat{\beta})=\left \hat{y}-\hat{X}\hat{\beta}\right)^T\left \hat{y}-\hat{X}\ha
 The function 
 !bt
 \[
-Q(\hat{\beta})=\left \hat{y}-\hat{X}\hat{\beta}\right)^T\left \hat{y}-\hat{X}\hat{\beta}\right),
+Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
 \]
 !et
 can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value of for example a numerical  experiment. When linking below with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value
@@ -178,7 +178,7 @@ where $\langle y_i \rangle$ is the mean value. Keep in mind also that till now
 In order to find the parameters $\beta_i$ we will then minimize the spread of $Q(\hat{\beta})$ by requiring
 !bt
 \[
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{ }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0, 
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}\right)^2\right]=0, 
 \]
 !et
 which results in
@@ -196,3 +196,56 @@ or in a matrix-vector form as
 
 
 !eblock
+
+
+!split
+===== Interpretations and optimizing our parameters  =====
+!bblock
+We can rewrite
+!bt
+\[
+\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),  
+\]
+!et
+as
+!bt
+\[
+\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},  
+\]
+!et
+and if the matrix $\hat{X}^T\hat{X}$ is invertible we have the solution
+!bt
+\[
+\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
+\]
+!et
+The residuals $\hat{\epsilon}$ are in turn given by
+!bt
+\[
+\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
+\]
+!et
+and with 
+!bt
+\[
+\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0, 
+\]
+!et
+we have
+!bt
+\[
+\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0, 
+\]
+!et
+meaning that the solution for $\hat{\beta}$ is the one which minimizes the residuals.  Later we will link this with the maximum likelihood approach.
+
+!eblock
+
+
+!split
+===== The singular value decompostion  =====
+!bblock
+Here we derive the equations for the SVD. 
+!eblock
+
+