-Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \hat{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \hat{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function.
-Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is
-$$
-y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_i x_i^j+\epsilon_i,
-$$
-
-where \( \epsilon_i \) is the error in our approximation.
+Consider an experiment in which \( p \) characteristics of \( n \) samples are
+measured. The data from this experiment are denoted \( \mathbf{X} \), with
+\( \mathbf{X} \) as above. The matrix \( \mathbf{X} \) is called the design
+matrix. Additional information of the samples is available in the
+form of \( \mathbf{Y} \) (also as above). The variable \( \mathbf{Y} \) is
+generally referred to as the response variable. The aim of
+regression analysis is to explain \( \mathbf{Y} \) in terms of
+\( \mathbf{X} \) through a functional relationship like \( Y_i =
+f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of
+\( f(\cdot) \) is available, it is common to assume a linear relationship
+between \( \mathbf{X} \) and \( \mathbf{Y} \). This assumption gives rise to
+the linear regression model where \( \beta = (\beta_1, \ldots,
+\beta_p)^{\top} \) is the regression parameter. The parameter
+\( \beta_j \), \( j=1, \ldots, p \), represents the effect size of covariate
+\( j \) on the response. That is, for each unit change in covariate \( j \)
+(while keeping the other covariates fixed) the observed change in the
+response is equal to \( \beta_j \).
@@ -240,7 +279,7 @@ where \( \epsilon_i \) is the error in our approximation.
Rewriting the fitting procedure as a linear algebra problem
+
General linear models
-For every set of values \( y_i,x_i \) we have thus the corresponding set of equations
+Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \hat{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \hat{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function.
+
+
+Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is
$$
-\begin{align*}
-y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\
-y_1&=\beta_0+\beta_1x_1^1+\beta_2x_1^2+\dots+\beta_{n-1}x_1^{n-1}+\epsilon_1\\
-y_2&=\beta_0+\beta_1x_2^1+\beta_2x_2^2+\dots+\beta_{n-1}x_2^{n-1}+\epsilon_2\\
-\dots & \dots \\
-y_{n-1}&=\beta_0+\beta_1x_{n-1}^1+\beta_2x_{n-1}^2+\dots+\beta_1x_{n-1}^{n-1}+\epsilon_{n-1}.\\
-\end{align*}
+y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_i x_i^j+\epsilon_i,
$$
+
+where \( \epsilon_i \) is the error in our approximation.
+
+
Generalizing the fitting procedure as a linear algebra problem
+
Rewriting the fitting procedure as a linear algebra problem, follows
-We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
-For every set of values \( y_i,x_i \) we can then generalize the equations to
+Defining the vectors
$$
-\begin{align*}
-y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
-y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
-y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
-\dots & \dots \\
-y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
-\dots & \dots \\
-y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
-\end{align*}
+\hat{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T,
+$$
+
+and
+$$
+\hat{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T,
+$$
+
+and
+$$
+\hat{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T,
+$$
+
+and the matrix
+$$
+\hat{X}=
+\begin{bmatrix}
+1& x_{0}^1 &x_{0}^2& \dots & \dots &x_{0}^{n-1}\\
+1& x_{1}^1 &x_{1}^2& \dots & \dots &x_{1}^{n-1}\\
+1& x_{2}^1 &x_{2}^2& \dots & \dots &x_{2}^{n-1}\\
+\dots& \dots &\dots& \dots & \dots &\dots\\
+1& x_{n-1}^1 &x_{n-1}^2& \dots & \dots &x_{n-1}^{n-1}\\
+\end{bmatrix}
+$$
+
+we can rewrite our equations as
+$$
+\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
$$
-We redefine in turn the matrix \( \hat{X} \) as
+We are obviously not limited to the above polynomial. We could replace the various powers of \( x \) with elements of Fourier series, that is, instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions.
+For every set of values \( y_i,x_i \) we can then generalize the equations to
$$
-\hat{X}=
-\begin{bmatrix}
-x_{00}& x_{01} &x_{02}& \dots & \dots &x_{0,n-1}\\
-x_{10}& x_{11} &x_{12}& \dots & \dots &x_{1,n-1}\\
-x_{20}& x_{21} &x_{22}& \dots & \dots &x_{2,n-1}\\
-\dots& \dots &\dots& \dots & \dots &\dots\\
-x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\
-\end{bmatrix}
+\begin{align*}
+y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
+y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
+y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_2\\
+\dots & \dots \\
+y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_i\\
+\dots & \dots \\
+y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
+\end{align*}
$$
-
-and without loss of generality we rewrite again our equations as
-$$
-\hat{y} = \hat{X}\hat{\beta}+\hat{\epsilon}.
-$$
-
-The left-hand side of this equation forms know. Our error vector \( \hat{\epsilon} \) and the parameter vector \( \hat{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?
@@ -251,7 +275,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
-We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
+We have defined the matrix \( \hat{X} \)
$$
-\hat{\tilde{y}}= \hat{X}\hat{\beta},
-$$
-
-and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
-$$
-Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
-$$
-
-or using the matrix \( \hat{X} \) as
-$$
-Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
+\begin{align*}
+y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\
+y_1&=\beta_0x_{10}+\beta_1x_{11}+\beta_2x_{12}+\dots+\beta_{n-1}x_{1n-1}+\epsilon_1\\
+y_2&=\beta_0x_{20}+\beta_1x_{21}+\beta_2x_{22}+\dots+\beta_{n-1}x_{2n-1}+\epsilon_1\\
+\dots & \dots \\
+y_{i}&=\beta_0x_{i0}+\beta_1x_{i1}+\beta_2x_{i2}+\dots+\beta_{n-1}x_{in-1}+\epsilon_1\\
+\dots & \dots \\
+y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_1x_{n-1,n-1}+\epsilon_{n-1}.\\
+\end{align*}
$$
-The function
+We well use this matrix to define the approximation \( \hat{\tilde{y}} \) via the unknown quantity \( \hat{\beta} \) as
$$
-Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
+\hat{\tilde{y}}= \hat{X}\hat{\beta},
$$
-can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
+and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parametrized values \( \tilde{y}_i \), namely
$$
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+Q(\hat{\beta})=\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\left(\hat{y}-\hat{\tilde{y}}\right)^T\left(\hat{y}-\hat{\tilde{y}}\right),
$$
-where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
-
-
-In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
+or using the matrix \( \hat{X} \) as
$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right).
$$
-
-which results in
-$$
-\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
-$$
-
-or in a matrix-vector form as
-$$
-\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
-$$
-
-
-We can rewrite
+The function
$$
-\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
+Q(\hat{\beta})=\left(\hat{y}-\hat{X}\hat{\beta}\right)^T\left(\hat{y}-\hat{X}\hat{\beta}\right),
$$
-as
+can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value of for example a numerical experiment. When linking below with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
-\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
+y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
$$
-and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
+where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
+
+
+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( Q(\hat{\beta}) \) by requiring
$$
-\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+$$
+
+which results in
+$$
+\frac{\partial Q(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
+$$
+
+or in a matrix-vector form as
+$$
+\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right).
$$
-The residuals \( \hat{\epsilon} \) are in turn given by
+We can rewrite
$$
-\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
+\frac{\partial Q(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right),
$$
-and with
+as
$$
-\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
+\hat{X}^T\hat{y} = \hat{X}^T\hat{X}\hat{\beta},
$$
-we have
+and if the matrix \( \hat{X}^T\hat{X} \) is invertible we have the solution
$$
-\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
+\hat{\beta} =\left(\hat{X}^T\hat{X}\right)^{-1}\hat{X}^T\hat{y}.
$$
-meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
-
@@ -255,7 +282,7 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
-Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
-
-
-Introducing the standard deviation \( \sigma_i \) for each measurement \( y_i \), we define now the \( \chi^2 \) function as
+The residuals \( \hat{\epsilon} \) are in turn given by
$$
-\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right),
+\hat{\epsilon} = \hat{y}-\hat{\tilde{y}} = \hat{y}-\hat{X}\hat{\beta},
$$
-where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements.
+and with
+$$
+\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
+$$
+
+we have
+$$
+\hat{X}^T\hat{\epsilon}=\hat{X}^T\left( \hat{y}-\hat{X}\hat{\beta}\right)= 0,
+$$
+
+meaning that the solution for \( \hat{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.
@@ -250,7 +284,7 @@ where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as
-In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\hat{\beta}) \) by requiring
+Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable.
+
+
+Introducing the standard deviation \( \sigma_i \) for each measurement \( y_i \), we define now the \( \chi^2 \) function as
$$
-\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
+\chi^2(\hat{\beta})=\sum_{i=0}^{n-1}\frac{\left(y_i-\tilde{y}_i\right)^2}{\sigma_i^2}=\left(\hat{y}-\hat{\tilde{y}}\right)^T\frac{1}{\hat{\Sigma^2}}\left(\hat{y}-\hat{\tilde{y}}\right),
$$
-which results in
-$$
-\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
-$$
+where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as matrix elements.
-or in a matrix-vector form as
-$$
-\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right).
-$$
-
-where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \hat{b} \) with elements \( b_i = y_i/\sigma_i \).
+
@@ -255,7 +279,7 @@ where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix
-We can rewrite
+In order to find the parameters \( \beta_i \) we will then minimize the spread of \( \chi^2(\hat{\beta}) \) by requiring
$$
-\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right),
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)^2\right]=0,
$$
-as
+which results in
$$
-\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta},
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_j} = -2\left[ \sum_{i=0}^{n-1}\frac{x_{ij}}{\sigma_i}\left(\frac{y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}}{\sigma_i}\right)\right]=0,
$$
-and if the matrix \( \hat{A}^T\hat{A} \) is invertible we have the solution
+or in a matrix-vector form as
$$
-\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}.
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right).
$$
+
+where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix elements \( a_{ij} = x_{ij}/\sigma_i \) and the vector \( \hat{b} \) with elements \( b_i = y_i/\sigma_i \).
-If we then introduce the matrix
+We can rewrite
$$
-\hat{H} = \left(\hat{A}^T\hat{A}\right)^{-1},
+\frac{\partial \chi^2(\hat{\beta})}{\partial \hat{\beta}} = 0 = \hat{A}^T\left( \hat{b}-\hat{A}\hat{\beta}\right),
$$
-we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \hat{H} \) are \( h_{ij} \))
+as
$$
-\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
+\hat{A}^T\hat{b} = \hat{A}^T\hat{A}\hat{\beta},
$$
-We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as (we leave this as an exercise)
+and if the matrix \( \hat{A}^T\hat{A} \) is invertible we have the solution
$$
-\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
-$$
-
-resulting in
-$$
-\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
+\hat{\beta} =\left(\hat{A}^T\hat{A}\right)^{-1}\hat{A}^T\hat{b}.
$$
@@ -258,7 +282,7 @@ $$
-The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write
+
+
+If we then introduce the matrix
$$
-y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
+\hat{H} = \left(\hat{A}^T\hat{A}\right)^{-1},
$$
-By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by
+we have then the following expression for the parameters \( \beta_j \) (the matrix elements of \( \hat{H} \) are \( h_{ij} \))
$$
-\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
+\beta_j = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}\frac{y_i}{\sigma_i}\frac{x_{ik}}{\sigma_i} = \sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}b_ia_{ik}
$$
-and
+We state without proof the expression for the uncertainty in the parameters \( \beta_j \) as (we leave this as an exercise)
$$
-\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
+\sigma^2(\beta_j) = \sum_{i=0}^{n-1}\sigma_i^2\left( \frac{\partial \beta_j}{\partial y_i}\right)^2,
+$$
+
+resulting in
+$$
+\sigma^2(\beta_j) = \left(\sum_{k=0}^{p-1}h_{jk}\sum_{i=0}^{n-1}a_{ik}\right)\left(\sum_{l=0}^{p-1}h_{jl}\sum_{m=0}^{n-1}a_{ml}\right) = h_{jj}!
$$
-For a linear fit we don't need to invert a matrix!!
-Defining
+The first step here is to approximate the function \( y \) with a first-order polynomial, that is we write
$$
-\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2},
+y=y(x) \rightarrow y(x_i) \approx \beta_0+\beta_1 x_i.
$$
-
+By computing the derivatives of \( \chi^2 \) with respect to \( \beta_0 \) and \( \beta_1 \) show that these are given by
$$
-\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2},
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{n-1}\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0,
$$
+and
$$
-\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right),
+\frac{\partial \chi^2(\hat{\beta})}{\partial \beta_0} = -2\left[ \sum_{i=0}^{n-1}x_i\left(\frac{y_i-\beta_0-\beta_1x_{i}}{\sigma_i^2}\right)\right]=0.
$$
-
-$$
-\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2},
-$$
-
-$$
-\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2},
-$$
-
-we obtain
-$$
-\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
-$$
-
-$$
-\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
-$$
-
-
-This approach (different linear and non-linear regression) suffers often from both being underdetermined and overdetermined in the unknown coefficients \( \beta_i \). A better approach is to use the Singular Value Decomposition (SVD) method discussed below. Or using Lasso and Ridge regression. See below.
@@ -273,7 +280,7 @@ This approach (different linear and non-linear regression) suffers often from bo
-We are now ready to write our first program which aims at solving the above linear regression equations. We start with data we have produced ourselves, in this case normally distributed random numbers along the \( x \)-axis. These numbers define then the value of a function \( y(x)=4+3x+N(0,1) \). Thereafter we order the \( x \) values and employ our linear regression algorithm to set up the best fit. Here we find it useful to use the numpy function \( c\_ \) arrays where arrays are stacked along their last axis after being upgraded to at least two dimensions with ones post-pended to the shape. The following examples help in understanding what happens
+
The \( \chi^2 \) function
+
+
+
+
+For a linear fit we don't need to invert a matrix!!
+Defining
+$$
+\gamma = \sum_{i=0}^{n-1}\frac{1}{\sigma_i^2},
+$$
+
+
+$$
+\gamma_x = \sum_{i=0}^{n-1}\frac{x_{i}}{\sigma_i^2},
+$$
+
+$$
+\gamma_y = \sum_{i=0}^{n-1}\left(\frac{y_i}{\sigma_i^2}\right),
+$$
+
+$$
+\gamma_{xx} = \sum_{i=0}^{n-1}\frac{x_ix_{i}}{\sigma_i^2},
+$$
+
+$$
+\gamma_{xy} = \sum_{i=0}^{n-1}\frac{y_ix_{i}}{\sigma_i^2},
+$$
+
+we obtain
+$$
+\beta_0 = \frac{\gamma_{xx}\gamma_y-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2},
+$$
+
+$$
+\beta_1 = \frac{\gamma_{xy}\gamma-\gamma_x\gamma_y}{\gamma\gamma_{xx}-\gamma_x^2}.
+$$
-
-
+This approach (different linear and non-linear regression) suffers often from both being underdetermined and overdetermined in the unknown coefficients \( \beta_i \). A better approach is to use the Singular Value Decomposition (SVD) method discussed below. Or using Lasso and Ridge regression. See below.
+
+We are now ready to write our first program which aims at solving the above linear regression equations. We start with data we have produced ourselves, in this case normally distributed random numbers along the \( x \)-axis. These numbers define then the value of a function \( y(x)=4+3x+N(0,1) \). Thereafter we order the \( x \) values and employ our linear regression algorithm to set up the best fit. Here we find it useful to use the numpy function \( c\_ \) arrays where arrays are stacked along their last axis after being upgraded to at least two dimensions with ones post-pended to the shape. The following examples help in understanding what happens
-We can repeat the above algorithm using scikit-learn as follows
+
+
+
-We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us.
-What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
-
-
-The Numpy functions are imported used the import numpy as np
-statement and the random number generator for the uniform distribution
-is called using the function np.random.rand(), where we specificy
-that we want \( 100 \) random variables. Using Numpy we define
-automatically an array with the specified number of elements, \( 100 \) in
-our case. With the Numpy function randn() we can compute random
-numbers with the normal distribution (mean value \( \mu \) equal to zero and
-variance \( \sigma^2 \) set to one) and produce the values of \( y \) assuming a linear
-dependence as function of \( x \)
-
-$$
-y = 2x+N(0,1),
-$$
-
-
-where \( N(0,1) \) represents random numbers generated by the normal
-distribution. From scikit-learn we import then the
-LinearRegression functionality and make a prediction \( \tilde{y} =
-\alpha + \beta x \) using the function fit(x,y). We call the set of
-data \( (\hat{x},\hat{y}) \) for our training data. The Python package
-scikit-learn has also a functionality which extracts the above
-fitting parameters \( \alpha \) and \( \beta \) (see below). Later we will
-distinguish between training data and test data.
-
-
-For plotting we use the Python package
-matplotlib which produces publication
-quality figures. Feel free to explore the extensive
-gallery of examples. In
-this example we plot our original values of \( x \) and \( y \) as well as the
-prediction ypredict (\( \tilde{y} \)), which attempts at fitting our
-data with a straight line.
-
-
-The Python code follows here.
+We can repeat the above algorithm using scikit-learn as follows
-This example serves several aims. It allows us to demonstrate several
-aspects of data analysis and later machine learning algorithms. The
-immediate visualization shows that our linear fit is not
-impressive. It goes through the data points, but there are many
-outliers which are not reproduced by our linear regression. We could
-now play around with this small program and change for example the
-factor in front of \( x \) and the normal distribution. Try to change the
-function \( y \) to
+We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us.
+What follows is a simple Python code where we have defined function \( y \) in terms of the variable \( x \). Both are defined as vectors of dimension \( 1\times 100 \). The entries to the vector \( \hat{x} \) are given by random numbers generated with a uniform distribution with entries \( x_i \in [0,1] \) (more about probability distribution functions later). These values are then used to define a function \( y(x) \) (tabulated again as a vector) with a linear dependence on \( x \) plus a random noise added via the normal distribution.
+
+
+The Numpy functions are imported used the import numpy as np
+statement and the random number generator for the uniform distribution
+is called using the function np.random.rand(), where we specificy
+that we want \( 100 \) random variables. Using Numpy we define
+automatically an array with the specified number of elements, \( 100 \) in
+our case. With the Numpy function randn() we can compute random
+numbers with the normal distribution (mean value \( \mu \) equal to zero and
+variance \( \sigma^2 \) set to one) and produce the values of \( y \) assuming a linear
+dependence as function of \( x \)
$$
-y = 10x+0.01 \times N(0,1),
+y = 2x+N(0,1),
$$
-where \( x \) is defined as before.
+where \( N(0,1) \) represents random numbers generated by the normal
+distribution. From scikit-learn we import then the
+LinearRegression functionality and make a prediction \( \tilde{y} =
+\alpha + \beta x \) using the function fit(x,y). We call the set of
+data \( (\hat{x},\hat{y}) \) for our training data. The Python package
+scikit-learn has also a functionality which extracts the above
+fitting parameters \( \alpha \) and \( \beta \) (see below). Later we will
+distinguish between training data and test data.
+
+For plotting we use the Python package
+matplotlib which produces publication
+quality figures. Feel free to explore the extensive
+gallery of examples. In
+this example we plot our original values of \( x \) and \( y \) as well as the
+prediction ypredict (\( \tilde{y} \)), which attempts at fitting our
+data with a straight line.
+
+
-Does the fit look better? Indeed, by
-reducing the role of the normal distribution we see immediately that
-our linear prediction seemingly reproduces better the training
-set. However, this testing 'by the eye' is obviouly not satisfactory in the
-long run. Here we have only defined the training data and our model, and
-have not discussed a more rigorous approach to the cost function.
+This example serves several aims. It allows us to demonstrate several
+aspects of data analysis and later machine learning algorithms. The
+immediate visualization shows that our linear fit is not
+impressive. It goes through the data points, but there are many
+outliers which are not reproduced by our linear regression. We could
+now play around with this small program and change for example the
+factor in front of \( x \) and the normal distribution. Try to change the
+function \( y \) to
+
+$$
+y = 10x+0.01 \times N(0,1),
+$$
+
+
+where \( x \) is defined as before.
@@ -239,7 +277,7 @@ have not discussed a more rigorous approach to the cost function.
-We need more rigorous criteria in defining whether we have succeeded or
-not in modeling our training data. You will be surprised to see that
-many scientists seldomly venture beyond this 'by the eye' approach. A
-standard approach for the cost function is the so-called \( \chi^2 \)
-function
-
-$$ \chi^2 = \frac{1}{n}
-\sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2},
-$$
-
-
-where \( \sigma_i^2 \) is the variance (to be defined later) of the entry
-\( y_i \). We may not know the explicit value of \( \sigma_i^2 \), it serves
-however the aim of scaling the equations and make the cost function
-dimensionless.
+Does the fit look better? Indeed, by
+reducing the role of the normal distribution we see immediately that
+our linear prediction seemingly reproduces better the training
+set. However, this testing 'by the eye' is obviouly not satisfactory in the
+long run. Here we have only defined the training data and our model, and
+have not discussed a more rigorous approach to the cost function.
-Minimizing the cost function is a central aspect of
-our discussions to come. Finding its minima as function of the model
-parameters (\( \alpha \) and \( \beta \) in our case) will be a recurring
-theme in these series of lectures. Essentially all machine learning
-algorithms we will discuss center around the minimization of the
-chosen cost function. This depends in turn on our specific
-model for describing the data, a typical situation in supervised
-learning. Automatizing the search for the minima of the cost function is a
-central ingredient in all algorithms. Typical methods which are
-employed are various variants of gradient methods. These will be
-discussed in more detail later. Again, you'll be surprised to hear that
-many practitioners minimize the above function ''by the eye', popularly dubbed as
-'chi by the eye'. That is, change a parameter and see (visually and numerically) that
-the \( \chi^2 \) function becomes smaller.
+We need more rigorous criteria in defining whether we have succeeded or
+not in modeling our training data. You will be surprised to see that
+many scientists seldomly venture beyond this 'by the eye' approach. A
+standard approach for the cost function is the so-called \( \chi^2 \)
+function
+
+$$ \chi^2 = \frac{1}{n}
+\sum_{i=0}^{n-1}\frac{(y_i-\tilde{y}_i)^2}{\sigma_i^2},
+$$
+
+
+where \( \sigma_i^2 \) is the variance (to be defined later) of the entry
+\( y_i \). We may not know the explicit value of \( \sigma_i^2 \), it serves
+however the aim of scaling the equations and make the cost function
+dimensionless.
@@ -247,7 +277,7 @@ the \( \chi^2 \) function becomes smaller.
-There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
-the relative error as
-
-$$
-\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
-$$
-
-We can modify easily the above Python code and plot the relative error instead
-
-Depending on the parameter in front of the normal distribution, we may
-have a small or larger relative error. Try to play around with
-different training data sets and study (graphically) the value of the
-relative error.
+Minimizing the cost function is a central aspect of
+our discussions to come. Finding its minima as function of the model
+parameters (\( \alpha \) and \( \beta \) in our case) will be a recurring
+theme in these series of lectures. Essentially all machine learning
+algorithms we will discuss center around the minimization of the
+chosen cost function. This depends in turn on our specific
+model for describing the data, a typical situation in supervised
+learning. Automatizing the search for the minima of the cost function is a
+central ingredient in all algorithms. Typical methods which are
+employed are various variants of gradient methods. These will be
+discussed in more detail later. Again, you'll be surprised to hear that
+many practitioners minimize the above function ''by the eye', popularly dubbed as
+'chi by the eye'. That is, change a parameter and see (visually and numerically) that
+the \( \chi^2 \) function becomes smaller.
-As mentioned above, scikit-learn has an impressive functionality.
-We can for example extract the values of \( \alpha \) and \( \beta \) and
-their error estimates, or the variance and standard deviation and many
-other properties from the statistical data analysis.
+There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
+the relative error as
-
-Here we show an
-example of the functionality of scikit-learn.
+$$
+\epsilon_{\mathrm{relative}}= \frac{\vert \hat{y} -\hat{\tilde{y}}\vert}{\vert \hat{y}\vert}.
+$$
+
+We can modify easily the above Python code and plot the relative error instead
+Depending on the parameter in front of the normal distribution, we may
+have a small or larger relative error. Try to play around with
+different training data sets and study (graphically) the value of the
+relative error.
+
-The function coef gives us the parameter \( \beta \) of our fit while intercept yields
-\( \alpha \). Depending on the constant in front of the normal distribution, we get values near or far from \( alpha =2 \) and \( \beta =5 \). Try to play around with different parameters in front of the normal distribution. The function meansquarederror gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
-$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
-\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
-$$
+As mentioned above, scikit-learn has an impressive functionality.
+We can for example extract the values of \( \alpha \) and \( \beta \) and
+their error estimates, or the variance and standard deviation and many
+other properties from the statistical data analysis.
-The smaller the value, the better the fit. Ideally we would like to
-have an MSE equal zero. The attentive reader has probably recognized
-this function as being similar to the \( \chi^2 \) function defined above.
+Here we show an
+example of the functionality of scikit-learn.
+
-The r2score function computes \( R^2 \), the coefficient of
-determination. It provides a measure of how well future samples are
-likely to be predicted by the model. Best possible score is 1.0 and it
-can be negative (because the model can be arbitrarily worse). A
-constant model that always predicts the expected value of \( \hat{y} \),
-disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
+The function coef gives us the parameter \( \beta \) of our fit while intercept yields
+\( \alpha \). Depending on the constant in front of the normal distribution, we get values near or far from \( alpha =2 \) and \( \beta =5 \). Try to play around with different parameters in front of the normal distribution. The function meansquarederror gives us the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error or loss defined as
+$$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
+\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
+$$
-If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
-$$
-R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
-$$
-
-where we have defined the mean value of \( \hat{y} \) as
-$$
-\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
-$$
+The smaller the value, the better the fit. Ideally we would like to
+have an MSE equal zero. The attentive reader has probably recognized
+this function as being similar to the \( \chi^2 \) function defined above.
The mean absolute error and other functions in scikit-learn
+
Other functions in scikit-learn
-Another quantity will meet again in our discussions of regression analysis is
- mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
-The MAE is defined as follows
-$$
-\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
-$$
-
-Finally we present the
-squared logarithmic (quadratic) error
-$$
-\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
-$$
+The r2score function computes \( R^2 \), the coefficient of
+determination. It provides a measure of how well future samples are
+likely to be predicted by the model. Best possible score is 1.0 and it
+can be negative (because the model can be arbitrarily worse). A
+constant model that always predicts the expected value of \( \hat{y} \),
+disregarding the input features, would get a \( R^2 \) score of \( 0.0 \).
-where \( \log_e (x) \) stands for the natural logarithm of \( x \). This error
-estimate is best to use when targets having exponential growth, such
-as population counts, average sales of a commodity over a span of
-years etc.
+If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as
+$$
+R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+$$
+
+where we have defined the mean value of \( \hat{y} \) as
+$$
+\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+$$
The mean absolute error and other functions in scikit-learn
-We will discuss in more
-detail these and other functions in the various lectures. We conclude this part with another example. Instead of
-a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+Another quantity will meet again in our discussions of regression analysis is
+ mean absolute error (MAE), a risk metric corresponding to the expected value of the absolute error loss or what we call the \( l1 \)-norm loss. In our discussion above we presented the relative error.
+The MAE is defined as follows
+$$
+\text{MAE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1} \left| y_i - \tilde{y}_i \right|.
+$$
+
+Finally we present the
+squared logarithmic (quadratic) error
+$$
+\text{MSLE}(\hat{y}, \hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n - 1} (\log_e (1 + y_i) - \log_e (1 + \tilde{y}_i) )^2,
+$$
-Similarly, using R, we can perform similar studies.
-(more details on R will be inserted later).
+where \( \log_e (x) \) stands for the natural logarithm of \( x \). This error
+estimate is best to use when targets having exponential growth, such
+as population counts, average sales of a commodity over a span of
+years etc.
@@ -271,6 +277,10 @@ Similarly, using R, we can perform similar studies.
-Add info about the equations, play around with different learning rates
+
Cubic polynomial in scikit-learn
+
+
+We will discuss in more
+detail these and other functions in the various lectures. We conclude this part with another example. Instead of
+a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+Add description of the various python commands.
+
Linking the regression analysis with a statistical interpretation
-
Polynomial Regression
+Before we proceed, and to link with our discussions of Bayesian statistics to come, it is useful the derive the standard regression analysis equations using a statistical interpretation. This allows us also to derive quantities like the variance and other expectation values in a rather straightforward way.
-
-
+It is assumed that \( \varepsilon_i
+\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are
+independent, i.e.:
+$$
+\begin{align*}
+\mbox{Cov}(\varepsilon_{i_1},
+\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if}
+& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right.
+\end{align*}
+$$
-m =100
-x =2*np.random.rand(m,1)+4.
-y =4+3*x*x++x-np.random.randn(m,1)
+The randomness of \( \varepsilon_i \) implies that
+\( \mathbf{Y}_i \) is also a random variable. In particular,
+\( \mathbf{Y}_i \) is normally distributed, because \( \varepsilon_i \sim
+\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \beta \) is a
+non-random scalar. To specify the parameters of the distribution of
+\( \mathbf{Y}_i \) we need to calculate its first two moments.
-xb = np.c_[np.ones((m,1)), x]
-theta = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)
-xnew = np.array([[0],[2]])
-xbnew = np.c_[np.ones((2,1)), xnew]
-ypredict = xbnew.dot(theta)
-
-plt.plot(xnew, ypredict, "r-")
-plt.plot(x, y ,'ro')
-plt.axis([0,2.0,0, 15.0])
-plt.xlabel(r'$x$')
-plt.ylabel(r'$y$')
-plt.title(r'Random numbers ')
-plt.show()
-
-How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
-$$
-\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
-$$
-
-
-
-Add codes and discuss this in connection with lasso and ridge, show example where the standard inversion of a matrix fails and where SVD comes to rescue
+A general
+\( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal
+matrix \( \hat{D} \) of dimensionality \( n\times n \) and two orthognal
+matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality
+\( m \times m \) and the last dimensionality \( n\times n \).
+We have then
+$$
+\hat{A} = \hat{U}\hat{D}\hat{V}^T
+$$
+
+
+
@@ -236,6 +269,15 @@ Add codes and discuss this in connection with lasso and ridge, show example wher
-Add discussion about classification versus regression, show examples of more than two cases and why regression is not the best approach. Motivate for k-nearest neighbors
+
From standard regression to Ridge regressions
-Add examples on classification problems
+One of the typical problems we encounter with linear regression, in particular
+when the matrix \( \hat{X} \) (our so-called design matrix) is high-dimensional,
+are problems with near singular or singular matrices. The column vectors of \( \hat{X} \)
+may be linearly dependent, normally referred to as super-collinearity.
+This means that the matrix may be rank deficient and it is basically impossible to
+to model the data using linear regression. As an example, consider the matrix
+$$
+\begin{align*}
+\mathbf{X} & = \left[
+\begin{array}{rrr}
+1 & -1 & 2
+\\
+1 & 0 & 1
+\\
+1 & 2 & -1
+\\
+1 & 1 & 0
+\end{array} \right]
+\end{align*}
+$$
+The columns of \( \hat{X} \) are linearly dependent. We se this easily since the
+the first column is the row-wise sum of the other two columns. The rank (more correct,
+the column rank) of a matrix is the dimension of the space spanned by the
+column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number
+of linearly independent columns. In this particular case the matrix has rank 2.
+
+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies
+that the inverse of the matrix \( \hat{X}^T\hat{x} \) (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
+$$
+\begin{align*}
+\hat{X} & = \left[
+\begin{array}{rr}
+1 & -1
+\\
+1 & -1
+\end{array} \right].
+\end{align*}
+$$
+
+We see easily that \( \mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined.
+This is equivalent to saying that the matrix \( \hat{X} \) has at least an eigenvalue which is zero.
+
+
@@ -226,6 +296,15 @@ Add examples on classification problems
+If our design matrix \( \hat{X} \) which enters the linear regression problem
+$$
+\begin{align}
+\hat{\beta} & = (\hat{X}^{T} \hat{X})^{-1} \hat{X}^{T} \hat{y},
+\tag{1}
+\end{align}
+$$
+
+has linearly dependent column vectors, we will not be able to compute the inverse
+of \( \hat{X}^T\hat{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+The estimators are only well-defined if \( (\hat{X}^{T}\hat{X})^{-1} \) exits.
+This is more likely to happen when the matrix \( \hat{X} \) is high-dimensional. In this case it is likely to encounter a situation where
+the regression parameters \( \beta_i \) cannot be estimated.
+
+
+The ad hoc approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change
+$$
+\hat{X}^{T} \hat{X} \rightarrow \hat{X}^{T} \hat{X}+\lambda \hat{I},
+$$
+
+where \( \hat{I} \) is the identity matrix.
+
+
Fitting vs. predicting when data is in the model class
+
+
+We start by considering the case
+\( f(x)=2x \).
+
+
+Then the data is clearly generated by a model that is contained within
+all three model classes we are using to make predictions (linear
+models, third order polynomials, and tenth order polynomials).
+
+
+Run the code for the following cases:
+
+
+
For \( f(x)=2x \) , \( Ntrain=10 \) and \( \sigma =0 \) (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when \( x \in [0,1] \) . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?
+
Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?
+
Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example \( x \in [0,1.2] \) ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?
+
Repeat the above for \( f(x)=2x \) , \( Ntrain=10 \) , and \( \sigma=1 \) . What changes?
+
+
+Repeat the exercises above for \( f(x)=2x \) , \( Ntrain=100 \) , and \( \sigma=1 \) . What changes?
+Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.
+
+
Fitting versus predicting when data is not in the model class
+
+
+Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider \( f(x)=2x-10x^5+15x^{10} \) . Notice that the for linear and third-order polynomial the true model \( f(x) \) is not contained in model class.
+
+
+
Do better fits lead to better predictions?
+
What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points \( Ntrain \) and \( \sigma \)?
+
+
+Summarize what you think you learned about the relationship of knowing the true model class and predictive power.
+
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Aug 24, 2018
+
Sep 6, 2018
@@ -182,7 +182,35 @@ A regression model aims at finding a likelihood function \( p(y\vert \hat{x}) \)
-
General linear models
+
Regression analysis, overarching aims II
+
+
+
+Consider an experiment in which \( p \) characteristics of \( n \) samples are
+measured. The data from this experiment are denoted \( \mathbf{X} \), with
+\( \mathbf{X} \) as above. The matrix \( \mathbf{X} \) is called the design
+matrix. Additional information of the samples is available in the
+form of \( \mathbf{Y} \) (also as above). The variable \( \mathbf{Y} \) is
+generally referred to as the response variable. The aim of
+regression analysis is to explain \( \mathbf{Y} \) in terms of
+\( \mathbf{X} \) through a functional relationship like \( Y_i =
+f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of
+\( f(\cdot) \) is available, it is common to assume a linear relationship
+between \( \mathbf{X} \) and \( \mathbf{Y} \). This assumption gives rise to
+the linear regression model where \( \beta = (\beta_1, \ldots,
+\beta_p)^{\top} \) is the regression parameter. The parameter
+\( \beta_j \), \( j=1, \ldots, p \), represents the effect size of covariate
+\( j \) on the response. That is, for each unit change in covariate \( j \)
+(while keeping the other covariates fixed) the observed change in the
+response is equal to \( \beta_j \).
+
+
+
+
+
+
+
+
General linear models
@@ -204,7 +232,7 @@ where \( \epsilon_i \) is the error in our approximation.
-
Rewriting the fitting procedure as a linear algebra problem
+
Rewriting the fitting procedure as a linear algebra problem
@@ -225,7 +253,7 @@ $$
-
Rewriting the fitting procedure as a linear algebra problem, follows
+
Rewriting the fitting procedure as a linear algebra problem, follows
@@ -275,7 +303,7 @@ $$
-
Generalizing the fitting procedure as a linear algebra problem
+
Generalizing the fitting procedure as a linear algebra problem
@@ -299,7 +327,7 @@ $$
-
Generalizing the fitting procedure as a linear algebra problem
+
Generalizing the fitting procedure as a linear algebra problem
@@ -330,7 +358,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
-
Optimizing our parameters
+
Optimizing our parameters
@@ -353,7 +381,7 @@ $$
-
Optimizing our parameters, more details
+
Optimizing our parameters, more details
@@ -382,7 +410,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -430,7 +458,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -461,7 +489,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -494,7 +522,7 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -517,7 +545,7 @@ where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -548,7 +576,7 @@ where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -577,7 +605,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -613,7 +641,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -642,7 +670,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -698,7 +726,7 @@ This approach (different linear and non-linear regression) suffers often from bo
-
Simple regression model
+
Simple regression model
We are now ready to write our first program which aims at solving the above linear regression equations. We start with data we have produced ourselves, in this case normally distributed random numbers along the \( x \)-axis. These numbers define then the value of a function \( y(x)=4+3x+N(0,1) \). Thereafter we order the \( x \) values and employ our linear regression algorithm to set up the best fit. Here we find it useful to use the numpy function \( c\_ \) arrays where arrays are stacked along their last axis after being upgraded to at least two dimensions with ones post-pended to the shape. The following examples help in understanding what happens
@@ -738,7 +766,7 @@ We see that, as expected, a linear fit gives a seemingly (from the graph) good r
-
Simple regression model, now using scikit-learn
+
Simple regression model, now using scikit-learn
We can repeat the above algorithm using scikit-learn as follows
@@ -770,7 +798,7 @@ plt.show()
-
Simple linear regression model using scikit-learn
+
Simple linear regression model using scikit-learn
We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us.
@@ -841,7 +869,7 @@ plt.show()
-
Simple linear regression model
+
Simple linear regression model
This example serves several aims. It allows us to demonstrate several
@@ -865,7 +893,7 @@ where \( x \) is defined as before.
-
Less noise
+
Less noise
Does the fit look better? Indeed, by
@@ -878,7 +906,7 @@ have not discussed a more rigorous approach to the cost function.
-
How to study our fits
+
How to study our fits
We need more rigorous criteria in defining whether we have succeeded or
@@ -902,7 +930,7 @@ dimensionless.
-
Minimizing the cost function
+
Minimizing the cost function
Minimizing the cost function is a central aspect of
@@ -923,7 +951,7 @@ the \( \chi^2 \) function becomes smaller.
-
Relative error
+
Relative error
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
@@ -965,7 +993,7 @@ relative error.
-
The richness of scikit-learn
+
The richness of scikit-learn
As mentioned above, scikit-learn has an impressive functionality.
@@ -1011,7 +1039,7 @@ plt.show()
-
Functions in scikit-learn
+
Functions in scikit-learn
The function coef gives us the parameter \( \beta \) of our fit while intercept yields
@@ -1030,7 +1058,7 @@ this function as being similar to the \( \chi^2 \) function defined above.
-
Other functions in scikit-learn
+
Other functions in scikit-learn
The r2score function computes \( R^2 \), the coefficient of
@@ -1058,7 +1086,7 @@ $$
-
The mean absolute error and other functions in scikit-learn
+
The mean absolute error and other functions in scikit-learn
Another quantity will meet again in our discussions of regression analysis is
@@ -1087,12 +1115,13 @@ years etc.
-
Cubic polynomial in scikit-learn
+
Cubic polynomial in scikit-learn
We will discuss in more
detail these and other functions in the various lectures. We conclude this part with another example. Instead of
-a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+Add description of the various python commands.
Linking the regression analysis with a statistical interpretation
+
+
+Before we proceed, and to link with our discussions of Bayesian statistics to come, it is useful the derive the standard regression analysis equations using a statistical interpretation. This allows us also to derive quantities like the variance and other expectation values in a rather straightforward way.
+
+
+It is assumed that \( \varepsilon_i
+\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are
+independent, i.e.:
+
+
+The randomness of \( \varepsilon_i \) implies that
+\( \mathbf{Y}_i \) is also a random variable. In particular,
+\( \mathbf{Y}_i \) is normally distributed, because \( \varepsilon_i \sim
+\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \beta \) is a
+non-random scalar. To specify the parameters of the distribution of
+\( \mathbf{Y}_i \) we need to calculate its first two moments.
+
+A general
+\( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal
+matrix \( \hat{D} \) of dimensionality \( n\times n \) and two orthognal
+matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality
+\( m \times m \) and the last dimensionality \( n\times n \).
+We have then
+
+$$
+\hat{A} = \hat{U}\hat{D}\hat{V}^T
+$$
+
+
+
+
+
+
+
Code examples for Ridge and Lasso Regression
@@ -1320,37 +1364,403 @@ plt.show()
-
The singular value decompostion
-
-
-
-How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
-
-$$
-\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
-$$
-
-
+
From standard regression to Ridge regressions
-Add codes and discuss this in connection with lasso and ridge, show example where the standard inversion of a matrix fails and where SVD comes to rescue
+One of the typical problems we encounter with linear regression, in particular
+when the matrix \( \hat{X} \) (our so-called design matrix) is high-dimensional,
+are problems with near singular or singular matrices. The column vectors of \( \hat{X} \)
+may be linearly dependent, normally referred to as super-collinearity.
+This means that the matrix may be rank deficient and it is basically impossible to
+to model the data using linear regression. As an example, consider the matrix
+
+The columns of \( \hat{X} \) are linearly dependent. We se this easily since the
+the first column is the row-wise sum of the other two columns. The rank (more correct,
+the column rank) of a matrix is the dimension of the space spanned by the
+column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number
+of linearly independent columns. In this particular case the matrix has rank 2.
+
+
+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies
+that the inverse of the matrix \( \hat{X}^T\hat{x} \) (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
+
+
+We see easily that \( \mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined.
+This is equivalent to saying that the matrix \( \hat{X} \) has at least an eigenvalue which is zero.
-
Lasso and Ridge regression
+
Fixing the singularity
-Discuss the mathematics here
+If our design matrix \( \hat{X} \) which enters the linear regression problem
+
+
+has linearly dependent column vectors, we will not be able to compute the inverse
+of \( \hat{X}^T\hat{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+The estimators are only well-defined if \( (\hat{X}^{T}\hat{X})^{-1} \) exits.
+This is more likely to happen when the matrix \( \hat{X} \) is high-dimensional. In this case it is likely to encounter a situation where
+the regression parameters \( \beta_i \) cannot be estimated.
+
+
+The ad hoc approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change
+
-Add discussion about classification versus regression, show examples of more than two cases and why regression is not the best approach. Motivate for k-nearest neighbors
+
A second-order polynomial with Ridge and Lasso
+
+
+
+
importnumpyasnp
+importmatplotlib.pyplotasplt
+fromsklearn.linear_modelimport Ridge
+fromsklearn.metricsimport r2_score
+
+np.random.seed(4155)
+
+n_samples = 100
+
+x = np.random.rand(n_samples,1)
+y = 5*x*x + 0.1*np.random.rand(n_samples,1)
+
+# Centering x and y.
+x_ = x - np.mean(x)
+y_ = y - np.mean(y) # beta_0 = mean(y)
+
+X = np.c_[np.ones((n_samples,1)), x, x**2]
+X_ = np.c_[x_, x_**2]
+
+
+### 1.
+lmb_values = [1e-4, 1e-3, 1e-2, 10, 1e2, 1e4]
+num_values = len(lmb_values)
+
+## Ridge-regression of centered and not centered data
+beta_ridge = np.zeros((3,num_values))
+beta_ridge_centered = np.zeros((3,num_values))
+
+I3 = np.eye(3)
+I2 = np.eye(2)
+
+for i,lmb inenumerate(lmb_values):
+ beta_ridge[:,i] = (np.linalg.inv( X.T @ X + lmb*I3) @ X.T @ y).flatten()
+ beta_ridge_centered[1:,i] = (np.linalg.inv( X_.T @ X_ + lmb*I2) @ X_.T @ y_).flatten()
+
+# sett beta_0 = np.mean(y)
+beta_ridge_centered[0,:] = np.mean(y)
+
+## OLS (ordinary least squares) solution
+beta_ls = np.linalg.inv( X.T @ X ) @ X.T @ y
+
+## Evaluate the models
+pred_ls = X @ beta_ls
+pred_ridge = X @ beta_ridge
+pred_ridge_centered = X_ @ beta_ridge_centered[1:] + beta_ridge_centered[0,:]
+
+## Plot the results
+
+# Sorting
+sort_ind = np.argsort(x[:,0])
+
+x_plot = x[sort_ind,0]
+x_centered_plot = x_[sort_ind,0]
+
+pred_ls_plot = pred_ls[sort_ind,0]
+pred_ridge_plot = pred_ridge[sort_ind,:]
+pred_ridge_centered_plot = pred_ridge_centered[sort_ind,:]
+
+# Plott not centered
+plt.plot(x_plot,pred_ls_plot,label='ls')
+
+for i inrange(num_values):
+ plt.plot(x_plot,pred_ridge_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x,y,'ro')
+
+plt.title('linear regression on un-centered data')
+plt.legend()
+
+# Plott centered
+plt.figure()
+
+for i inrange(num_values):
+ plt.plot(x_centered_plot,pred_ridge_centered_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x_,y,'ro')
+
+plt.title('linear regression on centered data')
+plt.legend()
+
+
+# 2.
+
+pred_ridge_scikit = np.zeros((n_samples,num_values))
+for i,lmb inenumerate(lmb_values):
+ pred_ridge_scikit[:,i] = (Ridge(alpha=lmb,fit_intercept=False).fit(X,y).predict(X)).flatten() # fit_intercept=False fordi bias er allerede i X
+
+plt.figure()
+
+plt.plot(x_plot,pred_ls_plot,label='ls')
+
+for i inrange(num_values):
+ plt.plot(x_plot,pred_ridge_scikit[sort_ind,i],label='scikit-ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x,y,'ro')
+plt.legend()
+plt.title('linear regression using scikit')
+
+plt.show()
+
+### R2-score of the results
+for i inrange(num_values):
+ print('lambda = %g'%lmb_values[i])
+ print('r2 for scikit: %g'%r2_score(y,pred_ridge_scikit[:,i]))
+ print('r2 for own code, not centered: %g'%r2_score(y,pred_ridge[:,i]))
+ print('r2 for own, centered: %g\n'%r2_score(y,pred_ridge_centered[:,i]))
+
+
+
+
+
+
Fitting vs. predicting when data is in the model class
-Add examples on classification problems
+We start by considering the case
+\( f(x)=2x \).
+
+
+Then the data is clearly generated by a model that is contained within
+all three model classes we are using to make predictions (linear
+models, third order polynomials, and tenth order polynomials).
+
+
+Run the code for the following cases:
+
+
+
For \( f(x)=2x \) , \( Ntrain=10 \) and \( \sigma =0 \) (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when \( x \in [0,1] \) . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?
+
Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?
+
Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example \( x \in [0,1.2] \) ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?
+
Repeat the above for \( f(x)=2x \) , \( Ntrain=10 \) , and \( \sigma=1 \) . What changes?
+
+
+
+Repeat the exercises above for \( f(x)=2x \) , \( Ntrain=100 \) , and \( \sigma=1 \) . What changes?
+Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.
+
+
+
+
+
Fitting versus predicting when data is not in the model class
+
+
+Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider \( f(x)=2x-10x^5+15x^{10} \) . Notice that the for linear and third-order polynomial the true model \( f(x) \) is not contained in model class.
+
+
+
Do better fits lead to better predictions?
+
What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points \( Ntrain \) and \( \sigma \)?
+
+
+
+Summarize what you think you learned about the relationship of knowing the true model class and predictive power.
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Aug 24, 2018
+
Sep 6, 2018
@@ -205,7 +226,38 @@ A regression model aims at finding a likelihood function \( p(y\vert \hat{x}) \)
-
General linear models
+
Regression analysis, overarching aims II
+
+
+
+
+
+Consider an experiment in which \( p \) characteristics of \( n \) samples are
+measured. The data from this experiment are denoted \( \mathbf{X} \), with
+\( \mathbf{X} \) as above. The matrix \( \mathbf{X} \) is called the design
+matrix. Additional information of the samples is available in the
+form of \( \mathbf{Y} \) (also as above). The variable \( \mathbf{Y} \) is
+generally referred to as the response variable. The aim of
+regression analysis is to explain \( \mathbf{Y} \) in terms of
+\( \mathbf{X} \) through a functional relationship like \( Y_i =
+f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of
+\( f(\cdot) \) is available, it is common to assume a linear relationship
+between \( \mathbf{X} \) and \( \mathbf{Y} \). This assumption gives rise to
+the linear regression model where \( \beta = (\beta_1, \ldots,
+\beta_p)^{\top} \) is the regression parameter. The parameter
+\( \beta_j \), \( j=1, \ldots, p \), represents the effect size of covariate
+\( j \) on the response. That is, for each unit change in covariate \( j \)
+(while keeping the other covariates fixed) the observed change in the
+response is equal to \( \beta_j \).
+
+
+
+
+
+
+
+
+
General linear models
@@ -226,7 +278,7 @@ where \( \epsilon_i \) is the error in our approximation.
-
Rewriting the fitting procedure as a linear algebra problem
+
Rewriting the fitting procedure as a linear algebra problem
@@ -246,7 +298,7 @@ $$
-
Rewriting the fitting procedure as a linear algebra problem, follows
+
Rewriting the fitting procedure as a linear algebra problem, follows
@@ -287,7 +339,7 @@ $$
-
Generalizing the fitting procedure as a linear algebra problem
+
Generalizing the fitting procedure as a linear algebra problem
@@ -310,7 +362,7 @@ $$
-
Generalizing the fitting procedure as a linear algebra problem
+
Generalizing the fitting procedure as a linear algebra problem
@@ -338,7 +390,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
-
Optimizing our parameters
+
Optimizing our parameters
@@ -360,7 +412,7 @@ $$
-
Optimizing our parameters, more details
+
Optimizing our parameters, more details
@@ -384,7 +436,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -423,7 +475,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -449,7 +501,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -477,7 +529,7 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -500,7 +552,7 @@ where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -528,7 +580,7 @@ where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -554,7 +606,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -585,7 +637,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -609,7 +661,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -655,7 +707,7 @@ This approach (different linear and non-linear regression) suffers often from bo
-
Simple regression model
+
Simple regression model
We are now ready to write our first program which aims at solving the above linear regression equations. We start with data we have produced ourselves, in this case normally distributed random numbers along the \( x \)-axis. These numbers define then the value of a function \( y(x)=4+3x+N(0,1) \). Thereafter we order the \( x \) values and employ our linear regression algorithm to set up the best fit. Here we find it useful to use the numpy function \( c\_ \) arrays where arrays are stacked along their last axis after being upgraded to at least two dimensions with ones post-pended to the shape. The following examples help in understanding what happens
@@ -695,7 +747,7 @@ We see that, as expected, a linear fit gives a seemingly (from the graph) good r
-
Simple regression model, now using scikit-learn
+
Simple regression model, now using scikit-learn
We can repeat the above algorithm using scikit-learn as follows
@@ -726,7 +778,7 @@ plt.show()
-
Simple linear regression model using scikit-learn
+
Simple linear regression model using scikit-learn
We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us.
@@ -794,7 +846,7 @@ plt.show()
-
Simple linear regression model
+
Simple linear regression model
This example serves several aims. It allows us to demonstrate several
@@ -816,7 +868,7 @@ where \( x \) is defined as before.
-
Less noise
+
Less noise
Does the fit look better? Indeed, by
@@ -829,7 +881,7 @@ have not discussed a more rigorous approach to the cost function.
-
How to study our fits
+
How to study our fits
We need more rigorous criteria in defining whether we have succeeded or
@@ -851,7 +903,7 @@ dimensionless.
-
Minimizing the cost function
+
Minimizing the cost function
Minimizing the cost function is a central aspect of
@@ -872,7 +924,7 @@ the \( \chi^2 \) function becomes smaller.
-
Relative error
+
Relative error
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
@@ -912,7 +964,7 @@ relative error.
-
The richness of scikit-learn
+
The richness of scikit-learn
As mentioned above, scikit-learn has an impressive functionality.
@@ -957,7 +1009,7 @@ plt.show()
-
Functions in scikit-learn
+
Functions in scikit-learn
The function coef gives us the parameter \( \beta \) of our fit while intercept yields
@@ -974,7 +1026,7 @@ this function as being similar to the \( \chi^2 \) function defined above.
-
Other functions in scikit-learn
+
Other functions in scikit-learn
The r2score function computes \( R^2 \), the coefficient of
@@ -998,7 +1050,7 @@ $$
-
The mean absolute error and other functions in scikit-learn
+
The mean absolute error and other functions in scikit-learn
Another quantity will meet again in our discussions of regression analysis is
@@ -1023,12 +1075,13 @@ years etc.
-
Cubic polynomial in scikit-learn
+
Cubic polynomial in scikit-learn
We will discuss in more
detail these and other functions in the various lectures. We conclude this part with another example. Instead of
-a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+Add description of the various python commands.
Linking the regression analysis with a statistical interpretation
+
+
+Before we proceed, and to link with our discussions of Bayesian statistics to come, it is useful the derive the standard regression analysis equations using a statistical interpretation. This allows us also to derive quantities like the variance and other expectation values in a rather straightforward way.
+
+
+It is assumed that \( \varepsilon_i
+\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are
+independent, i.e.:
+$$
+\begin{align*}
+\mbox{Cov}(\varepsilon_{i_1},
+\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if}
+& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right.
+\end{align*}
+$$
+
+The randomness of \( \varepsilon_i \) implies that
+\( \mathbf{Y}_i \) is also a random variable. In particular,
+\( \mathbf{Y}_i \) is normally distributed, because \( \varepsilon_i \sim
+\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \beta \) is a
+non-random scalar. To specify the parameters of the distribution of
+\( \mathbf{Y}_i \) we need to calculate its first two moments.
+
+A general
+\( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal
+matrix \( \hat{D} \) of dimensionality \( n\times n \) and two orthognal
+matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality
+\( m \times m \) and the last dimensionality \( n\times n \).
+We have then
+$$
+\hat{A} = \hat{U}\hat{D}\hat{V}^T
+$$
+
+
+
+
+
+
+
Code examples for Ridge and Lasso Regression
@@ -1252,38 +1317,390 @@ plt.show()
-
The singular value decompostion
-
-
-
-How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
-$$
-\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
-$$
-
-
+
From standard regression to Ridge regressions
-Add codes and discuss this in connection with lasso and ridge, show example where the standard inversion of a matrix fails and where SVD comes to rescue
+One of the typical problems we encounter with linear regression, in particular
+when the matrix \( \hat{X} \) (our so-called design matrix) is high-dimensional,
+are problems with near singular or singular matrices. The column vectors of \( \hat{X} \)
+may be linearly dependent, normally referred to as super-collinearity.
+This means that the matrix may be rank deficient and it is basically impossible to
+to model the data using linear regression. As an example, consider the matrix
+$$
+\begin{align*}
+\mathbf{X} & = \left[
+\begin{array}{rrr}
+1 & -1 & 2
+\\
+1 & 0 & 1
+\\
+1 & 2 & -1
+\\
+1 & 1 & 0
+\end{array} \right]
+\end{align*}
+$$
+
+
+The columns of \( \hat{X} \) are linearly dependent. We se this easily since the
+the first column is the row-wise sum of the other two columns. The rank (more correct,
+the column rank) of a matrix is the dimension of the space spanned by the
+column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number
+of linearly independent columns. In this particular case the matrix has rank 2.
+
+
+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies
+that the inverse of the matrix \( \hat{X}^T\hat{x} \) (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
+$$
+\begin{align*}
+\hat{X} & = \left[
+\begin{array}{rr}
+1 & -1
+\\
+1 & -1
+\end{array} \right].
+\end{align*}
+$$
+
+We see easily that \( \mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined.
+This is equivalent to saying that the matrix \( \hat{X} \) has at least an eigenvalue which is zero.
-
Lasso and Ridge regression
+
Fixing the singularity
-Discuss the mathematics here
+If our design matrix \( \hat{X} \) which enters the linear regression problem
+$$
+\begin{align}
+\hat{\beta} & = (\hat{X}^{T} \hat{X})^{-1} \hat{X}^{T} \hat{y},
+\label{_auto1}
+\end{align}
+$$
+
+has linearly dependent column vectors, we will not be able to compute the inverse
+of \( \hat{X}^T\hat{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+The estimators are only well-defined if \( (\hat{X}^{T}\hat{X})^{-1} \) exits.
+This is more likely to happen when the matrix \( \hat{X} \) is high-dimensional. In this case it is likely to encounter a situation where
+the regression parameters \( \beta_i \) cannot be estimated.
+
+
+The ad hoc approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change
+$$
+\hat{X}^{T} \hat{X} \rightarrow \hat{X}^{T} \hat{X}+\lambda \hat{I},
+$$
+
+where \( \hat{I} \) is the identity matrix.
-
Logistic regression
-Add discussion about classification versus regression, show examples of more than two cases and why regression is not the best approach. Motivate for k-nearest neighbors
+
A second-order polynomial with Ridge and Lasso
+
+
+
+
importnumpyasnp
+importmatplotlib.pyplotasplt
+fromsklearn.linear_modelimport Ridge
+fromsklearn.metricsimport r2_score
+
+np.random.seed(4155)
+
+n_samples = 100
+
+x = np.random.rand(n_samples,1)
+y = 5*x*x + 0.1*np.random.rand(n_samples,1)
+
+# Centering x and y.
+x_ = x - np.mean(x)
+y_ = y - np.mean(y) # beta_0 = mean(y)
+
+X = np.c_[np.ones((n_samples,1)), x, x**2]
+X_ = np.c_[x_, x_**2]
+
+
+### 1.
+lmb_values = [1e-4, 1e-3, 1e-2, 10, 1e2, 1e4]
+num_values = len(lmb_values)
+
+## Ridge-regression of centered and not centered data
+beta_ridge = np.zeros((3,num_values))
+beta_ridge_centered = np.zeros((3,num_values))
+
+I3 = np.eye(3)
+I2 = np.eye(2)
+
+for i,lmb inenumerate(lmb_values):
+ beta_ridge[:,i] = (np.linalg.inv( X.T @ X + lmb*I3) @ X.T @ y).flatten()
+ beta_ridge_centered[1:,i] = (np.linalg.inv( X_.T @ X_ + lmb*I2) @ X_.T @ y_).flatten()
+
+# sett beta_0 = np.mean(y)
+beta_ridge_centered[0,:] = np.mean(y)
+
+## OLS (ordinary least squares) solution
+beta_ls = np.linalg.inv( X.T @ X ) @ X.T @ y
+
+## Evaluate the models
+pred_ls = X @ beta_ls
+pred_ridge = X @ beta_ridge
+pred_ridge_centered = X_ @ beta_ridge_centered[1:] + beta_ridge_centered[0,:]
+
+## Plot the results
+
+# Sorting
+sort_ind = np.argsort(x[:,0])
+
+x_plot = x[sort_ind,0]
+x_centered_plot = x_[sort_ind,0]
+
+pred_ls_plot = pred_ls[sort_ind,0]
+pred_ridge_plot = pred_ridge[sort_ind,:]
+pred_ridge_centered_plot = pred_ridge_centered[sort_ind,:]
+
+# Plott not centered
+plt.plot(x_plot,pred_ls_plot,label='ls')
+
+for i inrange(num_values):
+ plt.plot(x_plot,pred_ridge_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x,y,'ro')
+
+plt.title('linear regression on un-centered data')
+plt.legend()
+
+# Plott centered
+plt.figure()
+
+for i inrange(num_values):
+ plt.plot(x_centered_plot,pred_ridge_centered_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x_,y,'ro')
+
+plt.title('linear regression on centered data')
+plt.legend()
+
+
+# 2.
+
+pred_ridge_scikit = np.zeros((n_samples,num_values))
+for i,lmb inenumerate(lmb_values):
+ pred_ridge_scikit[:,i] = (Ridge(alpha=lmb,fit_intercept=False).fit(X,y).predict(X)).flatten() # fit_intercept=False fordi bias er allerede i X
+
+plt.figure()
+
+plt.plot(x_plot,pred_ls_plot,label='ls')
+
+for i inrange(num_values):
+ plt.plot(x_plot,pred_ridge_scikit[sort_ind,i],label='scikit-ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x,y,'ro')
+plt.legend()
+plt.title('linear regression using scikit')
+
+plt.show()
+
+### R2-score of the results
+for i inrange(num_values):
+ print('lambda = %g'%lmb_values[i])
+ print('r2 for scikit: %g'%r2_score(y,pred_ridge_scikit[:,i]))
+ print('r2 for own code, not centered: %g'%r2_score(y,pred_ridge[:,i]))
+ print('r2 for own, centered: %g\n'%r2_score(y,pred_ridge_centered[:,i]))
+
+
+
+
+
Fitting vs. predicting when data is in the model class
-Add examples on classification problems
+We start by considering the case
+\( f(x)=2x \).
+Then the data is clearly generated by a model that is contained within
+all three model classes we are using to make predictions (linear
+models, third order polynomials, and tenth order polynomials).
+
+
+Run the code for the following cases:
+
+
+
For \( f(x)=2x \) , \( Ntrain=10 \) and \( \sigma =0 \) (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when \( x \in [0,1] \) . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?
+
Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?
+
Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example \( x \in [0,1.2] \) ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?
+
Repeat the above for \( f(x)=2x \) , \( Ntrain=10 \) , and \( \sigma=1 \) . What changes?
+
+
+Repeat the exercises above for \( f(x)=2x \) , \( Ntrain=100 \) , and \( \sigma=1 \) . What changes?
+Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.
+
+
+
+
+
Fitting versus predicting when data is not in the model class
+
+
+Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider \( f(x)=2x-10x^5+15x^{10} \) . Notice that the for linear and third-order polynomial the true model \( f(x) \) is not contained in model class.
+
+
+
Do better fits lead to better predictions?
+
What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points \( Ntrain \) and \( \sigma \)?
+
+
+Summarize what you think you learned about the relationship of knowing the true model class and predictive power.
+
+
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
Aug 24, 2018
+
Sep 6, 2018
@@ -210,7 +231,38 @@ A regression model aims at finding a likelihood function \( p(y\vert \hat{x}) \)
-
General linear models
+
Regression analysis, overarching aims II
+
+
+
+
+
+Consider an experiment in which \( p \) characteristics of \( n \) samples are
+measured. The data from this experiment are denoted \( \mathbf{X} \), with
+\( \mathbf{X} \) as above. The matrix \( \mathbf{X} \) is called the design
+matrix. Additional information of the samples is available in the
+form of \( \mathbf{Y} \) (also as above). The variable \( \mathbf{Y} \) is
+generally referred to as the response variable. The aim of
+regression analysis is to explain \( \mathbf{Y} \) in terms of
+\( \mathbf{X} \) through a functional relationship like \( Y_i =
+f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of
+\( f(\cdot) \) is available, it is common to assume a linear relationship
+between \( \mathbf{X} \) and \( \mathbf{Y} \). This assumption gives rise to
+the linear regression model where \( \beta = (\beta_1, \ldots,
+\beta_p)^{\top} \) is the regression parameter. The parameter
+\( \beta_j \), \( j=1, \ldots, p \), represents the effect size of covariate
+\( j \) on the response. That is, for each unit change in covariate \( j \)
+(while keeping the other covariates fixed) the observed change in the
+response is equal to \( \beta_j \).
+
+
+
+
+
+
+
+
+
General linear models
@@ -231,7 +283,7 @@ where \( \epsilon_i \) is the error in our approximation.
-
Rewriting the fitting procedure as a linear algebra problem
+
Rewriting the fitting procedure as a linear algebra problem
@@ -251,7 +303,7 @@ $$
-
Rewriting the fitting procedure as a linear algebra problem, follows
+
Rewriting the fitting procedure as a linear algebra problem, follows
@@ -292,7 +344,7 @@ $$
-
Generalizing the fitting procedure as a linear algebra problem
+
Generalizing the fitting procedure as a linear algebra problem
@@ -315,7 +367,7 @@ $$
-
Generalizing the fitting procedure as a linear algebra problem
+
Generalizing the fitting procedure as a linear algebra problem
@@ -343,7 +395,7 @@ The left-hand side of this equation forms know. Our error vector \( \hat{\epsilo
-
Optimizing our parameters
+
Optimizing our parameters
@@ -365,7 +417,7 @@ $$
-
Optimizing our parameters, more details
+
Optimizing our parameters, more details
@@ -389,7 +441,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -428,7 +480,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -454,7 +506,7 @@ $$
-
Interpretations and optimizing our parameters
+
Interpretations and optimizing our parameters
@@ -482,7 +534,7 @@ meaning that the solution for \( \hat{\beta} \) is the one which minimizes the r
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -505,7 +557,7 @@ where the matrix \( \hat{\Sigma} \) is a diagonal matrix with \( \sigma_i \) as
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -533,7 +585,7 @@ where we have defined the matrix \( \hat{A} =\hat{X}/\hat{\Sigma} \) with matrix
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -559,7 +611,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -590,7 +642,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -614,7 +666,7 @@ $$
-
The \( \chi^2 \) function
+
The \( \chi^2 \) function
@@ -660,7 +712,7 @@ This approach (different linear and non-linear regression) suffers often from bo
-
Simple regression model
+
Simple regression model
We are now ready to write our first program which aims at solving the above linear regression equations. We start with data we have produced ourselves, in this case normally distributed random numbers along the \( x \)-axis. These numbers define then the value of a function \( y(x)=4+3x+N(0,1) \). Thereafter we order the \( x \) values and employ our linear regression algorithm to set up the best fit. Here we find it useful to use the numpy function \( c\_ \) arrays where arrays are stacked along their last axis after being upgraded to at least two dimensions with ones post-pended to the shape. The following examples help in understanding what happens
@@ -700,7 +752,7 @@ We see that, as expected, a linear fit gives a seemingly (from the graph) good r
-
Simple regression model, now using scikit-learn
+
Simple regression model, now using scikit-learn
We can repeat the above algorithm using scikit-learn as follows
@@ -731,7 +783,7 @@ plt.show()
-
Simple linear regression model using scikit-learn
+
Simple linear regression model using scikit-learn
We start with perhaps our simplest possible example, using scikit-learn to perform linear regression analysis on a data set produced by us.
@@ -799,7 +851,7 @@ plt.show()
-
Simple linear regression model
+
Simple linear regression model
This example serves several aims. It allows us to demonstrate several
@@ -821,7 +873,7 @@ where \( x \) is defined as before.
-
Less noise
+
Less noise
Does the fit look better? Indeed, by
@@ -834,7 +886,7 @@ have not discussed a more rigorous approach to the cost function.
-
How to study our fits
+
How to study our fits
We need more rigorous criteria in defining whether we have succeeded or
@@ -856,7 +908,7 @@ dimensionless.
-
Minimizing the cost function
+
Minimizing the cost function
Minimizing the cost function is a central aspect of
@@ -877,7 +929,7 @@ the \( \chi^2 \) function becomes smaller.
-
Relative error
+
Relative error
There are many ways to define the cost function. A simpler approach is to look at the relative difference between the training data and the predicted data, that is we define
@@ -917,7 +969,7 @@ relative error.
-
The richness of scikit-learn
+
The richness of scikit-learn
As mentioned above, scikit-learn has an impressive functionality.
@@ -962,7 +1014,7 @@ plt.show()
-
Functions in scikit-learn
+
Functions in scikit-learn
The function coef gives us the parameter \( \beta \) of our fit while intercept yields
@@ -979,7 +1031,7 @@ this function as being similar to the \( \chi^2 \) function defined above.
-
Other functions in scikit-learn
+
Other functions in scikit-learn
The r2score function computes \( R^2 \), the coefficient of
@@ -1003,7 +1055,7 @@ $$
-
The mean absolute error and other functions in scikit-learn
+
The mean absolute error and other functions in scikit-learn
Another quantity will meet again in our discussions of regression analysis is
@@ -1028,12 +1080,13 @@ years etc.
-
Cubic polynomial in scikit-learn
+
Cubic polynomial in scikit-learn
We will discuss in more
detail these and other functions in the various lectures. We conclude this part with another example. Instead of
-a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+a linear \( x \)-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn.
+Add description of the various python commands.
Linking the regression analysis with a statistical interpretation
+
+
+Before we proceed, and to link with our discussions of Bayesian statistics to come, it is useful the derive the standard regression analysis equations using a statistical interpretation. This allows us also to derive quantities like the variance and other expectation values in a rather straightforward way.
+
+
+It is assumed that \( \varepsilon_i
+\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are
+independent, i.e.:
+$$
+\begin{align*}
+\mbox{Cov}(\varepsilon_{i_1},
+\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if}
+& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right.
+\end{align*}
+$$
+
+The randomness of \( \varepsilon_i \) implies that
+\( \mathbf{Y}_i \) is also a random variable. In particular,
+\( \mathbf{Y}_i \) is normally distributed, because \( \varepsilon_i \sim
+\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \beta \) is a
+non-random scalar. To specify the parameters of the distribution of
+\( \mathbf{Y}_i \) we need to calculate its first two moments.
+
+A general
+\( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal
+matrix \( \hat{D} \) of dimensionality \( n\times n \) and two orthognal
+matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality
+\( m \times m \) and the last dimensionality \( n\times n \).
+We have then
+$$
+\hat{A} = \hat{U}\hat{D}\hat{V}^T
+$$
+
+
+
+
+
+
+
Code examples for Ridge and Lasso Regression
@@ -1257,38 +1322,390 @@ plt.show()
-
The singular value decompostion
-
-
-
-How can we use the singular value decomposition to find the parameters \( \beta_j \)? More details will come. We first note that a general \( m\times n \) matrix \( \hat{A} \) can be written in terms of a diagonal matrix \( \hat{\Sigma} \) of dimensionality \( n\times n \) and two orthognal matrices \( \hat{U} \) and \( \hat{V} \), where the first has dimensionality \( m \times n \) and the last dimensionality \( n\times n \). We have then
-$$
-\hat{A} = \hat{U}\hat{\Sigma}\hat{V}
-$$
-
-
+
From standard regression to Ridge regressions
-Add codes and discuss this in connection with lasso and ridge, show example where the standard inversion of a matrix fails and where SVD comes to rescue
+One of the typical problems we encounter with linear regression, in particular
+when the matrix \( \hat{X} \) (our so-called design matrix) is high-dimensional,
+are problems with near singular or singular matrices. The column vectors of \( \hat{X} \)
+may be linearly dependent, normally referred to as super-collinearity.
+This means that the matrix may be rank deficient and it is basically impossible to
+to model the data using linear regression. As an example, consider the matrix
+$$
+\begin{align*}
+\mathbf{X} & = \left[
+\begin{array}{rrr}
+1 & -1 & 2
+\\
+1 & 0 & 1
+\\
+1 & 2 & -1
+\\
+1 & 1 & 0
+\end{array} \right]
+\end{align*}
+$$
+
+
+The columns of \( \hat{X} \) are linearly dependent. We se this easily since the
+the first column is the row-wise sum of the other two columns. The rank (more correct,
+the column rank) of a matrix is the dimension of the space spanned by the
+column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number
+of linearly independent columns. In this particular case the matrix has rank 2.
+
+
+Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies
+that the inverse of the matrix \( \hat{X}^T\hat{x} \) (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this
+$$
+\begin{align*}
+\hat{X} & = \left[
+\begin{array}{rr}
+1 & -1
+\\
+1 & -1
+\end{array} \right].
+\end{align*}
+$$
+
+We see easily that \( \mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined.
+This is equivalent to saying that the matrix \( \hat{X} \) has at least an eigenvalue which is zero.
-
Lasso and Ridge regression
+
Fixing the singularity
-Discuss the mathematics here
+If our design matrix \( \hat{X} \) which enters the linear regression problem
+$$
+\begin{align}
+\hat{\beta} & = (\hat{X}^{T} \hat{X})^{-1} \hat{X}^{T} \hat{y},
+\label{_auto1}
+\end{align}
+$$
+
+has linearly dependent column vectors, we will not be able to compute the inverse
+of \( \hat{X}^T\hat{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+The estimators are only well-defined if \( (\hat{X}^{T}\hat{X})^{-1} \) exits.
+This is more likely to happen when the matrix \( \hat{X} \) is high-dimensional. In this case it is likely to encounter a situation where
+the regression parameters \( \beta_i \) cannot be estimated.
+
+
+The ad hoc approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change
+$$
+\hat{X}^{T} \hat{X} \rightarrow \hat{X}^{T} \hat{X}+\lambda \hat{I},
+$$
+
+where \( \hat{I} \) is the identity matrix.
-
Logistic regression
-Add discussion about classification versus regression, show examples of more than two cases and why regression is not the best approach. Motivate for k-nearest neighbors
+
A second-order polynomial with Ridge and Lasso
+
+
+
+
importnumpyasnp
+importmatplotlib.pyplotasplt
+fromsklearn.linear_modelimport Ridge
+fromsklearn.metricsimport r2_score
+
+np.random.seed(4155)
+
+n_samples =100
+
+x = np.random.rand(n_samples,1)
+y =5*x*x +0.1*np.random.rand(n_samples,1)
+
+# Centering x and y.
+x_ = x - np.mean(x)
+y_ = y - np.mean(y) # beta_0 = mean(y)
+
+X = np.c_[np.ones((n_samples,1)), x, x**2]
+X_ = np.c_[x_, x_**2]
+
+
+### 1.
+lmb_values = [1e-4, 1e-3, 1e-2, 10, 1e2, 1e4]
+num_values =len(lmb_values)
+
+## Ridge-regression of centered and not centered data
+beta_ridge = np.zeros((3,num_values))
+beta_ridge_centered = np.zeros((3,num_values))
+
+I3 = np.eye(3)
+I2 = np.eye(2)
+
+for i,lmb inenumerate(lmb_values):
+ beta_ridge[:,i] = (np.linalg.inv( X.T @ X + lmb*I3) @ X.T @ y).flatten()
+ beta_ridge_centered[1:,i] = (np.linalg.inv( X_.T @ X_ + lmb*I2) @ X_.T @ y_).flatten()
+
+# sett beta_0 = np.mean(y)
+beta_ridge_centered[0,:] = np.mean(y)
+
+## OLS (ordinary least squares) solution
+beta_ls = np.linalg.inv( X.T @ X ) @ X.T @ y
+
+## Evaluate the models
+pred_ls = X @ beta_ls
+pred_ridge = X @ beta_ridge
+pred_ridge_centered = X_ @ beta_ridge_centered[1:] + beta_ridge_centered[0,:]
+
+## Plot the results
+
+# Sorting
+sort_ind = np.argsort(x[:,0])
+
+x_plot = x[sort_ind,0]
+x_centered_plot = x_[sort_ind,0]
+
+pred_ls_plot = pred_ls[sort_ind,0]
+pred_ridge_plot = pred_ridge[sort_ind,:]
+pred_ridge_centered_plot = pred_ridge_centered[sort_ind,:]
+
+# Plott not centered
+plt.plot(x_plot,pred_ls_plot,label='ls')
+
+for i inrange(num_values):
+ plt.plot(x_plot,pred_ridge_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x,y,'ro')
+
+plt.title('linear regression on un-centered data')
+plt.legend()
+
+# Plott centered
+plt.figure()
+
+for i inrange(num_values):
+ plt.plot(x_centered_plot,pred_ridge_centered_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x_,y,'ro')
+
+plt.title('linear regression on centered data')
+plt.legend()
+
+
+# 2.
+
+pred_ridge_scikit = np.zeros((n_samples,num_values))
+for i,lmb inenumerate(lmb_values):
+ pred_ridge_scikit[:,i] = (Ridge(alpha=lmb,fit_intercept=False).fit(X,y).predict(X)).flatten() # fit_intercept=False fordi bias er allerede i X
+
+plt.figure()
+
+plt.plot(x_plot,pred_ls_plot,label='ls')
+
+for i inrange(num_values):
+ plt.plot(x_plot,pred_ridge_scikit[sort_ind,i],label='scikit-ridge, lmb=%g'%lmb_values[i])
+
+plt.plot(x,y,'ro')
+plt.legend()
+plt.title('linear regression using scikit')
+
+plt.show()
+
+### R2-score of the results
+for i inrange(num_values):
+ print('lambda = %g'%lmb_values[i])
+ print('r2 for scikit: %g'%r2_score(y,pred_ridge_scikit[:,i]))
+ print('r2 for own code, not centered: %g'%r2_score(y,pred_ridge[:,i]))
+ print('r2 for own, centered: %g\n'%r2_score(y,pred_ridge_centered[:,i]))
+
+
+
+
+
Fitting vs. predicting when data is in the model class
-Add examples on classification problems
+We start by considering the case
+\( f(x)=2x \).
+Then the data is clearly generated by a model that is contained within
+all three model classes we are using to make predictions (linear
+models, third order polynomials, and tenth order polynomials).
+
+
+Run the code for the following cases:
+
+
+
For \( f(x)=2x \) , \( Ntrain=10 \) and \( \sigma =0 \) (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when \( x \in [0,1] \) . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?
+
Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?
+
Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example \( x \in [0,1.2] \) ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?
+
Repeat the above for \( f(x)=2x \) , \( Ntrain=10 \) , and \( \sigma=1 \) . What changes?
+
+
+Repeat the exercises above for \( f(x)=2x \) , \( Ntrain=100 \) , and \( \sigma=1 \) . What changes?
+Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.
+
+
+
+
+
Fitting versus predicting when data is not in the model class
+
+
+Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider \( f(x)=2x-10x^5+15x^{10} \) . Notice that the for linear and third-order polynomial the true model \( f(x) \) is not contained in model class.
+
+
+
Do better fits lead to better predictions?
+
What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points \( Ntrain \) and \( \sigma \)?
+
+
+Summarize what you think you learned about the relationship of knowing the true model class and predictive power.
+
+
# Generate Test Data
+
+#Number of test data
+N_test=20
+
+sigma_test=sigma_train
+
+max_x=1.2
+x_test=max_x*np.random.random(N_test)
+# Draw random noise
+s_test = sigma_test*np.random.randn(N_test)
+
+#Linear
+y_test=2*x_test+s_test
+#Tenth order
+#y_test=2*x_test-10*x_test**5+15*x_test**10+s_test
+
+#Make design matrices for prediction
+x_plot=np.linspace(0,max_x, 200)
+X3 = poly3.fit_transform(x_plot[:,np.newaxis])
+X10 = poly10.fit_transform(x_plot[:,np.newaxis])
+
+%matplotlib notebook
+
+fig = plt.figure()
+p1=plt.plot(x_test,y_test.transpose(), 'o', ms=12, label='data')
+p2=plt.plot(x_plot,clf.predict(x_plot[:,np.newaxis]), label='linear')
+p3=plt.plot(x_plot,clf3.predict(X3), label='3rd order')
+p10=plt.plot(x_plot,clf10.predict(X10), label='10th order')
+
+
+plt.legend(loc=2)
+plt.xlabel('$x$')
+plt.ylabel('$y$')
+plt.legend(loc='best')
+plt.title(Title+" (pred.)")
+plt.tight_layout()
+plt.show()
+
+#Linear Filename
+#filename_test=Title+"pred-linear.pdf"
+#Tenth Order Filename
+#filename_test=Title+"pred-o10.pdf"
+#plt.savefig(filename_test)
+#plt.ylim((-6,12))
+
+
+
+
+
Lasso regression
+
+
+
+
+
Logistic regression
diff --git a/doc/pub/Regression/ipynb/Regression.ipynb b/doc/pub/Regression/ipynb/Regression.ipynb
index a4a85095f..af7e36ea4 100644
--- a/doc/pub/Regression/ipynb/Regression.ipynb
+++ b/doc/pub/Regression/ipynb/Regression.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Aug 24, 2018**\n",
+ "Date: **Sep 6, 2018**\n",
"\n",
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -33,6 +33,28 @@
"\n",
"\n",
"\n",
+ "## Regression analysis, overarching aims II\n",
+ "\n",
+ "\n",
+ "Consider an experiment in which $p$ characteristics of $n$ samples are\n",
+ "measured. The data from this experiment are denoted $\\mathbf{X}$, with\n",
+ "$\\mathbf{X}$ as above. The matrix $\\mathbf{X}$ is called the *design\n",
+ "matrix*. Additional information of the samples is available in the\n",
+ "form of $\\mathbf{Y}$ (also as above). The variable $\\mathbf{Y}$ is\n",
+ "generally referred to as the *response variable*. The aim of\n",
+ "regression analysis is to explain $\\mathbf{Y}$ in terms of\n",
+ "$\\mathbf{X}$ through a functional relationship like $Y_i =\n",
+ "f(\\mathbf{X}_{i,\\ast})$. When no prior knowledge on the form of\n",
+ "$f(\\cdot)$ is available, it is common to assume a linear relationship\n",
+ "between $\\mathbf{X}$ and $\\mathbf{Y}$. This assumption gives rise to\n",
+ "the *linear regression model* where $\\beta = (\\beta_1, \\ldots,\n",
+ "\\beta_p)^{\\top}$ is the *regression parameter*. The parameter\n",
+ "$\\beta_j$, $j=1, \\ldots, p$, represents the effect size of covariate\n",
+ "$j$ on the response. That is, for each unit change in covariate $j$\n",
+ "(while keeping the other covariates fixed) the observed change in the\n",
+ "response is equal to $\\beta_j$.\n",
+ "\n",
+ "\n",
"\n",
"## General linear models\n",
"Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\\hat{y}=[y_0,y_1,\\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\\hat{x}=[x_0,x_1,\\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. \n",
@@ -897,19 +919,10 @@
{
"cell_type": "code",
"execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[[1 4]\n",
- " [2 5]\n",
- " [3 6]]\n",
- "[[1 2 3 0 0 4 5 6]]\n"
- ]
- }
- ],
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import numpy as np\n",
"print(np.c_[np.array([1,2,3]), np.array([4,5,6])])\n",
@@ -918,20 +931,11 @@
},
{
"cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHmlJREFUeJzt3XmUZGWd5vHvQxYFZlW1QFUhILVAi/TBbh0wxwUXlKJHGmhpp+0+cAqaTashRw+OuE6do4y2y9hHFGQKKWwQqRwUEWawD3aLoMMoiyaI7MhiJTsUFFuxFLX85o/3hnkziMx8M2O5kZnP55w8FXHvjRu/vATx5Hvf975XEYGZmdl4tqm6ADMzmxocGGZmlsWBYWZmWRwYZmaWxYFhZmZZHBhmZpbFgWFdQ9K7JN1VdR3TgaTFkjZI6qm6Fps+HBjWcZLWSjqofnlE/L+I2LuKmupJOlXSpuJL92lJ10h6e9V15YqI+yNibkRsqboWmz4cGDbjSZo1yqofRMRcYAHwc+CHHX5/s67iwLCuIek9kh4sPV8r6ROSbpb0jKQfSNq+tP4wSTeVWgBvLK37jKR7JT0n6XZJHyitO1bSryR9Q9KTwKlj1RURm4EB4LWSFma+/36Sflu8/w+L2v+p/HtK+rSkR4HzMvb3aUkPFfu7S9KyYvlbJA1KelbSY5JOK5YvlRS1MJK0m6TLJK2XdI+kD5f2faqkiyR9r9j/bZL6sv/D2YzhwLBu9/fAwcAewBuBYwEk7QucC/wjMB84G7hM0nbF6+4F3gW8GvjvwBpJu5b2+1bgPuA1wJfGKkDSbOAfgCeBp8Z7/2L7S4HvAjsBFwIfqNvtLsW6JcCKcfa3N/AR4D9GxDzgfcDaYj+nA6dHxJ8AfwpcNMqv8X3gQWA34IPAlyUdWFr//mKbHYDLgDPHOiY2MzkwrNudEREPR8R64MfAfyiWrwDOjojrI2JLRJwPbATeBhARPyxetzUifgDcDbyltN+HI+JbEbE5Il4c5b3/XtLTwIvAh4EPFq2N8d7/bcCsovZNEXEJ8Ou6fW8FPh8RG4v3H2t/W4DtgH0kbRsRayPi3mI/m4DXSVoQERsi4rr6X0LSIuAdwKcj4qWIuAn4DikEa34ZEZcXfR4XAG8a5ZjYDObAsG73aOnxC8Dc4vES4JTi9M3TxRf7ItJf0Ej6h9LpnaeBPyf1RdQ8kPHeF0XEDqRWyK3Am0vrxnr/3YCHYuTMnvXvty4iXsrZX0TcA3yMdOrscUnfl7Rb8boTgNcDd0r6jaTDGvweuwHrI+K50rIh4LWl5/XHeXv3rVg9B4ZNVQ8AX4qIHUo/vRFxoaQlwDmk0zjziy/9WwGVXp89TXNEPEFqAZxaOq016vsDj5D6O8rvt6h+t7m/T1HD/4qId5KCJYD/USy/OyKOBHYull0saU7dvh8GdpI0r7RsMfBQ7jEwAweGVWdbSduXfib61+w5wImS3qpkjqRDiy/FOaQv1XUAko4jtTAmLSLuAv4d+FTG+19LOo30EUmzJB3OyNNhE/p9JO0t6cCif+Yl0imyrcXvdpSkhRGxFXi62NfWutofAK4BvlIc6zeSWiZrmjkmNvM4MKwql5O++Go/p07kxRExSOpXOJPUEX0PRYd4RNwOfJ30xf0Y8BfAr1pQ8z+TOqh3Huf9Xwb+M+lL+WngKOBfSX0SE/59SP0XXwWeIJ062hn4bLHuYOA2SRtIHeBHjNIncySwlNTauJTUf/KzCf7+NsPJN1Ayaz9J1wPfjojzqq7FbLLcwjBrA0kHSNqlOCV1DGlI8L9VXZdZMzwKwqw99iZdEzGHdL3HByPikWpLMmuOT0mZmVkWn5IyM7MsU+qU1IIFC2Lp0qVVl2FmNqXccMMNT0TEwvG3HNuUCoylS5cyODhYdRlmZlOKpKFW7MenpMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyytD0wJJ0r6XFJtzZYd4qkkLSg3XWYmVlzOtHC+C7pNpIjSFoE/Cfg/g7UYGZmTWp7YETE1cD6Bqu+AXwK8A05zMymgEr6MCQdDjwUEb/L2HaFpEFJg+vWretAdWZm1kjHA0NSL/DfgM/lbB8RqyOiLyL6Fi5sejp3MzObpCpaGH8K7AH8TtJaYHfgRkm7VFCLmZll6vgNlCLiFmDn2vMiNPoi4olO12JmZvk6Maz2QuBaYG9JD0o6od3vaWZmrdf2FkZEHDnO+qXtrsHMzJrnK73NzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLK0PTAknSvpcUm3lpb9s6Q7Jd0s6VJJO7S7DjMza04nWhjfBQ6uW3YF8OcR8Ubg98BnO1CHmZk1oe2BERFXA+vrlv00IjYXT68Ddm93HWZm1pxu6MM4HvjJaCslrZA0KGlw3bp1HSzLzMzKKg0MSSuBzcDAaNtExOqI6IuIvoULF3auODMzG2FWVW8s6VjgMGBZRERVdZiZWZ5KAkPSwcCngAMi4oUqajAzs4npxLDaC4Frgb0lPSjpBOBMYB5whaSbJH273XWYmVlz2t7CiIgjGyz+l3a/r5mZtVY3jJIyM7MpwIFhZmZZHBhmZpbFgWFmZlkcGGZmlsWBYWZmWRwYZmaWxYFhZmZZHBhmZpbFgWFmZlkcGGZmlsWBYWZmWRwYZmaWxYFhZmZZHBhmZpbFgWFmZlkcGGZmlsWBYWZmWRwYZmaWxYFhZmZZ2h4Yks6V9LikW0vLdpJ0haS7i393bHcdZmbWnE60ML4LHFy37DPAlRGxF3Bl8dzMzFotomW7antgRMTVwPq6xYcD5xePzwf+pt11mJlNSwMDsHQpbLNN+ndgAB5+GNasgeOPT8taZFbL9jQxr4mIR4rHjwKvGW1DSSuAFQCLFy/uQGlmZl1sYABWroT774eddoJnn4VNm9K6oSE4+ujhVsWOO8J735u2bYHKO70jIoBR20wRsToi+iKib+HChR2szMysIo1aDbXlK1akYIiAJ58cDouaCNhhB7jhBli3Dn70o5aVVVUL4zFJu0bEI5J2BR6vqA4zs+7S3w/f/vZwK2FoKIXEXXfBGWfACy+Mv49nnoH99mt5aVW1MC4DjikeHwP8n4rqMDOrzsAALFgAUvqZOxfOOuuVHdUvvABf/GIKghxtOn3fiWG1FwLXAntLelDSCcBXgb+UdDdwUPHczGz66++HWbNSQBx1VDqtVPP886O/ToLddx9//7298KUvNV9nA20/JRURR46yalm739vMrDK1zumhoeFl0uSHuS5enIJgxYqRp6Vmz4Z582D9+uFtli9vrvZRVNWHYWY2/ZRDolE4TDYspJFBUBsl1eaAqOfAMDObjPLw1sWL4XWvg6uuGg6FVl0wJ8GJJw6HwvLlHQuIeg4MM7OJqg1vrZ0aGhoaeeqpGT09aVhsB04xTVTl12GYmXW1RtdErFyZN7x1oubPh/PPhyeegK1bYe3argkLcGCYmQ2rD4f+/pEXyg0NwXHHtbY1cdJJad8RKSi6KCDq+ZSUmc1c/f1w9tnpr/l6Q0MjL6Crqb+yeiJ6elIArVo1+X1UyC0MM5uZ+vvTRXKNwqJmrI7r3t6x9y+NbD1EwObNUzYswIFhZjPV6tWTf+2SJen1S5akYFiyJIVD+fkFF0zpcGjEp6TMbPooXwfR0wNbtqQv79pIow0b4Oqr0/DXLVvy9ll/PUXtSuoKh7dWxS0MM5u6yp3UCxaM7JCuBUKto/r1r0/TfR96KHzrW3n77+1N10CUWw6rV8+4oKhxYJjZ1FEfEMcfP/ZU3zWbNqXtPvlJuOIKePrpdAppLLVwWLUqDW/twmGuneZTUmY2NdRfLFeetC/Hpk3w5S8PP6/1L5RHSc2Zk57P4FAYi1sYZtZdaq0IaXhW16VL4eSTm7tYrtGU36tWpVNXtVFMGzY4LMbgFoaZdY/6mweV+yGa0cYpv2cStzDMrBq5Nw+aiNmz0/QakEZJwYzvqG4lB4aZtV+jKTeOOy7/5kGj2XbbFBC1EUznnpum16hdJBcx4zuqW8mnpMys9QYGUp9Do47poaHUkpiM+fNTS6SCe0GYA8PMWqF8wdw224w93cZk9fbC6ac7ICrkU1Jm1pzacNdax3Q7wsL9EF3BgWFmzWnlvSF6ekb2SaxZ436ILlJpYEj6r5Juk3SrpAslbV9lPWbWQKMbCK1bBxddlKbNmMyQ197edKV1bUQTdP3NgwwUrbrv7ETfWHot8Etgn4h4UdJFwOUR8d3RXtPX1xeDg4OdKtHM6q+uhpGT8c2bl0Yjvfhi/j7LkwFaR0i6ISL6mt1P1aekZgGvkjQL6AUerrges5mlUesB4KWX0oyu/f2vPN0Uke45fe216b7T55wz/r0h5s/36aVpoLIWBoCkk4EvAS8CP42IV3yKJK0AVgAsXrz4zUOtujWi2Uw01nBXSNc17LUX3HsvbNw4+n6kkZ3b400rbpWa8i0MSTsChwN7ALsBcyQdVb9dRKyOiL6I6Fu4cGGnyzSbuhpdLHf88WNP2rdpE9x9d9r2xz+GRYsab1c/L9Py5anl4AvmprUqr8M4CPhDRKwDkHQJsD+wpsKazKau/v409HTLltQC6OlJX94w+v2pG9m8GU47LT1+5plX9mF4XqYZq8o+jPuBt0nqlSRgGXBHhfWYTU0DA6nz+ayzhifrq/2lX5Z7+rnceli+/JW3IvX1EDNWZS2MiLhe0sXAjcBm4LdAEzfZNZuBGo1iakaj1sMMvBWpNVbp1CAR8Xng81XWYDZlPf88fPzjrQsLd1LbOMY9JSXpCklv6kQxZjaGl1+Gq6+GU0+Fd70r3Z/68cfzXz/axXIe7mqZcvowPg18U9J5knZtd0FmM0b9/akXLBh5PcSWLTA4CF/7GrzvfSkgDjgAvvCFdJ3Exz8OO+889nvMnTuy72HVquHpvyPSY4eEZcq+DkPS3wKfAy4BvhYRE7i0szV8pbdNG+P1PfT0pJsB1a6g3mcfWLYMDjwwhcaOO469HylN21G7b7XNaK26DiOrD6MYxXQXcBbwT8CHJX02Ii5otgCzGaM87HU8W7ak1sbAQAqJXXZpvF2tdbBype8RYW03bgtD0q9IF9fdBlwHXA/cCZwMbBcRK9pdZI1bGDZl1F9Rvd12Y1853Uj91dRmk9TJFsYK4PZ4ZbJ8VJKvmzCDka2HRjcQmmhYwCuvpjar2LiBERG3jbH60BbWYjY19fePvOVoK1oFvpraulBTV3pHxH2tKsRsStq0Cc4+u7l9SGl4a/nGQb6a2rqQ7+ltNhFbt8LNN6epv6+8Ml0X0UyL4qSTPJLJpoyq74dh1l0a3R/i979PE/f93d+l6x723RdOOQXuuQeOPjptm0Ma3ranx2FhU45bGGY19dc0DA2lQKiN99h9dzjssDTM9cAD0/Oach9GzezZ6epsSKebTj/dp5lsSnNgmD3xBPziF+lCt0Z3l9tpJ7juOnjd61IroV6tlVAbJdXTk4LHrQebZnxKyqav0W4/+txzcPnl6bTSvvvCwoXpdNOGDY3389RT6S50jcKiZtWq4RsHbd7ssLBpyYFh01Pt9NLQUPoSHxqC446D178+tRgOPRTOPDPdm/qLX4Rrrhn9ugdfD2EG+JSUTUebN8MnPvHK00ubNqXg+OQn07xM++8Pr3rV8Povf9l3lzMbg1sYNnXUTjFJMGtW+rd2qunWW+GMM+Dww9Osr48+2ngfmzalYFi2bGRYgO8uZzaO7Nlqu4Hnkpph6udjyrHnnikMLr00dWbXW7Ik3ffBbAbp6Gy1Zm03mXAomz8/3Tti6dL0/IADfHrJrMV8Ssqq1d+fhqEeddTkwwJg/frhsACfXjJrA7cwrHMGBtJ9G4aGUkjk3BciV6ORTMuXOyDMWqjSFoakHSRdLOlOSXdIenuV9VgL1d9+dN681IoYGkrrWxkWPtVk1hFVtzBOB/4tIj4oaTbQW3E91qxGfRHNnGoaTa2FsmSJ7zBn1iGVBYakVwPvBo4FiIiXgZerqsea1N+fJuhr56g7z8dkVqkqT0ntAawDzpP0W0nfkTSnfiNJKyQNShpct25d56u0kRpdCzF3bpp8r5VhUZvVdckSWLMm7fuJJxwWZhWqMjBmAfsBZ0XEvsDzwGfqN4qI1RHRFxF9Cxcu7HSNVt8Xcfzxr+yHeP751rxXbcrviLTviHTNhEPCrCtUGRgPAg9GxPXF84tJAWJVK7cijj56eD6mJ58cnq67WbWJ/MotCE/aZ9bVKguMiHgUeEDS3sWiZcDtVdUzY9XP6NrfPzxpH7S+T6IWEFu3ugVhNsVUPUrqo8BAMULqPuC4iuuZGcrXQ0jDoTA01PhGQJNV27dHMplNC5UGRkTcBDQ9v4llGG3qjVa1IHp60r62bvUNhMymKU8NMhMMDKR7QbTieohtt03DWyEFA6QWxPnnD3dUuy/CbFpyYEx1o91Vrubhh1PLYtOm/H3W31mu3EF93nlpeGstGNwPYTZjVN2HYc2ov1huaAg+/GH4zW9SQFx1Fdx558T22dsLxxyTbmF6//1pjib3P5gZDoypodZJXf4Ch8ZXVr/4Yroaes4cePe74YQT4LTT4JFHRt+/O6fNLIMDo5s16qgeGkohsM02o3dYS2m679mz0/Ndd019GI1OS3m6DTPL5D6MbtCoH2JgII00atRRvXFjakmMZvHi4bCAFAbnnTfcWQ3p8Zo1nm7DzLL5Fq1VGm2o6/bbpxFIk5lyQ4ILLnAImNkf+RatU12tBVG+hWjNSy+N//r581Mro/x6CU480WFhZm3hU1LtMN5Q18ceg499rHFY5OjtTf0O9bcgveACX/9gZm3jFkarveENcHtpSqzaUNcbb0zXLVx5Jdx22/j7adSCqC0vd1K7NWFmHeIWRiuUZ3e9vcH8iS++mIa2nnMO7LYbfOUrsMsuo+9vtBaEO6nNrEJuYTRrrL6IMgmeegq22y49X7So8evcgjCzLuUWRiONpvwuP1+zBm66Cb7+dfjQh/L6IhYvHg4LSEHgFoSZTSEeVlsz2pTfrbJmjYPAzCrRqmG1bmFAakHU7iwHeWExfz488EBqGYxnn30cFmY25c2swBjtiupGczKNZ/162H33NPdSb+/o2y1bljcqysysy82cTu+DDkpDWmtqczJN9vTT4sXp31rLoX5yQLcozGyamT6BUe6DqKnd+Q1GhkXNxo2Te6/e3uEZYyGFgwPCzKa56REYow1t3bKl+XtUL1kChxzi+0OY2Yw3tQLjllvSKaSenhQGtfs3rFw5+Wk2xpqTydNsmJn9UeXDaiX1AIPAQxFx2Fjb9knxikG1s2fDyy9PvoA1a9K/7oMws2lqOs1WezJwB/Ank3p1M2GxbJmvqDYzy1TpsFpJuwOHAt9pemdjDW096aT009OTnvf0pOc/+1nTb2tmNlNUfR3GN4FPAVtH20DSCkmDkka/xHvJkuFpNspqwbBqVfrZvDkNod282f0TZmYTVFlgSDoMeDwibhhru4hYHRF9o55/qw1xXb4c1q5NgVD7cTCYmbVMlS2MdwDvl7QW+D5woKQ1E9pDrWXh/gczs7arLDAi4rMRsXtELAWOAK6KiKPGfWFtVteI1KJwWJiZdUQ3jJLK9+Y3Q7tmqzUzszF1RWBExC+AX1RchpmZjaHqUVJmZjZFODDMzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLI4MMzMLIsDw8zMsjgwzMwsiwPDzMyyODDMzCxLZYEhaZGkn0u6XdJtkk6uqhYzMxvfrArfezNwSkTcKGkecIOkKyLi9gprMjOzUVTWwoiIRyLixuLxc8AdwGurqsfMzMbWFX0YkpYC+wLXN1i3QtKgpMF169Z1ujQzMytUHhiS5gI/Aj4WEc/Wr4+I1RHRFxF9Cxcu7HyBZmYGVBwYkrYlhcVARFxSZS1mZja2KkdJCfgX4I6IOK2qOszMLE+VLYx3AEcDB0q6qfg5pMJ6zMxsDJUNq42IXwKq6v3NzGxiKu/0NjOzqcGBYWZmWRwYZmaWxYFhZmZZHBhmZpbFgWFmZlkcGGZmlsWBYWZmWRwYZmaWxYFhZmZZHBhmZpbFgWFmZlkcGGZmlsWBYWZmWRwYZmaWxYFhZmZZHBhmZpbFgWFmZlkcGGZmlsWBYWZmWSoNDEkHS7pL0j2SPlNlLWZmNrbKAkNSD/A/gb8C9gGOlLRPVfWYmdnYqmxhvAW4JyLui4iXge8Dh1dYj5mZjWFWhe/9WuCB0vMHgbfWbyRpBbCieLpR0q0dqK1ZC4Anqi4ig+tsnalQI7jOVpsqde7dip1UGRhZImI1sBpA0mBE9FVc0rhcZ2tNhTqnQo3gOlttKtXZiv1UeUrqIWBR6fnuxTIzM+tCVQbGb4C9JO0haTZwBHBZhfWYmdkYKjslFRGbJX0E+HegBzg3Im4b52Wr219ZS7jO1poKdU6FGsF1ttqMqlMR0Yr9mJnZNOcrvc3MLIsDw8zMsnRNYIw3TYik7ST9oFh/vaSlpXWfLZbfJel9Fdb4cUm3S7pZ0pWSlpTWbZF0U/HT1s79jDqPlbSuVM+HSuuOkXR38XNMxXV+o1Tj7yU9XVrXkeMp6VxJj492/Y+SM4rf4WZJ+5XWdfJYjlfn8qK+WyRdI+lNpXVri+U3tWr4ZRN1vkfSM6X/tp8rrevYVEIZdX6yVOOtxedxp2JdR46npEWSfl5859wm6eQG27T28xkRlf+QOr3vBfYEZgO/A/ap26Yf+Hbx+AjgB8XjfYrttwP2KPbTU1GN7wV6i8cn1Wosnm/oomN5LHBmg9fuBNxX/Ltj8XjHquqs2/6jpIERnT6e7wb2A24dZf0hwE8AAW8Dru/0scysc//a+5Om47m+tG4tsKBLjud7gH9t9vPS7jrrtv1r4KpOH09gV2C/4vE84PcN/l9v6eezW1oYOdOEHA6cXzy+GFgmScXy70fExoj4A3BPsb+O1xgRP4+IF4qn15GuLem0ZqZceR9wRUSsj4ingCuAg7ukziOBC9tUy6gi4mpg/RibHA58L5LrgB0k7Upnj+W4dUbENUUdUN1nM+d4jqajUwlNsM6qPpuPRMSNxePngDtIM2iUtfTz2S2B0WiakPpf/I/bRMRm4BlgfuZrO1Vj2QmkZK/ZXtKgpOsk/U0b6qvJrfNviybqxZJqF1B26lhO6L2KU3t7AFeVFnfqeI5ntN+jk8dyouo/mwH8VNINSlPxVO3tkn4n6SeS3lAs68rjKamX9EX7o9Lijh9PpVP0+wLX161q6eez66cGmYokHQX0AQeUFi+JiIck7QlcJemWiLi3mgr5MXBhRGyU9I+kltuBFdWS4wjg4ojYUlrWTcdzypD0XlJgvLO0+J3FsdwZuELSncVf2FW4kfTfdoOkQ4D/DexVUS05/hr4VUSUWyMdPZ6S5pIC62MR8Wy73ge6p4WRM03IH7eRNAt4NfBk5ms7VSOSDgJWAu+PiI215RHxUPHvfcAvSH8NtMO4dUbEk6XavgO8Ofe1nayz5AjqmvwdPJ7jGe336LqpbyS9kfTf+/CIeLK2vHQsHwcupT2ndLNExLMRsaF4fDmwraQFdOHxLIz12Wz78ZS0LSksBiLikgabtPbz2e6OmczOm1mkTpc9GO7QekPdNv+FkZ3eFxWP38DITu/7aE+nd06N+5I65vaqW74jsF3xeAFwN23qsMusc9fS4w8A18VwR9gfinp3LB7vVFWdxXZ/RupEVBXHs3iPpYzeSXsoIzsVf93pY5lZ52JS/97+dcvnAPNKj68BDq6wzl1q/61JX7T3F8c26/PSqTqL9a8m9XPMqeJ4Fsfle8A3x9impZ/Pth3sSfzyh5B6+e8FVhbLvkD6Sx1ge+CHxYf+18CepdeuLF53F/BXFdb4M+Ax4Kbi57Ji+f7ALcWH/BbghIqP5VeA24p6fg78Wem1xxfH+B7guCrrLJ6fCny17nUdO56kvx4fATaRzvOeAJwInFisF+lGYPcWtfRVdCzHq/M7wFOlz+ZgsXzP4jj+rvhMrKy4zo+UPpvXUQq4Rp+XquostjmWNOCm/LqOHU/SacUAbi79dz2knZ9PTw1iZmZZuqUPw8zMupwDw8zMsjgwzMwsiwPDzMyyODDMzCyLA8PMzLI4MMzMLIsDw6wJxf0I/rJ4/E+SvlV1TWbt4skHzZrzeeALxURz+wLvr7ges7bxld5mTZL0f4G5wHsi3ZfAbFryKSmzJkj6C9Kdz152WNh058Awm6TizmUDpLuabZDUtjvqmXUDB4bZJBR3WrsEOCUi7gC+SOrPMJu23IdhZmZZ3MIwM7MsDgwzM8viwDAzsywODDMzy+LAMDOzLA4MMzPL4sAwM7Ms/x/AARw+KHcZ5gAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
@@ -941,7 +945,7 @@
"import matplotlib.pyplot as plt\n",
"\n",
"x = 2*np.random.rand(100,1)\n",
- "y = 4+3*x+0.01*np.random.randn(100,1)\n",
+ "y = 4+3*x+np.random.randn(100,1)\n",
"\n",
"xb = np.c_[np.ones((100,1)), x]\n",
"beta = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
@@ -975,20 +979,11 @@
},
{
"cell_type": "code",
- "execution_count": 17,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHtpJREFUeJzt3XmYXHWd7/H3hyxCCCLpxAESOoERccCr4mTUUa8iiCKIOHeZCzcgS64ZyIjgqAjkXnFc7ujjDAbEwORKQKAvLogjet0iy3hVFhMmbAICgSxs2dghJNDf+eN3yj5dXVV9uruqTnX35/U8/fSpc06d+tZJpb792xURmJmZDWaHsgMwM7PRwQnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwrAxS9JBktaXHUezSQpJryk7Dht/nDCsrSQ9JOkFSc9KekzSpZKmlh2XmQ3OCcPKcGRETAXeBBwInFVyPOOSpIllx2CjixOGlSYiHgN+TkocAEg6QtK/SXpa0jpJn8sdm5NVxxwvaa2kTZIW5Y7vlJVYnpD0e+Av8q8n6c8k3SDpSUl3SfpQ7tilkpZI+mlW+vmNpN0lLc6ud4+kA+u9lyyukyXdl13/G5KUHfucpCtqvI+J2eMbJH1R0m+z1/6RpC5JPdl9+J2kOVUvebik1dk9+KqkHXLXP0nS3VncP5c0uyrOv5V0H3Cfkq9J2pC91h2SXj/Yv52NT04YVhpJs4APAPfndj8HfAR4FXAEcIqkD1c99Z3AfsAhwGcl/Vm2/xzgT7Of9wPH515rEvAj4BfAq4FTgR5J++Wu+9fA/wSmAy8CNwK3Zo+vAs4d5C19kJSk3pBd6/2DnJ93NHAcMDOL/0bgEmAacHf23vL+CpgLvBk4Cjgpe59HAWcD/wmYAfx/4Mqq534YeCuwP/A+4F3Aa4Fds7g3DyFuG0ecMKwM/yLpGWAdsIHcl2FE3BARd0REb0TcTvqye3fV8/8+Il6IiNuA24A3Zvv/GvhSRGyJiHXA+bnnvA2YCnw5IrZFxHXAj4Fjcuf8ICJWRsRW4AfA1oi4LCJeBr5Dqj5r5MsR8WRErAWuJ1dyKuCSiHggIp4Cfgo8EBG/jIiXgO/VeO2vZO9zLbA49z5OBv4hIu7Onvu/gTflSxnZ8S0R8QKwHdgFeB2g7HmPDiFuG0ecMKwMH46IXYCDSF9U0ysHJL1V0vWSNkp6ivQFOL3q+Y/ltp8nJQKAPUlJqGJNbntPYF1E9FYdn5l7/Hhu+4UajwdrnK8XVxFDfe3q97lntj0bOC+rFnsS2AKI/u/zj8/NEucFwDeADZKWSnrlEOK2ccQJw0oTEf8KXAr8Y273/wWuAfaKiF2Bi0hfeEU8CuyVe9yd234E2Ctf158df3iIYQ/Hc8CU3OPdm3DN6vf5SLa9DvibiHhV7meniPht7vx+U1RHxPkR8eekKqrXAp9uQnw2BjlhWNkWA4dKqlQr7QJsiYitkt4C/PchXOu7wFmSdsvaR07NHbuZ9Ff/GZImSToIOBL49ojfweBWAe+S1C1pV5rTK+zT2fvcCziNVGUGKcGeJekAAEm7Svqv9S4i6S+yUt0kUmLbCvTWO9/GNycMK1VEbAQuAz6b7VoIfD5r4/gsKQkU9fek6pkHSY3bl+deZxspQXwA2AQsAT4SEfeM9D0MJiKWk77QbwdWktpORuqH2bVWAf8PuDh7rR8AXwG+Lelp4E7Se67nlcD/AZ4g3bvNwFebEJ+NQfICSmZmVoRLGGZmVogThpmZFeKEYWZmhThhmJlZIaNq8rHp06fHnDlzyg7DzGxUWbly5aaImDHS64yqhDFnzhxWrFhRdhhmZqOKpDWDnzU4V0mZmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVogThpmZFeKEYWZmhThhmJlZIU4YZmZWSMsThqRlkjZIurPGsU9KCknTWx2HmZmNTDtKGJcCh1XvzNYifh+wtg0xmJnZCLU8YUTEr4AtNQ59DTgD8BqxZmajQCltGJKOAh6OiNsKnLtA0gpJKzZu3NiG6MzMrJa2JwxJU4Czgc8WOT8ilkbE3IiYO2PGiKdzNzOzYSqjhPGnwN7AbZIeAmYBt0ravYRYzMysoLYvoBQRdwCvrjzOksbciNjU7ljMzKy4dnSrvRK4EdhP0npJ81v9mmZm1nwtL2FExDGDHJ/T6hjMzGzkPNLbzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK6TlCUPSMkkbJN2Z2/dVSfdIul3SDyS9qtVxmJnZyLSjhHEpcFjVvuXA6yPiDcAfgLPaEIeZmY1AyxNGRPwK2FK17xcR8VL28CZgVqvjMDOzkemENoyTgJ/WOyhpgaQVklZs3LixjWGZmVleqQlD0iLgJaCn3jkRsTQi5kbE3BkzZrQvODMz62diWS8s6QTgg8AhERFlxWFmZsWUkjAkHQacAbw7Ip4vIwYzMxuadnSrvRK4EdhP0npJ84ELgF2A5ZJWSbqo1XGYmdnItLyEERHH1Nh9catf18zMmqsTekmZmdko4IRhZmaFOGGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVogThpmZFeKEYWZmhThhmJlZIU4YZmZWiBOGmZkV4oRhZmaFOGGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVkjLE4akZZI2SLozt2+apOWS7st+79bqOMzMbGTaUcK4FDisat+ZwLURsS9wbfbYzMxGoqcH5syBHXZIv3t6mnr5lieMiPgVsKVq91HAt7LtbwEfbnUcZmZjRk8PTJ8OUvqZPh0WLoQFC2DNGohIv48/HmbMaNrLTmzalYbmTyLi0Wz7MeBP6p0oaQGwAKC7u7sNoZmZdbCFC+HCC/vv27x54D6Al1+G555r2kuX3ugdEQFEg+NLI2JuRMyd0cRMaWbWsepVLfX01E4MjWzd2rSwyiphPC5pj4h4VNIewIaS4jAz6ww9PbBoUapKklK1EqTHCxbAvffC+ecP/brd3ekaTVBWCeMa4Phs+3jghyXFYWZWjnwpYvp0OPHEvi/2qKp0ef55+MIX4KmnGl9T6v94yhT40peaFnI7utVeCdwI7CdpvaT5wJeBQyXdB7w3e2xmNrZVkoQExx3X10C9eTNs3974uRLMmtX4nJNPhtmz07mzZ8PSpTBvXtPCb3mVVEQcU+fQIa1+bTOzjtHTk6qWnn8+Pa4uRQymuzuVFk48sXZyOeUUWLJk5HE2UHqjt5nZuHDmmX3JYqgqVUvz5sEll0BXV9+xri644oqWJwtwwjAzG55aYyHyA+Weegp+9CM4/XR4wxtg/fqhXb/SHlFdtTRvHmzalEooEWm7idVOjZTVS8rMbPTq6YGTToJt2/r2bd6cBsp997vw2GOwYgX09sKOO8I73wnr1sGTT9a/5uTJsMsusGVLX/VTmxJBUS5hmJkN1aJF/ZNFxcsvwzXXwMSJ6Zzrr4cnnoDly+GCC1LVUl6+FLFsWSot9PbCQw91XLIAJwwzs/7qDZrr7YVVq+DccxuPa5DgN7+Bz38eDjoolTAgJYClS/v3Yrr88lSt1KEJoppiqC31JZo7d26sWLGi7DDMbKyq7skEqaroTW+CBx5I1U6QShAvvVT7GrNnpwTQQSStjIi5I72OSxhmNr7kx0JMnJh+V0oSn/nMwJ5M27bBypXwwQ/CZZeltohLL02JpNqkSU0dKNdpXMIws/GjVgmiIj8dR61jvb0Dr3XaaX2ljq4uOO+8jqxacgnDzKyeeu0QZ51VfyxERDq/llozZZfYvbUsThhmNnZUxkYce2z/dSFOOAH23TdVJzXS2zuwJ1OT52MazZwwzGxsqFQ3VaqI8l56CdauhVe+svE1KoPkWjgf02jmhGFmna9RV9fbb4fFi+GjH2089cb27Wn6jOoSREV++o2HHuro8RBl8UhvM+s8lbUh1q6FadPgmWf6BsqtWZNGWX/967B6NWzcWOya3d19X/6VdScmTEiD7WbP7siR1Z3GJQwzK1d16aF6berNmweOqt62LU29cdhhaTK+NWvSl349+XaISgkiIlVVjaKBc2Vzt1oza7/3vheuvXZk16ju6lqvy2wHd3dtF3erNbPRI1+K2GmnkScLGNjVtdbUG1dcMS66u7aL2zDMrLWq//LfunXk16zX1XXePCeHFnIJw8yao976EGefPfyFgyomTUpVS+7qWiqXMMxs5Hp6Bi4dunlzWrd6OO2kO++cEs7atR27NsR45BKGmQ1NrTERZ5xRe53piL41H4qaOBH++Z89FqIDlZowJH1C0l2S7pR0paQdy4zHzBhYtZSvXqq0R+Sn3TjuOHjkkfrXixg4WK7enE1dXWkmWCeIjlRawpA0E/g4MDciXg9MAI4uKx4zo2/p0erpNTZvTvMxzZ8/sD2i0aR9UHu6jcsu65u0L//jHk0drew2jInATpK2A1OABn+mmFnL1Vt6FNIgt3qLBvX2pobp6mqpyZP72h+cCEa90koYEfEw8I/AWuBR4KmI+EX1eZIWSFohacXGolMAmFltCxf2LRo0cWJ6DClJ/PrXjZcebWT27DTiuqurb19XV1qn2olizChtpLek3YDvA/8NeBL4HnBVRFxR7zke6W1WUPVcTFB7FleAWbNgy5ZiXV+7uuCFF/qfO2WKu7l2uLEw0vu9wIMRsTEitgNXA28vMR6zsaHSxTU/F1O9ZAGwfn1qt7j66tQ7qZHzzvP03+NYmW0Ya4G3SZoCvAAcArj4YFZUvfmYGi01Ws/Xv963vWoVXHjhwHNOOaUvMThBjEtltmHcDFwF3ArckcWytKx4zEaVRpP3DTVZTJjQ//GSJWkOpuo5mZYsGV6sNmaU2ksqIs4BzikzBrOOVmmLqF67YbiN07UsWDBwn3s1WQ2DljAkLZf0xnYEY2Y5+UFykJIFNC9ZTJiQqplccrCCilRJfQZYLOkSSXu0OiCzcaWnB6ZO7RtRPWFCX1fXZkzaV61SvVRZPMjJwoZg0IQREbdGxHuAHwM/k3SOpJ1aH5rZGJSfh2nqVDj2WHjuub7jvb2pwbm7O3WJHa7q2V0rScLzMtkIFGr0liTgXuBC4FTgPknHtTIwszEjP1ju2GP7urvmE0W1detSQmnkFa+ovb8yiG7TJk/eZ01VpA3jN8DDwNeAmcAJwEHAWyS5V5NZIwsXphJDpf1hKC66aOCkfRVTpsDFF9eej8kJwlqkSAljATAzIg6NiP8VET+OiPsj4lTgP7Y4PrPO16gdYukw/6aaMKH/kqOVfeDBclaaEU0NImmfiFjdxHga8tQg1jHy3V3rmTYtTbkxHO69ZE3UEVODtDNZmLVdrYWCKvvz3V3r2bJl6IsHSU4W1rG84p5ZLdXzMa1Zkx5ffjl86lPFu7uefHLj45XG8EpPpt5eJwvrWE4YNr5Vry5XWVnutNMGru2wfTt85CPw2GPFrj1hQvryP+WUgdNvABxySLqmezLZKOGEYeNXrdXlKivLNZrddfr0YtevTLmxZEkaJFfdm+mXvxx26GZlcMKwsa9eW0S91eXqrSpXsXhx/e6ukF7H7RA2BpW9RKtZa+R7MeWn+16zBj76UbjlluHNydTV1Vd1VFmgqLu7bxlSszGstBX3hsPdaq2ufILYYYfULjCYRutGdHXBM8/0L4FMnuwlR21U6ohutWalqlQ1SXDccX0lhqLJ4tJLUxKoNmlSWllu2bL+a0I4Wdg45yopG30qvZjyDdNDLSl3d6ceTxMm9L9WV1dKFl5ZzmwAlzCss1U3WC9cmHofNerFNJgpU1KbA6SEsGlTX8+lTZucJMzqcAnDOldlRHVlkNyaNbXXmi6i0l4xe7YbqM2GySUMK0+97q4Ajz4Kp5/enAWEurrSCG3P5Go2Ik4Y1h71qpbyU2/Mnw/vex8ccADsuWeqHhqKyrxN+Vldr7jC1UxmTVJqwpD0KklXSbpH0t2S/rLMeKyJ8j2Ydtih/8JBa9aktR6qSw8vvphGP++1F3zlK7D77rWvXWtCv3wpojKq2qUJs6Yquw3jPOBnEfFfJE0GGgyftVFj4cKUECo9l2r1YGrUq+lnP0u/Z87s34YBqcH6+OPhJz/xoDmzNistYUjaFXgXaQU/ImIbUGOeBhtVenr6J4uh6u7u2/aIarOOUmYJY29gI3CJpDcCK4HTIqLfQseSFpBW/aM7/2VinSUC7rkHTj21eLKoHmmd7+5aMW+eE4RZhyizDWMi8Gbgwog4EHgOOLP6pIhYGhFzI2LujBkz2h2j1Zv+G1JbxLJlqX1i5kzYf3944oli150yJa0VkR9J7WVHzTpamSWM9cD6iLg5e3wVNRKGtVFlPqZK9c/hh8M3v9l/XYjNm9MI6U98AjZuTPte/Wo4+OC0vsM558AjjzR+nerR1GY2KpSWMCLiMUnrJO0XEfcChwC/Lyueca3WVBuNBsn19sKzz6Yv/YMPTt1gKz2XdtppYEN1hQfNmY1qZfeSOhXoyXpIrQZOLDme8ad6NHVRW7fCxz8+cL8bqs3GLE9vPp5t357GPDz++NCfO3t2GudgZh3P05tbffWm3OjthVWr4J/+CY44AqZNG16ymDx5YG8mMxvznDBGu/yI6spP9ajqk06Ct741NU4feCB86lPwwANpDYki61PvkPuYdHV5XQizcarsNgwbrloN1fVs2wYrV6ZEcsgh8J73wKxZ6dg73tG4DWPKFHd3NTPACWN0Gk5DdW9vWmGuWnUj9bRp6fGWLW6wNrN+XCXViRpN+/3002kMxFB7NTUaJT9vXmrA7u1NM7tu2pS2PXmfmeW4hNFpai0aNH8+fO97qYH6d7+Dl18e2jVrTblhZjZELmF0mrPPrj3t9w9/mLbPPDM1XhfV1eU2CDNrCieMVmtUvQSp6uf222HxYjjyyNSOUIsEN94IX/winHtuKjU0svPOXjzIzJrKVVKtUJmTac2a/jOyrlmTqpsefzx9oV93HVx/fd+cTK95DUydmqbdqOZpv82sZB7p3SxD6eZaseeeqZvrwQenn+7u2j2g3LXVzEagWSO9XcIYrvzMrtOmwZNPDq0x+p574LWvHbjcqEsPZtahnDCKqk4QzzyTBsTB0EoVkOZh2m+/+se9aJCZdSAnjCKqq4mGmiDy3MXVzEYp95IqolZX16GoVDt5VTkzG8XGd8Ko1+U1Au68E84/H446qn5X1yK6uuDyy9M1PXLazEax8VMlVb386Gtek7q15ru8nnQSXHABrF4NGzak/fvsU7+ra96kSWna7+eeS4+9DKmZjTFjv4TR05Om8K6e8vvaa/uSRcW2bXDLLXDooXDxxfDgg2ka8IsuGjhQbtKklBSkVNV0ySUpqUSkHw+YM7MxZmyWMOoNnCsiIo2QznNXVzOzUZww6iWFqVPT3Evbt6fHQx2YWG9WV3d1NbNxbnQmjOpurvmkMFhbQyOSu7yamdVRehuGpAmS/k3Sjws/adGikXVzrR0InHyySxFmZnWUnjCA04C7C525dSt84xupGmokurrglFNSY3Wl0fryy2HJkpFd18xsDCu1SkrSLOAI4EvA3w36hLvugo99DCZMGNq8TZU2jtmz3VhtZjZMZZcwFgNnAL31TpC0QNIKSSuenTYtdXP91rcarwcxcWL/Lq8eOGdmNmKllTAkfRDYEBErJR1U77yIWAoshTS9OfvskwbTQe1eUh4wZ2bWEmVWSb0D+JCkw4EdgVdKuiIiji30bHdzNTNrq9KqpCLirIiYFRFzgKOB6wonCzMza7uy2zDMzGyU6IiBexFxA3BDyWGYmVkDLmGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVogThpmZFeKEYWZmhThhmJlZIU4YZmZWiBOGmZkV4oRhZmaFOGGYmVkhThhmZlaIE4aZmRXihGFmZoU4YZiZWSFOGGZmVogThpmZFeKEYWZmhZSWMCTtJel6Sb+XdJek08qKxczMBjexxNd+CfhkRNwqaRdgpaTlEfH7EmMyM7M6SithRMSjEXFrtv0McDcws6x4zMyssY5ow5A0BzgQuLnGsQWSVkhasXHjxnaHZmZmmdIThqSpwPeB0yPi6erjEbE0IuZGxNwZM2a0P0AzMwNKThiSJpGSRU9EXF1mLGZm1liZvaQEXAzcHRHnlhWHmZkVU2YJ4x3AccDBklZlP4eXGI+ZmTVQWrfaiPg1oLJe38zMhqb0Rm8zMxsdnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCnHCMDOzQpwwzMysECcMMzMrxAnDzMwKccIwM7NCnDDMzKwQJwwzMyvECcPMzApxwjAzs0KcMMzMrBAnDDMzK8QJw8zMCik1YUg6TNK9ku6XdGaZsZiZWWOlJQxJE4BvAB8A9geOkbR/WfGYmVljZZYw3gLcHxGrI2Ib8G3gqBLjMTOzBiaW+NozgXW5x+uBt1afJGkBsCB7+KKkO9sQ20hNBzaVHUQBjrN5RkOM4DibbbTEuV8zLlJmwigkIpYCSwEkrYiIuSWHNCjH2VyjIc7RECM4zmYbTXE24zplVkk9DOyVezwr22dmZh2ozITxO2BfSXtLmgwcDVxTYjxmZtZAaVVSEfGSpI8BPwcmAMsi4q5Bnra09ZE1heNsrtEQ52iIERxns42rOBURzbiOmZmNcR7pbWZmhThhmJlZIR2TMAabJkTSKyR9Jzt+s6Q5uWNnZfvvlfT+EmP8O0m/l3S7pGslzc4de1nSquynpY37BeI8QdLGXDz/I3fseEn3ZT/Hlxzn13Ix/kHSk7ljbbmfkpZJ2lBv/I+S87P3cLukN+eOtfNeDhbnvCy+OyT9VtIbc8ceyvavalb3yxHEeZCkp3L/tp/NHWvbVEIF4vx0LsY7s8/jtOxYW+6npL0kXZ9959wl6bQa5zT38xkRpf+QGr0fAPYBJgO3AftXnbMQuCjbPhr4Tra9f3b+K4C9s+tMKCnG9wBTsu1TKjFmj5/toHt5AnBBjedOA1Znv3fLtncrK86q808ldYxo9/18F/Bm4M46xw8HfgoIeBtwc7vvZcE43155fdJ0PDfnjj0ETO+Q+3kQ8OORfl5aHWfVuUcC17X7fgJ7AG/OtncB/lDj/3pTP5+dUsIoMk3IUcC3su2rgEMkKdv/7Yh4MSIeBO7Prtf2GCPi+oh4Pnt4E2lsSbuNZMqV9wPLI2JLRDwBLAcO65A4jwGubFEsdUXEr4AtDU45CrgskpuAV0nag/bey0HjjIjfZnFAeZ/NIveznrZOJTTEOMv6bD4aEbdm288Ad5Nm0Mhr6uezUxJGrWlCqt/4H8+JiJeAp4Cugs9tV4x580mZvWJHSSsk3STpwy2Ir6JonP85K6JeJakygLJd93JIr5VV7e0NXJfb3a77OZh676Od93Koqj+bAfxC0kqlqXjK9peSbpP0U0kHZPs68n5KmkL6ov1+bnfb76dSFf2BwM1Vh5r6+ez4qUFGI0nHAnOBd+d2z46IhyXtA1wn6Y6IeKCcCPkRcGVEvCjpb0glt4NLiqWIo4GrIuLl3L5Oup+jhqT3kBLGO3O735ndy1cDyyXdk/2FXYZbSf+2z0o6HPgXYN+SYiniSOA3EZEvjbT1fkqaSkpYp0fE0616HeicEkaRaUL+eI6kicCuwOaCz21XjEh6L7AI+FBEvFjZHxEPZ79XAzeQ/hpohUHjjIjNudi+Cfx50ee2M86co6kq8rfxfg6m3vvouKlvJL2B9O99VERsruzP3csNwA9oTZVuIRHxdEQ8m23/BJgkaTodeD8zjT6bLb+fkiaRkkVPRFxd45Tmfj5b3TBTsPFmIqnRZW/6GrQOqDrnb+nf6P3dbPsA+jd6r6Y1jd5FYjyQ1DC3b9X+3YBXZNvTgftoUYNdwTj3yG3/FXBT9DWEPZjFu1u2Pa2sOLPzXkdqRFQZ9zN7jTnUb6Q9gv6Nire0+14WjLOb1L739qr9OwO75LZ/CxxWYpy7V/6tSV+0a7N7W+jz0q44s+O7kto5di7jfmb35TJgcYNzmvr5bNnNHsabP5zUyv8AsCjb93nSX+oAOwLfyz70twD75J67KHvevcAHSozxl8DjwKrs55ps/9uBO7IP+R3A/JLv5T8Ad2XxXA+8Lvfck7J7fD9wYplxZo8/B3y56nltu5+kvx4fBbaT6nnnAycDJ2fHRVoI7IEslrkl3cvB4vwm8ETus7ki279Pdh9vyz4Ti0qO82O5z+ZN5BJcrc9LWXFm55xA6nCTf17b7iepWjGA23P/roe38vPpqUHMzKyQTmnDMDOzDueEYWZmhThhmJlZIU4YZmZWiBOGmZkV4oRhZmaFOGGYmVkhThhmI5CtR3Botv1FSV8vOyazVvHkg2Yjcw7w+WyiuQOBD5Ucj1nLeKS32QhJ+ldgKnBQpHUJzMYkV0mZjYCk/0Ba+Wybk4WNdU4YZsOUrVzWQ1rV7FlJLVtRz6wTOGGYDUO20trVwCcj4m7gC6T2DLMxy20YZmZWiEsYZmZWiBOGmZkV4oRhZmaFOGGYmVkhThhmZlaIE4aZmRXihGFmZoX8OwP7A67UJe89AAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"# Importing various packages\n",
"from random import random, seed\n",
@@ -997,7 +992,7 @@
"from sklearn.linear_model import LinearRegression\n",
"\n",
"x = 2*np.random.rand(100,1)\n",
- "y = 4+3*x+0.01*np.random.randn(100,1)\n",
+ "y = 4+3*x+np.random.randn(100,1)\n",
"linreg = LinearRegression()\n",
"linreg.fit(x,y)\n",
"xnew = np.array([[0],[2]])\n",
@@ -1068,20 +1063,11 @@
},
{
"cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcXGWd7/HPLyEBshCyAApJJyAykU0gAYIgIlsISRAuygsn4DAyNig4oyJ4hbljGIwwekdEr4pBuCi04HLVV0JIgJHVCEqCAQMRQSAhLJIdskCW/t0/ntNTnU5Vd1V1naXO+b5fr3p196nlPOf0qef3nGc1d0dERIqrT9oJEBGRdCkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIFJwCgXTLzKaZ2b0xffatZvbVBn9mbOktGjP7oJk9m3Y6JH4KBIKZHWdmvzOzdWa22szmm9mRAO7e5u6npp3GrszMzWz/rtuzlN4o0G02s/XReb3PzMamna5qufsj7v53aadD4qdAUHBmthtwF/AdYBiwD3A18E6a6Wo2ZrZThae+7u6DCOf1FeDmhPcv0iMFAjkAwN3vcPdt7r7J3e9196cAzOwCM/ttx4ujkvhnzOw5M3vLzK4xs/dEdxRvmtnPzKx/9NoTzGy5mV1pZivN7CUzm1YpIWY2xcwWmdna6PMOrfVgKqT34ii9a83su2ZmnZ7/pJktMbM1ZnaPmY3u9NwNZvZydFwLzeyDnZ6bbma/MLPbzexN4ILu0uXum4CfAYd1SW93+z/VzJ6N7tS+Z2YPmdk/dTrO+WZ2vZmtAqZ393kWXG9mb0TH8yczOzh67nQzeyb6f75iZl+Mtp9gZss7ped9ZvZgdB6fNrMzOj13a3Ru50Sf83sze081/zNJnwKB/AXYZmY/MrNJZja0ivdMBMYBE4ArgJnAecAo4GDg451e+y5gBKFE/A/ATDPbobrBzA4HbgEuAoYDPwBmmdnO9R5YJ1OAI4FDgXOi9GNmHwGuBP4HsAfwCHBHp/c9Tsi4hwE/AX5uZrt0ev4jwC+A3YG27hJgZgMJ5+X5Ttsq7t/MRkSf/WXC+XgW+ECXjz0aeAHYC5jRw/GcChxPCPxDovOwKnruZuAidx9M+P/dXyb9/YDZwL3AnsBngbYu/8tzCXeTQ6PjnNHdOZHsUCAoOHd/EzgOcOAmYIWZzTKzvbp529fd/U13fxpYDNzr7i+4+zpgLnB4l9f/L3d/x90fAuYQMqGuWoEfuPvvozuTHxGqpyb07ggBuM7d17r7MuABSqXyi4Fr3X2Ju28FvgYc1lGKdvfb3X2Vu2919/8EdgY6Z3yPuvuv3b09KvGX80UzWwu8RTjP53d6rrv9nw487e6/jJ77NvB6l89+1d2/E6VvUw+ftwUYDIwFLHrNa9HnbAEONLPd3H2Nuz9R5jgmAIOic7nZ3e8nVCl2Dvq/cvc/RPtuo8vdj2SXAoEQZQoXuPtIQolwb+Bb3bzlb51+31Tm70Gd/l7j7hs6/b00+vyuRgOXRdUOa6PMc1SF19aqcwa6sVP6RgM3dNrfasAIdy+Y2RejapZ10fNDCHc3HV6uYt//2913B8YQzk3nQNLd/vfu/PkeZodczva67r/i50UZ9/8Bvgu8YWYzLbQPAZxNCDxLo+qnY8ocx97Ay+7e3mnb0iitHSqdZ8k4BQLZjrv/GbiVEBAaYWhULdKhBXi1zOteBma4++6dHgPc/Y4yr22UlwlVIp33uau7/y5qD7iCcPcyNMrM1xEy1g5VT90b3Y38CyGj3rWn/QOvASM73h+1a4zs+rHVHk+Uhm+7+zjgQEIV0eXR9sfd/SOEKp9fE9oyunoVGGVmnfOMFkIDuDQ5BYKCM7OxZnaZmY2M/h5FuN1/rIG7udrM+keZ6xTg52VecxNwsZkdHTVsDjSzyWY2uJvP7W9mu3R69K0xXTcCXzazgwDMbIiZfSx6bjCwFVgB7GRm/wbsVv5jquPu9xEy1NYq9j8HOMTMzrTQI+gSQntLXcdjZkdG57YfsAF4G2iP/i/TzGyIu28B3gTay3z27wml/CvMrJ+ZnQBMBe6s41RIxigQyFuERsffm9kGQgBYDFzWoM9/HVhDyADbgIuju47tuPsC4FOE6os1hMbGC3r47KcJ1S0dj3+sJWHu/ivgP4A7LfT8WQxMip6+B5hHaExfSsg4q6kK6sk3CJnpzt3t391XAh8Dvk5o1D0QWEA33Xp7OJ7dCMF2TXQ8q6K0QGi3eCl6z8XADj273H0zIeOfBKwEvgd8otz/UpqPaWEaiUtUarw9anuQXoiqZJYD09z9gbTTI/miOwKRjDKziWa2e9SF9kpC+0Qjq+xEAEhkNKKZvUSogtgGbHX38UnsV6TJHUMYv9AfeAY4s5tuqiJ1S6RqKAoE46N6TxERyRBVDYmIFFxSdwQvEnorOGH06Mwyr2kl6lY3cODAcWPHNs0kjSIiqVu4cOFKd9+jnvcmFQj2cfdXzGxP4D7gs+7+cKXXjx8/3hcsWBB7ukRE8sLMFtbb/ppI1ZC7vxL9fAP4FXBUEvsVEZGexR4IohGigzt+J8yCuDju/YqISHWS6D66F/CrMFUKOwE/cfd5CexXRESqEHsgcPcXgPfHvR8REamPuo+KiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUXGKBwMz6mtkfzeyupPYpIiI9S/KO4F+AJQnuT0REqpBIIDCzkcBk4IdJ7E9ERKqX1B3Bt4ArgPZKLzCzVjNbYGYLVqxYkVCyREQk9kBgZlOAN9x9YXevc/eZ7j7e3cfvsccecSdLREQiSdwRHAucYWYvAXcCJ5rZ7QnsV0R6o60NxoyBPn3Cz7a2tFMkMYk9ELj7l919pLuPAc4F7nf38+Ler4j0QlsbtLbC0qXgHn62tioY5JTGEYjIjq66CjZu3H7bxo1hu+TOTknuzN0fBB5Mcp8iUodly2rbLk1NdwQisqOWltq2S1NTIBCRHc2YAQMGbL9twICwXXJHgUBEdjRtGsycCaNHg1n4OXNm2C65k2gbgYg0kWnTlPEXhO4IREQKToFARKTgFAhERApOgUBEpOAUCERECk6BQESk4BQIREQKToFARKTgFAhERApOgUBEpOAUCERECk6BQESk4BQIREQKToFARKTgFAhEJNva2mDMGOjTJ/xsa0s7Rbmj9QhEJLva2qC1FTZuDH8vXRr+Bq2V0EC6IxCR7LrqqlIQ6LBxY9guDaNAICLZtWxZbdulLgoEIpJdLS21bZe6KBCISHbNmAEDBmy/bcCAsF0aRoFARLJr2jSYORNGjwaz8HPmTDUUN5h6DYlItk2bpow/ZrojEBEpOAUCEZGCUyAQESk4BQIRkYJTIBBJQ97mz8nb8RSMAoFI0jrmz1m6FNxL8+c0a+aZt+NJU0oB1dw9kR3VYvz48b5gwYK0kyESjzFjQmbZ1ejR8NJLSaem9/J2PGnpOsEehMFzVY6bMLOF7j6+nl0rEIgkrU+fUHLuygza25NPT2/l7XjS0suA2ptAEHvVkJntYmZ/MLMnzexpM7s67n2KZFre5s/J2/GkJcUJ9pJoI3gHONHd3w8cBpxmZhMS2K9INuVt/px6jkeNyztKMaDGHgg8WB/92S96ZK8+SiQpeZs/p9bjUeNyeSkWEBJpIzCzvsBCYH/gu+7+pTKvaQVaAVpaWsYtLVdXJiLNT43LlbW1hUV3li0LdwIzZlRdQGiaxmIz2x34FfBZd19c6XVqLBbJMTUuxyLTjcWdufta4AHgtCT3KyIJqabuX43LmZNEr6E9ojsBzGxX4BTgz3HvV3JOjY3ZU23df94ay3MgiTuCdwMPmNlTwOPAfe5+VwL7lbxSY2M2VbvQfN4ay3NAA8qk+aixMZtU95+qpmkjEGmIFAfeSDdU99+0FAik+SjDySbV/TctBQJpPspwskl1/01Li9dL8+nIWOoceCMx0kLzTUmBQJqTMhyRhlHVkIhIwSkQiIgUnAKBiEjBKRCIiBScAoGISMEpEIiIpCUjkyeq+6iISBo6Jk/smKivY/JESLxrtO4IROqRkZKcNLFqZ2tNgO4IRGqVoZKcNLEMTZ6oOwKRWmWoJCdNLEOTJ/YYCMzsPjN7fxKJEWkKGSrJSRPL0OSJ1dwRfAn4lpn9XzN7d9wJEsm8DJXkYqV2kHhlaLbWHgOBuz/h7h8G7gLmmdlXorWHRYopQyW52Gg50GRMmxZW1WtvDz9TamOqqo3AzAx4Fvg+8FngOTM7P86EiWRWhkpysammHUR3DLlRTRvBfOAV4HpgH+AC4ATgKDObGWfiRDIrIyW52PTUDlKUO4aCBLseF683s4OAZ7zMC81sibu/r9GJ0uL1IikbMyZk7l2NHh0CX0/P50HXbsIQqgAzevcX6+L17v50uSAQmVzPTqXBClJqkQT11A5ShJ5TBeom3KtxBO7+QqMSInUqyi26JKundpAi9JwqQrCLaEBZFtVSwi9QqUUS1l07SBF6ThUh2EUUCLKm1hJ+gUotkiF57TnVuRC2fj3077/983kLdpEeG4vTUOjG4lob4YrQaCeShHKNw/36wW67werV4U5gxozMBrveNBZr0rmsqbWEP2NG+Z4NOSy1iMSqXDXrli0waBCsXJlOmhKiqqGsqbVeMq+36CJxKtcOV+BqVgWCrKmnES7vg5tEGqlSO9ywYeVfn8PG4a4UCLJGJXyReFXqaQf57wlVgQJBFqmELxKfSlU9q1cXthCmxmIRKZaWlvI97VpaQqbfLBn/1q3w6KNw113h0Qu6IxCRYmnmwXDr1sFPfwrnnw977QXHHw/f/Ca8u3dLxeiOQESKpaPEf9VVoZoo4+MDeO65UOKfPRseeSTcCQwfDpMnw9SpcOqpMGRIqM6qkwaUiUhjtbU1TyabRVu3wvz5pcz/2WfD9oMOChn/lCkwYQL07bvd2zI9oMzMRgE/BvYCHJjp7jfEvV8RSUHX0bkdXTNBwaA7a9bAvHkh4587F9auDaOaTzgBLrkkZP777hvb7mO/I4jWOX63uz9hZoOBhcCZ7v5MpffojkAkIY0uvWvKk+o9+2yp1P/b38K2bbDHHqHKZ8qUUOUzeHDVH5fpOwJ3fw14Lfr9LTNbQljprGIgEJEEVFt6ryVYFHh0bo+2bAkZ/uzZIQA891zYfuih8KUvhWqfI4/coconCYm2EZjZGOBh4GB3f7PLc61AK0BLS8u4peVKFSLSONWU3mtdpUt3BNtbtapU5TNvXuj1078/nHhiKPVPmRLOTQP05o4gsUBgZoOAh4AZ7v7L7l6rqiGRBPTpE6ZY6MosDGaE2jP2JlveseHc4c9/LpX6588P53KvvUpVPqecEiaya7BMVw0BmFk/4P8BbT0FARFJSHcDqzrUWtXTbF0zG2Hz5tCtc/bs8HghWrjxsMPgyitDlc/48SHwZlQSvYYMuBlY4u7fjHt/IlKlaqYwryZYdNVMo3PrtXIl3H13KPXfcw+8+SbsvDOcdBJcfnko/Y8alXYqq5bEHcGxwPnAn8xsUbTtSne/O4F9i0gl1ZTetd5F4A5PP13q5fPoo2Hbu94F55wTSv0nnQQDB6ad0rpoQJmIdK+oA8TeeQceeqiU+Xe0iRxxRGlg1xFHpFfl0+X/ssfSpS+ucN+vno/KbqWVZFu5hT0kn7IwG25S19sbb8Ctt8LZZ8OIETBxItx0Exx8MPzgB7B8OSxcCNOnp1vvX2ZNhRaov/uRu2fuMW7cOJcMu/129wED3MMlGB4DBoTteXb77e6jR7ubhZ95P96siPN6a293f/JJ9xkz3CdMCP9bcN9nH/eLLnKfPdt9w4be76dWPV1ro0dvfz7Ax4F7nXlu6pl+uYcCQcaVuQgdwvZGyVqmW9TglwWNvt42bXKfO9f95JPd+/Ytfd6++7pffbX7E0+EAJGWaq61joDVoECgNgKpXTX9z3sji33RNVAqPY243l5/PfTymT0b7rsPNmzY8TVpX2MdqrnWyrxmPLDAva4pSNVGILWr1HWwUWu7VlpK8KqrGvP59ejN1AlqT+mdeq43d1i0CK65Bo4+OszXf+GFoX7/E5+APffc8T1pX2MdqrnWyqyp4FB/KazeW4k4H6oayri4q0nK3PY6hO1pqbd6QlVKvVftOdy40X3OHPeLL3YfObJ0zRx9tPs117gvWlSq8sniNdah2mutS/XpCHjB68xzU8/0yz0UCJpAnHX4SbRB1KreDD2rx5Kl9pdqVErzq6+6z5zpfsYZpf/PwIHuZ53lfsst7q+/Xv7zsvh/6VDntQYscAUCyY2slqLryUCzVvLM6rmtVnu7+8KF7tOnu48bVzqGlhb3Sy5xnzfP/e23e/6crJ+HOq41BQLJn2YstZaTtZJn1tJTjQ0b3GfNcm9tdd9771IgPeYY9699zf2pp+rr5ZOXayyiQCC1y9mXILOyVvLM2h1KJcuXu994o/vkye677BLSOGiQ+9lnu996q/sbb6SdwspS+m4pEEhtspY5pSHJL2uWgm5W7wi2bXP/wx/c/+3f3A8/vJSuffd1/+d/dr/33uqqfNLWqO+WqoYUCGKX1cwgKUUOhFk69vXr3X/9a/cLL3R/17tCWvr0cT/2WPfrrnNfvDjdgV31aMR3K4XGYg0oK6K4B4RlXdEHh6U5idyyZTBnThjYdf/9YWK33XaD004Lk7hNmhTm+GlWjfhu1Xl9NsUKZbVQIIhZ0TPCogfCJLW3w+OPl1bsevLJsP097wkzeE6dCscdF5ZvTFsjAmQjvlt1Xp+9CQQaWVxEZUYl5maO+WpG8cY9Mrro1q+HX/4SPvnJMKJ3wgS49loYMgS+/nVYsiQs3H799WHt3qwEgS6zedLaWvso8EZ8t9K4PuutU4rzoTaCBGSpAbNRqq1bzVI9eV689JL7d77jPnGie//+4ZwOGeJ+7rnubW3uq1alncLuNbLdrLffLQ0o6xQI8phRSbxq+TLr+uqdrVvdf/c79y9/2f3gg0vn+oAD3L/wBfcHHnDfvDntVFYva91qE+41lM02gv328wV/+1u2Zp+U7FPdf7zefBPuvTfU9d99N6xYAX37wgc/WFqx64AD0k5l9Tq3CfTpA9u27fiaJmo3600bQRJrFtfulVdg8+btt3XMDKhAIJXUs9C6dO/FF0sNvQ8+CFu2wNChoXfP1Kmht8/uu6edytp1neq8XBDIS7tZFbIZCLoGgQ7VTPkrxaWF1ntv27awMHvHOr3PPBO2jx0Ln/tcKPV/4AOwUzazjqqVm+ocwh1Oe3ux1mYmq4Ggf//ywUAlO+lOx5e2iAut98a6dXDPPSHjnzsXVq0KGf2HPgSf+lTI/PffP+1UNlalQmV7eyGrEbMZCPbZB8q1ERShZJfmYJ88mDZN56sazz9fKvU//DBs3QrDh8Ppp4eMf+LE0N0zr1SNuJ1sjiMYNiw0DI8eHRr6Ro8uRkNxo/oyi3S1dWvI8C+/HN73Pnjve+Hznw8Frssug9/+Nvz+4x/DOefkOwhAvsfS1KPe7kZxPgo7jiDNOYDUnTJ/Vq92v+MO97//e/ehQ8O11K+f+ymnuN9wg/tf/5p2CtOVs2ue3HUfLeoUE2l1f8ziYvFxyXvV21/+Uurl88gjofF3xAiYPDlU+Zx6apjbR3JHcw3lRVpzABVl7qE8BrwtW2D+/JD5z54dpm4AOOSQkPFPnQpHHRV6w0iuKRA0s84l1GHD4K23tu8xlURGVZSBWHkJeKtXh949d90F8+bB2rWhp92HPxwy/smTw7FKoeRvQFlRdC2hrloF/fqF3hurVydXdVGUHhSVugxmfXyKO/z5z6VePvPnhwC9555w1lkh8z/5ZBg8OO2USpNSIEhTuUEtW7bAoEGwcmVy6SjKQKxmCnibN4c6/o7M/69/Ddvf/3648spQ7XPkkeFuTqSXin0VVTNlcZyyUkKdNq15u+vW8j/MepfBVavgttvg6KNhl11CKf9b34KBA+F73wvXxaJFcM014TUKAtIo9XY3ivORSPfRNKYi7tpdbfjw9LqL5kE9/8MsdRlsbw/LMV53XViesU+f8teDpsiWKqDuo3VIuuGwXI+V/v3DV33LltK2Zu/FkqRmbPzdvBkeeqjUxfPFF8P2ww8Pdf0zZ8Lrr+/4viwfk2SCViirVVtb+QwE4quWKdcesHlz6NOd1SqZtKvOepKVqrWerFgBP/oRfPSjoU//qafCTTfBQQfBjTfC8uXwxBNw9dVhdG85WTsmyZXiNRZ3lMwriavhsNIXefXqZBuGq9X1DqZjugvITqDqrvE3zYFj7rB4canU/9hjYdvee8PHPx5K/ieeuGN7RUfam6VBW/Kj3jqlah/ALcAbwOJq3xNrG0GlaRzirotNc/qIeqSR3lrr7yu1EXz608m3/7z9tvu8ee6XXLL9uRs/3n36dPeFC0ObQE+0jKbUiSwvVQkcDxyRmUBQaUk6iL+hOIkveKMaQ5Neuq/e81PueJMKYq+/7n7zze5nneU+cGDYx667up9xhvtNN7m/8kp9n5ulBm1pGpkOBCF9jMlMIMjzxG6NDDZJn6dG7i+uINbe7r5okfs117gfdVTpc0eOdL/4Yvc5c9w3buzdPkTqlItAALQCC4AFLS0tcZynIM+33o3MTJM+T43MvBt5HjZtChn8pz/tPmpU6bOOOioEhD/+sboqH5GY5SIQdH7EPo4gr7fejS4JJ3meshTEXn01VO2ccUbpcwYOdD/zzFAV9NprtadJJGYKBBJktUG6moDS6DuQWoJYe3tozL366tC427H/lpbQ+Dt3brgzEMkwBQIJsljtVUuakrwD2bDBffZs99ZW9733Lt05TZjgPmOG+5NPqspHmkpvAkHsI4vN7A7gBGAE8DfgK+5+c3fvKdQ01I2WtYVXsjT695VXYM6c0L//N7+BTZvCBH8TJ4ZJ3E4/PczoKdKEtB6BZFeaax20t4cRux0Du554ImwfMyYM6po6FY4/HnbeOd50pClrBQOJjaaYkOyqNCI2rpGyGzbArFnwqU/ByJFhquavfhV23RWuvTaM+H3hBfj2t+GUU/IfBFpbwx2Ze2l0eNamCimyrEzjUm+dUpwPtRHkSBLtFsuWuX/ve+6nn+6+yy5hH4MHu3/sY+4//rH7ihWN21cz6anzQF57zzWLBn83yHIbQT1UNZQzja6eaG+HBQtKVT6LFoXt++1XqvL54AfD7K5F1l213G235W/95mbT4PYztRFI/q1fD/fdFzL+OXPCLJ19+sCxx4aMf8oUGDs2ZHISdJfRQHYa8Yuqwe1nWrNY8mnp0tJSjQ88EKbtHjIETjstZP6TJsGwYWmnMru6W4L0/PPLv0fTXScnQzPNqrFYsmPbNnj00VCNdOihoUR76aWhcffSS+H++8Pc/nfeGaovFAS6190SpJUym2HDstF4WQQZWjpVVUOSrrfegnvvDaX+u+8OGX3fvqGOf8qUUPI/4IC0U5k/WjEvGxrYfqY2AmkuL75YqvJ58MGQ8QwdGqp6pkwJVT9Dh6adyvzrmgmtXw+rVu34ujy1G+R4XIUCQb1yfFFkyrZtYZWujl4+Tz8dto8dWyr1f+ADsJOarFKV5uC/JJS7C8rRHU9vAkHqYwbKPRIZR5DFeXkapdH9w+v5vLVr3X/2M/fzz3cfPjyc3512cj/xRPdvftP9ued6l6Zm0Gz99JOYtDDNc5LVSRkbhKxPOlfrI5FAkNeLorcBrusXtZZlH59/3v36691POilk+uA+bJj7eee533mn+5o1jTzSbGvGgkbcaU77nCS96l7CFAjqkdeLojcBrtwXtdJ5Gj3afcsW94cfdr/8cvexY0vPHXig+xVXuD/ySHhNs6unFNusBY04S+xpn5O09x8zBYJ65PWi6E2Aq3ROKj2GDQs/+/VzP/lk9xtuCHcFeVJvKTYPBY1GB4W0z0nadyQxUyCoR14vit4EuEpf1HKPPn3cP/EJ95//3H3duriPKj31ns9mL2jE8f3IwjlptnabGigQ1OP220uNmBB+z8NF0ZsvcLV3BDvvHCZzK4J6S7HNXtCII9Nu9nOScb0JBMUcWdzRjaxzn+lNm9JLTyN1N5q0O2vWwOTJYTBXZ336wHHHwT77lD7v5psrT1GQN/VOo13v/yErKk010ZspKJr9nORZvREkzkfsdwRZuEVNW3u7+5Il7t/4hvvxx7v37RvOweDBYaF2M/dRo5qjtBbn7X5RS7H6jjQdVDVUo7QbrdKyebP7b37j/rnPue+/f+m4Dz3U/aqr3B97zH3btrRTWZskMuoc1ytXlKUAWMTzXwcFgloVqbSzcqX7bbe5n3OO+267+X/X8U+a5P7d77ovXZp2CnunSP/LpGUhA85SQMq43gSCYk4xkeeh5u6wZElpOoff/S5MD7DXXqXpHE46KSzangd5nxah6Bq8eEueaT2CWnVk9nmZZ2jzZnj44VLm/8ILYfvhh4djnDoVxo0LmWbeZGhOd4lBHI3WsoNiBgIImX6zZvwQpmu+++6Q8d9zT5jOeZddQmn/iitCD6CRI9NOZfy6W3xFmp8CfSJyWETMKXdYvBiuvTbM1LnXXnDBBTB/Ppx7LsyaFbrD3nUXXHRR8wSBtrbeLYSS9S6JvT2+osvQ4i25Vm/jQpyPRAaUNYO333afN8/90kvdx4wpNZaNG+c+fbr7ggWhG2g5WWjo60neGwLzfnxJaYZrOQNQY3GO/O1vocpn9uywcteGDbDrrnDyyaGuf/Jk2Hvv7j+jWRrD894QmPfjk0zRwjRZVc3CN+7w1FOlFbv+8IewbZ99QsY/ZQqceGIIBtVqlgwo7z1+8n58kim9CQRqI4hLR6l86dKQGSxdGv5ua4O334a5c+GSS0Kmfdhh8K//GjKHq6+GJ57V07sAAAAHFElEQVSAl1+G738/3AHUEgSgeXpa1Dt9Q7Oo5fjUliBpqrdOKc5HLtoIKg102nXXUr3xgAHuZ57p/sMfur/2Wvz7TnqQVU91u3mvQ6/2+PJ+HiQRaGRxBnU3pfNnPuM+d677pk3x7DsLGUstmWCeGwKrOb5KgXv48IQTK82sN4FAbQSNtGkT3H9/qOv/4Q/Dou1dVeoX3WjVtE/EqVnaKbKgUlsCwO23Z6uBXzJLjcVpevXV0NB7113wX/8VgsGgQTB2bGgE3ry59Nos9tyJixpKq1cpaIICp1RNjcVJam+HhQth+nQYPz707rnoopDpX3hhGOW7ciU8/jjcckt2BzrFLe8NwY3U3eCorDXwSy7pjqAaGzeG0n5Hyf+110LmfswxpS6eBx0UtknQLGMZsmLEiO0XSuqgOwKpkiadi8Py5aW+/fffH7p8Dh4MEyeGzH/SJNhjj7RTmV15m9gvbjfcoDmTJDUKBB3a22HBglLmv2hR2L7ffqHqZ8oUOP546N8/3XQ2k2af2C9JjQycaXcUkKaTSNWQmZ0G3AD0BX7o7td19/rEqoY2bID77gsZ/5w5YXqHPn3g2GNLc/ePHasqn6xRRleZquQKK9O9hsysL/AX4BRgOfA48HF3f6bSe2INBMuWlUr9DzwA77wDQ4bAaaeFzH/SJBg+PJ59S+8po+ueuu0WVtYDwTHAdHefGP39ZQB3v7bSexoaCNrbw/w9HYu2PPVU2L7//qHEP3UqHHcc9OvXmP1JvJTRdU/ddgsr64Hgo8Bp7v5P0d/nA0e7+6VdXtcKtEZ/HgwsjjVhzWMEsDLtRGTAiBHQPhr2rfSChbAwyQSlqOI1cSgc0g92aMjaApufgj/FnrLk6ftR8nfuPrieN2amsdjdZwIzAcxsQb2RLW90LgIzW7BC5wHQNdGZzkWJmdVdjZLEgLJXgFGd/h4ZbRMRkQxIIhA8DrzXzPY1s/7AucCsBPYrIiJViL1qyN23mtmlwD2E7qO3uPvTPbxtZtzpaiI6F4HOQ4nORYnORUnd5yKTU0yIiEhyNOmciEjBKRCIiBRcaoHAzE4zs2fN7Hkz+59lnt/ZzH4aPf97MxuTfCqTUcW5+IKZPWNmT5nZb8xsdBrpTEJP56LT6842Mzez3HYdrOZcmNk50bXxtJn9JOk0JqWK70iLmT1gZn+Mvienp5HOJJjZLWb2hpmVHWtlwbejc/WUmR3R44fWu7RZbx6ERuO/AvsRBr88CRzY5TWfAW6Mfj8X+Gkaac3IufgwMCD6/dNFPhfR6wYDDwOPAePTTneK18V7gT8CQ6O/90w73Smei5nAp6PfDwReSjvdMZ6P44EjgMUVnj8dmAsYMAH4fU+fmdYdwVHA8+7+grtvBu4EPtLlNR8BfhT9/gvgJLNczv7W47lw9wfcvWNynccIYzHyqJrrAuAa4D+At5NMXMKqORefAr7r7msA3P2NhNOYlGrOhQO7Rb8PAV5NMH2JcveHgdXdvOQjwI89eAzY3cze3d1nphUI9gFe7vT38mhb2de4+1ZgHZDH2eCqORedXUiI9nnU47mIbnNHufucJBOWgmquiwOAA8xsvpk9Fs3ym0fVnIvpwHlmthy4G/hsMknLpFrzlOxMMSE9M7PzgPHAh9JOSxrMrA/wTeCClJOSFTsRqodOINwlPmxmh7j72lRTlY6PA7e6+39GE13eZmYHu7tm2qtCWncE1Uw78d+vMbOdCLd7Zdbya3pVTcFhZicDVwFnuPs7CaUtaT2di8GECQkfNLOXCPWfs3LaYFzNdbEcmOXuW9z9RcJ07+9NKH1JquZcXAj8DMDdHwV2IUxIV0Q1T+uTViCoZtqJWcA/RL9/FLjfo5aQnOnxXJjZ4cAPCEEgr/XA0MO5cPd17j7C3ce4+xhCe8kZ7p6hBa4bpprvyK8JdwOY2QhCVdELSSYyIdWci2XASQBm9j5CIFiRaCqzYxbwiaj30ARgnbu/1t0bUqka8grTTpjZvwML3H0WcDPh9u55QsPIuWmkNW5VnotvAIOAn0ft5cvc/YzUEh2TKs9FIVR5Lu4BTjWzZ4BtwOXunru75irPxWXATWb2eULD8QU5LThiZncQCgAjojaRrwD9ANz9RkIbyenA88BG4B97/MycnisREamSRhaLiBScAoGISMEpEIiIFJwCgYhIwSkQiIgUnAKBiEjBKRCIiBScAoFIFaK57k+Jfv+qmX0n7TSJNIomnROpzleAfzezPYHDgdyN7Jbi0shikSqZ2UOEqT5OcPe30k6PSKOoakikCmZ2CPBuYLOCgOSNAoFID6LVndoIKz+tz/ECMFJQCgQi3TCzAcAvgcvcfQlhmcyvpJsqkcZSG4GISMHpjkBEpOAUCERECk6BQESk4BQIREQKToFARKTgFAhERApOgUBEpOD+PwPBHVsGaYvcAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"# Importing various packages\n",
"import numpy as np\n",
@@ -1216,27 +1202,18 @@
},
{
"cell_type": "code",
- "execution_count": 19,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHcVJREFUeJzt3X2QZXV95/H3ZwZBG4gPMGZlhulhdYghkESnRbJbMWzQCCQObplsgY0JKZJZZFmJDylJMJJFx1pjfKjdwsUxcX2YVlCSssaVLLXGpyQLLo0PGKAwIzDMgMrgU0iBjDDf/eOctu809/Y9597zfD6vqq7ue/rcc3/39O3zPb/f9/egiMDMzGw1a+ougJmZNZ+DhZmZjeVgYWZmYzlYmJnZWA4WZmY2loOFmZmN5WBhvSLpdEn7pnj+1ZL+pMgymbWBg4W1jqR7JD0i6V8kfVvSByUdVcLrXCDp7we3RcRFEfGWol/LrOkcLKytXhYRRwG/CDwP+KOay1MZSYdl2Zb3GGarcbCwVouIbwM3kAQNACQdIenPJd0r6Ttp09FThj1f0mWSvinpIUm3S/r36fafBa4Gfimtwfwg3f5BSW9Nf75D0m8MHOswSfslPT99fJqk/yvpB5K+Jun0Ue9D0nGS/ip9/t2SXjPwuz+VdJ2knZL+GbhgxLYjJL1H0v3p13skHZEe43RJ+yS9UdK3gf852Rm3vnKwsFaTtAE4C9g9sPm/AieSBJDnAOuBN484xDeBXwaeCvwXYKekZ0XEHcBFwI0RcVREPG3Icz8GnDfw+KXAgxHxZUnrgU8DbwWeAbwB+CtJ64a8hzXAp4CvpWU9A/gDSS8d2O0c4DrgacDCiG2XA6el7/sXgFOBNw0c41+lZZkFto04H2ZDOVhYW31S0kPAXuAB4AoASSK5EL42Ir4XEQ8BbwPOHXaQiPhERNwfEQcj4lrgn0gusll8FNgqaSZ9/EqSAAJwPnB9RFyfHvv/AIvA2UOO8wJgXURcGREHIuIu4P0rynxjRHwyPdYjI7bNA1dGxAMRsZ8k+L1q4BgHgSsi4tGBY5hl4nZLa6uXR8RnJP0KyUX7WOAHwDpgBrgliRsACFg77CCSfht4HbAp3XRUeqyxImK3pDuAl0n6FLCVJH8Cyd37b0l62cBTngR8bsihZoHjlpq6UmuBvxt4vHfI81ZuOw7YM/B4T7ptyf6I+NGo92O2GgcLa7WI+IKkDwJ/DrwceBB4BPi5iLhvtedKmiW5gz+D5C79cUlfJQkuAFmmZF5qiloD3B4RS81he4GPRMTvZzjGXuDuiNi8yj7DyrJy2/0kgee29PHGdNtqxzDLxM1Q1gXvAV4i6Rci4iBJAHi3pGcCSFq/ov1/yZEkF9D96X6/C5w88PvvABskHb7Ka18D/BrwapIazpKdJDWOl0paK+nJaZJ5w5Bj/D/goTT5/JR0/5MlvSDTu1/2MeBNktZJOpYkT7Mz5zHMhnKwsNZL2+c/zHIS+40kCe+b0p5CnwF+ZsjzbgfeCdxIEhhOAf5hYJfPktylf1vSgyNe+1vp8/8NcO3A9r0kCeg/JglGe4E/ZMj/XEQ8DvwGSWL6bpLa0V+QJN3zeCtJXuRW4OvAl9NtZlOTFz8yM7NxXLMwM7OxKgkWks6UdKek3ZIuG/L7C9LBSF9Nv36vinKZmVk2pfeGkrQWuAp4CbAPuFnSrrS9eNC1EXFJ2eUxM7P8qqhZnArsjoi7IuIASe+Rcyp4XTMzK0gV4yzWc+jgoX3AC4fs9wpJLwK+QTL69gmDkCRtI52m4Mgjj9zy3Oc+t4Timpl11y233PJgRDxh2plxmjIo71PAxyLiUUn/EfgQ8Ksrd4qIHcAOgLm5uVhcXKy2lGZmLSdpz/i9nqiKZqj7gOMHHm9It/1ERHw3Ih5NH/4FsKWCcpmZWUZVBIubgc2STkhHwp4L7BrcQdKzBh5uBe6ooFxmZpZR6c1QEfGYpEtI1hxYC3wgIm6TdCWwGBG7gNdI2go8BnwPuKDscpmZWXatHcHtnIWZWX6SbomIubzP8whuMzMbq1vBYmEBNm2CNWuS7wsL455hZmYZNKXr7PQWFmDbNnj44eTxnj3JY4D5+frKZWbWAd2pWVx++XKgWPLww8l2MzObSneCxb335ttuZmaZdSdYbNyYb7uZTcc5wl7pTrDYvh1mZg7dNjOTbDezYi3lCPfsgYjlHKEDRmd1J1jMz8OOHTA7C1LyfccOJ7fNyuAcYe94UJ6Z5bdmTVKjWEmCgwerL49l5kF5ZlYd5wh7x8GiLZxMtCZxjrB3HCzawMlEaxrnCHvHOYs22LQpCRArzc7CPfdUXRozazHnLLrMAw7NrGYOFm3gZKLVzTmz3nOwaAMnE61OzpkZDhbt4GSi1ckD8AwHi/aYn0+S2QcPJt+7Fii63szR5vfnnJnRpfUsrL26vhZJ29/fxo3De+M5Z9YrrllY/brezNH29+ecmeFgYU3Q9WaOtr8/58wMBwubRlHt8F3vGtyF99f1nJmN5WBhkymyO2XXmzm6/v6sFxwsbDJFtsN3vZlj1PuD9vaQst7x3FA2Ga9nMJ2VPaQgqW10KUhaI3luKKtWF9rh69T2HlLWOw4WNhm3w0+n7T2krHccLGwyXc8zlM01M2sZBwubnLtTTs41M2sZBwuzOrhmZi3juaHM6jI/7+BgreGaRVHaPKuomdkYrlkUoe2zipqZjeGaRRHcZ97MOs7BogjuM2/WHG4SLkUlwULSmZLulLRb0mWr7PcKSSEp91D0WrnPvFkzeL3w0pQeLCStBa4CzgJOAs6TdNKQ/Y4GLgW+VHaZCuc+82bN4Cbh0lRRszgV2B0Rd0XEAeAa4Jwh+70FeDvwowrKVCz3mTdrBjcJl6aKYLEe2DvweF+67SckPR84PiI+vdqBJG2TtChpcf/+/cWXdBoezWxWPzcJl6b2BLekNcC7gNeP2zcidkTEXETMrVu3rvzCmVm7uEm4NFUEi/uA4wceb0i3LTkaOBn4vKR7gNOAXa1LcptZ/dwkXJoqBuXdDGyWdAJJkDgXeOXSLyPih8CxS48lfR54Q0R4ZSMzy8/TqJSi9JpFRDwGXALcANwBfDwibpN0paStZb++mZlNr5KcRURcHxEnRsSzI2J7uu3NEbFryL6nu1ZhZo3V00F/tSe4zXqhpxeYzunxoD8HC7Oy9fgC0zk9HvTnYGFWth5fYDqnx4P+HCzMytbjC0zn9HjQn4OFWdl6fIHpnB4P+nOwMCtbjy8wndPjQX8OFn1TVK8c9+7JrscXmE7q6Txwioi6yzCRubm5WFz0cIxcVi7/Cskdbt4LV1HHMbPKSbolInJPp+Rg0SebNiXdNleanU3ukKo+jplVbtJg4WaoPimqV45795j1joNFnxTVK8e9e8x6x8GiT4rqlePePWa942DRJ0X1ynHvHrPecYLbrGoLC8lUH/femzTdbd/uQGuVmTTBXcXiR2a2ZGW346VJBcEBwxrNzVBmVfKkgtZSDhZmVXK3Y2spBwuzKrnbsbWUg4VZldzt2FrKwcKsSu52bC3l3lBmVZufd3Cw1nHNwszMxnKwMDOzsRwszMxsLAcLM6ueV1psHSe4zaxanvKklVyzMLNqecqTVnKwsGzcbGBF8ZQnreRgYeMtNRvs2QMRy80GDhg2CU950koOFjaemw2sSJ7ypJUcLGw8NxtYkTzlSSu5N5SNt3Fj0vQ0bLvZJDzlSeu4ZmHjudnArPccLGw8NxuY9V4lwULSmZLulLRb0mVDfn+RpK9L+qqkv5d0UhXlshzm5+Gee+DgweS7A4VZr5QeLCStBa4CzgJOAs4bEgw+GhGnRMQvAn8GvKvscpmZWXZV1CxOBXZHxF0RcQC4BjhncIeI+OeBh0cCUUG5zMwsoyp6Q60H9g483ge8cOVOkv4T8DrgcOBXhx1I0jZgG8BG98QxM6tMYxLcEXFVRDwbeCPwphH77IiIuYiYW7duXbUFNDPrsSqCxX3A8QOPN6TbRrkGeHmpJTIzs1yqCBY3A5slnSDpcOBcYNfgDpI2Dzz8deCfKiiXmZllVHrOIiIek3QJcAOwFvhARNwm6UpgMSJ2AZdIejHwY+D7wO+UXS4zM8uukuk+IuJ64PoV29488POlVZTDzMwm05gEt5mZFaCktWc8kaCZWVeUuGStaxZmZl1R4tozuYKFpEslvT/9+U+mfnUzMytOiWvP5K1ZPJvl0dhHT/3qZmZWnBKXrM0bLAJ4iqSTgeOmfnUzMytOiWvP5A0W7wQEvAr446lf3czMilPi2jN5g8Xbgd3A2yKiuwswl9T1zMysdCWtPZM3WLwSuBt4m6QPFVKCplnqerZnD0Qsdz1zwDDrD98wPkHeYPE0YDPwDOBbxRenAUrsemZmLeAbxqEUkX2dIUn/A7gW+ELkeWIJ5ubmYnFxsfgDr1mTfEBWkpJqnZl126ZNSYBYaXY2adZpOUm3RMRc3udlrllI+rfA/wJmgLMknZ33xVqhxK5nZq3Xh+aZYYECChmr0GZ5mqGeARwDrEu/ji2lRHUrseuZWav1oXlmYSFpRRgm6w1jmQG1zmAdEZm/gLMHft6a57lFf23ZsiVKs3NnxOxshJR837mzvNcya4vZ2YgkTBz6NTtbd8mKM+o9StmuAzt3RszMHPrcmZliriEFHZtkaYjc19zMOQtJvw6cB3yUdKxFRJxbTggbr7SchZkN14d83qj3CKO3Dyoz31HQsUvPWZA0Oz2Sfn864HYZsz7pQz5v1HuZnc32/BLnZir12BlkDhYR8SHgL4EXAmcAry2rUGbWQH3I5037HssMqDUH67zjLH6PZNnTK0gG55lZX5Q4lURjTPseywyoNQfrvIsffQd4MnAQ+Onii2NmjTY/363gMMw073HpeZdfnjQPbdyYXMyLOGdlHjuDvIPyTgIOAK8BPhsRnyyrYOM4wW1mlt+kCe7MNYsVg/BuIJmu3MzMeiBPM9Q6lgOEcLAwM+uNvL2hng78cvpzxr5kZmbWdl5W1czMxvKyqmY2Xh8mELRVeVlVM1tdHyYQtLHyTFEu4OURcVlEvDG6vKyqmS3zgmBGvgR3AC+QdJ6kszu7noVZGdrcjDPNnERtft92iLwjuD8DHM6h3WjNbDVLzThLd+dLzTjQjtHQGzcOn+10tTmJFhbg0kvhu99d3ta2922HGFuzkHTj0s9pl9m/Bm6NiA+XWTCzzmh7M07eOYmWguNgoFjSpvdth8jSDHUEgKR3AUTEQ8B7yyyUWafUPLX01PJOrjcsOA5qy/uuQ4Ob7bIEC0n6aeD8NMkN8JQSy2TWLV1YB2J+Pllg5+DB5PtqzUjjgkGb3neVGt7rLEuw+CPg70hWyHu3pIszPs/MoPappQuV5c53tWDQ1vddhYY3V4696EfE/46IEyPiD4BrgecAF5ZeMrOu6Mo6EFnvfIcFR4Bjjmnn+65Kw5src01R3iSeotysYnnWgF5YqG3dhdYqc/3uAVWswT0xSWdKulPSbkmXDfn96yTdLulWSX8ryZMUmjVNnjvfPDkOSzS8ubL0YCFpLXAVcBZwEnBeuojSoK8AcxHx88B1wJ+VXS4zy6kLifoma3hzZRU1i1OB3RFxV0QcAK4BzhncISI+FxFLmZ2bgA0VlMtsMg3u3liqht/5dkKDa2RVBIv1LE9rDrAv3TbKhcDfDPuFpG2SFiUt7t+/v8AiWq9Mc7FvePfGUjX8ztfK1agusJLOB+aAdwz7fUTsiIi5iJhbt25dtYXru67cTU97sW9498bSNfjO18pVRbC4Dzh+4PGGdNshJL0YuBzYGhGPVlAuy6pLd9PTXuwb3r3RrCxVBIubgc2STpB0OHAusGtwB0nPA95HEigeqKBMlkeX7qanvdg7yWs9VXqwiIjHgEuAG4A7gI9HxG2SrpS0Nd3tHcBRwCckfVXSrhGHszp06W562ou9k7zWU3mnKJ9IRFwPXL9i25sHfn5xFeWwCU0yRXVTbd9+6HThkO9iv9RG7wFn1jONSnBbQ3XpbrqIHj1O8loPVVKzsJbr2t30/Hx7y25WEwcLy8YXWLNeczOUmVkTNHwsk4NFFRr+IbCe8eexeVowlsnBomwt+BBYj/jz2EzTjmWq4AbA61mUraI56s0y8eexmdasSYL3SlLS6241SzcAK7uDj+jl1+j1LHqtSwParP38eWymaQaLVjTDgoNF2Tw9hDWJP49P1IQczjRjmSq6AXCwKFuXBrRZ+/nzeKim5HCmGSxa1Q1ARLTya8uWLdEaO3dGzM5GSMn3nTvrLpH1WdmfxzZ93mdnI5IwcejX7GzdJctu586ImZlDyz8zM/K8A4sxwTXXCW4zK07OZGvtpkksN8nCQuYZFiZNcDtYmNl0Bi9Ua9bA448/cZ+m9rbqYe8w94Yys+qtbPMfFiigub2tnMPJzMGiLk3ogWE2rWHdNodpam8rryuemScSrMPKdt2lHhjgD6m1y7AmnJWafqfuSTIzcc2iDl1aptTap8ha7dq1o3/nO/VOcc2iDh5Fa3UpulY7KkcB7epNZGO5ZlEHj6K1IkxSQyi6Vjs7m2+7tZaDRR3cA8OmNWzk8ateBRdfvPrziq7V+rPcGw4WdXAPDJvWsBpCBFx99eo1jKJrtf4sF6+hPSU9KM+sjUaNPIbVB5S1bYR131Tw9/GgPLM+Wa0msFqTkmsCzdbgnpIOFmZttH17crEfZlyT0vx8UvM4eDD57kDRHA3uKelgYTapOtuW5+fhooueGDCcXG63BveUdLDog4YmzFqtCesgvPe98JGPuEmpSHX/rzS5d9kk85o34atV61nUKedc95ZRF9ZBsEM15X+l5PVA8HoWNlQPp2CuRFfWQbBlPflfcW8oG67BCbPaq/zTaHDbcmHa/PeZRJP/VxrAwaLrmnpRa0Kb/zSa3LZchLb/fSbR1P+VhnCw6LqmXtQa3J88k66PV2j732cSTf1faQgHi65r6kWtC1X+Lo9X6MLfJ68m/a8U3QQ4cLyfh1MmOYQT3FaPniQTW8t/n/oUPeXHiuPNAYsRI0Z0juaahdXDVf5m898nm7w1gCz7F90EmHXp23Em6W+b9ws4E7gT2A1cNuT3LwK+DDwG/GaWY3qcRQeU3J/cpuS/z+ryjsvIur80fAyPNFk5VxxvC0Q0cZyFpLXAN4CXAPuAm4HzIuL2gX02AT8FvAHYFRHXjTuum6HMrFZ5m+qy7l90E+CK4zW5GepUYHdE3BURB4BrgHMGd4iIeyLiVsCjmcysOtMkkvN2Asi6vegmwGHHm0AVwWI9sHfg8b50W26StklalLS4f//+QgpnZj017ViSvOMysm4vulfWiuP9GA5McphWJbgjYkdEzEXE3Lp16+oujpm12bSJ5Lw1gDz7F90te+B4t8LXJzlEFcHiPuD4gccb0m1mZvWZdixJ3hpAk8ZxTOCwCl7jZmCzpBNIgsS5wCsreF0zs9E2bhyeSM4zvcf8fL6Lfd79G6T0mkVEPAZcAtwA3AF8PCJuk3SlpK0Akl4gaR/wW8D7JN1WdrnMrOc8liSXSnIWEXF9RJwYEc+OiO3ptjdHxK7055sjYkNEHBkRx0TEz1VRLhuhb7ONWj+1vFmoaq1KcFsJVgaGiy/u32yj1l9tnN+rpps5zw3VZ8PmoJGGL+rjOYHM6lfAvFGTLn7kYNFno0aKDuMV4MzqV8Dobq+UZ/nlmW7aC8CYTaeI5qMap453sOizUQFAK6aNcQ8Rs+kUtfJgjav5OVj02aiugxdd5B4iZkuKqBEUNe14jd19qxiUZ021FAAuvzypxm7cmHzoHBjMEisTyks1Asj3f1JU81GN/7NOcJuZjVLUdOENWnnQCW4zs6IVVSPowGhxBwszs1GKSih3YLS4g4WZ2ShF1gjaOFp8gIOFmdkoHagRFMXBwqwJPHljc7W8RlAUd501q1tR3TPNSuSahdlqqrjjL2rAllmJXLMwG6WqO/4a5/sxy8o1C7NRqrrjr3G+H7OsHCysOZqW5K3qjr8DA7as+xwsrBmKmpWzSFXd8bt7prWA54ayZmjQ3Dk/UcCqZGZN47mhrN2amOT1Hb/ZT7g3lDXDxo3DaxZ1J3nn5x0czHDNwlZTZcLZSV6rQ97PeJ79m9ZhY1oR0cqvLVu2hJVo586ImZmIJN2cfM3MJNvLfM3Z2Qgp+V7ma5nl/Yzn2X+1fWv+nAOLMcE11wluG66JCWezIuX9jOfZf9S+xxwDjzxSa6eJSRPcDhY23Jo1yf3QSlIyoZpZ2+X9jOfZf9S+o1R4E+beUFasNo0q7lrbsFUj72c8z/a8/yctmNrFwcKGa0vCuejBfA48/ZH3M55n/1H7HnPM8GM38SZspUkSHU34coK7Am1IOM/OHppEXPqanc1/rDqS+lavvJ/xPPsP27cBnzGc4LZeKjK34qS+VWFhIZmM8t57kxrF9u2jk9t59s3IOQvrpyJzK00cRZ6Xm9GaL+vKew2bL83BwtqtyNxKm5L6wzTs4mJTatiiWA4W1m5Fzt/UlqT+KA27uGTimtBoDavpOmdhNqiENuLKtG1sjGf1XV1JOTTnLMyKkLU9uYna1ozWxppQlRpW060kWEg6U9KdknZLumzI74+QdG36+y9J2lRFucw6pWEXl7Ea1szSOA2bIr/0YCFpLXAVcBZwEnCepJNW7HYh8P2IeA7wbuDtZZfLrHMadnEZq201oTo0qKZbRc3iVGB3RNwVEQeAa4BzVuxzDvCh9OfrgDMkqYKymXVLgy4uY7WtJtRzVSx+tB7YO/B4H/DCUftExGOSfggcAzw4uJOkbcC29OGjkv6xlBK3z7GsOFc95nOxrPHn4lh4xnGw/klw+I/hwP0PP3zfg+ef/z3OP7+El2r2uajQz0zypFatlBcRO4AdAJIWJ8nod5HPxTKfi2U+F8t8LpZJmqgbaRXNUPcBxw883pBuG7qPpMOApwLfraBsZmaWQRXB4mZgs6QTJB0OnAvsWrHPLuB30p9/E/hstHUAiJlZB5XeDJXmIC4BbgDWAh+IiNskXUky++Eu4C+Bj0jaDXyPJKCMs6O0QrePz8Uyn4tlPhfLfC6WTXQuWjuC28zMquMR3GZmNpaDhZmZjdX4YOGpQpZlOBevk3S7pFsl/a2k2TrKWYVx52Jgv1dICkmd7DaZ5TxI+g/p5+I2SR+tuoxVyfD/sVHS5yR9Jf0fObuOclZB0gckPTBqLJoS/y09V7dKev7Yg06yvF5VXyQJ8W8C/xo4HPgacNKKfS4Grk5/Phe4tu5y13gu/h0wk/786j6fi3S/o4EvAjcBc3WXu6bPxGbgK8DT08fPrLvcNZ6LHcCr059PAu6pu9wlno8XAc8H/nHE788G/gYQcBrwpXHHbHrNwlOFLBt7LiLicxGxNI3nTSRjWrooy+cC4C0k84z9qMrCVSjLefh94KqI+D5ARDxQcRmrkuVcBPBT6c9PBe6vsHyViogvkvQsHeUc4MORuAl4mqRnrXbMpgeLYVOFrB+1T0Q8BixNFdI1Wc7FoAtJ7hy6aOy5SKvVx0fEp6ssWMWyfCZOBE6U9A+SbpJ0ZmWlq1aWc/GnwPmS9gHXA/+5mqI1Ut7rSbum+7BsJJ0PzAG/UndZ6iBpDfAu4IKai9IEh5E0RZ1OUtP8oqRTIuIHtZaqHucBH4yId0r6JZKxXSdHRANXhmqeptcsPFXIsiznAkkvBi4HtkbEoxWVrWrjzsXRwMnA5yXdQ9Imu6uDSe4sn4l9wK6I+HFE3A18gyR4dE2Wc3Eh8HGAiLgReDLJBIN9lOl6MqjpwcJThSwbey4kPQ94H0mg6GrbNIw5FxHxw4g4NiI2RcQmkvzN1ojo2jq8Wf4/PklSq0DSsSTNUndVWciKZDkX9wJnAEj6WZJgsb/SUjbHLuC3015RpwE/jIhvrfaERjdDRXlThbROxnPxDuAo4BNpjv/eiNhaW6FLkvFcdF7G83AD8GuSbgceB/4wIjpX8854Ll4PvF/Sa0mS3Rd09MYSSR8juUk4Ns3RXAE8CSAiribJ2ZwN7AYeBn537DE7eq7MzKxATW+GMjOzBnCwMDOzsRwszMxsLAcLMzMby8HCzMzGcrAwM7OxHCzMzGwsBwuzgqRrJbwk/fmtkv573WUyK0qjR3CbtcwVwJWSngk8D+jc6HnrL4/gNiuQpC+QTLlyekQ8VHd5zIriZiizgkg6BXgWcMCBwrrGwcKsAOkqYwskK5D9S4cXGbKecrAwm5KkGeCvgddHxB0ky7leUW+pzIrlnIWZmY3lmoWZmY3lYGFmZmM5WJiZ2VgOFmZmNpaDhZmZjeVgYWZmYzlYmJnZWP8fxnUyTh8MM1IAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.linear_model import LinearRegression\n",
"\n",
"x = np.random.rand(100,1)\n",
- "y = 5*x+np.random.randn(100,1)\n",
+ "y = 5*x+0.01*np.random.randn(100,1)\n",
"linreg = LinearRegression()\n",
"linreg.fit(x,y)\n",
"ypredict = linreg.predict(x)\n",
@@ -1272,34 +1249,11 @@
},
{
"cell_type": "code",
- "execution_count": 21,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "The intercept alpha: \n",
- " [2.00137415]\n",
- "Coefficient beta : \n",
- " [[4.99698001]]\n",
- "Mean squared error: 0.00\n",
- "Variance score: 1.00\n",
- "Mean squared log error: 0.00\n",
- "Mean absolute error: 0.01\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHVRJREFUeJzt3XuUXeV53/HvTyPJYoRsmpHAXKQZx41xiG2KMwW8nAtBdkodipvGywtHEoI4ltHUBFpacKK1apKGum5WHCtJkRAYENIxBrvQKK4vNTaulx2DPRRMuJga6EgIMEgQMCo3XZ7+8Z7jOXN0LnsuZ5/L/n3WmjVzzr69szXaz3neqyICMzMrrnmdLoCZmXWWA4GZWcE5EJiZFZwDgZlZwTkQmJkVnAOBmVnBORDYnJH0q5Ie7nQ5+oGkFZL2SRpow7nfLenH5fP/S0lfkbR2rq9jvUMeR2DTJWkC+P2IuL3TZWlE0hXABuBV4ADwIHBpRHyvk+XqBpK+AeyIiI11tp1P+rf9ldwLZh3jjMB6nqT5DTbdHBFHAkuBO4Av5Hz9bjUMPNDpQlj3cCCwOSPpDEm7q15PSPp3ku6T9IKkmyUtqtp+tqR7JT0v6e8kvaNq28clPSrpRUkPSvrtqm3nS/qupL+Q9CxwRbNyRcQBoAQcL2lZxuu/U9I95et/oVz2P63+PSVdLuknwPUZzne5pCfK53tY0sry+6dKGpf0U0lPS/p0+f0RSVEJMpKOk7RD0nOSHpH0kapzXyHpFkk3ls//gKTRBv9GjwI/D/xtuWrodZK+Jen3Jf0isBl4V3nb883uq/UPBwJrtw8CZwFvAt4BnA8g6RTgOuCjwBBwNbBD0uvKxz0K/CrwBuCPge2Sjq0672nAY8AxwJXNCiBpIXAe8CzwD62uX97/NuAG4OeAm4DfrjntG8vbhoF1Lc53IvAx4J9GxBLgnwET5fNsBDZGxOuBNwO3NPg1Pg/sBo4DPgD8J0lnVm0/p7zPUcAO4K/rnSQi3gzsAv5FRBwZEa9WbXsIuBD4XnnbUQ3KYn3GgcDa7S8j4smIeA74W+CflN9fB1wdEXdFxMGI2Eqqzz8dICK+UD7uUETcDPwYOLXqvE9GxF9FxIGIeLnBtT9Y/lT7MvAR4APl7KDV9U8H5pfLvj8ibgW+X3PuQ8AnIuLV8vWbne8g8DrgJEkLImIiIh4tn2c/8I8lLY2IfRFxZ+0vIWk58G7g8oh4JSLuBa4lBbeK70TElyPiILANOLnBPTE7jAOBtdtPqn5+CTiy/PMwcGm5GuX58gN7OekTL5LOq6pmeR54G6muv+LxDNe+pfyp9hjgfuCXq7Y1u/5xwBMxtSdF7fX2RMQrWc4XEY8Al5CqsJ6R9HlJx5WP+zDwFuBHkn4g6ew6v8dxwHMR8WLVezuB46te197nRT3YdmEd4kBgnfI4cGVEHFX1NRgRN0kaBq4hVacMlR/m9wOqOj5zd7eI2Ev6xH5FVfVSw+sDT5HaE6qvt7z2tFl/n3IZPlfuiTNcPvZT5fd/HBEfAo4uv/dFSYtrzv0k8HOSllS9twJ4Ius9mAZ3IywgBwKbqQWSFlV9TffT5zXAhZJOU7JY0m+VH3aLSQ+kPQCSLiBlBDMWEQ8DXwMuy3D975Gqcz4mab6k9zO1Wmpav4+kEyWdWW7/eIVUVXWo/LutlrQsIg4BlcbZQzVlfxz4O+CT5Xv9DlImsX0296SBp4ETyu0kVhAOBDZTXyY90CpfV0zn4IgYJ9Xb/zWpAfcRyg3JEfEg8OekB/LTwNuB785Bmf+M1LB7dIvrvwb8K9LD9nlgNfAlUp3/tH8fUvvAfwb2kqpwjgb+sLztLOABSftIDcfnNmjz+BAwQsoObiO1T7RjHMc3SV1LfyJpbxvOb13IA8rMMpB0F7A5Iq7vdFnM5pozArM6JP26pDeWq4bWkrq+frXT5TJrh7YHgnL96L1VXz+VdEm7r2s2SycCPyRVDV1K6nr6VGeLZNYeuVYNKU2g9QRwWkTszO3CZmbWUN5VQyuBRx0EzMy6R94DTs4lDdc/jKR1pL7eLF68+Jff+ta35lkuM7Pe8vd/D6+99rOXE8DeCDXcv4ncqobK/ZKfBH4pIp5utu/o6GiMj4/nUi4zs56kqc/8UWB8hoEgz6qhfw7871ZBwMzMmnjsscOCwGzlGQg+RINqITMza2H/fjjtNHjzmyffW7So8f7TkEsgKM+d8l7g1jyuZ2bWs0olGBmBefPS91IJPvMZWLgQvl+eBPeGGyACrr0WhodBYj+81uSsTXXlyGK3EZhZIZVKsG4dvPRS/e1nnw1/8zcpSNSQdHdE1F2QqBVPU2tm1g1KJVi7Fg4erL/9ySfh2GPrb5slTzFhZtYJlSogCQYGYPXqxkFAalsQAGcEZmb5GxuDzZtTPT/AoUPN91+xoq3FcSAwM8tTqQSbNmXff3AQrmy6LPesuWrIzCwvpRKcd17r/SoGBmDLFli1qn1lwoHAzCwflR5BraqBKgYHYevWtgcBcCAwM8vHhg2Nu4XWGhrKJROocCAwM2u3fftgZ4ZJl4eHYft22Ls3tyAADgRmZnOrdmSwBEuWND9mwYIUACYmcg0AFQ4EZmZzpdIOsHNn6hpamwUcccThxwwNwfXXdyQAVDgQmJnNRnUGsHZt/XaAhQtTYLjmmp/NDdSpaqB6PI7AzGymaucGajQyeP/+9H3Vqo4/9OtxRmBmNlNZewK1eWTwbDkQmJllVT0/0Pz52XoC5TAyeLYcCMzMsqhuCIbG1UCQRgRX2gFyHA8wU24jMDPLIms10OBgTzz8qzkjMDNrpLpHUNYBYT0WBMAZgZlZfa1WC6s1PJwGhPUgZwRmZvVMZ26gHmgQbsYZgZlZLan1PgMDqcF4eDgFgR6rDqrmjMDMrLotoFUQGB5Oo4QPHEjfOzQ/0FxyRmBmxdasLWBwcOr7PV4F1IgzAjMrttWr6weBSg+g6rmBerBHUBYOBGZWTDfd1LwaaNeu9NCfmEirivVBFVAjDgRm1t9KJVi6ND30q79+93ebH9fl8wPNJQcCM+tfY2Op6ufZZxvvs21bqvuv1qdtAY04EJhZfyqVYPPm5vsMD6dAUZC2gEYUEZ0uw2FGR0djfHy808Uws142MtJ6Wggp1f/3AUl3R8ToTI51RmBm/SnL3EAFagdoxoHAzPpD9VoBWUYGL1hQqHaAZhwIzKx3VT/816zJlgVAVywY301yCQSSjpL0RUk/kvSQpHflcV0z62O1C8XUa+8cGjp8sfiIrlgwvpvkNcXERuCrEfEBSQuBwVYHmJk1lWV20OeeSw99a6rtGYGkNwC/BnwWICJei4jn231dM+sj1ZPCjYzAH/2RG4PnUB5VQ28C9gDXS7pH0rWSFtfuJGmdpHFJ43v27MmhWGbWE6qrgCLS909+svVxBRsUNht5BIL5wDuBTRFxCvD/gI/X7hQRWyJiNCJGly1blkOxzKwnNKsCOuKIqa8rvYUKOChsNvIIBLuB3RFxV/n1F0mBwcysvrExmD8/PdgbVQFJcM01UxuDt23rmzUC8tT2xuKI+ImkxyWdGBEPAyuBB9t9XTPrUWNjsGlT6/1WrEgPez/wZy2vXkMXAaVyj6HHgAtyuq6Z9YJSKVUBZR0H4Pr/OZVLIIiIe4EZzYFhZn2u2QphtaSUCfT4GsHdxktVmllnZRkPAGmx+AMH2l+eAvIUE2aWn9rxAKVS9uqgdevaWbJCc0ZgZvmorQLauTOtBZDF+vVw1VXtK1vBORCYWT6yVgFVGxz0eIAcuGrIzPKxa1f996U0GdzwcHo9MJC+e1BYbpwRmFn7NVsfwOMBOs4ZgZm1z8aNzYOAxwN0BQcCM2sPCS65ZOp7lSqggi4S361cNWRms1MZFbxrV6rmqdcd9OWXYdGi9LMf/F3HgcDMZm5sDDZvnlwdrF4QqLdymHUVBwIzm5lSaWoQqOUA0DPcRmBmM7NhQ+OHfbMGYus6zgjMbPpaPei9RGRPcUZgZtm9+GLrICC5S2iPcUZgZtnUCwCDg1OnjZDgwgvdM6jHOCMws0mV2UGlyaUiK1/Vtm5N7QNbthy+VKQnh+s5zgjMLKmdHfTgwfr7VTcQe2qIvuCMwMySVrODDg+7S2ifciAws6TR7KBZt1vPciAwK6LalcLOOKP1p313Ce1bbiMwK5p6K4W1Wi7Ss4T2NWcEZkXTrC1g2zYvEFNADgRm/a62GqjRp38prSE8MZGqiQ4cSN8nJhwE+pyrhsz6Wb1qoEbcBlBYzgjM+lnWBePdBlBoDgRm/axZBuCVwqzMVUNm/Wj3bli+vPH24eFU92+GMwKz/lE9T1CzIOBqIKvhQGDWq6p7Ay1dmnr81FYFXXaZF4y3lhRdOHfI6OhojI+Pd7oYZt2rtjdQI64CKgxJd0fE6EyOzaWNQNIE8CJwEDgw08KaGSkIrF3beHbQap4fyDLIs7H4NyJib47XM+s/lUwgSxAAjw2wTNxryKyXZM0EwI3ClllejcUB/E9Jd0taV28HSeskjUsa37NnT07FMushUvMgsGABDA25UdimLa+M4Fci4glJRwNfl/SjiPh29Q4RsQXYAqmxOKdymXW/VovFQ5og7vrr/eC3GcklI4iIJ8rfnwFuA07N47pmPa3eWsGQqnxqX2/d6iBgM9b2QCBpsaQllZ+B3wTub/d1zXrWK6/UDwAR9ReMdxWQzVIeVUPHALcp/WHPBz4XEV/N4bpmvadRAKjmBeNtjrU9I4iIxyLi5PLXL0WEuzGY1apXDXTZZV4s3nLhKSbM8lQqpekgKg/+o45qnAV86lP5l88KyeMIzPJSKsEFF8D+/ZPvvfDC1H2cAVgHOCMwy0NlWojqIFDtjW90ELCOcSAwa5fqaaHXrGk+GOzpp3MrllktVw2ZtcN73gPf+Mbk61af9j0nkHWQMwKzuTY2NjUItLJwoecEso5yIDCbreoqoPnzYdOm7McODcF113lcgHWUq4bMZqN2gZjpzAzqEcHWJZwRmM3Ghg2tVwmr5WkhrMs4IzCbjdo1gptZuRJuv719ZTGbIWcEZtNRvWD88uXZjhkYgPXrHQSsazkjMMuqtj1g9+7m+7sdwHqEMwKzZqozgLVrs7cHDAw4CFjPcEZg1oh7BFlBOCMwa+Tii7NlAENDXijGepozArN6SiV49tnW+w0OwsaNfvBbT3MgMKt18slw332Ntw8MwKFDaX6gK690ELCe56ohK67qhuCRkfRaah4EIC0Uf+gQTEw4CFhfcEZgxVS7SMzOnbB6devjhob88Le+44zAiuniixsvEgOwfXuq/69WaQ8w6zPOCKyYmjUEV68dsGED7Nrl9gDraw4EVjz1FouvZ9UqP/itEBwIrDiyBIChofaXw6zLtGwjkPR1SSfnURizOVcqwXHH1Q8CCxce/tptAFZAWRqLLwc+I+l6Sce2u0Bmc6ZUSj2Bnnpq6vvbt6d2gOuumzoi2CuFWUEpWi2qXdlR+h3gPwC3Av8lIl5uV6FGR0djfHy8Xae3flQqTW3YbbZOwPBwGgNg1kck3R0RozM5NlP3UUkCHgY2ARcBP5a0ZiYXNJtzlTEBO3emT/qtFovZtSufcpn1iCxtBN8FngD+AjgeOB84AzhV0pZ2Fs4sk1ZjAmqtWNG+spj1oCy9htYBD8bhdUgXSXqoDWUym55mYwIGB6fOIDo4mMYDmNnPtMwIIuKBOkGg4rfmuDxm2Umtu4Ru2eIpos1amNUUExHx2FwVxKyhsTGYPz89zOfPT6+zjglYtSo1DHuSOLOGcptrSNKApHskfSmva1ofGBuDTZsmVwc7eDC9rrZ9u8cEmM1CniOLLwYeAl6f4zWt1119dfPtnhfIbNZyyQgknUBqT7g2j+tZHzl0qPG26iDgKiCzGcuraugzwGVAw//VktZJGpc0vmfPnpyKZV0t6+RwZjYrbQ8Eks4GnomIu5vtFxFbImI0IkaXLVvW7mJZt6luEM7SG2jx4nzKZVYAeWQE7wbOkTQBfB44U9L2HK5rvaBUgiVLpjYI1xoYOPx1q7YDM8us7YEgIv4wIk6IiBHgXOCbEZFhTUDre6USrFsH+/bV3z4wkNoBtm6dOhZg61a3AZjNIa9HYJ2zYcPUUb+1KhmCF4gxa6tc1yyOiG9FxNl5XtO6SKkEIyMwbx4sWtR6crjaKiEzawtnBJaPSjVQJQN49dXWx6xb194ymRmQc0ZgBVOdAaxdW78aqF7vIAnWr4errmp7Ec3MgcDaoVSCpUvT6mCVNQIa9QiKmNoQvH17GhTmIGCWG1cN2dyqLBKTdX0ArxZm1nEOBDZ3SiVYs2bq1A/NeG0As67gqiGbG5XG4FZBYGDAawOYdRlnBDY3Lr+8+ZgASBmAH/5mXceBwGYvy+Rw8+Y5CJh1KVcN2fRUdwnNMjkcpEVibrzRQcCsSzkQWDb1uoTWGhw8/L2hIbjuOgcBsy7mQGCtVRqCn322/vbh4RQYaheK374d9u51EDDrcoqsXf1yNDo6GuPj450uhlWMjDSfF0hqvpKYmbWdpLsjYnQmxzojsMNVtwMMDLSeHG7FilyKZWbt4V5DNtXYGGzePNkG0Cpj9KAws57njMAmlUpTg0C1er2DhobcJdSsDzgQ2KTVqxtnAPUmh3NDsFlfcNWQpQniFi5svo8nhzPrW84Iiqa6IXhkJH3CbxUEJLcDmPUxB4IiqYwHqAwIq+0NNDp6+KAwCS680FVAZn3MgaBImi0WHwE/+MHhg8K2bfMiMWZ9zm0ERdJoPEB1j6BVq/zp36xgnBH0m9o2gFKp9eRwHhBmVmjOCPpJpQ2gUv2zc2fqEtqMB4SZFZ4zgn7SrA0AUjvA9u1T2wA8IMys8BwIelllauhK1U+zNoDKQLFVq9J4gEOH0ncHAbPCc9VQryqV4Pd+D157rfW+bgMwsyacEfSqDRuyBQG3AZhZCw4EvaK6N9DSpa2nhq5wG4CZteCqoV5Q2xuo0UphtYaHHQTMrCVnBL2gVW+gelwlZGYZtT0QSFok6fuSfijpAUl/3O5r9p1du7LtVxk05m6hZjYNeVQNvQqcGRH7JC0AviPpKxFxZw7X7n3NRgRX8zTRZjZDbc8IItlXfrmg/NVi/UPjjjuyBwFXA5nZLOTSRiBpQNK9wDPA1yPirjr7rJM0Lml8z549eRSre9RbI+DMM6fuUz0ieGgofXl0sJnNAUWrxcnn8mLSUcBtwEURcX+j/UZHR2N8fDy3cnVUbY+gWhMT6WFvZtaEpLsjYnQmx+baaygingfuAM7K87pdqZIFrF7dfI0ABwEza7M8eg0tK2cCSDoCeC/wo3Zft6tVrxTWSNb2ATOzWcqj19CxwFZJA6TAc0tEfCmH63afUimNCcgyKtjzA5lZTtoeCCLiPuCUdl+na42NpcbcgwezH+NeQGaWI48sbqexMdi0aXpBwL2AzCxnnmuonbZsyb7v4KADgJl1hDOCdsqaCTgLMLMOciCYC2NjMDAwuVLY4sXZev0MDqaBYl4pzMw6yIFgtirtAIcOTb6XZaZQZwFm1iUcCGajVEpBoJFjjoH161O2AOn7+vVpoJizADPrErlOMZFV108xUSrBxRe3XiBGmpopmJm1yWymmHCvoelqNTdQNQ8KM7Me4Kqh6cq6Wti8eR4UZmY9wYGgkdqpoUsl+IM/yDY9xOLFcOONbgMws57gqqF6SiW44ALYvz+93rkzzRLaigeFmVkPckZQq1SCNWsmg0A9g4OHvzc05CBgZj3JgaBapSG4WU+qiPTAr6wWNjycBoXt3esgYGY9yVVDFaUSrF2bbVqIVav80DezvuGMACYzgVZBYGgon/KYmeWouIGguldQs+UiKxYuhI0bcymamVmeilk1NJ1BYZAygY0bXR1kZn2peIFgOm0BAwOwdasDgJn1tWJUDY2Nwfz5qZfP6tXZgsDgoIOAmRVCfweCUgmOPDL7cpGVNQU8RbSZFUj/Vg1Ntx3Ao4LNrKD6KyOo7gm0dm32IDAw4CBgZoXVPxlBbQaQdb1gZwJmVnC9nxFUsoAsYwFquS3AzKzHM4LptgNAqjb66EfhqqvaVy4zsx7SWxlBqQRLl6aePRKcd17rIFDdE2j79lRl5CBgZvYzvZMRjI0dvlB8q/WAXf9vZtZSb2QEpdLhQaAV1/+bmWXSGxnBhg3Z93UWYGY2Lb2REeza1Xy7RwSbmc1Yb2QEK1Y0XzTecwKZmc1Y2zMCScsl3SHpQUkPSLp42ie58so0aVw9K1c6CJiZzUIeVUMHgEsj4iTgdOBfSzppWmdYtQpuuCFNIFchwfr1cPvtc1hUM7PiaXvVUEQ8BTxV/vlFSQ8BxwMPTutEXifYzKwtcm0sljQCnALcVWfbOknjksb37NmTZ7HMzAott0Ag6UjgvwGXRMRPa7dHxJaIGI2I0WXLluVVLDOzwsslEEhaQAoCpYi4NY9rmplZNnn0GhLwWeChiPh0u69nZmbTk0dG8G5gDXCmpHvLX+/L4bpmZpZBHr2GvgOo3dcxM7OZ6Y0pJszMrG0cCMzMCs6BwMys4BwIzMwKzoHAzKzgHAjMzArOgcDMrOAcCMzMCs6BwMys4BwIzMwKzoHAzKzgHAjMzArOgcDMrOAcCMzMCk4R0ekyHEbSi8DDnS5Hl1gK7O10IbqA78Mk34tJvheTToyIJTM5sO3rEczQwxEx2ulCdANJ474Xvg/VfC8m+V5MkjQ+02NdNWRmVnAOBGZmBdetgWBLpwvQRXwvEt+HSb4Xk3wvJs34XnRlY7GZmeWnWzMCMzPLiQOBmVnBdSwQSDpL0sOSHpH08TrbXyfp5vL2uySN5F/KfGS4F/9W0oOS7pP0DUnDnShnHlrdi6r9fkdSSOrbroNZ7oWkD5b/Nh6Q9Lm8y5iXDP9HVki6Q9I95f8n7+tEOfMg6TpJz0i6v8F2SfrL8r26T9I7W540InL/AgaAR4GfBxYCPwROqtlnDNhc/vlc4OZOlLVL7sVvAIPln9cX+V6U91sCfBu4ExjtdLk7+HfxC8A9wD8qvz660+Xu4L3YAqwv/3wSMNHpcrfxfvwa8E7g/gbb3wd8BRBwOnBXq3N2KiM4FXgkIh6LiNeAzwPvr9nn/cDW8s9fBFZKUo5lzEvLexERd0TES+WXdwIn5FzGvGT5uwD4j8CngFfyLFzOstyLjwD/NSL+ASAinsm5jHnJci8CeH355zcAT+ZYvlxFxLeB55rs8n7gxkjuBI6SdGyzc3YqEBwPPF71enf5vbr7RMQB4AVgKJfS5SvLvaj2YVK070ct70U5zV0eEf8jz4J1QJa/i7cAb5H0XUl3Sjort9LlK8u9uAJYLWk38GXgonyK1pWm+0zp2ikmrA5Jq4FR4Nc7XZZOkDQP+DRwfoeL0i3mk6qHziBlid+W9PaIeL6jpeqMDwE3RMSfS3oXsE3S2yLiUKcL1gs6lRE8ASyven1C+b26+0iaT0r3ns2ldPnKci+Q9B5gA3BORLyaU9ny1upeLAHeBnxL0gSp/nNHnzYYZ/m72A3siIj9EfF/gf9DCgz9Jsu9+DBwC0BEfA9YRJqQrogyPVOqdSoQ/AD4BUlvkrSQ1Bi8o2afHcDa8s8fAL4Z5ZaQPtPyXkg6BbiaFAT6tR4YWtyLiHghIpZGxEhEjJDaS86JiBlPttXFsvwf+e+kbABJS0lVRY/lWcicZLkXu4CVAJJ+kRQI9uRayu6xAziv3HvodOCFiHiq2QEdqRqKiAOSPgZ8jdQj4LqIeEDSnwDjEbED+CwpvXuE1DBybifK2m4Z78WfAUcCXyi3l++KiHM6Vug2yXgvCiHjvfga8JuSHgQOAv8+Ivoua854Ly4FrpH0b0gNx+f36QdHJN1E+gCwtNwm8glgAUBEbCa1kbwPeAR4Cbig5Tn79F6ZmVlGHllsZlZwDgRmZgXnQGBmVnAOBGZmBedAYGZWcA4EZmYF50BgZlZwDgRmGZTnun9v+ec/lfRXnS6T2VzxpHNm2XwC+BNJRwOnAH03stuKyyOLzTKS9L9IU32cEREvdro8ZnPFVUNmGUh6O3As8JqDgPUbBwKzFsqrO5VIKz/t6+MFYKygHAjMmpA0CNwKXBoRD5GWyfxEZ0tlNrfcRmBmVnDOCMzMCs6BwMys4BwIzMwKzoHAzKzgHAjMzArOgcDMrOAcCMzMCu7/A6jE3yWzv6FTAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import numpy as np \n",
"import matplotlib.pyplot as plt \n",
@@ -1307,7 +1261,7 @@
"from sklearn.metrics import mean_squared_error, r2_score, mean_squared_log_error, mean_absolute_error\n",
"\n",
"x = np.random.rand(100,1)\n",
- "y = 2.0+ 5*x+0.01*np.random.randn(100,1)\n",
+ "y = 2.0+ 5*x+0.5*np.random.randn(100,1)\n",
"linreg = LinearRegression()\n",
"linreg.fit(x,y)\n",
"ypredict = linreg.predict(x)\n",
@@ -1446,32 +1400,17 @@
"\n",
"We will discuss in more\n",
"detail these and other functions in the various lectures. We conclude this part with another example. Instead of \n",
- "a linear $x$-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn."
+ "a linear $x$-dependence we study now a cubic polynomial and use the polynomial regression analysis tools of scikit-learn. \n",
+ "Add description of the various python commands."
]
},
{
"cell_type": "code",
- "execution_count": 22,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX5+PHPM5NAEggQIOxLAFEBZY0UEdwX3IpoRa1UtCgqorWtRW391dqvbdVqcakbCioqoFhx1wqoBcSAYZVNiKxhCSEECCQkmcz5/XEnYZLMJLPPZPK8Xy9eM3PvmTvnJuS5Z55z7jlijEEppVT8skW7AkoppcJLA71SSsU5DfRKKRXnNNArpVSc00CvlFJxTgO9UkrFOQ30SikV5zTQK6VUnNNAr5RScS4h2hUAaNu2rcnIyIh2NZRSqkFZsWLFAWNMen3lYiLQZ2RkkJ2dHe1qKKVUgyIiO3wpV2/qRkRmiMh+EVlXY/vdIrJJRNaLyBNu2x8UkRwR+VFELvG/6koppULJlxb968C/gZmVG0TkPGA0MMAYUyoi7Vzb+wLXA/2ATsACETnZGFMR6oorpZTyTb0temPMIuBgjc13Ao8ZY0pdZfa7to8G5hhjSo0x24AcYGgI66uUUspPgeboTwZGisjfgOPAfcaY74HOQJZbuVzXNr+Vl5eTm5vL8ePHA6yiqk9SUhJdunQhMTEx2lVRSoVRoIE+AWgNDAPOAN4VkZ7+HEBEJgITAbp161Zrf25uLqmpqWRkZCAiAVZTeWOMoaCggNzcXHr06BHt6iilwijQcfS5wPvGshxwAm2B3UBXt3JdXNtqMcZMM8ZkGmMy09Nrjw46fvw4bdq00SAfJiJCmzZt9BuTUo1AoIH+A+A8ABE5GWgCHAA+Aq4XkaYi0gPoDSwPtHIa5MNLf75KNQ71pm5EZDZwLtBWRHKBh4EZwAzXkMsyYLyx1iRcLyLvAhsAB3CXjrhRSqno8mXUzQ3GmI7GmERjTBdjzHRjTJkxZpwx5jRjzGBjzFdu5f9mjOlljDnFGPN5eKsfXvv27eP666+nV69eDBkyhMsuu4zNmzfX+Z6MjAwOHDhQa/tLL73EzJkzPbzDM7vdzsCBA6v+bd++nezsbO655x4AvvnmG5YuXerfCSmlglOSB/PPhndbWI8ledGukU9i4s7YWGSMYcyYMYwfP545c+YAsGbNGvLy8jj55JP9Pt4dd9zhV/nk5GRWr15dbVtGRgaZmZmAFeibN2/O8OHD/a6LUipAS66FA9+BcViPS66FixZZAX/JtVC4GtIGwoi5kNw+2rWtopOaefH111+TmJhYLUAPGDCAkSNH8s0333DFFVdUbZ88eTKvv/561esnnniC008/naFDh5KTkwPAX/7yF5588kkAcnJyuPDCCxkwYACDBw/mp59+8qlOlZ+7fft2XnrpJaZOncrAgQNZvHhxCM5YKVWvwtVWkAfrsdDVGKu8ADiKTlwAYkiDaNE/8vF6Nuw5EtJj9u3Ugoev7Od1/7p16xgyZEhAx27ZsiU//PADM2fO5N577+WTTz6ptv/GG2/kgQceYMyYMRw/fhyn01nrGCUlJQwcOBCAHj16MG/evKp9GRkZ3HHHHTRv3pz77rsvoDoqpQKQNvBEi14SrNfg/QIQIxpEoG9obrjhhqrH3/72t9X2FRUVsXv3bsaMGQNYNy154il1o5SKshFza6dowPsFIEY0iEBfV8s7XPr168d7773ncV9CQkK1VnjNsejuwxZ1CKNScSS5vZWTr8nbBSBGaI7ei/PPP5/S0lKmTZtWtW3t2rUsXryY7t27s2HDBkpLSzl06BALFy6s9t533nmn6vHMM8+sti81NZUuXbrwwQcfAFBaWkpxcbHf9UtNTaWoqMjv9ymlwqDyAjD2iPUYQx2xoIHeKxFh3rx5LFiwgF69etGvXz8efPBBOnToQNeuXRk7diynnXYaY8eOZdCgQdXeW1hYSP/+/XnmmWeYOnVqrWO/+eabPPvss/Tv35/hw4ezb98+v+t35ZVXMm/ePO2MVUrVS6z7nKIrMzPT1Fx4ZOPGjfTp0ydKNWo89OesVMMlIiuMMZn1ldMWvVJKxTkN9EopFec00CulVJzTQK+UUnFOA71SSsU5DfRKKRXnNNB7UVBQUDVFcIcOHejcuXPV67KyspB+VlZWFiNGjODUU09l0KBBTJw4kZKSEq/lc3JyqubBqemWW27hxx9/DGn9lFINW4OYAiEa2rRpUzXXzF/+8hePE4gZYzDGYLMFfr3cu3cv1113HXPnzmXo0KEYY3j33Xc5evQoycnJfh/vtddeC7guSqn4VG+EEpEZIrLftZpUzX2/FxEjIm1dr0VEnhWRHBFZKyKDw1HpaMrJyaFv377ceOON9OvXj127dtGqVauq/XPmzOHWW28FIC8vj6uvvprMzEyGDh1KVlZWreM999xzTJgwgaFDhwLWHbnXXXcd6enpPPTQQzz99NNVZU899VRyc3MBKC8v5/rrr6dPnz6MHTu26hvAiBEjqi5Qn376KYMHD2bAgAFcfPHF4fmBKKVini8t+teBfwPVlkcSka7AxcBOt82XYq0T2xv4GfCi6zE4994LoZ7JceBAcAui/ti0aRMzZ84kMzMTh8Phtdw999zDlClTGDZsGNu3b+eKK65g3brq18t169Zx++23+12HDRs2MH36dIYNG8ZNN93Eyy+/zL333lu1f9++fdx5551Vc/McPHjQ789QqtGI8YVDguXLUoKLAE9RYiowBXCfQ2E0MNNYsoBWItIxJDWNIb169apa6akuCxYs4I477mDgwIFcddVVFBYW1pl790ePHj0YNmwYAOPGjWPJkiXV9n/33Xecd955dO/eHYDWrVuH5HOViksxvnBIsALK0YvIaGC3MWZNjWl4OwO73F7nurbtDbiGEHDLO1yaNWtW9dxms+E+X5D7lMXGGJYvX06TJk28Hqtfv36sWLGCyy+/vNa+uqZDrjn9sU6HrFQQYnzhkGD53YsoIinAH4E/B/PBIjJRRLJFJDs/Pz+YQ0WVzWYjLS2NLVu24HQ6q60EdeGFF/L8889Xvfa0kMjdd9/N9OnTqZzUzRjD3Llzyc/PJyMjgxUrVgCwfPlydu06cQ3dtm0b33//PQCzZs1ixIgR1Y47fPhwvv76a3bs2AGgqRul6pI20FowBGJy4ZBgBTJcpBfQA1gjItuBLsBKEekA7Aa6upXt4tpWizFmmjEm0xiTmZ6eHkA1Ysfjjz/OJZdcwvDhw+nSpUvV9ueff55vv/2W/v3707dvX1555ZVa7+3UqROzZs3iN7/5Daeeeip9+/blq6++onnz5lx77bXk5eVx2mmnMW3aNHr27Fn1vj59+vCvf/2LPn36UFxczMSJE6sdt3379rz44ouMHj2aAQMGcOONN4bvB6BUQzdiLrQ9ExJSrccYWzgkWD5NUywiGcAnxpjTPOzbDmQaYw6IyOXAZOAyrE7YZ40xQ+s7vk5THD36c1YqTCLQwRuyaYpFZDbwHXCKiOSKyIQ6in8GbAVygFeAST7WVyml4ksMdfDW2xlrjLmhnv0Zbs8NcFfw1VJKqQYuhjp4dQoEpZQKhxjq4NVAr5RS4TBiLrQeghVmDTjLrLx9FGigV0qpcEhuD7YmIDYwFXBwRdTy9BrolVIqXGIkT6+Bvg52u52BAwfSr18/BgwYwFNPPVXtTlVPtm/fzqxZsyJUQ6VUTIuRPL0G+jokJyezevVq1q9fz/z58/n888955JFH6nyPBnqlVJUYuRFLA72P2rVrx7Rp0/j3v/+NMYbt27czcuRIBg8ezODBg1m6dCkADzzwAIsXL2bgwIFMnTrVazmlVCOQ3B4uWgRjj1iPUZoRM34WHonAXWg9e/akoqKC/fv3065dO+bPn09SUhJbtmzhhhtuIDs7m8cee4wnn3ySTz75BIDi4mKP5ZRSKlLiJ9BX3oVmHCfuQrtoUdg+rry8nMmTJ7N69WrsdjubN28OqpxSSoVL/AT6CPRub926FbvdTrt27XjkkUdo3749a9aswel0kpSU5PE9U6dO9amcUkqFS/zk6MPcu52fn88dd9zB5MmTEREOHz5Mx44dsdlsvPnmm1RUVACQmppKUVFR1fu8lVNKqUiJn0Afht7tkpKSquGVF154IRdffDEPP/wwAJMmTeKNN95gwIABbNq0qWoxkv79+2O32xkwYABTp071Wk4ppSLFp2mKw02nKY4e/Tkr1XCFbJpipZSKKyV5MP9seLeF9Ril+WciSQO9UqpxiaF54iMlpgN9LKSV4pn+fFWjFCPzz0SSLytMzRCR/SKyzm3bP0Vkk4isFZF5ItLKbd+DIpIjIj+KyCWBViwpKYmCggINRmFijKGgoECHe6rGJ0bmn4kkX8bRvw78G5jptm0+8KAxxiEijwMPAveLSF/geqAf0AlYICInG2P8HlPYpUsXcnNzyc/P9/etykdJSUnVFjNXKu6V5FnzwhsniN2aLz7OFgL3xJelBBe5Fgd33/al28ss4Beu56OBOcaYUmCbiOQAQ7HWnPVLYmIiPXr08PdtSinl3ZJrrXnhcQIJ1nzxnqZKicCUKpEUihz9r4HPXc87A7vc9uW6timlVPT5mp+Psw7boAK9iPwJcABvB/DeiSKSLSLZmp5RSkWEr/n5OOuwDTjQi8jNwBXAjeZEj+luoKtbsS6ubbUYY6YZYzKNMZnp6emBVkMppXzn6x30cdZhG9CkZiIyCpgCnGOMKXbb9REwS0T+hdUZ2xtYHnQtlVIqFCrnh6/PiLm1c/QNWL2BXkRmA+cCbUUkF3gYa5RNU2C+iABkGWPuMMasF5F3gQ1YKZ27Ahlxo5RSUeXrBaGBiNm5bpRSStVN57pRSikFaKBXSqngNIBJ0jTQK6VUMBrAmHsN9EopFYwGMOZeA71SSgWjAYy510CvlFLBCMMypqEW0A1TSimlXBrAmHtt0Sul4lMDGA0TKRrolVLxqQGMhokUDfRKqfjUAEbDRIoGeqVUfGoAo2EiRQO9Uio+NYDRMJGio26UUvEpWqNhYnAZQm3RK6VUKMVgJ7AGeqWUCqUY7ATWQK+Uig+xMm7eh07g4+WRXY+p3kAvIjNEZL+IrHPb1lpE5ovIFtdjmmu7iMizIpIjImtFZHA4K6+UUlViJWVSTydweYWTX76Sxd8/2xixKvnSon8dGFVj2wPAQmNMb2Ch6zXApVjrxPYGJgIvhqaaSilVj1hJmVR2Ao89Yj3W6Ih97PNNrNx5iNM7t4xYleoN9MaYRcDBGptHA2+4nr8BXOW2faaxZAGtRKRjqCqrlFJeNYBx81+s28v0JdsYf2Z3rhzQKWKfG2iOvr0xZq/r+T6g8pLVGdjlVi7XtU0ppcIrxsfN5+w/yn1z1zKgayv+eHmfiH520OPojTFGRPxeYVxEJmKld+jWrVuw1VBKNXYxPIvksVIHd7y1gqYJNl4aN5imCfaIfn6gLfq8ypSM63G/a/tuoKtbuS6ubbUYY6YZYzKNMZnp6ekBVkMppWKbMYYp761la/5RnrthEB1bJke8DoEG+o+A8a7n44EP3bbf5Bp9Mww47JbiUUqpRmf6km18+sNepow6leEntY1KHepN3YjIbOBcoK2I5AIPA48B74rIBGAHMNZV/DPgMiAHKAZuCUOdlVKqQcjaWsA/Pt/EqH4duP3snlGrR72B3hhzg5ddF3goa4C7gq2UUko1dPsOH2fyrJVktEnhn9f2R0SiVhed1EwppUKszOFk0tsrKCmrYM7EYaQmJUa1PjoFglJKhdgjH69n5c5DPPGLAZzULrV2gXnzICcnYvXRQK+UUiH0VtYO3l62kzvP7cXl/T3cL/r113DddfDggxGrkwZ6pZQKkaytBfzlo/Wcf2o77rv4lNoF1q+HMWOgd2+YNi1i9dJAr5RSIbDrYDGT3l5J9zYpPH39QOy2Gp2ve/fCZZdBcjJ89hmkpUWsbhrolVINW6inJw7geMdKHdw2MxtHhZNXx59Bi5qdr0VFcMUVUFAAn34K3bsHV0c/aaBXSjVsoZ6e2M/jOZ2G++auYXNeEf/+5WB6tG1WvUBZGVxzDaxZA3PnwuDIz96ugV4p1bCFenpib8fz0tJ/7qscPl+3jz9e1oezT64xnYvTCbfcAvPnw/TpcOmlUVkgRQO9UqphC2Z6Yk9B19vxPLT0v1i3l6kLNnPN4C5MGNGj9vGnTIFZs+Af/4Dx470eJ9w00CulGrZgpif2FHS9Ha9GS39N7hHufWc1g7q14m9jTqt95+tTT1n/7r4b7r//xPYoLJCid8YqpRq2YKYn9hR0vR0vbaB1MTAOdpd34NZtD5Ke2pRXbsokKbHGtMNvvw333QfXXgtTp4L7RcDtOJFaIEVb9EqpxsuftI+rpV8k7fj1zn9wXFoyY/wZtG3etHq5L7+Em2+Gc8+FN98Eu93jcSK5QIq26JVSjdeIuVa6pnC1FeTrCrrJ7XGc/w2T38jmp5IDvH5LJr3b15jeYNkyuPpq6NsXPvgAmjb1eJxIL5CigV4p1TiV5NUO8jUW8nZnjOGRjzfwv835PHb16YzoXWNu+R9+sEbVdOgAX3wBLSO3+Hd9NHWjlGqc/Bz9MuPb7byZtYPbz+nJ9UNrLH+akwMXX2zd9bpgAXT0MMdNFGmgV0pFRxTGk1fjx+iX+RvyePTTDYzq14H7Lzm1+s7cXLjwQigvt8bLZ2SEr84BCirQi8hvRWS9iKwTkdkikiQiPURkmYjkiMg7ItIkVJVVSsWRKIwnr8bHjtgVOwq5e/ZK+ndpxdTrBmJzn8MmPx8uuggOHrTSNX37RqDi/gs40ItIZ+AeINMYcxpgB64HHgemGmNOAgqBCaGoqFIqzkRhPHk1Pox+ydl/lAlvfE+HFknMGJ9JchO3ETRHjlg5+e3b4eOPITMzcnX3U7CpmwQgWUQSgBRgL3A+8J5r/xvAVUF+hlIqHgVzR2soVI5+GXvEeqzREZt35DjjZywnwSbM/PXPaOM+jLK4GK680pq/5r334Jxz6v6sKKepAg70xpjdwJPATqwAfxhYARwypvIyTS7QOdhKKqXiUBTGk/vqyPFybn7tew4Vl/HazUPp1iblxM6SEhg9GpYsscbJX355/QeMcpoqmNRNGjAa6AF0ApoBo/x4/0QRyRaR7Pz8/ECroZRqqJLbW8E9baCVtllybeQ7ZD0odVRwx5sr2JJXxIvjhnB6F7dhkqWl1jj5hQvhtdfg+ut9O2iU01TBpG4uBLYZY/KNMeXA+8BZQCtXKgegC7Db05uNMdOMMZnGmMz09HRPRZRSsS7YlES0O2RrcDoNf5i7lqU/FfD4Nf2rz0ZZVmZNafDFF/DKK3DTTb4fOMppqmAC/U5gmIikiDWbzwXABuBr4BeuMuOBD4OrolIqZgUbqKPdIevGGMOjn27kozV7mDLqFK4Z0uXEzvJyq/X+8cfw4oswwc8xJlFOUwV8Z6wxZpmIvAesBBzAKmAa8CkwR0QedW2bHoqKKqViULCBOgoTfHnz7MIcZny7jZuHZ3DnOb1O7HA44Fe/gnnz4Jln4I47/D94FKY9cBfUFAjGmIeBh2ts3goMDea4SqkGIthA7c9cM2H02rfbquaV//MVfU9MOVxRYS0c8s478OSTcM89UalfsHSuG6VU4IIN1FFu6QK8tyKXRz7ewCX92vP4NaefuCHK4bDy8LNnw9//Dr//fVTrGQwN9EqpwMVAoA7GF+v2MeW9NYw4qS3P3jCIBLur27K8HG680Vrj9bHHqi8c0gBpoFdKNUpLthzgntmrGNC1FS//aghNE1x3vZaVWR2v8+ZZK0T97nfRrWgIaKBXSjU6K3cWMvHNbHqmN+P1m4fSrKkrFJaWWkMoP/7Y6nhtoDn5mjTQK6UalbW5hxg/fTnpqU2ZOWEoLVMSrR3Hj8M118Bnn8Hzz8OkSdGtaAhpoFdKNRrrdh9m3KvLaNUskdm3DaNdapK1o7gYxoyxlgF8+WWYODG6FQ0xDfRKqUZhw54jjJu+jNSkRGbdOoxOrZKtHYcPwxVXwLffwowZ1nDKOKOBXikV937cV8S46ctITrQz+7ZhdG3tmqQsPx9GjYK1a2HOHBg7NroVDRNdYUopFR7RXkHKJWd/ETe+mkWiXZh127ATM1Hm5sLZZ8OGDfDhh3Eb5EEDvVIqXGJgwrKt+Ue54ZVliFhBvkfbZtaOnBwYMQJ274b//hcuuyzidYskDfRKKd/420KP8oRlOfuLuH5aFk6nYdatP6NXenNrx7p1MHIkHD0KX31lterjnAZ6pZRv/G2hR3Fq3o17j3Ddy1kYYM7EYfRun2rt+O47K7DbbLBoUUwv/xdKGuiVUr7xt4Uepal51+0+zA2vZJFot/GOe5D/+GO44AJo08ZaHSpGF/IOBx11o5Tyjb8zVUZhHpxVOwu5acZyWiRZ4+SrOl5ffRVuvx2GDIFPPoF27SJar2jTFr1SyjcxvMYrwPJtBxn36jJaN2vCO7e7grwx8OijcNttcNFFVk6+kQV50Ba9UspXoWihl+TVntY4uX3QVVuac4AJb2TTsVUSs24dRoeWSdZc8nffba0IdWE7uOlb+O6ykH1mQ6IteqVU5IRhyOV/1+/j5te+p2vrZOZMdAX5khJrXPyLL8J1XeCWAuBoTKxLGw1BBXoRaSUi74nIJhHZKCJnikhrEZkvIltcj2mhqqxSqoEL8ZDLd7N3cedbK+jbqQXvTDyTdgmH4d1hMKA5zHsf/vl/cPVhoCJknwnEzM1gvgq2Rf8M8IUx5lRgALAReABYaIzpDSx0vVZKqZAOuXxl0VamvLeWs05qy9u3/oy0Zk1g1uUweRnscMK9NhjwZXiGecbAzWD+CDjQi0hL4Gxci38bY8qMMYeA0cAbrmJvAFcFW0mlVANX2QI+uBLsKZDQPOAOXWMMT3yxib99tpHLT+/Iq+Mzrfnkv/oK7lkBZcBDQKbTar372onsTys9yjeD+SuYFn0PIB94TURWicirItIMaG+M2esqsw/w2OshIhNFJFtEsvPz84OohlIqLEKZnqhsAVccg4piSBtkdez62Sla4TT8cd46XvjmJ24Y2o1nbxhkrQz12mtwySWQngJ/tUMvTrTeKzuRxx6p+zP9aaV7+5YQoymdYAJ9AjAYeNEYMwg4Ro00jTHGAMbTm40x04wxmcaYzPT09CCqoZQKi1CmJ0LQAj5eXsHkWSuZvXwnk87txd/HnIYdAw89BL/+NZx3HmSthj7DAxsC6k8dvX1LiNGUTjDDK3OBXGPMMtfr97ACfZ6IdDTG7BWRjsD+YCuplIqCUKYn/L3ZqmZVjpVx28xssncU8tDlfbh1ZE9rrprx4+H99+HWW+GFFyAxEToEOATUnzp6G2oaoymdgFv0xph9wC4ROcW16QJgA/ARMN61bTzwYVA1VEpFRyg7MYO42WpnQTHXvLSUtbsP8/wvB1tBfvt2OOss+OADawHvadOsIB+MUNwQFsX5feoiVnYlwDeLDAReBZoAW4FbsC4e7wLdgB3AWGPMwbqOk5mZabKzswOuh1IqDMJ0c5M/1uw6xIQ3vsfhNLxyUyZnZLSGxYvh6quhvNxaLGTUqIjWqU4R/pmJyApjTL0zswUV6ENFA71SqqaFG/OYPGsVbVOb8PotQ61phl95xVq0u2dP+OgjOOWU+g8Ux3wN9HpnrFIq5ryVtYPbZmbTu31z3r/zLHq1ampNZzBxojUD5bJljT7I+0PnulFKxQxHhZNHP93I60u3c8Gp7Xjul4NIOXgALh9rpWx+9zt4/HFI0NDlD/1pKRWPvOWKYyDv7s3hknImz1rJ4i0HuG1kDx64tA/275bCtdfCoUPw1ltw443RrmaDpKkbpeKRt/HcMTrOe9uBY4x54VuythbwxDX9+dNlfbD/+zk491xISYGsrOCCfIzeyBQpGuiVikfexnNHapy3H4F1ac4Brnr+WwqPlfHWhJ8xtm9rGDcOfvMbuPRSyM6G/v2Dq0+MXuAiRQO9UvHI23juSI3z9iGwGmN4K2sHN81YTrvUpnx41wh+5iiAM8+E2bOtBUM++ABatQq+PjF6I1OkaKBXKh55u/nH35uCAk151BNYj5dX8MB/fuChD9Yxsndb3p80nG5fzIPBg2H3bvj8c/jTn6xFvEMhRm9kihTtjFUqHnm7Rd/fVaIqW+bGcaJl7sv765hOYPehEu58awVrcw9z9/knce/wLtjvnmSt63rWWdZNUF26+F5HX4yYW7sTuhHRQK+U8i7QlIeXwLr0pwNMnrWKMoeTl381hEtshXDmMFi3Dh58EP761/AMnYzCQuWxRAO9Usq7QCcjqxFYjTG8umgr//h8Iz3Tm/Pyr4bQ6/P34c47oVkz+OILa5phX8XwMNFYpDl6pZR3IZjoq+h4OZNnr+Jvn23kkn4d+GDcafT63Z3WzJNnnAGrV/sX5KHRj6Lxl7bolVLeBZnyWLf7MJNnrWTnwWKmjDqFO2U3coarw/WRR+CPfwwsVdPIR9H4S1v0SqmQM8bwZtYOrn5xKcfLnbxz8xAmfTkDOe88azrhJUvgz38OPB/fyEfR+Etb9EqpkCo6Xs4D7//Ap2v3cs7J6Tw9IIm0X14OK1fChAnw9NPQvHlwH9LIR9H4SwO9UvEg3J2TPh6/MlWzq7CE+y85hds3fIlt5H3WNAbvvw9jxoTu8+tKKcViZ20U66Tz0SsVD+affWJ0DHZIaAYYK6BkvgDZk4ILMO7HlwSrY9Yt0DqdhleXbOXJ/26mdbMmvHR+ewb+3/3w2Wdw8cXW4t2dOoXm/Dx8ftDlIyEMdfJ1PvqgW/QiYgeygd3GmCtEpAcwB2gDrAB+ZYwpC/ZzlFJUbxW27GttO7wBKorBVLgKVYDjiPU0fyl8PtDaBv7d9OSujs7PvXm7+P3rH7G0MINL2m7iqabQfNT/QVkZPPMMTJ4c/B2u/na+xmJnbRTrFIrO2N8AG91ePw5MNcacBBQCE0LwGUopqD6ssGCZ9c9R5Bbka6qgKshD4AHGS+fnZz/sZdRzy1h1uAPPNnucl978A83vuQ8GDIC1a+GeewIP8u7TLyD+db4gG0Z5AAAYfUlEQVTGYmdtFOsUVKAXkS7A5VjrxiIiApwPvOcq8gZwVTCfoZRy494qrMVmjXdPaHEioHgSSICpMZ7+6BlzuG/uGia9vZKMxFwW75nAz/++GPnRwC1N4euv4aST/P8cd+4XNccxsKdYn996CDjL6p5/JxQLfYdaFOsUbOrmaWAKkOp63QY4ZEzV/8RcoLOnN4rIRGAiQLdu3YKshlKNhPudqu7cc77u6R3ECpKVrfqEFp4DTH0dhW7j6bO2FvCHaWvYXVjCH/slc+vTf8W28jD0BSbaoe/Q0ExGVu2iVgEYGHukeq7bWyoqFqc8iGKdAv5tiMgVwH5jzIpA3m+MmWaMyTTGZKanpwdaDaUaF/dWYZufWf9qthArA8rYI3DlZkgfbpVJH2m99tQR68OdpsVlDh7+cB3XT8siwTj5uukPTJw0GtvmMrjnJHgoBTo2g8JVoVncw1uqI5K57jhZsCSYFv1ZwM9F5DIgCWgBPAO0EpEEV6u+C7A7+GoqpQD/W4W+lq8neGZtLWDKe2vZebCYB9sd49a3/4x95Upr6oKXX4bu3X1rafvD21j5QOffCUSgs3fGmIADvTHmQeBBABE5F7jPGHOjiMwFfoE18mY88GEI6qlUfIqF8d4leYCceO0WPIvLHDzxxY+8vnQ7fZKdZO39mA7/nA7t2sE771jruYrrvaFuaXu7SEXyZqm6zikWfnc+CscNU/cDc0TkUWAVMD0Mn6FUfIiFFuOSa115fBd7CoyYy5ItB/jjvB/YWXCMJ9jML15+Etu+fXDXXdbqTy1bVj9OpFrakcx113VOsfC781FIAr0x5hvgG9fzrcDQUBxXqbgXC+O9C1fjPgSzoLw5j360l3mrdjNcDvPhstdI+98CGDQIPvzQmnHSk3iclqCuc4qF352PdAoEpaIp2Fawtxuo/EkluOpgnA7mHrqYv++9DWfpVmbvWcCw/8xAEhOt+WnuuuvEJGTe0hYx2qINWF3nFMm+giDpFAhKRVOwed5qUx+48ecW+5I8tn45kT/9cBbfFfXlt3lZTPryNRL37oFf/hIef7z20n6xOMVApMVAjj5iUyAopYIQbCvY2w1UPqYSisscPL+okFeW38Hg/Vv4ftnDpP+wEoYMgbnvWmu41ve5MZ62CJsG9A1GA71S4RTuVl9dN1DVkUowxvDZD/t49NMNOHbv4a117zH0fx9B+/YwY4a1+lNdNz01oLSF0kCvVHiFe2SGe2ehpxy9B5vzinj4w/Ws3rSbB378gnH/m429rAymTIE//QlatPDvc+Ol4zWOaaBXKlQ8td7DneLwI31wuLicZxZu4e0lOYzbsJDp380mpSAfRo+Gf/4TevcOy+f6JAby3fFMA71SoeKp9R6KFEeQQbDM4eTNrB08u2AzZ635hiXLZ5O+Z4eVf398nvc8fCQ1oDHpDZEGeqVCxVPr/cotwac4AgyCxhg+X7ePx7/YRMdVWbyf9Ra9tm2Avn3hxQ/hyitP3NUabdq5G1Ya6JUKFU+t95opjspJsvxpndcMggdX1nuMlTsL+dunGynLWs4/l89h6KZl1hDJ6dPhppsCX5Q7XLRzN6x0HL1SoeJLiiWQ8ec132NPca0oVfsYm/Yd4akvN7Nv4RKmLHuHkT9mYVq3Ru6/H+6+G5KTw3TyQdIcfUB0HL1SkeZLB2UgKYqaI1wKV9U6xvYDx5i6YDPbvvgfv/tuDuduXoZp3Rr+9jdk8mTfRtJEUwMak94QaaBXyptwtDIDSVHUDIJuLfw95R147sBkNv7hVX7z7SzO27IcZ1pawwnwKiI0daOUN+G4zd/TxQP8u6CU5JE7fwIv/diLbavbcVvWB5y7NRtnqzRs9/3eStFogG8UNHWjVLDCMRLEU4rCjwU7th84xksL93Dk3QFMzPoPA/f8SEXbtvDoo9g0wCsvNNAr5U3N6QUqiq2gHOqOQh8uKDn7j/LSlxtIfPtNJi6fR8+Du3Fk9IAXXsB+882x28mqYoIGeqW8qewEzf8WcIKp8N7iDiaf7yVvb4whe0chb322io5z3+L+7A9JP1pIef+B8MJTJFxzTewNk1QxKZjFwbuKyNciskFE1ovIb1zbW4vIfBHZ4npMC111lQoDbwtAV6ZZEpqdKOsthePD4tpeuS/43fZMHMPf5ZO1e7jnobfYctWNPHbv5Tzwzeu0PGMQzJ9P4uqVcN11/gf5OFnoWvkv4EAPOIDfG2P6AsOAu0SkL/AAsNAY0xtY6HqtVOyqL0inDbRa2uB9pEww+XzXBeXY6IO83nQG/+/OabS+8lKe+/tNjN34NYnjboTVq2ny1UK48MLA72YN5mLkK72YxKRgFgffC+x1PS8SkY1AZ2A0cK6r2BtYSwzeH1QtlQqn+oJ0zRkinWVWIHNP0fg7bNIt1bMz6Rz+U3Qv5vXZjF3+MTcf2U9Jx844//EPEm67Ddq0Cc955n9b+zyCFek5a/RGK58E06KvIiIZwCBgGdDedREA2AfoT13Ftvpa7JUpnLFHwNYEDq6o3SqukX6pb06bisVjWZBTzmP/vZqVDxYxafyl/G7BdFqemgF/7kPyU4exDfkMUjwsKhKK8wTAGfrWfTDfbAL5NhCJbylxIOhALyLNgf8A9xpjjrjvM9YgfY8D9UVkoohki0h2fn5+sNVQKnDuQbr1kBMtdk/Bxlsgc78YXLTIa6syv6iUGe8v46UnOtDzTzt5YMYbjNq6FMd5TWHNGlIfbQqnbgFzNPSBy/08xX5iu/t5BJt68SXN5U0gQVsnQ/NJUIFeRBKxgvzbxpj3XZvzRKSja39HYL+n9xpjphljMo0xmenp6cFUQ6nq/A1WvrTYK1VrFdsBqfdznE7Dtz/m8cL9T5N9xvmMu3YEdy18l1atj+K4XUh60UmzKYOgf3/vgSsUuW/382w73HNADraF7Oc3m2pqpZYW13+uwVxYGpGA74wVEcHKwR80xtzrtv2fQIEx5jEReQBobYyZUtex9M5YFVLB3NH6bgsryFVKSLUCY2Uu+OBKV2vYCdg8Ty7mKrt9717+t/lcEhYd5oJVC+lw9CDFKUmUjkgg7cJi6N4cMNVzy97qHuq7dL3ltr2dfyR4Wui8vnONRI4+hvsBfL0zNphAPwJYDPyA9b8e4I9Yefp3gW7ADmCsMeZgXcfSQK9CqmawAkgf6dsfqD+BtnB1raB4dPRBvn5xAoWfF3DGqvX0yd+Ow2bjwMgLaD3wO5oMOgqJJ8rXCqLRDsDhmPbBV5Xnnr+4+vZIXmw8iebPpB5hnwLBGLME8DbO64JAj6tU0DwtmO3rCBBva6F6Sqm4PqesAr7bP5Dd2T3IeGAol29bjQ3Dvu6tOTKuGS3OgQ63flk7YHhKM3ibxTFS87VHcy3YynP35ecUSXHQD6C31an4U3VHq1vL0Nc/UB8DbXnLwSxPfoLdHz1Ox++3cua21TRxZnOwXWfyrulDhzN/pEPHgydagO71CiSIRioAx8J0wbG28HgcLIqis1eq+BXKr9wleTgWjWXlpqPkZnWlQ3YuZ2xdTaKzgoL0Thy7cgwdb/0VicOGwvH9MZvTrSaGc88xJYZ/TmHP0YeSBnoVFnX9gfr4x3us1MGKBcsofOd9Oi1ewKAd60gwTvLbdaHoyqvofNuvaDr0jLrvVvU1UEQ6oMRw7ln5RgO9UnWpI8jtP3iUtXM/p/yDjzjl+2/oWZALwJ6uJ1F88aV0vm0cyfUFdx8/K6ByoRLNETYqJHQ+eqW8KcmDA0ut2SgBh9PJ+rWH2PvlYyR/8xX912Vx4fEiym12dp7Uia1XZtL1vufp1G/oifcvOMf3lrevnXmR7vSLg9yz8o0GehVdwaQrAn3vkmvJPdKaLWu64VhrJ2PzXgbk/8AAHuRQahp7R15A8YA9dDplBb1SdoHshT33Qb9FVe/3az4XXwNqpANvrHV6ehLD+fGGRFM3KrqCSVf48d79BUfY/PHXlCxYSOul8zlt5080rSin1J7Azh4dcVw2li43XE/q0MFgs9Wd1vAl5eEeoFr2tbYd3hBbOfqGQPsR6qSpGxWbagazwlWBpyvqSHXsyztEzsfzKVnwDW1WfEef7RsY4SgFYFeHjmwe2Z02gw7R8fRCenfJgIuerH7sulrXvrS83Vv9B1dYAaq+/HcsDG2MNXEwhj0WaKBXkVUz7WFPsYJlIOkKV8Atdxpy9nYnb9upVLw5nrbrVnHqro2MqHDgRNjVrTc5P7+G5r2207X3D3Tt1o2uAIfzIG2455RFXWkNX1IeGqBOCMPqW8o/mrpRwfPnD7lm2sPeDFoP9jkIGGPYs6+QXfMXceybb0j5fjEZO3bQsagAgDJ7ArndT+boGWfSctSFdLnyYuxtWlspgPylgNUBS0ILuHJz9c8KZepEUw4nBPOzCCQF1ojo8EoVOf78IftS1vXHXXFwDdudI9hz9AaKV/1I0x/W0O6njfTO20ai0wrY+9p0JD+jDdKzhHanFJB+UhHSaXjtY3qb/8a9XIhvsNJ8u0uohnHqxbMWzdGr8PDUwipYdmJ/5cpFJXmeA5uXtEfh0VK2r/+JQ0uX4/jqbVK2Habznmb0KPycXnwGwJFmLcnr1YdNF11Myjkj6DLqXDp06kiHmoHEU5okbWDtybJqlgsk3eItoIci3x4vF4ua6ZeWfa2g7e95aTosYNqiV/7xNJWsJzVbyyV5mMXXciBvKzsPZ1JYfAnlG7fSJOdH2u78iYy8HbQsPVZVfF9aaw52boGzu40WPY/R8dQSEk/qDyPfs4KCexBEwHGMetMyH58MDldL0lOLMJAWYzhbmfHSgq15wXKWWR3U/p5XvPw8QkhTNyo8PKVA3DiNsL88jdyD7Th0qB/HiwfC9lxSflpLu3159CjYQ7Py41XlDzdvxYFuJ1GaXkZC+0O06nmYtj0OY2vu4eDepg2mcrWkCreydmtxDX+mPQikBR3Ou0tj9c7VYL9pBHpe8fINJ4Q0ddPY1fyjyHwBsif51qlVxx9UWcvB5O3dwr6jLTm0P5Xi/CScB2w02V9O8wPFtCk4TLfCfWSWbQI2YS1ABgUtW1LYLpXtJ3ekaZdyWvcsI+22r2j54520LFztWsDDLVBjg4Rm1bcbh7Xwx/yza6Rh3N/nYipq38zkKZ0SbPAI56iQWB1xEuwC4IGelw4/DZi26Bs6b6MSkBOrH3lq8boTOxVtzqKgz7PkL/8rhTt/4lhBEqWHmlJemIAcTqZpUVOa799L24N5dCw6QOuS6i0wh81OQes0jrZpSnk7O4kdHLToWExaNwcJk/bDkktqf+0Gz2kgSbDWbrU1sfL9VevaYJ2LSO1ViOwpbufrpr7WYrDpgHC2MqPVgq3vc4P9pqEt85CJeupGREYBz2BFmVeNMY95K9voA70vsyweXElV8BaB1plw9ofVW1cuxkCxM4nCilQOVaRSVNyMY0VJHD/ShPKjCVQcsWOOCAmHK2hSVE6zohJaFh0l/dgh0o8V0rSivFYVj6WkcqRNO0pagbOlA3vnNJLPvIlWpw0kqWcGdO0KCQneA2d9nbhgnR9inZ89GSqO198XAFZ/QOU3FvcLgy+BO1bTI9FU38VPc+UxI6qBXkTswGbgIiAX+B64wRizwVP5BhHovbWcK58fWofHQAzV3+csPzFawJ5kBTO3FmuFsXHMmcKximSOOlM4WprM8eIESoubUFacQHlxAhUldipKbFBig+MGW4kTOW5IKKkgqbiMZsUltDh+jFYlRaQdLyLJUebxlJwiHE1J4ViLJEpTE6loaUdaGhLTHCSlldK8dTHJaeXQRiBZqNWyTmhGrTVPPV2YABJSrMe0QdU74yrV1Sqvi4dOX68pK0+tx1gNWtFs9dZ38dMWecyIdqA/E/iLMeYS1+sHAYwx//BUPqhAX5IH86+G3FVgBCpKwAC2plBReuK5o9T7duooI02goowKA2XliZRVJFBWnki5I4FyRyIOh53yigQc5XYcFXYcDjsVDjvOChsOhw1nhQ2nwwblIOUGKTPYy53YHE4SypzYHRUklDtILHeQ6KigiaOcJEcZTR1lJJeXemxde3I8MZHSpCaUpDSlrFkiFc1smGaCNDckpFbQJNVBUmopyS1KaJLqhJZAKieyOsHw1uqrOZyxsiymRj4eK2AXrgLHUT8+2AZj9tQdZOoL5LEatKJ5AYrVi5+qJdqdsZ2BXW6vc4GfheWTllxLyX9WkjzzeI0dx0P03LoQ2IFkykjGc+u4PuU2O2UJiZQlJlKWkIAjMQFHoh1Hop2KJjaczWzW6yZ2ShKbYmtikKZO7MlOElIqSEh20CSlnKYp5SSllJGYUgHJQJL1L8leThLltORY3RVJaFEjd+8lb+8PT2OavY1xNg7AVn3aA28jaSq/MVSler6nVlqmvqBc39jrWO3gi+aY8YYwq6XyS9RG3YjIRGAiQLdu3QI/UOFq/tvpZ2RdcjpGBKdYeV4jgkEwAk6xYQAjgmBIsFWQIBUk2BwkioMEWwWJNtdrm4NEWwWJUk6izUGCVJBoK6eJzUFCooNEewUJCQ4SEytoYi8jMcFBk8RymtodNEkso0lCGUmJZTRNKCcpsQx7ggE71jGpoBk1L0j+sLkenXWW8qgyMNb8Iw4kjSJ2a+qCyv2eRk54ukGp8tiVHa01A4mnAFPfaKD6xOrIlfpEs96xevFTAWv4qZv5Z7N19w7WF3cnSUpJspVZ/9yeN618LmUkisPnhYH8508gtp3I0YtAqwFgS6ye64fquW33PLj7zT/urd+ao2085dHduefUxW7VPW3Qidy2t05gqD8oLxrtaoV7OY9IiNXUTH0aar1VREU7R5+A1Rl7AbAbqzP2l8aY9Z7KB52j99T5V9XRGaLnlUFu6Kv1j0d3D3LuQTyUkzF5CwQaIJRqNGJheOVlwNNYieAZxpi/eSvbIEbdKKVUjIl2ZyzGmM/ANRuVUkqpqLHVX0QppVRDpoFeKaXinAZ6pZSKcxrolVIqzmmgV0qpOKeBXiml4lxMzEcvIvnADh+LtwUOhLE6sayxnntjPW/Qc2+M5+7PeXc3xqTXVygmAr0/RCTblxsE4lFjPffGet6g594Yzz0c562pG6WUinMa6JVSKs41xEA/LdoViKLGeu6N9bxBz70xCvl5N7gcvVJKKf80xBa9UkopP8RsoBeRUSLyo4jkiMgDHvY3FZF3XPuXiUhG5GsZej6c9+9EZIOIrBWRhSLSPRr1DIf6zt2t3DUiYkQkbkZk+HLuIjLW9btfLyKzIl3HcPDh/3s3EflaRFa5/s9fFo16hpqIzBCR/SKyzst+EZFnXT+XtSIyOKgPNMbE3D+sOex/AnoCTYA1QN8aZSYBL7meXw+8E+16R+i8zwNSXM/vjIfz9vXcXeVSgUVAFpAZ7XpH8PfeG1gFpLlet4t2vSN03tOAO13P+wLbo13vEJ372cBgYJ2X/ZcBnwMCDAOWBfN5sdqiHwrkGGO2GmPKgDnA6BplRgNvuJ6/B1wgEr5FAiOk3vM2xnxtjHEto0UW0CXCdQwXX37nAP8HPA5BLb4ba3w599uA540xhQDGmP0RrmM4+HLeBmjhet4S2BPB+oWNMWYRcLCOIqOBmcaSBbQSkY6Bfl6sBvrOwC6317mubR7LGGMcwGGgTURqFz6+nLe7CVhX/XhQ77m7vr52NcZ8GsmKRYAvv/eTgZNF5FsRyRKRURGrXfj4ct5/AcaJSC7WQkZ3R6ZqUedvLKhT2FaYUuElIuOATOCcaNclEkTEBvwLuDnKVYmWBKz0zblY3+IWicjpxphDUa1V+N0AvG6MeUpEzgTeFJHTjDHOaFesIYnVFv1uoKvb6y6ubR7LuBYjbwkURKR24ePLeSMiFwJ/An5ujCmNUN3Crb5zTwVOA74Rke1YecuP4qRD1pffey7wkTGm3BizDdiMFfgbMl/OewLwLoAx5jsgCWsumHjnUyzwVawG+u+B3iLSQ0SaYHW2flSjzEfAeNfzXwBfGVcvRgNW73mLyCDgZawgHw952kp1nrsx5rAxpq0xJsMYk4HVP/FzY0w8rCrvy//3D7Ba84hIW6xUztZIVjIMfDnvncAFACLSByvQ50e0ltHxEXCTa/TNMOCwMWZvoAeLydSNMcYhIpOB/2L1zM8wxqwXkb8C2caYj4DpWF/jcrA6Na6PXo1Dw8fz/ifQHJjr6nveaYz5edQqHSI+nntc8vHc/wtcLCIbgArgD8aYBv0N1sfz/j3wioj8Fqtj9uY4aNAhIrOxLtxtXf0PDwOJAMaYl7D6Iy4DcoBi4JagPi8OfmZKKaXqEKupG6WUUiGigV4ppeKcBnqllIpzGuiVUirOaaBXSqk4p4FeKaXinAZ6pZSKcxrolVIqzv1/KMXacW5YEmIAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "0.004999999999999991\n"
- ]
- }
- ],
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
@@ -1509,110 +1448,22 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Similarly, using **R**, we can perform similar studies. \n",
- "(more details on **R** will be inserted later).\n",
+ "Using **R**, we can perform similar studies. \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "## Simple regression model with gradient descent\n",
- "Add info about the equations, play around with different learning rates"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Importing various packages\n",
- "from math import exp, sqrt\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "\n",
- "x = 2*np.random.rand(100,1)\n",
- "y = 4+3*x+np.random.randn(100,1)\n",
- "\n",
- "xb = np.c_[np.ones((100,1)), x]\n",
- "theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
- "print(theta_linreg)\n",
- "theta = np.random.randn(2,1)\n",
- "\n",
- "eta = 0.1\n",
- "Niterations = 1000\n",
- "m = 100\n",
- "\n",
- "for iter in range(Niterations):\n",
- " gradients = 2.0/m*xb.T.dot(xb.dot(theta)-y)\n",
- " theta -= eta*gradients\n",
- "\n",
- "print(theta)\n",
- "xnew = np.array([[0],[2]])\n",
- "xbnew = np.c_[np.ones((2,1)), xnew]\n",
- "ypredict = xbnew.dot(theta)\n",
- "ypredict2 = xbnew.dot(theta_linreg)\n",
- "plt.plot(xnew, ypredict, \"r-\")\n",
- "plt.plot(xnew, ypredict2, \"b-\")\n",
- "plt.plot(x, y ,'ro')\n",
- "plt.axis([0,2.0,0, 15.0])\n",
- "plt.xlabel(r'$x$')\n",
- "plt.ylabel(r'$y$')\n",
- "plt.title(r'Random numbers ')\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Simple regression model with stochastic gradient descent\n",
- "Add info about the equations, play around with different learning rates"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [],
- "source": [
- "# Importing various packages\n",
- "from math import exp, sqrt\n",
- "from random import random, seed\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from sklearn.linear_model import SGDRegressor\n",
- "\n",
- "x = 2*np.random.rand(100,1)\n",
- "y = 4+3*x+np.random.randn(100,1)\n",
- "\n",
- "xb = np.c_[np.ones((100,1)), x]\n",
- "theta_linreg = np.linalg.inv(xb.T.dot(xb)).dot(xb.T).dot(y)\n",
- "print(theta_linreg)\n",
- "sgdreg = SGDRegressor(n_iter = 50, penalty=None, eta0=0.1)\n",
- "sgdreg.fit(x,y.ravel())\n",
- "print(sgdreg.intercept_, sgdreg.coef_)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
"## Polynomial Regression"
]
},
{
"cell_type": "code",
- "execution_count": 10,
- "metadata": {},
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": false
+ },
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -1644,72 +1495,131 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Ridge and Lasso Regression"
+ "\n",
+ "## Linking the regression analysis with a statistical interpretation\n",
+ "\n",
+ "Before we proceed, and to link with our discussions of Bayesian statistics to come, it is useful the derive the standard regression analysis equations using a statistical interpretation. This allows us also to derive quantities like the variance and other expectation values in a rather straightforward way. \n",
+ "\n",
+ "It is assumed that $\\varepsilon_i\n",
+ "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n",
+ "independent, i.e.:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*} \n",
+ "\\mbox{Cov}(\\varepsilon_{i_1},\n",
+ "\\varepsilon_{i_2}) & = \\left\\{ \\begin{array}{lcc} \\sigma^2 & \\mbox{if}\n",
+ "& i_1 = i_2, \\\\ 0 & \\mbox{if} & i_1 \\not= i_2. \\end{array} \\right.\n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The randomness of $\\varepsilon_i$ implies that\n",
+ "$\\mathbf{Y}_i$ is also a random variable. In particular,\n",
+ "$\\mathbf{Y}_i$ is normally distributed, because $\\varepsilon_i \\sim\n",
+ "\\mathcal{N}(0, \\sigma^2)$ and $\\mathbf{X}_{i,\\ast} \\, \\beta$ is a\n",
+ "non-random scalar. To specify the parameters of the distribution of\n",
+ "$\\mathbf{Y}_i$ we need to calculate its first two moments. \n",
+ "\n",
+ "## Expectation value and variance\n",
+ "\n",
+ "Its expectation equals:"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*} \n",
+ "\\mathbb{E}(Y_i) & =\n",
+ "\\mathbb{E}(\\mathbf{X}_{i, \\ast} \\, \\beta) + \\mathbb{E}(\\varepsilon_i)\n",
+ "\\, \\, \\, = \\, \\, \\, \\mathbf{X}_{i, \\ast} \\, \\beta, \n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "while\n",
+ "its variance is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*} \\mbox{Var}(Y_i) & = \\mathbb{E} \\{ [Y_i\n",
+ "- \\mathbb{E}(Y_i)]^2 \\} \\, \\, \\, = \\, \\, \\, \\mathbb{E} ( Y_i^2 ) -\n",
+ "[\\mathbb{E}(Y_i)]^2 \\\\ & = \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\,\n",
+ "\\beta + \\varepsilon_i )^2] - ( \\mathbf{X}_{i, \\ast} \\, \\beta)^2 \\\\ &\n",
+ "= \\mathbb{E} [ ( \\mathbf{X}_{i, \\ast} \\, \\beta)^2 + 2 \\varepsilon_i\n",
+ "\\mathbf{X}_{i, \\ast} \\, \\beta + \\varepsilon_i^2 ] - ( \\mathbf{X}_{i,\n",
+ "\\ast} \\, \\beta)^2 \\\\ & = ( \\mathbf{X}_{i, \\ast} \\, \\beta)^2 + 2\n",
+ "\\mathbb{E}(\\varepsilon_i) \\mathbf{X}_{i, \\ast} \\, \\beta +\n",
+ "\\mathbb{E}(\\varepsilon_i^2 ) - ( \\mathbf{X}_{i, \\ast} \\, \\beta)^2 \n",
+ "\\\\ & = \\mathbb{E}(\\varepsilon_i^2 ) \\, \\, \\, = \\, \\, \\,\n",
+ "\\mbox{Var}(\\varepsilon_i) \\, \\, \\, = \\, \\, \\, \\sigma^2. \n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Hence, $Y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\beta, \\sigma^2)$. \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## The singular value decompostion\n",
+ "\n",
+ "\n",
+ "A general\n",
+ "$m\\times n$ matrix $\\hat{A}$ can be written in terms of a diagonal\n",
+ "matrix $\\hat{D}$ of dimensionality $n\\times n$ and two orthognal\n",
+ "matrices $\\hat{U}$ and $\\hat{V}$, where the first has dimensionality\n",
+ "$m \\times m$ and the last dimensionality $n\\times n$. \n",
+ "We have then"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\hat{A} = \\hat{U}\\hat{D}\\hat{V}^T\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Code examples for Ridge and Lasso Regression"
]
},
{
"cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "X_train: (37, 1)\n",
- "y_train: (37,)\n",
- "X_test: (13, 1)\n",
- "y_test: (13,)\n",
- "------------------------------------\n",
- "Ordinary Least Squares\n",
- "Prediction Shape: (13,)\n",
- "Coefficients: \n",
- " [0.54090544]\n",
- "Mean squared error: 4.92\n",
- "Variance score: 0.11\n"
- ]
- },
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "/usr/local/lib/python3.7/site-packages/sklearn/linear_model/base.py:509: RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver.\n",
- " linalg.lstsq(X, y)\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "------------------------------------\n",
- "Ridge Regression\n",
- "Ridge Coefficient: [0.54089262]\n",
- "Ridge Intercept: 4.6076494432587065\n",
- "------------------------------------\n",
- "Lasso\n",
- "Lasso Coefficient: [0.54002825]\n",
- "Lasso Intercept: 4.623208095009552\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
@@ -1789,8 +1699,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## The singular value decompostion\n",
- "How can we use the singular value decomposition to find the parameters $\\beta_j$? More details will come. We first note that a general $m\\times n$ matrix $\\hat{A}$ can be written in terms of a diagonal matrix $\\hat{\\Sigma}$ of dimensionality $n\\times n$ and two orthognal matrices $\\hat{U}$ and $\\hat{V}$, where the first has dimensionality $m \\times n$ and the last dimensionality $n\\times n$. We have then"
+ "## From standard regression to Ridge regressions\n",
+ "\n",
+ "One of the typical problems we encounter with linear regression, in particular \n",
+ "when the matrix $\\hat{X}$ (our so-called design matrix) is high-dimensional, \n",
+ "are problems with near singular or singular matrices. The column vectors of $\\hat{X}$ \n",
+ "may be linearly dependent, normally referred to as super-collinearity. \n",
+ "This means that the matrix may be rank deficient and it is basically impossible to \n",
+ "to model the data using linear regression. As an example, consider the matrix"
]
},
{
@@ -1798,7 +1714,18 @@
"metadata": {},
"source": [
"$$\n",
- "\\hat{A} = \\hat{U}\\hat{\\Sigma}\\hat{V}\n",
+ "\\begin{align*}\n",
+ "\\mathbf{X} & = \\left[\n",
+ "\\begin{array}{rrr}\n",
+ "1 & -1 & 2\n",
+ "\\\\\n",
+ "1 & 0 & 1\n",
+ "\\\\\n",
+ "1 & 2 & -1\n",
+ "\\\\\n",
+ "1 & 1 & 0\n",
+ "\\end{array} \\right]\n",
+ "\\end{align*}\n",
"$$"
]
},
@@ -1806,39 +1733,414 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Add codes and discuss this in connection with lasso and ridge, show example where the standard inversion of a matrix fails and where SVD comes to rescue\n",
+ "The columns of $\\hat{X}$ are linearly dependent. We se this easily since the \n",
+ "the first column is the row-wise sum of the other two columns. The rank (more correct,\n",
+ "the column rank) of a matrix is the dimension of the space spanned by the\n",
+ "column vectors. Hence, the rank of $\\mathbf{X}$ is equal to the number\n",
+ "of linearly independent columns. In this particular case the matrix has rank 2.\n",
+ "\n",
+ "Super-collinearity of an $(n \\times p)$-dimensional design matrix $\\mathbf{X}$ implies\n",
+ "that the inverse of the matrix $\\hat{X}^T\\hat{x}$ (the matrix we needto invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\begin{align*}\n",
+ "\\hat{X} & = \\left[\n",
+ "\\begin{array}{rr}\n",
+ "1 & -1\n",
+ "\\\\\n",
+ "1 & -1\n",
+ "\\end{array} \\right].\n",
+ "\\end{align*}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We see easily that $\\mbox{det}(\\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \\times (-1) - 1 \\times (-1) = 0$. Hence, $\\mathbf{X}$ is singular and its inverse is undefined.\n",
+ "This is equivalent to saying that the matrix $\\hat{X}$ has at least an eigenvalue which is zero.\n",
+ "\n",
+ "## Fixing the singularity\n",
+ "\n",
+ "If our design matrix $\\hat{X}$ which enters the linear regression problem"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "\n",
+ "$$\n",
+ "\\begin{equation}\n",
+ "\\hat{\\beta} = (\\hat{X}^{T} \\hat{X})^{-1} \\hat{X}^{T} \\hat{y},\n",
+ "\\label{_auto1} \\tag{1}\n",
+ "\\end{equation}\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "has linearly dependent column vectors, we will not be able to compute the inverse\n",
+ "of $\\hat{X}^T\\hat{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n",
+ "The estimators are only well-defined if $(\\hat{X}^{T}\\hat{X})^{-1}$ exits. \n",
+ "This is more likely to happen when the matrix $\\hat{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n",
+ "the regression parameters $\\beta_i$ cannot be estimated.\n",
+ "\n",
+ "The *ad hoc* approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\hat{X}^{T} \\hat{X} \\rightarrow \\hat{X}^{T} \\hat{X}+\\lambda \\hat{I},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where $\\hat{I}$ is the identity matrix.\n",
"\n",
"\n",
- "## Lasso and Ridge regression\n",
"\n",
- "Discuss the mathematics here\n",
+ "## A second-order polynomial with Ridge and Lasso"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt\n",
+ "from sklearn.linear_model import Ridge\n",
+ "from sklearn.metrics import r2_score\n",
"\n",
- "## Logistic regression\n",
- "Add discussion about classification versus regression, show examples of more than two cases and why regression is not the best approach. Motivate for k-nearest neighbors\n",
+ "np.random.seed(4155)\n",
"\n",
- "Add examples on classification problems"
+ "n_samples = 100\n",
+ "\n",
+ "x = np.random.rand(n_samples,1)\n",
+ "y = 5*x*x + 0.1*np.random.rand(n_samples,1)\n",
+ "\n",
+ "# Centering x and y.\n",
+ "x_ = x - np.mean(x)\n",
+ "y_ = y - np.mean(y) # beta_0 = mean(y)\n",
+ "\n",
+ "X = np.c_[np.ones((n_samples,1)), x, x**2]\n",
+ "X_ = np.c_[x_, x_**2]\n",
+ "\n",
+ "\n",
+ "### 1.\n",
+ "lmb_values = [1e-4, 1e-3, 1e-2, 10, 1e2, 1e4]\n",
+ "num_values = len(lmb_values)\n",
+ "\n",
+ "## Ridge-regression of centered and not centered data\n",
+ "beta_ridge = np.zeros((3,num_values))\n",
+ "beta_ridge_centered = np.zeros((3,num_values))\n",
+ "\n",
+ "I3 = np.eye(3)\n",
+ "I2 = np.eye(2)\n",
+ "\n",
+ "for i,lmb in enumerate(lmb_values):\n",
+ " beta_ridge[:,i] = (np.linalg.inv( X.T @ X + lmb*I3) @ X.T @ y).flatten()\n",
+ " beta_ridge_centered[1:,i] = (np.linalg.inv( X_.T @ X_ + lmb*I2) @ X_.T @ y_).flatten()\n",
+ "\n",
+ "# sett beta_0 = np.mean(y)\n",
+ "beta_ridge_centered[0,:] = np.mean(y)\n",
+ "\n",
+ "## OLS (ordinary least squares) solution \n",
+ "beta_ls = np.linalg.inv( X.T @ X ) @ X.T @ y\n",
+ "\n",
+ "## Evaluate the models\n",
+ "pred_ls = X @ beta_ls\n",
+ "pred_ridge = X @ beta_ridge\n",
+ "pred_ridge_centered = X_ @ beta_ridge_centered[1:] + beta_ridge_centered[0,:]\n",
+ "\n",
+ "## Plot the results\n",
+ "\n",
+ "# Sorting\n",
+ "sort_ind = np.argsort(x[:,0])\n",
+ "\n",
+ "x_plot = x[sort_ind,0]\n",
+ "x_centered_plot = x_[sort_ind,0]\n",
+ "\n",
+ "pred_ls_plot = pred_ls[sort_ind,0]\n",
+ "pred_ridge_plot = pred_ridge[sort_ind,:]\n",
+ "pred_ridge_centered_plot = pred_ridge_centered[sort_ind,:]\n",
+ "\n",
+ "# Plott not centered\n",
+ "plt.plot(x_plot,pred_ls_plot,label='ls')\n",
+ "\n",
+ "for i in range(num_values):\n",
+ " plt.plot(x_plot,pred_ridge_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])\n",
+ "\n",
+ "plt.plot(x,y,'ro')\n",
+ "\n",
+ "plt.title('linear regression on un-centered data')\n",
+ "plt.legend()\n",
+ "\n",
+ "# Plott centered\n",
+ "plt.figure()\n",
+ "\n",
+ "for i in range(num_values):\n",
+ " plt.plot(x_centered_plot,pred_ridge_centered_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])\n",
+ "\n",
+ "plt.plot(x_,y,'ro')\n",
+ "\n",
+ "plt.title('linear regression on centered data')\n",
+ "plt.legend()\n",
+ "\n",
+ "\n",
+ "# 2.\n",
+ "\n",
+ "pred_ridge_scikit = np.zeros((n_samples,num_values))\n",
+ "for i,lmb in enumerate(lmb_values):\n",
+ " pred_ridge_scikit[:,i] = (Ridge(alpha=lmb,fit_intercept=False).fit(X,y).predict(X)).flatten() # fit_intercept=False fordi bias er allerede i X\n",
+ "\n",
+ "plt.figure()\n",
+ "\n",
+ "plt.plot(x_plot,pred_ls_plot,label='ls')\n",
+ "\n",
+ "for i in range(num_values):\n",
+ " plt.plot(x_plot,pred_ridge_scikit[sort_ind,i],label='scikit-ridge, lmb=%g'%lmb_values[i])\n",
+ "\n",
+ "plt.plot(x,y,'ro')\n",
+ "plt.legend()\n",
+ "plt.title('linear regression using scikit')\n",
+ "\n",
+ "plt.show()\n",
+ "\n",
+ "### R2-score of the results\n",
+ "for i in range(num_values):\n",
+ " print('lambda = %g'%lmb_values[i])\n",
+ " print('r2 for scikit: %g'%r2_score(y,pred_ridge_scikit[:,i]))\n",
+ " print('r2 for own code, not centered: %g'%r2_score(y,pred_ridge[:,i]))\n",
+ " print('r2 for own, centered: %g\\n'%r2_score(y,pred_ridge_centered[:,i]))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Fitting vs. predicting when data is in the model class\n",
+ "\n",
+ "We start by considering the case\n",
+ "$f(x)=2x$.\n",
+ "\n",
+ "Then the data is clearly generated by a model that is contained within\n",
+ "all three model classes we are using to make predictions (linear\n",
+ "models, third order polynomials, and tenth order polynomials).\n",
+ "\n",
+ "Run the code for the following cases:\n",
+ "\n",
+ "1. For $f(x)=2x$ , $Ntrain=10$ and $\\sigma =0$ (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when $x \\in [0,1]$ . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?\n",
+ "\n",
+ "2. Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?\n",
+ "\n",
+ "3. Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example $x \\in [0,1.2]$ ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?\n",
+ "\n",
+ "4. Repeat the above for $f(x)=2x$ , $Ntrain=10$ , and $\\sigma=1$ . What changes?\n",
+ "\n",
+ "Repeat the exercises above for $f(x)=2x$ , $Ntrain=100$ , and $\\sigma=1$ . What changes?\n",
+ "Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.\n",
+ "\n",
+ "\n",
+ "## Fitting versus predicting when data is not in the model class\n",
+ "\n",
+ "Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider $f(x)=2x-10x^5+15x^{10}$ . Notice that the for linear and third-order polynomial the true model $f(x)$ is not contained in model class.\n",
+ "\n",
+ "1. Do better fits lead to better predictions?\n",
+ "\n",
+ "2. What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points $Ntrain$ and $\\sigma$?\n",
+ "\n",
+ "Summarize what you think you learned about the relationship of knowing the true model class and predictive power.\n",
+ "\n",
+ "## The code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\n",
+ "import sklearn as sk\n",
+ "from sklearn import datasets, linear_model\n",
+ "from sklearn.preprocessing import PolynomialFeatures\n",
+ "\n",
+ "import matplotlib as mpl\n",
+ "from matplotlib import pyplot as plt\n",
+ "\n",
+ "%matplotlib notebook\n",
+ "\n",
+ "# The Training Data\n",
+ "\n",
+ "N_train=100\n",
+ "\n",
+ "sigma_train=1;\n",
+ "\n",
+ "# Train on integers\n",
+ "x=np.linspace(0.05,0.95,N_train)\n",
+ "# Draw random noise\n",
+ "s = sigma_train*np.random.randn(N_train)\n",
+ "\n",
+ "#linear\n",
+ "y=2*x+s\n",
+ "\n",
+ "#Tenth Order\n",
+ "#y=2*x-10*x**5+15*x**10+s\n",
+ "\n",
+ "p1=plt.plot(x,y, \"o\",ms=15, label='Training')\n",
+ "\n",
+ "#Linear Regression\n",
+ "# Create linear regression object\n",
+ "clf = linear_model.LinearRegression()\n",
+ "\n",
+ "# Train the model using the training sets\n",
+ "clf.fit(x[:, np.newaxis], y)\n",
+ "# The coefficients\n",
+ "\n",
+ "xplot=np.linspace(0.02,0.98,200)\n",
+ "linear_plot=plt.plot(xplot, clf.predict(xplot[:, np.newaxis]),label='Linear')\n",
+ "\n",
+ "#Polynomial Regression\n",
+ "\n",
+ "\n",
+ "poly3 = PolynomialFeatures(degree=3)\n",
+ "X = poly3.fit_transform(x[:,np.newaxis])\n",
+ "clf3 = linear_model.LinearRegression()\n",
+ "clf3.fit(X,y)\n",
+ "\n",
+ "\n",
+ "Xplot=poly3.fit_transform(xplot[:,np.newaxis])\n",
+ "poly3_plot=plt.plot(xplot, clf3.predict(Xplot), label='Poly 3')\n",
+ "\n",
+ "\n",
+ "\n",
+ "#poly5 = PolynomialFeatures(degree=5)\n",
+ "#X = poly5.fit_transform(x[:,np.newaxis])\n",
+ "#clf5 = linear_model.LinearRegression()\n",
+ "#clf5.fit(X,y)\n",
+ "\n",
+ "#Xplot=poly5.fit_transform(xplot[:,np.newaxis])\n",
+ "#plt.plot(xplot, clf5.predict(Xplot), 'r--',linewidth=1)\n",
+ "\n",
+ "poly10 = PolynomialFeatures(degree=10)\n",
+ "X = poly10.fit_transform(x[:,np.newaxis])\n",
+ "clf10 = linear_model.LinearRegression()\n",
+ "clf10.fit(X,y)\n",
+ "\n",
+ "Xplot=poly10.fit_transform(xplot[:,np.newaxis])\n",
+ "poly10_plot=plt.plot(xplot, clf10.predict(Xplot), label='Poly 10')\n",
+ "\n",
+ "axes = plt.gca()\n",
+ "axes.set_ylim([-7,7])\n",
+ "\n",
+ "handles, labels=axes.get_legend_handles_labels()\n",
+ "plt.legend(handles,labels, loc='lower center')\n",
+ "plt.xlabel(\"$x$\")\n",
+ "plt.ylabel(\"$y$\")\n",
+ "Title=\"$N=$\"+str(N_train)+\", $\\sigma=$\"+str(sigma_train)\n",
+ "plt.title(Title+\" (train)\")\n",
+ "plt.tight_layout()\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "## Generating test data"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "# Generate Test Data\n",
+ "\n",
+ "#Number of test data\n",
+ "N_test=20\n",
+ "\n",
+ "sigma_test=sigma_train\n",
+ "\n",
+ "max_x=1.2\n",
+ "x_test=max_x*np.random.random(N_test)\n",
+ "# Draw random noise\n",
+ "s_test = sigma_test*np.random.randn(N_test)\n",
+ "\n",
+ "#Linear\n",
+ "y_test=2*x_test+s_test\n",
+ "#Tenth order\n",
+ "#y_test=2*x_test-10*x_test**5+15*x_test**10+s_test\n",
+ "\n",
+ "#Make design matrices for prediction\n",
+ "x_plot=np.linspace(0,max_x, 200)\n",
+ "X3 = poly3.fit_transform(x_plot[:,np.newaxis])\n",
+ "X10 = poly10.fit_transform(x_plot[:,np.newaxis])\n",
+ "\n",
+ "%matplotlib notebook\n",
+ "\n",
+ "fig = plt.figure() \n",
+ "p1=plt.plot(x_test,y_test.transpose(), 'o', ms=12, label='data')\n",
+ "p2=plt.plot(x_plot,clf.predict(x_plot[:,np.newaxis]), label='linear')\n",
+ "p3=plt.plot(x_plot,clf3.predict(X3), label='3rd order')\n",
+ "p10=plt.plot(x_plot,clf10.predict(X10), label='10th order')\n",
+ "\n",
+ "\n",
+ "plt.legend(loc=2)\n",
+ "plt.xlabel('$x$')\n",
+ "plt.ylabel('$y$')\n",
+ "plt.legend(loc='best')\n",
+ "plt.title(Title+\" (pred.)\")\n",
+ "plt.tight_layout()\n",
+ "plt.show()\n",
+ "\n",
+ "#Linear Filename\n",
+ "#filename_test=Title+\"pred-linear.pdf\"\n",
+ "#Tenth Order Filename\n",
+ "#filename_test=Title+\"pred-o10.pdf\"\n",
+ "#plt.savefig(filename_test)\n",
+ "#plt.ylim((-6,12))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Lasso regression\n",
+ "\n",
+ "\n",
+ "## Logistic regression"
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.7.0"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 2
}
diff --git a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz
index 511f8a4c0..11e567611 100644
Binary files a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz and b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf
index 9a78bb544..c60178739 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf and b/doc/pub/Regression/pdf/Regression-beamer-handouts2x3.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-beamer.pdf b/doc/pub/Regression/pdf/Regression-beamer.pdf
index 5d6b26295..84cdc8a34 100644
Binary files a/doc/pub/Regression/pdf/Regression-beamer.pdf and b/doc/pub/Regression/pdf/Regression-beamer.pdf differ
diff --git a/doc/pub/Regression/pdf/Regression-minted.pdf b/doc/pub/Regression/pdf/Regression-minted.pdf
index d21efc3c3..e2ebb70a7 100644
Binary files a/doc/pub/Regression/pdf/Regression-minted.pdf and b/doc/pub/Regression/pdf/Regression-minted.pdf differ
diff --git a/doc/src/Regression/Regression.do.txt b/doc/src/Regression/Regression.do.txt
index 216fda678..6d7b7b469 100644
--- a/doc/src/Regression/Regression.do.txt
+++ b/doc/src/Regression/Regression.do.txt
@@ -17,6 +17,9 @@ A regression model aims at finding a likelihood function $p(y\vert \hat{x})$, th
The goal of the regression analysis is to extract/exploit relationship between $y_i$ and $\hat{x}_i$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions .
!eblock
+!split
+===== Regression analysis, overarching aims II =====
+!bblock
Consider an experiment in which $p$ characteristics of $n$ samples are
@@ -30,26 +33,14 @@ $\mathbf{X}$ through a functional relationship like $Y_i =
f(\mathbf{X}_{i,\ast})$. When no prior knowledge on the form of
$f(\cdot)$ is available, it is common to assume a linear relationship
between $\mathbf{X}$ and $\mathbf{Y}$. This assumption gives rise to
-the *linear regression model*:
-
-In model (ref{form.linRegressionModel}) $\beta = (\beta_1, \ldots,
+the *linear regression model* where $\beta = (\beta_1, \ldots,
\beta_p)^{\top}$ is the *regression parameter*. The parameter
$\beta_j$, $j=1, \ldots, p$, represents the effect size of covariate
$j$ on the response. That is, for each unit change in covariate $j$
(while keeping the other covariates fixed) the observed change in the
-response is equal to $\beta_j$. The second summand on the right-hand
-side of the model, $\varepsilon_i$, is referred to as the error. It
-represents the part of the response not explained by the functional
-part $\mathbf{X}_{i,\ast} \, \beta$ of the model
-(ref{form.linRegressionModel}). In contrast to the functional part,
-which is considered to be systematic (i.e. non-random), the error is
-assumed to be random. Consequently, $Y_{i_1,\ast}$ need not equal
-$Y_{i_2,\ast}$ for $i_1 \not= i_2$, even if $\mathbf{X}_{i_1,\ast}=
-\mathbf{X}_{i_2,\ast}$. To complete the formulation of model
-(ref{form.linRegressionModel}) we need to specify the probability
-distribution of $\varepsilon_i$.
-
+response is equal to $\beta_j$.
+!eblock
!split
===== General linear models =====
@@ -900,225 +891,49 @@ Before we proceed, and to link with our discussions of Bayesian statistics to co
It is assumed that $\varepsilon_i
\sim \mathcal{N}(0, \sigma^2)$ and the $\varepsilon_{i}$ are
-independent, i.e.: \begin{align*} \mbox{Cov}(\varepsilon_{i_1},
+independent, i.e.:
+!bt
+\begin{align*}
+\mbox{Cov}(\varepsilon_{i_1},
\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if}
& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right.
-\end{align*} The randomness of $\varepsilon_i$ implies that
+\end{align*}
+!et
+The randomness of $\varepsilon_i$ implies that
$\mathbf{Y}_i$ is also a random variable. In particular,
$\mathbf{Y}_i$ is normally distributed, because $\varepsilon_i \sim
\mathcal{N}(0, \sigma^2)$ and $\mathbf{X}_{i,\ast} \, \beta$ is a
non-random scalar. To specify the parameters of the distribution of
-$\mathbf{Y}_i$ we need to calculate its first two moments. Its
-expectation equals: \begin{align*} \mathbb{E}(Y_i) & =
+$\mathbf{Y}_i$ we need to calculate its first two moments.
+
+!split
+===== Expectation value and variance =====
+
+Its expectation equals:
+!bt
+\begin{align*}
+\mathbb{E}(Y_i) & =
\mathbb{E}(\mathbf{X}_{i, \ast} \, \beta) + \mathbb{E}(\varepsilon_i)
-\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, \end{align*} while
-its variance is: \begin{align*} \mbox{Var}(Y_i) & = \mathbb{E} \{ [Y_i
+\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
+\end{align*}
+!et
+while
+its variance is
+!bt
+\begin{align*} \mbox{Var}(Y_i) & = \mathbb{E} \{ [Y_i
- \mathbb{E}(Y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( Y_i^2 ) -
-[\mathbb{E}(Y_i)]^2 # \\ # & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
+[\mathbb{E}(Y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \beta)^2 \\ &
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \beta)^2 + 2 \varepsilon_i
\mathbf{X}_{i, \ast} \, \beta + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
-\ast} \, \beta)^2 \\ # & = ( \mathbf{X}_{i, \ast} \, \beta)^2 + 2
+\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \beta)^2 + 2
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \beta +
-\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \beta)^2 #
+\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \beta)^2
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
-\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. \end{align*}
-Hence, $Y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \beta,
-\sigma^2)$. This formulation (in terms of the normal distribution) is
-equivalent to the formulation of model (ref{form.linRegressionModel}),
-as both capture the assumptions involved: the linearity of the
-functional part and the normality of the error.
-
-Model (ref{form.linRegressionModel}) is often written in a more condensed matrix form:
-
-
-!bt
-\begin{align}
-\mathbf{Y} & = \mathbf{X} \, \beta + \vvarepsilon, label{form.linRegressionModelinMatrix}
-\end{align}
-!et
-where $\vvarepsilon = (\varepsilon_1, \varepsilon_2, \ldots, \varepsilon_n)^{\top}$ and distributed as $\vvarepsilon \sim \mathcal{N}(\mathbf{0}_{p}, \sigma^2 \mathbf{I}_{nn})$. As above model (ref{form.linRegressionModelinMatrix}) can be expressed as a multivariate normal distribution: $\mathbf{Y} \sim \mathcal{N}(\mathbf{X} \, \beta, \sigma^2 \mathbf{I}_{nn})$.
-
-Model (ref{form.linRegressionModelinMatrix}) is a so-called
-hierarchical model. This terminology emphasizes that $\mathbf{X}$ and
-$\mathbf{Y}$ are not on a par, they play different roles in the
-model. The former is used to explain the latter. In model
-(ref{form.linRegressionModel}) $\mathbf{X}$ is referred as the
-*explanatory* or *independent* variable, while the variable
-$\mathbf{Y}$ is generally referred to as the *response* or *dependent*
-variable.
-
-The covariates, the columns of $\mathbf{X}$, may themselves be
-random. To apply the linear model they are temporarily assumed
-fixed. The linear regression model is then to be interpreted as
-$\mathbf{Y} \, | \, \mathbf{X} \sim \mathcal{N}(\mathbf{X} \, \beta,
-\sigma^2 \mathbf{I}_{nn})$
-
-
-The linear regression model (ref{form.linRegressionModel}) involves the unknown parameters: $\beta$ and $\sigma^2$, which need to be learned from the data. The parameters of the regression model, $\beta$ and $\sigma^2$ are estimated by means of likelihood maximization. Recall that $Y_i \sim \mathcal{N}( \mathbf{X}_{i,\ast} \, \beta, \sigma^2)$ with corresponding density: $ f_{Y_i}(y_i) = (2 \, \pi \, \sigma^2)^{-1/2} \, \exp[ - (y_i - \mathbf{X}_{i\ast} \, \beta)^2 / 2 \sigma^2 ]$. The likelihood thus is:
-
-
-!bt
-\begin{align*}
-L(\mathbf{Y}, \mathbf{X}; \beta, \sigma^2) & =
-# f_{\mathbf{Y}}(\mathbf{y}) \, \, \, = \, \, \, \prod_{i=1}^n f_{Y_i}(y_i) \, \, \, = \, \, \,
-\prod_{i=1}^n \frac{1}{\sqrt{2 \, \pi} \, \sigma} \, \exp[ - (Y_i - \mathbf{X}_{i, \ast} \, \beta)^2 / 2 \sigma^2 ],
+\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
\end{align*}
!et
-
-in which the independence of the observations has been used. Because
-of the concavity of the logarithm, the maximization of the likelihood
-coincides with the maximum of the logarithm of the likelihood (called
-the log-likelihood). Hence, to obtain maximum likelihood (ML)
-estimates of the parameter it is equivalent to find the maximum of the
-log-likelihood. The log-likelihood is:
-
-!bt
-\begin{align*}
-\mathcal{L}(\mathbf{Y}, \mathbf{X}; \beta, \sigma^2) & =
-\log[ L(\mathbf{Y}, \mathbf{X}; \beta, \sigma^2) ]
-\, \, \, = \, \, \, % \log \Big[ \prod_{i=1}^n f_{Y_i}(y_i) \Big]
-# \, \, \, = \, \, \, \sum_{i=1}^n \log[ f_{Y_i}(y_i) ]
-# \\
-# & = \sum_{i=1}^n [ -\log(\sqrt{2 \, \pi} \, \sigma) - (y_i - \mathbf{X}_{i\ast} \, \beta)^2 / 2 \sigma^2 ]
-# \\
-# & =
--n \, \log(\sqrt{2 \, \pi} \, \sigma) - \frac{1}{ 2 \sigma^2} \sum_{i=1}^n (y_i - \mathbf{X}_{i, \ast} \, \beta)^2.
-\end{align*}
-After noting that $\sum_{i=1}^n (Y_i - \mathbf{X}_{i, \ast} \, \beta)^2 = \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2 \, \, \, = \, \, \, (\mathbf{Y} - \mathbf{X} \, \beta)^{\top} \, (\mathbf{Y} - \mathbf{X} \, \beta)$, the log-likelihood can be written as:
-\begin{align*}
-\mathcal{L}(\mathbf{Y}, \mathbf{X}; \beta, \sigma^2) & = -n \, \log(\sqrt{2 \, \pi} \, \sigma) - \frac{1}{ 2 \sigma^2} \, \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2.
-\end{align*}
-!et
-
-In order to find the maximum of the log-likelihood, take its derivate with respect to $\beta$:
-!bt
-\begin{align*}
-\frac{\partial }{\partial \, \beta} \mathcal{L}(\mathbf{Y}, \mathbf{X}; \beta, \sigma^2) & = - \frac{1}{ 2 \sigma^2} \, \frac{\partial }{\partial \, \beta} \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2 \, \, \, = \, \, \, \frac{1}{\sigma^2} \, \mathbf{X}^{\top} (\mathbf{Y} - \mathbf{X} \, \beta).
-\end{align*}
-!et
-
-Equate this derivative to zero gives the estimating equation for $\beta$:
-
-!bt
-\begin{align} label{form.normalEquation}
-\mathbf{X}^{\top} \mathbf{X} \, \beta & = \mathbf{X}^{\top} \mathbf{Y}.
-\end{align}
-!et
-
-Equation (ref{form.normalEquation}) is called to the *normal equation*. Pre-multiplication of both sides of the normal equation by $(\mathbf{X}^{\top} \mathbf{X})^{-1}$ now yields the ML estimator of the regression parameter: $\hat{\beta} = (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y}$, in which it is assumed that $(\mathbf{X}^{\top} \mathbf{X})^{-1}$ is well-defined.
-
-Along the same lines one obtains the ML estimator of the residual variance. Take the partial derivative of the log-likelihood with respect to $\sigma^2$:
-
-!bt
-\begin{align*}
-\frac{\partial }{\partial \, \sigma} \mathcal{L}(\mathbf{Y}, \mathbf{X}; \beta, \sigma^2) & = - \frac{n}{\sigma} + \frac{1}{\sigma^3} \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2.
-\end{align*}
-!et
-
-
-Equate the right-hand side to zero and solve for $\sigma^2$ to find
-$\hat{\sigma}^2 = \tfrac{1}{n} \| \mathbf{Y} - \mathbf{X} \, \beta
-\|^2_2$. In this expression $\beta$ is unknown and the ML estimate of
-$\beta$ is plugged-in. \\ \\ With explicit expressions of the ML
-estimators at hand, we can study their properties. The expectation of
-the ML estimator of the regression parameter $\beta$ is:
-
-!bt
-\begin{align*}
-\mathbb{E}(\hat{\beta}) & = \mathbb{E}[
-(\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y}]
-\, \, \, = \, \, \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \,
-\mathbf{X}^{\top} \mathbb{E}[ \mathbf{Y}] # \\ \, \, \, = \, \, \,
-(\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{X} \,
-\beta \, \, \, \, \, = \, \, \, \beta.
-\end{align*}
-!et
-
-Hence, the ML
-estimator of the regression coefficients is unbiased.
-
-The variance of the ML estimator of $\beta$ is:
-
-!bt
-\begin{align*}
-\mbox{Var}(\hat{\beta}) & = \mathbb{E} \{ [\hat{\beta} - \mathbb{E}(\hat{\beta})] [\hat{\beta} - \mathbb{E}(\hat{\beta})]^{\top} \}
-\\
-& = \mathbb{E} \{ [(\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y} - \beta] \, [(\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y} - \beta]^{\top} \}
-\\
-# & = \mathbb{E} \{ [(\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y}] \, [(\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y}]^{\top} \} - \beta \, \beta^{\top}
-# \\
-# & = \mathbb{E} \{ (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y} \, \mathbf{Y}^{\top} \, \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \} - \beta \, \beta^{\top}
-# \\
-& = (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{\top} \} \, \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} - \beta \, \beta^{\top}
-\\
-& = (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \, \{ \mathbf{X} \, \beta \, \beta^{\top} \, \mathbf{X}^{\top} + \SSigma \} \, \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} - \beta \, \beta^{\top}
-# \\
-# & = (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \beta \, \beta^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
-# \\
-# & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \beta \beta^T
-\\
-& = \beta \, \beta^{\top} + \sigma^2 \, (\mathbf{X}^{\top} \mathbf{X})^{-1} - \beta \, \beta^{\top}
-\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{\top} \mathbf{X})^{-1},
-\end{align*}
-!et
-
-in which we have used that $\mathbb{E} (\mathbf{Y} \mathbf{Y}^{\top})
-= \mathbf{X} \, \beta \, \beta^{\top} \, \mathbf{X}^{\top} + \sigma^2
-\, \mathbf{I}_{nn}$. From $\mbox{Var}(\hat{\beta}) = \sigma^2 \,
-(\mathbf{X}^{\top} \mathbf{X})^{-1}$, one obtains an estimate of the
-variance of the estimate of the $j$-th regression coefficient:
-$\hat{\sigma}^2 (\hat{\beta}_j ) = \hat{\sigma}^2 \sqrt{
-[(\mathbf{X}^{\top} \mathbf{X})^{-1}]_{jj} }$. This may be used to
-construct a confidence interval for the estimates or test the
-hypothesis $H_0: \beta_j = 0$. In the latter $\hat{\sigma}^2$ should
-not be the maximum likelihood estimator, as it is biased. It is then
-to be replaced by the residual sum-of-squares divided by $n-p$ rather
-than $n$.
-
-
-The prediction of $Y_i$, denoted $\widehat{Y}_i$, is the expected
-value of $Y_i$ according the linear regression model (with its
-parameters replaced by their estimates). The prediction of $Y_i$ thus
-equals $\mathbb{E}(Y_i; \hat{\beta}, \hat{\sigma}^2) = \mathbf{X}_{i,
-\ast} \hat{\beta}$. In matrix notation the prediction is:
-
-!bt
-\begin{align*}
-\widehat{\mathbf{Y}} & = \mathbf{X} \, \hat{\beta} \, \, \, = \, \, \, \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} \, \, \, := \, \, \, \mathbf{H} \mathbf{Y},
-\end{align*}
-!et
-
-where $\mathbf{H}$ is the *hat matrix*, as it `puts the hat' on
-$\mathbf{Y}$. Note that the hat matrix is a projection matrix,
-i.e. $\mathbf{H}^2 = \mathbf{H}$ for
-
-!bt
-\begin{align*}
-\mathbf{H}^2 & = \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \, \, \, = \, \, \, \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top}.
-\end{align*}
-!et
-
-Thus, the prediction $\widehat{\mathbf{Y}}$ is an orthogonal
-projection of $\mathbf{Y}$ onto the space spanned by the columns of
-$\mathbf{X}$.
-
-With $\widehat{\beta}$ available, an estimate of the errors
-$\hat{\varepsilon}_i$, dubbed the *residuals* are obtained via:
-
-!bt
-\begin{align*}
-\hat{\vvarepsilon} & = \mathbf{Y} - \widehat{\mathbf{Y}} \, \, \, = \, \, \, \mathbf{Y} - \mathbf{X} \, \hat{\beta} \, \, \, = \, \, \, \mathbf{Y} - \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} \, \, \, = \, \, \, [ \mathbf{I} - \mathbf{X} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} ] \, \mathbf{Y}.
-\end{align*}
-!et
-
-Thus, the residuals are a projection of $\mathbf{Y}$ onto the
-orthogonal complement of the space spanned by the columns of
-$\mathbf{X}$. The residuals are to be used in diagnostics,
-e.g. checking of the normality assumption by means of a normal
-probability plot.
-
-
+Hence, $Y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \beta, \sigma^2)$.
@@ -1145,8 +960,7 @@ We have then
-!split
-===== Lasso regression =====
+
@@ -1232,13 +1046,6 @@ plt.show()
-!split
-===== Logistic regression =====
-
-
-
-
-
@@ -1287,7 +1094,7 @@ that the inverse of the matrix $\hat{X}^T\hat{x}$ (the matrix we needto invert t
\end{align*}
!et
We see easily that $\mbox{det}(\hat{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0$. Hence, $\mathbf{X}$ is singular and its inverse is undefined.
-This equivalent to saying that the matrix $\hat{X}$ has at least an eigenvalue which is zero.
+This is equivalent to saying that the matrix $\hat{X}$ has at least an eigenvalue which is zero.
!split
===== Fixing the singularity =====
@@ -1304,7 +1111,7 @@ The estimators are only well-defined if $(\hat{X}^{T}\hat{X})^{-1}$ exits.
This is more likely to happen when the matrix $\hat{X}$ is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters $\beta_i$ cannot be estimated.
-The *ad hoc* which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change
+The *ad hoc* approach which was introduced in the 70s was simply to add a diagonal component to the matrix to invert, that is we change
!bt
\[
\hat{X}^{T} \hat{X} \rightarrow \hat{X}^{T} \hat{X}+\lambda \hat{I},
@@ -1312,1739 +1119,293 @@ The *ad hoc* which was introduced in the 70s was simply to add a diagonal compon
!et
where $\hat{I}$ is the identity matrix.
+
+
!split
-=====
-The matrix $\mathbf{X}^{\top} \mathbf{X}$ is of rank $n$, while $\beta$ is a vector of length $p$. Hence, while there are $p$ unknowns, the system of linear equations from which these are to be solved effectively comprises $n$ degrees of freedom. If $p > n$, the vector $\beta$ cannot uniquely be determined from this system of equations. To make this more specific let $U$ be the $n$-dimensional space spanned by the columns of $\mathbf{X}$ and the $p-n$-dimensional space $V$ be orthogonal complement of $U$, i.e. $V = U^{\perp}$. Then, $\mathbf{X} \mathbf{v} = \mathbf{0}_{p}$ for all $\mathbf{v} \in V$. So, $V$ is the non-trivial null space of $\mathbf{X}$. Consequently, as $\mathbf{X}^{\top} \mathbf{X} \mathbf{v} =
-\mathbf{X}^{\top} \mathbf{0}_{p} = \mathbf{0}_{n}$, the solution of the normal equations is:
-
-!bt
-\begin{align*}
-\hat{\beta} & = ( \mathbf{X}^{\top} \mathbf{X})^{-} \mathbf{X}^{\top} \mathbf{Y} + \mathbf{v} \qquad \mbox{for all } \mathbf{v} \in V,
-\end{align*}
-!et
-
-where $\mathbf{A}^{-}$ denotes the Moore-Penrose inverse of the matrix $\mathbf{A}$, which is defined as:
-
-!bt
-\begin{align*}
-\mathbf{A}^{-} & = \sum_{j=1}^p \nu_j^{-1} \, I_{\{ \nu_j \not= 0 \} } \, \mathbf{v}_j \, \mathbf{v}_j^{\top}.
-\end{align*}
-!et
-
-The solution of the normal equations is thus only determined up to an
-element from a non-trivial space $V$, and there is no unique estimator
-of the regression parameter.
-
-To obtain an estimate of the regression parameter $\beta$ when
-$\mathbf{X}$ is (close to) super-collinearity, cite{Hoer1970} proposed
-an ad-hoc fix to resolve the (almost) singularity of
-$\mathbf{X}^{\top} \mathbf{X}$. Simply replace $\mathbf{X}^{\top}
-\mathbf{X}$ by $\mathbf{X}^{\top} \mathbf{X} + \lambda
-\mathbf{I}_{pp}$ with $\lambda \in [0, \infty)$. The scalar $\lambda$
-is a tuning parameter, henceforth called the *penalty parameter*.
-
-
-
-Recall the super-collinear design matrix $\mathbf{X}$ of Example ref{example.supercollinearity}. Then, for (say) $\lambda = 1$:
-!bt
-\begin{align*}
-\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} & = \left(
-\begin{array}{rrr}
-5 & 2 & 2
-\\
-2 & 7 & -4
-\\
-2 & -4 & 7
-\end{array} \right).
-\end{align*}
-!et
-The eigenvalues of this matrix are 11, 7, and 1. Hence, $\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}$ has no zero eigenvalue and its inverse is well-defined.
-
-With the ad-hoc fix for the singularity of $\mathbf{X}^{\top} \mathbf{X}$, cite{Hoer1970} proceed to define the *ridge regression estimator*:
-
-
-!bt
-\begin{align} label{form.ridgeRegressionEstimator}
-\hat{\beta}(\lambda) & = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y},
-\end{align}
-!et
-
-for $\lambda \in [0, \infty)$. Clearly, this is - for $\lambda$
-strictly positive - a well-defined estimator, even if $\mathbf{X}$ is
-high-dimensional. However, each choice of $\lambda$ leads to a
-different ridge regression estimate. The set of all ridge regression
-estimates $\{ \hat{\beta}(\lambda) \, : \, \lambda \in [0, \infty) \}$
-is called the *solution* or *regularization path* of the ridge
-estimator.
-
-
-
-
-Recall the super-collinear design matrix $\mathbf{X}$ of Example ref{example.supercollinearity}. Suppose that the corresponding response vector is $\mathbf{Y} = (1.3, -0.5, 2.6, 0.9)^{\top}$.
-The ridge regression estimates for, e.g. $\lambda = 1, 2$, and $10$ are then:
-\begin{align*}
-\hat{\beta}(1) & = (0.614, 0.548, 0.066)^{\top},
-\\
-\hat{\beta}(2) & = (0.537, 0.490, 0.048)^{\top},
-\\
-\hat{\beta}(10) & = (0.269, 0.267, 0.002)^{\top}.
-\end{align*}
-The full solution path of the ridge estimator is plotted in Figure ref{fig.ridgeSolPathPlusVar}.
-
-
-Having obtained an estimate of the regression parameter $\beta$, one can define the fit $\widehat{\mathbf{Y}}$. It is defined analogous to the standard case:
-
-!bt
-\begin{align*}
-\widehat{\mathbf{Y}}(\lambda) & = \mathbf{X} \hat{\beta}(\lambda)
-\, \, \, = \, \, \, \mathbf{X} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\, \, \, := \, \, \, \mathbf{H}(\lambda) \mathbf{Y}.
-\end{align*}
-!et
-
-Previously, when using the ML estimator, the fit could be understood as a projection of $\mathbf{Y}$ onto the subspace spanned by the columns of $\mathbf{X}$. The fit $\widehat{\mathbf{Y}}(\lambda)$ corresponding to the ridge estimator is not a projection of $\mathbf{Y}$ onto $\mathbf{X}$ (confer Exercise ref{question.ridgeResidualsProjection} $\!$a). Consequently, the `ridge residuals' $\mathbf{Y} - \widehat{\mathbf{Y}}(\lambda)$ are not orthogonal to the fit $\widehat{\mathbf{Y}}(\lambda)$ (confer Exercise ref{question.ridgeResidualsProjection} $\!$b).
-
-
-======= Eigenvalue shrinkage =======
-
-The effect of the ridge penalty may also studied from the perspective of singular values. Let the singular value decomposition of the $(n \times p)$-dimensional design matrix $\mathbf{X}$ be:
-
-!bt
-\begin{align*}
-\mathbf{X} & = \mathbf{U}_x \mathbf{D}_x \mathbf{V}_x^{\top},
-\end{align*}
-!et
-
-where $\mathbf{D}_x$ an $(n \times n)$-dimensional diagonal matrix
-with the singular values, $\mathbf{U}_x$ an $(n \times n)$-dimensional
-matrix with columns containing the left singular vectors (denoted
-$\mathbf{u}_i$), and $\mathbf{V}_x$ a $(p \times n)$-dimensional
-matrix with columns containing the right singular vectors (denoted
-$\mathbf{v}_i$). The columns of $\mathbf{U}_x$ and $\mathbf{V}_x$ are
-orthogonal: $\mathbf{U}_x^{\top} \mathbf{U}_x = \mathbf{I}_{nn} =
-\mathbf{V}_x^{\top} \mathbf{V}_x$.
-
-The OLS estimator can then be rewritten in terms of the SVD-matrices as:
-
-
-\begin{align*}
-\hat{\beta} & = (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{U}_x \mathbf{D}_x \mathbf{V}_x^{\top})^{-1} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{D}_x^2 \mathbf{V}_x^{\top})^{-1} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}
-\\
-& = \mathbf{V}_x \mathbf{D}_x^{-2} \mathbf{V}_x^{\top} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}
-\\
-& = \mathbf{V}_x \mathbf{D}_x^{-2} \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y},
-\end{align*}
-where $\mathbf{D}_x^{-2} \mathbf{D}_x$ is not simplified further to emphasize the effect of the ridge penalty. Similarly, the ridge estimator can be rewritten in terms of the SVD-matrices as:
-\begin{align*}
-\hat{\beta} & = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{U}_x \mathbf{D}_x \mathbf{V}_x^{\top} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{D}_x^2 \mathbf{V}_x^{\top} + \lambda \mathbf{V}_x \mathbf{V}_x^{\top})^{-1} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}
-\\
-& = \mathbf{V}_x (\mathbf{D}_x^{2} + \lambda \mathbf{I}_{nn})^{-1} \mathbf{V}_x^{\top} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}
-\\
-& = \mathbf{V}_x (\mathbf{D}_x^{2} + \lambda \mathbf{I}_{nn})^{-1} \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}.
-\end{align*}
-Combining the two results and writing $(\mathbf{D}_x)_{jj} = d_{x,jj}$ we have:
-\begin{align*}
-d_{x,jj}^{-1} & \geq & \frac{d_{x,jj}}{d_{x,jj}^2 + \lambda} \qquad \mbox{ for all } \lambda > 0.
-\end{align*}
-Thus, the ridge penalty shrinks the singular values.
-\\
-\\
-Return to the problem of the super-collinearity of $\mathbf{X}$ in the high-dimensional setting ($p > n$). The super-collinearity implies the singularity of $\mathbf{X}^{\top} \mathbf{X}$ and prevents the calculation of the OLS estimator of the regression coefficients. However, $\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}$ is non-singular, with inverse:
-\begin{align*}
-(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} & = \sum_{j=1}^p (d_{x,jj}^2 + \lambda)^{-1} \mathbf{v}_j \mathbf{v}_j^{\top}.
-\end{align*}
-The right-hand side is well-defined for $\lambda > 0$.
-
-
-===== Principal components regression =====
-
-Principal component regression is a close relative to ridge regression that can also be applied in a high-dimensional context. Principal components regression explains the response not by the covariates themselves but by linear combinations of the covariates as defined by the principal components of $\mathbf{X}$. Let $\mathbf{U} \mathbf{D} \mathbf{V}^{\top}$ be the singular value decomposition of $\mathbf{X}$. The $i$-th principal component of $\mathbf{X}$ is then $\mathbf{X} \mathbf{v}_i$, henceforth denoted $\mathbf{z}_i$. Let $\mathbf{Z}_k$ be the matrix of the first $k$ principal components, i.e. $\mathbf{Z}_k = \mathbf{X} \mathbf{V}_k$ where $\mathbf{V}_k$ contains the first $k$ right singular vectors as columns. Principal components regression then amounts to regressing the response $\mathbf{Y}$ onto $\mathbf{Z}_{k}$, that is, it fits the model $\mathbf{Y} = \mathbf{Z}_k \ggamma + \vvarepsilon$. The least squares estimator of $\ggamma$ then is (with some abuse of notation):
-\begin{align*}
-\hat{\ggamma} & = (\mathbf{Z}_k^{\top} \mathbf{Z}_k)^{-1} \mathbf{Z}_k^{\top} \mathbf{Y} \, \, \, = \, \, \, (\mathbf{V}_k^{\top} \mathbf{X}^{\top} \mathbf{X} \mathbf{V}_k)^{-1} \mathbf{V}_k^{\top} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_k^{\top} \mathbf{V} \mathbf{D} \mathbf{U}^{\top} \mathbf{U} \mathbf{D} \mathbf{V}^{\top} \mathbf{V}_k)^{-1} \mathbf{V}_k^{\top} \mathbf{V} \mathbf{D} \mathbf{U}^{\top} \mathbf{Y}
-\\
-& = (\mathbf{I}_{kn} \mathbf{D}^2 \mathbf{I}_{nk})^{-1} \mathbf{I}_{kn} \mathbf{D} \mathbf{U}^{\top} \mathbf{Y}
-\\
-& = \mathbf{D}_k^{-2} \widetilde{\mathbf{D}}_k \mathbf{U}^{\top} \mathbf{Y} \, \, \, = \, \, \ \widetilde{\mathbf{D}}_k^{-1} \mathbf{U}^{\top} \mathbf{Y},
-\end{align*}
-where $\mathbf{D}_k$ and $\widetilde{\mathbf{D}}_k$ are submatrices of $\mathbf{D}$. The matrix $\mathbf{D}_k$ is obtained from $\mathbf{D}$ by removal of the last $n-p$ rows and columsn, while for $\widetilde{\mathbf{D}}_k$ only the last $n-k$ rows are dropped. Similarly, $\mathbf{I}_{kn}$ and $\mathbf{I}_{nk}$ are obtained from $\mathbf{I}_{nn}$ by removal of the last $n-k$ rows and columns, respectively. The principal component regression estimator of $\beta$ then is $\hat{\beta}_{\mbox{{\tiny pcr}}} = \mathbf{V}_k \widetilde{\mathbf{D}}_k^{-1} \mathbf{U}^{\top} \mathbf{Y}$. When $k$ is set equal to the column rank of $\mathbf{X}$, and thus to the rank of $\mathbf{X}^{\top} \mathbf{X}$, the
-principal component regression estimator $\hat{\beta}_{\mbox{{\tiny pcr}}} = (\mathbf{X}^{\top} \mathbf{X})^- \mathbf{X}^{\top} \mathbf{Y}$, where $\mathbf{A}^-$ denotes the Moore-Penrose inverse of matrix $\mathbf{A}$.
-
-The relation between ridge and principal component regression becomes clear when their corresponding estimators are written in terms of the singular value decomposition of $\mathbf{X}$:
-\begin{align*}
-\hat{\beta}_{\mbox{{\tiny pcr}}} & = \mathbf{V}_x (\mathbf{I}_{nk} \mathbf{D}_x \mathbf{I}_{kn})^{-1} \mathbf{U}_x^{\top} \mathbf{Y},
-\\
-\hat{\beta} (\lambda) & = \mathbf{V}_x (\mathbf{D}_x^2 + \lambda \mathbf{I}_{nn})^{-1} \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{Y}.
-\end{align*}
-Both operate on the singular values of the design matrix. But where principal component regression thresholds the singular values of $\mathbf{X}$, ridge regression shrinks them (depending on their size). Hence, one applies a discrete map on the singular values while the other a continuous one.
-
-
-
-
-
-======= Moments =======
-
-The first two moments of the ridge regression estimator are derived. Next the performance of the ridge regression estimator is studied in terms of the mean squared error, which combines the first two moments.
-
-
-===== Expectation =====
-
-The left panel of Figure ref{fig.ridgeSolPathPlusVar} shows ridge estimates of the regression parameters converging to zero as the penalty parameter tends to infinity. This behaviour of the ridge estimator does not depend on the specifics of the data set. To see this study the expectation of the ridge estimator:
-\begin{align*}
-\mathbb{E} \big[ \hat{\beta}(\lambda) \big] & = \mathbb{E} \big[ (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y} \big]
-\\
-& = \mathbb{E} \big[ (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{X}^{\top} \mathbf{Y} \big]
-\\
-& = \mathbb{E} \big[ (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) \, \hat{\beta} \big]
-\\
-& = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) \, \mathbb{E} ( \hat{\beta} )
-\\
-& = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) \, \beta.
-\end{align*}
-Clearly, $\mathbb{E} \big[ \hat{\beta}(\lambda) \big] \not= \beta$ for any $\lambda > 0$. Hence, the ridge estimator is biased.
-
-From the expression above it is clear that the expectation of the ridge estimator vanishes as $\lambda$ tends to infinity:
-\begin{align*}
-\lim_{\lambda \rightarrow \infty} \mathbb{E} \big[ \hat{\beta}(\lambda) \big] & = \lim_{\lambda \rightarrow \infty} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) \, \beta \, \, \, = \, \, \, \mathbf{0}_{p}.
-\end{align*}
-Hence, all regression coefficients are shrunken towards zero as the penalty parameter increases. This also holds for $\mathbf{X}$ with $p > n$. Furthermore, this behaviour is not strictly monotone in $\lambda$: $\lambda_{a} > \lambda_b$ does not necessarily imply $|\hat{\beta}_j (\lambda_a) | < |\hat{\beta}_j (\lambda_b) |$. Upon close inspection this can be witnessed from the ridge solution path of $\beta_3$ in Figure ref{fig.ridgeSolPathPlusVar}.
-
-
-\begin{example} *Orthonormal design matrix* label{example.orthoronormalDesign}
-\\
-Consider an orthonormal design matrix $\mathbf{X}$, i.e.:
-\begin{align*}
-\mathbf{X}^{\top} \mathbf{X} & = \mathbf{I}_{pp} \, \, \, = \, \, \, (\mathbf{X}^{\top} \mathbf{X})^{-1}.
-\end{align*}
-An example of an orthonormal design matrix would be:
-\begin{align*}
-\mathbf{X} & = \frac{1}{2} \left(
-\begin{array}{rr}
--1 & -1
-\\
--1 & 1
-\\
-1 & -1
-\\
-1 & 1
-\end{array} \right).
-\end{align*}
-This design matrix is orthonormal as $\mathbf{X}^{\top} \mathbf{X} = \mathbf{I}_{22}$, which is easily verified:
-\begin{align*}
-\mathbf{X}^{\top} \mathbf{X} & = \frac{1}{4}
-\left(
-\begin{array}{rrrr}
--1 & -1 & 1 & 1
-\\
--1 & 1 & -1 & 1
-\end{array} \right)
-\left(
-\begin{array}{rr}
--1 & -1
-\\
--1 & 1
-\\
-1 & -1
-\\
-1 & 1
-\end{array} \right) \, \, \, = \, \, \,
-\frac{1}{4}
-\left(
-\begin{array}{rr}
-4 & 0
-\\
-0 & 4
-\end{array} \right) \, \, \, = \, \, \, \mathbf{I}_{22}.
-\end{align*}
-In case of an orthonormal design matrix the relation between the OLS and ridge estimator is:
-\begin{align*}
-\hat{\beta}(\lambda) & = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\, \, \, = \, \, \, (\mathbf{I}_{pp} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (1 + \lambda)^{-1} \mathbf{I}_{pp} \mathbf{X}^{\top} \mathbf{Y}
-\qquad \, \, = \, \, \, (1 + \lambda)^{-1} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (1 + \lambda)^{-1} \hat{\beta}.
-\end{align*}
-Hence, the ridge estimator scales the OLS estimator by a factor. When taking the expectation on both sides, it is evident that the ridge estimator converges to zero as $\lambda \rightarrow \infty$.
-\end{example}
-
-
-
-===== Variance =====
-
-As for the ML estimate of the regression parameter $\beta$ of model (ref{form.linRegressionModelinMatrix}), we derive the second moment of the ridge estimator. Hereto define:
-\begin{align*}
-\mathbf{W}_{\lambda} & = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{X}.
-\end{align*}
-Using $\mathbf{W}_{\lambda}$ the ridge estimator $\hat{\beta}(\lambda)$ can be expressed as $\mathbf{W}_{\lambda} \hat{\beta}$ for:
-\begin{align*}
-\mathbf{W}_{\lambda} \hat{\beta} & = \mathbf{W}_{\lambda} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = \{ (\mathbf{X}^{\top} \mathbf{X})^{-1} [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ] \}^{-1} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = \hat{\beta}(\lambda).
-\end{align*}
-The linear operator $\mathbf{W}_{\lambda}$ thus transforms the ML estimator of the regression parameter into the ridge estimator.
-
-It is now easily seen that:
-\begin{align*}
-\mbox{Var}[ \hat{\beta}(\lambda) ] & = \mbox{Var}[ \mathbf{W}_{\lambda} \hat{\beta} ] \qquad \qquad \, \, \, \, \, \, = \, \, \, \mathbf{W}_{\lambda} \mbox{Var}[\hat{\beta} ] \mathbf{W}_{\lambda}^{\top}
-\\
-& = \sigma^2 \mathbf{W}_{\lambda} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{W}_{\lambda}^{\top} \, \, \, = \, \, \, \sigma^2 [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1}\}^{\top},
-\end{align*}
-in which we have used $\mbox{Var}(\mathbf{A} \mathbf{Y}) = \mathbf{A} \mbox{Var}( \mathbf{Y}) \mathbf{A}^{\top}$ for a non-random matrix $\mathbf{A}$, the fact that $\mathbf{W}_{\lambda}$ is non-random, and $ \mbox{Var}[\hat{\beta} ] = \sigma^2 (\mathbf{X}^{\top} \mathbf{X})^{-1}$.
-
-
-Like the expectation the variance of the ridge estimator vanishes as $\lambda$ tends to infinity:
-\begin{align*}
-\lim_{\lambda \rightarrow \infty} \mbox{Var} \big[ \hat{\beta}(\lambda) \big] & = \lim_{\lambda \rightarrow \infty} \sigma^2 \mathbf{W}_{\lambda} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{W}_{\lambda}^{\top} \, \, \, = \, \, \, \mathbf{0}_{pp}.
-\end{align*}
-Hence, the variance of the ridge regression coefficient estimates decreases towards zero as the penalty parameter becomes large. This is illustrated in the right panel of Figure ref{fig.ridgeSolPathPlusVar} for the data of Example ref{example.supercollinearity}.
-
-
-With an explicit expression of the variance of the ridge estimator at hand, we can compare it to that of the OLS estimator:
-\begin{align*}
-\mbox{Var}[ \hat{\beta} ] - \mbox{Var}[ \hat{\beta}(\lambda) ] & = \sigma^2 [(\mathbf{X}^{\top} \mathbf{X})^{-1} - \mathbf{W}_{\lambda} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{W}_{\lambda}^{\top} ]
-\\
-& = \sigma^2 \mathbf{W}_{\lambda} \{ [\mathbf{I} + \lambda (\mathbf{X}^{\top} \mathbf{X})^{-1} ] (\mathbf{X}^{\top} \mathbf{X})^{-1} [\mathbf{I} + \lambda (\mathbf{X}^{\top} \mathbf{X})^{-1} ]^{\top} - (\mathbf{X}^{\top} \mathbf{X})^{-1} \} \mathbf{W}_{\lambda}^{\top}
-\\
-& = \sigma^2 \mathbf{W}_{\lambda} [ 2 \, \lambda \, (\mathbf{X}^{\top} \mathbf{X})^{-2} + \lambda^2 (\mathbf{X}^{\top} \mathbf{X})^{-3} ] \mathbf{W}_{\lambda}^{\top}
-\\
-& = \sigma^2 [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} [ 2 \, \lambda \, \mathbf{I}_{pp} + \lambda^2 (\mathbf{X}^{\top} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1}\}^{\top}.
-\end{align*}
-The difference is non-negative definite as each component in the matrix product is non-negative definite. Hence, the variance of the ML estimator exceeds (in the positive definite ordering) that of the ridge estimator:
-
-!bt
-
-!bt
-\begin{align} label{form.VarInequalityMLandRidge}
-\mbox{Var}[ \hat{\beta} ] & \succeq & \mbox{Var}[ \hat{\beta}(\lambda) ],
-\end{align}
-!et
-!et
-with the inequality being strict if $\lambda > 0$. In other words, the variance of the ML estimator is larger than that of the ridge estimator (in the sense that their difference is non-negative definite). The variance inequality (ref{form.VarInequalityMLandRidge}) can be interpreted in terms of the stochastic behaviour of the estimator. This is illustrated by the next example.
-
-# original latex figure with scale=0.45, angle=0
-
-FIGURE: [varMLellipses.eps, width=400 frac=1.0] Level sets of the distribution of the ML (left panel) and ridge (right panel) regression estimators.} label{fig.varOLSandRidge
-# \afterpage{}
-
-
-\begin{example} *Variance comparison*
-\\
-Consider the design matrix:
-\begin{align*}
-\mathbf{X} & = \left(
-\begin{array}{rr}
--1 & 2
-\\
-0 & 1
-\\
-2 & -1
-\\
-1 & 0
-\end{array} \right).
-\end{align*}
-The variances of the ML and ridge (with $\lambda=1$) estimates of the regression coefficients then are:
-\begin{align*}
-\mbox{Var}(\hat{\beta}) & = \sigma^2 \left(
-\begin{array}{rr}
-0.3 & 0.2
-\\
-0.2 & 0.3
-\end{array} \right)
-\qquad \mbox{and} \qquad
-\mbox{Var}[\hat{\beta}(\lambda)] \, \, \, = \, \, \, \sigma^2 \left(
-\begin{array}{rr}
-0.1524 & 0.0698
-\\
-0.0698 & 0.1524
-\end{array} \right).
-\end{align*}
-These variances can be used to construct levels sets of the distribution of the estimates. The level sets that contain 50\%, 75\% and 95\% of the distribution of the ML and ridge estimates are plotted in Figure ref{fig.varOLSandRidge}. In line with inequality (ref{form.VarInequalityMLandRidge}) the level sets of the ridge estimate are smaller than that of the ML estimate: it thus varies less.
-\end{example}
-
-
-
-
-\begin{contexample}_ref{example.orthoronormalDesign_} *Orthonormal design matrix (continued)*
-\\
-Assume the design matrix $\mathbf{X}$ is orthonormal. Then, $\mbox{Var}[ \hat{\beta} ] = \sigma^2 \mathbf{I}_{pp}$ and
-\begin{align*}
-\mbox{Var}[ \hat{\beta}(\lambda) ] & = \sigma^2 \mathbf{W}_{\lambda} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{W}_{\lambda}^{\top}
-\, \, \, = \, \, \, \sigma^2 [\mathbf{I}_{pp} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{I}_{pp} \{ [\mathbf{I}_{pp} + \lambda \mathbf{I}_{pp} ]^{-1} \}^{\top}
-\, \, \, = \, \, \, \sigma^2 (1 + \lambda )^{-2} \mathbf{I}_{pp} .
-\end{align*}
-As the penalty parameter $\lambda$ is non-negative the former exceeds the latter. In particular, this expression vanishes as $\lambda \rightarrow \infty$.
-\end{contexample}
-
-# #if FORMAT in ("latex", "pdflatex")
-\noindent
-# #endif
-The full distribution of the ridge regression estimator is now known. The estimator, $\hat{\beta}(\lambda) = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}$ is a linear estimator, linear in $\mathbf{Y}$. As $\mathbf{Y}$ is normally distributed, so is $\hat{\beta}(\lambda)$. Moreover, the normal distribution is fully characterized by its first two moments, which are available. Hence:
-\begin{align*}
-\hat{\beta}(\lambda) & \sim & \mathcal{N} \big( (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{X} \, \beta, \sigma^2 [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1}\}^{\top} \big).
-\end{align*}
-Given $\lambda$ and $\beta$, the random behavior of the estimator is thus known.
-
-
-
-===== Mean squared error} label{sect:ridgeMSE =====
-Previously, we motivated the ridge estimator as an ad hoc solution to collinearity. An alternative motivation comes from studying the Mean Squared Error (MSE) of the ridge regression estimator: for a suitable choice of $\lambda$ the ridge regression estimator may outperform the ML regression estimator in terms of the MSE. Before we prove this, we first derive the MSE of the ridge estimator and quote some auxiliary results.
-
-Recall that (in general) for any estimator of a parameter $\theta$:
-\begin{align*}
-\mbox{MSE}( \hat{\theta} ) & = \mathbb{E} [ ( \hat{\theta} - \theta)^2 ]
-\, \, \, = \, \, \, \mbox{Var}( \hat{ \theta} ) + [\mbox{Bias} ( \hat{\theta} )]^2.
-\end{align*}
-Hence, the MSE is a measure of the quality of the estimator.
-
-The MSE of the ridge estimator is:
-
-!bt
-
-!bt
-\begin{align}
-\mbox{MSE}[\hat{\beta}(\lambda)] & = \mathbb{E} [ (\mathbf{W}_{\lambda} \, \hat{\beta} - \beta)^{\top} \, (\mathbf{W}_{\lambda} \, \hat{\beta} - \beta) ] \nonumber
-\\
-& = \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, \hat{\beta} ) - \mathbb{E} ( \beta^{\top} \, \mathbf{W}_{\lambda} \, \hat{\beta}) - \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \, \beta) + \mathbb{E} ( \beta^{\top} \beta) \nonumber
-\\
-& = \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, \hat{\beta} ) - \mathbb{E} ( \beta^{\top} \, \mathbf{W}_{\lambda}^{\top} \mathbf{W}_{\lambda} \, \hat{\beta}) - \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \, \mathbf{W}_{\lambda} \beta) + \mathbb{E} ( \beta^{\top} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, \beta ) \nonumber
-\\
-& & - \mathbb{E} ( \beta^{\top} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, \beta ) + \mathbb{E} ( \beta^{\top} \, \mathbf{W}_{\lambda}^{\top} \mathbf{W}_{\lambda} \, \hat{\beta}) + \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \, \mathbf{W}_{\lambda} \beta) \nonumber
-\\
-& & - \mathbb{E} ( \beta^{\top} \, \mathbf{W}_{\lambda} \, \hat{\beta}) - \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \, \beta) + \mathbb{E} ( \beta^{\top} \beta) \nonumber
-\\
-& = \mathbb{E} [ ( \hat{\beta} - \beta )^{\top} \mathbf{W}_{\lambda}^{\top} \, \mathbf{W}_{\lambda} \, (\hat{\beta} - \beta) ] \nonumber
-\\
-& & - \beta^{\top} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, \beta + \beta^{\top} \, \mathbf{W}_{\lambda}^{\top} \mathbf{W}_{\lambda} \, \beta + \beta^{\top} \mathbf{W}_{\lambda}^{\top} \, \mathbf{W}_{\lambda} \beta \nonumber
-\\
-& & - \beta^{\top} \, \mathbf{W}_{\lambda} \, \beta - \beta^{\top} \mathbf{W}_{\lambda}^{\top} \, \beta + \beta^{\top} \beta \nonumber
-\\
-& = \mathbb{E} \big\{ ( \hat{\beta} - \beta )^{\top} \mathbf{W}_{\lambda}^{\top} \, \mathbf{W}_{\lambda} \, (\hat{\beta} - \beta) \big\} + \beta^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp})^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp}) \, \beta \nonumber
-\\
-& = \sigma^2 \, \mbox{tr}\big\{ \mathbf{W}_{\lambda} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{W}_{\lambda}^{\top} \big\} + \beta^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp})^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp}) \, \beta. label{form.ridgeMSE}
-\end{align}
-!et
-!et
-In the last step we have used $\hat{\beta} \sim \mathcal{N}( \beta, \sigma^2 \, [\mathbf{X}^{\top} \mathbf{X}]^{-1} )$ and the expectation of the quadratic form of a multivariate random variable $\vvarepsilon \sim \mathcal{N}(\mmu_{\varepsilon}, \SSigma_{\varepsilon})$ for some nonrandom symmetric positive definite matrix $\LLambda$ is (cf. \citealt{Math1992}):
-\begin{align*}
-\mathbb{E} ( \vvarepsilon^{\top} \, \LLambda \, \vvarepsilon) & = \mbox{tr} ( \LLambda \, \SSigma_{\varepsilon}) + \mmu_{\varepsilon}^{\top} \, \LLambda \, \mmu_{\varepsilon},
-\end{align*}
-of course replacing $\vvarepsilon$ by $\hat{\beta}$ in this expectation. The first summand in the final derived expression for $\mbox{MSE}[\hat{\beta}(\lambda)]$ is the sum of the variances of the ridge estimator, while the second summand can be thought of the ``squared bias'' of the ridge estimator. In particular, $\lim_{\lambda \rightarrow \infty} \mbox{MSE}[\hat{\beta}(\lambda)] = \beta^{\top} \beta$, which is the squared biased for an estimator that equals zero (as does the ridge estimator in the limit).
-
-
-
-\begin{example} *Orthonormal design matrix*
-\\
-Assume the design matrix $\mathbf{X}$ is orthonormal. Then, $\mbox{MSE}[ \hat{\beta} ] = p \, \sigma^2$ and
-\begin{align*}
-\mbox{MSE}[ \hat{\beta}(\lambda) ] & = \frac{p \, \sigma^2}{ (1+ \lambda)^{2}} + \frac{\lambda^2}{ (1+ \lambda)^{2}} \beta^{\top} \beta.
-\end{align*}
-The latter achieves its minimum at: $\lambda = p \sigma^2 / \beta^{\top} \beta$.
-\end{example}
-
-
-
-The following theorem and proposition are required for the proof of the main result.
-
-\begin{theorem} *(Theorem 1 of \citealp{Theo1974*}) label{theo.Theobald1}
-\\
-Let $\hat{\ttheta}_1$ and $\hat{\ttheta}_2$ be (different) estimators of $\ttheta$ with second order moments:
-\begin{align*}
-\mathbf{M}_k & = \mathbb{E} [ (\hat{\ttheta}_k - \ttheta) (\hat{\ttheta}_k - \ttheta)^{\top} ] \qquad \mbox{for } k=1,2,
-\end{align*}
-and
-\begin{align*}
-\mbox{MSE}(\hat{\ttheta}_k) & = \mathbb{E} [ (\hat{\ttheta}_k - \ttheta)^{\top} \mathbf{A} (\hat{\ttheta}_k - \ttheta) ] \qquad \mbox{for } k=1,2,
-\end{align*}
-where $\mathbf{A} \succeq 0$. Then, $\mathbf{M}_1 - \mathbf{M}_2 \succeq 0$ if and only if $\mbox{MSE}(\hat{\ttheta}_1) - \mbox{MSE}(\hat{\ttheta}_2) \geq 0$ for all $\mathbf{A} \succeq 0$.
-\end{theorem}
-
-
-\begin{proposition} *(\citealp{Fare1976*}) label{prop.Farebrother}
-\\
-Let $\mathbf{A}$ be a $(p \times p)$-dimensional, positive definite matrix, $\mathbf{b}$ be a nonzero $p$ dimensional vector, and $c \in \mathbb{R}_+$. Then, $c \mathbf{A} - \mathbf{b} \mathbf{b}^{\top} \succ 0$ if and only if $\mathbf{b}^{\top} \mathbf{A}^{-1} \mathbf{b} > c$.
-\end{proposition}
-
-We are now ready to proof the main result, formalized as Theorem ref{theo.Theobald2}, that for some $\lambda$ the ridge regression estimator yields a lower MSE than the ML regression estimator.
-
-\begin{theorem} *(Theorem 2 of \citealp{Theo1974*}) label{theo.Theobald2}
-\\
-There exists $\lambda > 0$ such that $\mbox{MSE}[\hat{\beta}(\lambda)] < \mbox{MSE}[\hat{\beta}(0)] = \mbox{MSE}[\hat{\beta}]$.
-\end{theorem}
-
-\begin{proof}
-The second order moment matrix of the ridge estimator is:
-\begin{align*}
-\mathbf{M} (\lambda) & := & \mathbb{E} [ (\hat{\beta}(\lambda) - \beta) (\hat{\beta} (\lambda) - \beta)^{\top} ]
-\\
-& = \mathbb{E} \{ \hat{\beta}(\lambda) [\hat{\beta}(\lambda)]^{\top} \} - \mathbb{E} [ \hat{\beta}(\lambda) ] \{ \mathbb{E} [ \hat{\beta}(\lambda) ] \}^{\top} + \mathbb{E} [\hat{\beta} (\lambda) - \beta)] \{ \mathbb{E} [\hat{\beta} (\lambda) - \beta)] \}^{\top}
-\\
-& = \mbox{Var}[ \hat{\beta}(\lambda) ] + \mathbb{E} [\hat{\beta} (\lambda) - \beta)] \{ \mathbb{E} [\hat{\beta} (\lambda) - \beta)] \}^{\top}.
-\end{align*}
-Then:
-\begin{align*}
-\mathbf{M} ( 0 ) - \mathbf{M}(\lambda) & = \sigma^2 (\mathbf{X}^{\top} \mathbf{X})^{-1} - \sigma^2 \mathbf{W}_{\lambda} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{W}_{\lambda}^{\top} ]
-\\
-& & - (\mathbf{W}_{\lambda} - \mathbf{I}_{pp}) \beta \beta^{\top} (\mathbf{W}_{\lambda} -\mathbf{I}_{pp})^{\top}
-\\
-& = \sigma^2 \mathbf{W}_{\lambda} [ 2 \, \lambda \, (\mathbf{X}^{\top} \mathbf{X})^{-2} + \lambda^2 (\mathbf{X}^{\top} \mathbf{X})^{-3} ] \mathbf{W}_{\lambda}^{\top}
-\\
-& & - \lambda^2 [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} \beta \beta^{\top} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} \}^{\top}
-\\
-& = \sigma^2 [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} [ 2 \, \lambda \, \mathbf{I}_{pp} + \lambda^2 (\mathbf{X}^{\top} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} \}^{\top}
-\\
-& & - \lambda^2 [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} \beta \beta^{\top} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} \}^{\top}
-\\
-& = \lambda [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} [ 2 \, \sigma^2 \, \mathbf{I}_{pp} + \lambda \sigma^2 (\mathbf{X}^{\top} \mathbf{X})^{-1} - \lambda \beta \beta^{\top} ] \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}]^{-1} \}^{\top}.
-\end{align*}
-This is positive definite if and only if $ 2 \, \sigma^2 \, \mathbf{I}_{pp} + \lambda \sigma^2 (\mathbf{X}^{\top} \mathbf{X})^{-1} - \lambda \beta \beta^{\top} \succ 0$. Hereto it suffices to show that $2 \, \sigma^2 \, \mathbf{I}_{pp} - \lambda \beta \beta^{\top} \succ 0$. By Proposition ref{prop.Farebrother} this holds for $\lambda$ such that $2 \sigma^2 (\beta^{\top} \beta)^{-1} > \lambda$. For these $\lambda$, we thus have $\mathbf{M} ( 0 ) - \mathbf{M}(\lambda)$. Application of Theorem ref{theo.Theobald1} now concludes the proof.
-\end{proof}
-This result of cite{Theo1974} is generalized by cite{Fare1976} to the class of design matrices $\mathbf{X}$ with $\mbox{rank}(\mathbf{X}) < p$.
-\\
-\\
-Theorem ref{theo.Theobald2} can be used to illustrate that the ridge regression estimator strikes a balance between the bias and variance. This is illustrated in the left panel of Figure ref{fig.MSEridge}. For small $\lambda$, the variance of the ridge estimator dominates the MSE. This may be understood when realizing that in this domain of $\lambda$ the ridge estimator is close to the unbiased ML regression estimator. For large $\lambda$, the variance vanishes and the bias dominates the MSE. For small enough values of $\lambda$, the decrease in variance of the ridge regression estimator exceeds the increase in its bias. As the MSE is the sum of these two, the MSE first decreases as $\lambda$ moves away from zero. In particular, as $\lambda = 0$ corresponds to the ML regression estimator, the ridge regression estimator yields a lower MSE for these values of $\lambda$. In the right panel of Figure ref{fig.MSEridge} $\mbox{MSE}[ \hat{\beta}(\lambda)] < \mbox{MSE}[ \hat{\beta}(0)]$ for $\lambda < 7$ (roughly) and the ridge estimator outperforms the ML estimator.
-# original latex figure with scale=0.45, angle=0
-
-FIGURE: [MSE_ridge.eps, width=400 frac=1.0] Left panel: mean squared error, and its `bias' and `variance' parts, of the ridge regression estimator (for artificial data). Right panel: mean squared error of the ridge and ML estimator of the regression coefficient vector (for the same artificial data).} label{fig.MSEridge
-# \afterpage{}
-\\
-\\
-# #if FORMAT in ("latex", "pdflatex")
-\noindent
-# #endif
-Besides another motivation behind the ridge regression estimator, the use of Theorem ref{theo.Theobald2} is limited. The optimal choice of $\lambda$ depends on the quantities $\beta$ and $\sigma^2$. These are unknown in practice. Then, the penalty parameter is chosen in a data-driven fashion by means of cross-validation (see Section ref{subsect.crossvalidation}).
-\\
-\\
-\begin{remark} \mbox{ }
-\\
-Theorem ref{theo.Theobald2} can also be used to conclude on the biasedness of the ridge regression estimator. The Gauss-Markov theorem \citep{Rao1973} states (under some assumptions) that the ML regression estimator is the best linear unbiased estimator (BLUE) with the smallest MSE. As the ridge regression estimator is a linear estimator and outperforms (in terms of MSE) this ML estimator, it must be biased (for it would otherwise refute the Gauss-Markov theorem).
-\end{remark}
-
-
-
-
-======= Constrained estimation} label{sect.constrainedEstimation =======
-The ad-hoc fix of cite{Hoer1970} to super-collinearity of the design matrix (and, consequently the singularity of the matrix $\mathbf{X}^{\top} \mathbf{X}$) has been motivated post-hoc.
-The ridge estimator minimizes the *ridge loss function*, which is defined as:
-
-!bt
-
-!bt
-\begin{align}
-\mathcal{L}_{\mbox{{\footnotesize ridge}}}(\beta; \lambda) & = \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2 + \lambda \| \beta \|^2_2
-\, \, \, = \, \, \, \sum_{i=1}^n (Y_i - \mathbf{X}_{i\ast} \, \beta)^2 + \lambda \sum_{j=1}^p \beta_j^2. label{form.ridgeLossFunction}
-\end{align}
-!et
-!et
-This loss function is the traditional sum-of-squares augmented with a *penalty*. The particular form of the penalty, $\lambda \| \beta \|^2_2$ is referred to as the *ridge penalty* and $\lambda$ as the *penalty parameter*. For $\lambda=0$, minimization of the ridge loss function yields the ML estimator. For any $\lambda > 0$, the ridge penalty contributes to the loss function, affecting its minimum and its location. The minimum of the sum-of-squares is well-known. The minimum of the ridge penalty is attained at $\beta = \mathbf{0}_{p}$ whenever $\lambda > 0$. The $\beta$ that minimizes $\mathcal{L}_{\mbox{{\footnotesize ridge}}}(\beta; \lambda)$ then balances the sum-of-squares and the penalty. The effect of the penalty in this balancing act is to shrink the regression coefficients towards zero, its minimum. In particular, the larger $\lambda$, the larger the contribution of the penalty to the loss function, the stronger the tendency to shrink non-zero regression coefficients to zero (and decrease the contribution of the penalty to the loss function). This motivates the name `penalty' as non-zero elements of $\beta$ increase (or penalize) the loss function.
-
-
-To verify that the ridge estimator indeed minimizes the ridge loss function, proceed as usual. Take the derivative with respect to $\beta$:
-\begin{align*}
-\frac{\partial}{\partial \beta} \mathcal{L}_{\mbox{{\footnotesize ridge}}}(\beta; \lambda) & = -2 \, \mathbf{X}^{\top} (\mathbf{Y} - \mathbf{X} \beta) + 2 \, \lambda \, \mathbf{I}_{pp} \, \beta \, \, \, = \, \, \, -2 \, \mathbf{X}^{\top} \mathbf{Y} + 2 \, ( \mathbf{X}^{\top} \mathbf{X} + \lambda \, \mathbf{I}_{pp}) \beta.
-\end{align*}
-Equate the derivative to zero and solve for $\beta$. This yields the ridge regression estimator.
-
-The ridge estimator is thus a stationary point of the ridge loss function. A stationary point corresponds to a minimum if the Hessian matrix with second order partial derivatives is positive definite. The Hessian of the ridge loss function is
-\begin{align*}
-\frac{\partial^2}{\partial \beta \, \partial \beta^{\top}} \mathcal{L}_{\mbox{{\footnotesize ridge}}}(\beta; \lambda) & = 2 \, ( \mathbf{X}^{\top} \mathbf{X} + \lambda \, \mathbf{I}_{pp}).
-\end{align*}
-This Hessian is the sum of the semi-positive definite matrix $\mathbf{X}^{\top} \mathbf{X}$ and the positive definite matrix $\lambda \, \mathbf{I}_{pp}$. Lemma 14.2.4 of cite{Harv2008} then states that the sum of these matrices is itself a positive definite matrix. Hence, the Hessian is positive definite and the ridge loss function has a stationary point at the ridge estimator, which is a minimum.
-
-The ridge regression estimator minimizes the ridge loss function. It rests to verify that it is a global minimum. To this end we introduce the concept of a convex function. As a prerequisite, a set $\mathcal{S} \subset \mathbb{R}^p$ is called *convex* if for all $\beta_1, \beta_2 \in \mathcal{S}$ their weighted average $\beta_{\theta} = (1 - \theta) \beta_1 + \theta \beta_2$ for all $\theta \in [0, 1]$ is itself an element of $\mathcal{S}$, thus $\beta_{\theta} \in \mathcal{S}$. If for all $\theta \in (0, 1)$, the weighted average $\beta_{\theta}$ is inside $\mathcal{S}$ and not on its boundary, the set is called *strict convex*. Examples of (strict) convex and nonconvex sets are depicted in Figure ref{fig.ridgeAsConstrainedEst}. A function $f(\cdot)$ is *(strict) convex* if the set $\{ y \, : \, y \geq f(\beta) \mbox{ for all } \beta \in \mathcal{S} \mbox{ for any convex } \mathcal{S} \}$, called the epigraph of $f(\cdot)$, is (strict) convex. Examples of (strict) convex and nonconvex functions are depicted in Figure ref{fig.ridgeAsConstrainedEst}. The ridge loss function is the sum of two parabola's: one at least convex and the other a strict convex function in $\beta$. The sum of convex and strict convex function is itself strict convex (confer Lemma 9.4.2 of \citealt{Flet2008}). The ridge loss function is thus strict convex. Theorem 9.4.1 of \citealt{Flet2008} then warrants, by the strict convexity of the ridge loss function, that the ridge estimator is a global minimum.
-\\
-\\
-From the ridge loss function the limiting behavior of the variance of the ridge regression estimator can be understood. The ridge penalty with its minimum $\beta = \mathbf{0}_{p}$ does not involve data and, consequently, the variance of its minimum equals zero. With the ridge regression being a compromise between the ML estimator and the minimum of the penalty, so is its variance a compromise of their variances. As $\lambda$ tends to infinity, the ridge estimator and its variance converge to the minimum and the variance of the minimum, respectively. Hence, in the limit (large $\lambda$) the variance of the ridge regression estimator vanishes. Understandably, as the penalty now fully dominates the loss function and, consequently, it does no longer involve data (i.e. randomness).
-# original latex figure with scale=0.38, angle=0
-
-FIGURE: [convexSets.eps, width=400 frac=1.0] Top panels show examples of convex (left) and nonconvex (right) sets. Middle panels show examples of convex (left) and nonconvex (right) functions. The left bottom panel illustrates the ridge estimation as a constrained estimation problem. The ellipses represent the contours of the ML loss function, with the blue dot at the center the ML estimate. The circle is the ridge parameter constraint. The red dot is the ridge estimate. It is at the intersection of the ridge constraint and the smallest contour with a non-empty intersection with the constraint. The right bottom panel shows the data corresponding to Example ref{exam.overfitting}. The grey line represents the `true' relationship, while the black line the fitted one. } label{fig.ridgeAsConstrainedEst
-\afterpage{}
-\\
-\\
-Above it has been shown that the ridge estimator can be defined as:
-
-!bt
-
-!bt
-\begin{align} label{form.ridgeEstViaPenEst}
-\hat{\beta}(\lambda) & = \arg \min_{\beta} \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2 + \lambda \| \beta \|^2_2.
-\end{align}
-!et
-!et
-This minimization problem can be reformulated into the following constrained optimization problem (illustrated in Figure ref{fig.ridgeAsConstrainedEst}):
-
-!bt
-
-!bt
-\begin{align} label{form.constrEstProblemRidge}
-\hat{\beta}(\lambda) & = \arg \min_{\| \beta \|_2^2 \leq c} \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2,
-\end{align}
-!et
-!et
-for some suitable $c > 0$. The constrained optimization problem (ref{form.constrEstProblemRidge}) can be solved by means of the Karush-Kuhn-Tucker (KKT) multiplier method, which minimizes a function subject to inequality constraints. The KKT multiplier method states that, under some regularity conditions (all met here), there exists a constant $\nu \geq 0$, called the *multiplier*, such that the solution $\hat{\beta}(\nu)$ of the constrained minimization problem (ref{form.constrEstProblemRidge}) satisfies the so-called KKT conditions. The first KKT condition (referred to as the stationarity condition) demands that the gradient (with respect to $\beta$) of the Lagrangian associated with the minimization problem equals zero at the solution $\hat{\beta}(\nu)$. The Lagrangian for problem (ref{form.constrEstProblemRidge}) is:
-\begin{align*}
-\| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2 + \nu ( \| \beta \|^2_2 - c).
-\end{align*}
-The second KKT condition (the complementarity condition) requires that $\nu (\| \hat{\beta}(\nu) \|_2^2 - c) = 0$. If $\nu = \lambda$ and $c = \| \hat{\beta}(\lambda) \|_2^2$, the ridge estimator $\beta (\lambda)$ satisfies both KKT conditions. Hence, both problems have the same solution when $c = \| \hat{\beta}(\lambda) \|_2^2$.
-\\
-\\
-The relevance of viewing the ridge regression estimator as the solution to a constrained estimation problem becomes obvious when considering a typical threat to high-dimensional data analysis: overfitting. *Overfitting* refers to the phenomenon of modelling the noise rather than the signal. In case the true model is parsimonious (few covariates driving the response) and data on many covariates are available, it is likely that a linear combination of all covariates yields a higher likelihood than a combination of the few that are actually related to the response. As only the few covariates related to the response contain the signal, the model involving all covariates then cannot but explain more than the signal alone: it also models the error. Hence, it overfits the data. In high-dimensional settings overfitting is a real threat. The number of explanatory variables exceeds the number of observations. It is thus possible to form a linear combination of the covariates that perfectly explains the response, including the noise.
-
-Large estimates of regression coefficients are often an indication of overfitting. Augmentation of the estimation procedure with a constraint on the regression coefficients is a simple remedy to large parameter estimates. As a consequence it decreases the probability of overfitting. Overfitting is illustrated in the next example.
-
-
-\begin{example} *(Overfitting)* label{exam.overfitting}
-\\
-Consider an artificial data set comprising of ten observations on a response $Y_i$ and nine covariates $X_{i,j}$. All covariate data are sampled from the standard normal distribution: $X_{i,j} \sim \mathcal{N}(0, 1)$. The response is generated by $Y_i = X_{i,1} + \varepsilon_i$ with $\varepsilon_{i} \sim \mathcal{N}(0, 1/4)$. Hence, only the first covariate contributes to the response.
-
-The regression model $Y_i = \sum_{j=1}^9 X_{i,j} \beta_j+ \varepsilon_i$ is fitted to the artificial data using `R`. This yields the regression parameter estimates:
-\begin{align*}
-\hat{\beta}^{\top} & = (0.048, -2.386, -5.528, 6.243, -4.819, 0.760, -3.345, -4.748, 2.136).
-\end{align*}
-As $\beta^{\top} = (1, 0, \ldots, 0)$, many regression coefficient are clearly over-estimated.
-
-The fitted values $\widehat{Y}_i = \mathbf{X}_i \hat{\beta}$ are plotted against the values of the first covariates in the right bottom panel of Figure ref{fig.ridgeAsConstrainedEst}. As a reference the line $x=y$ is added, which represents the `true' model. The fitted model follows the `true' relationship. But it also captures the deviations from this line that represent the errors.
-\end{example}
-
-
-
-======= Bayesian regression} label{sect:Bayes =======
-Ridge regression has a close connection to Bayesian linear regression. Bayesian linear regression assumes the parameters $\beta$ and $\sigma^2$ to be the random variables, while at the same time considering $\mathbf{X}$ and $\mathbf{Y}$ as fixed. Within the regression context, the conjugate priors of $\beta$ and $\sigma^2$ are:
-\begin{align*}
-\beta \, | \, \sigma^2 \sim \mathcal{N}(\mathbf{0}_p, \sigma^2 \lambda^{-1} \mathbf{I}_{pp}) \qquad \mbox{and} \qquad \sigma^2 \sim \mathcal{IG}(\alpha_0, \beta_0),
-\end{align*}
-where $\mathcal{IG}$ denotes the inverse Gamma distribution with shape parameter $\alpha_0$ and scale parameter $\beta_0$. The penalty parameter can be interpreted as the precision of the prior, determining how informative the prior should be. A smaller penalty (i.e. precision) corresponds to a wider prior, and a larger penalty to a more informative, concentrated prior (Figure ref{fig.ridgePriorOfBeta}).
-# original latex figure with scale=0.45, angle=0
-
-FIGURE: [ridgePriorOfBeta.eps, width=400 frac=1.0] Conjugate prior of the regression parameter $\beta$ for various choices of $\lambda$, the penalty parameters c.q. precision.} label{fig.ridgePriorOfBeta
-
-Under the assumption of the conjugate priors above, the joint posterior distribution of $\beta$ and $\sigma^2$ is then:
-\begin{align*}
-f_{\beta, \sigma^2} (\beta, \sigma^2 \, | \, \mathbf{Y}, \mathbf{X}) & = f_Y (\mathbf{Y} \, | \, \mathbf{X}, \beta, \sigma^2) \, f_{\beta}(\beta | \sigma^2) \, f_{\sigma}(\sigma^2)
-\\
-& \propto & \sigma^{-n} \exp \Big[ - \frac{1}{2\sigma^2} ( \mathbf{Y} - \mathbf{X} \beta)^{\top} ( \mathbf{Y} - \mathbf{X} \beta) \Big]
-\\
-& & \times \, \, \sigma^{-p} \exp \Big[ - \frac{1}{2\sigma^2} \lambda \beta^{\top} \beta \Big] \, \times \, \, [\sigma^2]^{-\alpha_0-1} \exp \Big[ - \frac{\beta_0}{2\sigma^2} \Big].
-\end{align*}
-As
-\begin{align*}
-& & \hspace{-1.5cm} ( \mathbf{Y} - \mathbf{X} \beta)^{\top} ( \mathbf{Y} - \mathbf{X} \beta) + \lambda \beta^{\top} \beta
-\\
-& = \mathbf{Y}^{\top} \mathbf{Y} -
-\beta^{\top} \mathbf{X}^{\top} \mathbf{Y} - \mathbf{Y}^{\top} \mathbf{X} \beta + \beta^{\top} \mathbf{X}^{\top} \mathbf{X} \beta + \lambda \beta^{\top} \beta
-\\
-& = \mathbf{Y}^{\top} \mathbf{Y} - \beta^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& & - \, \mathbf{Y}^{\top} \mathbf{X} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) \beta + \beta^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) \beta
-\\
-& = \mathbf{Y}^{\top} \mathbf{Y} - \beta^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})
-\hat{\beta} (\lambda)
-\\
-& & - \, [ \hat{\beta} (\lambda) ]^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) \beta + \beta^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) \beta
-\\
-& = \mathbf{Y}^{\top} \mathbf{Y} - \mathbf{Y}^{\top} \mathbf{X} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& & + \, \big[ \beta - \hat{\beta}(\lambda) \big]^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) \big[ \beta - \hat{\beta}(\lambda) \big],
-\end{align*}
-the posterior distribution can be rewritten to:
-\begin{align*}
-f_{\beta, \sigma^2} (\beta, \sigma^2 \, | \, \mathbf{Y}, \mathbf{X})
-& \propto & g_{\beta} (\beta \, | \, \sigma^2, \mathbf{Y}, \mathbf{X}) \,
-g_{\sigma^2} (\sigma^2 \, | \, \mathbf{Y}, \mathbf{X})
-\end{align*}
-with
-\begin{align*}
-g_{\beta} (\beta \, | \, \sigma^2, \mathbf{Y}, \mathbf{X})
-& \propto & \exp \Big\{ - \frac{1}{2\sigma^2} \big[ \beta - \hat{\beta}(\lambda) \big]^{\top} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) \big[ \beta - \hat{\beta}(\lambda) \big] \Big\}.
-\end{align*}
-Then, clearly the conditional posterior mean of $\beta$ is $\mathbb{E}(\beta \, | \, \sigma^2, \mathbf{Y}, \mathbf{X}) = \hat{\beta}(\lambda)$. Hence, the ridge regression estimator can be viewed as the Bayesian posterior mean estimator of $\beta$ when imposing a Gaussian prior on the regression parameter.
-
-With little extra work we may also obtain the conditional posterior of $\sigma^2$ from the joint posterior distribution:
-\begin{align*}
-f_{\sigma^2} (\sigma^2 \, | \, \beta, \mathbf{Y}, \mathbf{X}) & \propto & (\sigma^2)^{-[(n+p)/2 + \alpha_0 + 1]}
-\exp [ - \frac{1}{2\sigma^2} ( \| \mathbf{Y} - \mathbf{X} \beta \|_2^2 +
-\lambda \| \beta \|_2^2 + \beta_0) ],
-\end{align*}
-in which one can recognize the shape of an inverse gamma distribution.
-\\
-\\
-A Bayesian estimator of a parameter $\ttheta$ is the estimator that minimizes the Bayes risk over a prior distribution of the parameter $\ttheta$. The Bayes risk is defined as $\int_{\ttheta} \mathbb{E} [(\hat{\ttheta} - \ttheta)^{\top} (\hat{\ttheta} - \ttheta)] \pi_{\ttheta}(\ttheta; \aalpha) d\ttheta$, where $\pi_{\ttheta}(\ttheta; \aalpha)$ is the prior distribution of $\ttheta$ with hyperparameter $\aalpha$. It is thus a weighted average of the Mean Squared Error, with weights specified through the prior. The Bayes risk is minimized by the mean posterior $\mathbb{E}_{\ttheta}(\ttheta \, | \, \mbox{data})$ (cf., e.g., \citealp{Bijm2017}). The Bayesian estimator of $\ttheta$ thus yields the smallest possible expected MSE, under the assumption of the employed prior.
-
-The Bayes risk of the ridge estimator over the normal prior $\beta \sim \mathcal{N}(\mathbf{0}_p, \sigma^2 \lambda^{-1} \mathbf{I}_{pp})$ is:
-\begin{align*}
-\mathbb{E}_{\beta} \{\mbox{MSE}[\hat{\beta}(\lambda)]
-\, | \, \sigma^2, \mathbf{Y}, \mathbf{X} \} & = \sigma^2 \, \mbox{tr}\big\{ \mathbf{W}_{\lambda} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{W}_{\lambda}^{\top} \big\} + \mathbb{E}_{\beta} [ \beta^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp})^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp}) \, \beta]
-\\
-& = \sigma^2 \, \big\{ \mbox{tr} \big[ \mathbf{W}_{\lambda} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \, \mathbf{W}_{\lambda}^{\top} \big] +
-\lambda^{-1} \mbox{tr} [(\mathbf{W}_{\lambda} - \mathbf{I}_{pp})^{\top} (\mathbf{W}_{\lambda} - \mathbf{I}_{pp})] \big\}
-\\
-& = \sigma^2 \sum_{j=1}^p (d_{jj}^2 + \lambda)^{-1},
-\end{align*}
-in which we have used *i)* the previously derived explicit expression (ref{form.ridgeMSE}) of the ridge estimator's MSE, *ii)* the expectation of the quadratic form of a multivariate random variable \citep{Math1992}, *iii)* the singular value decomposition of $\mathbf{X}$ with singular values $d_{jj}$, and *iv)* the fact that the trace of a square matrix equals the sum of its eigenvalues. As the ridge estimator coincides with the posterior mean, this is the minimal achievable MSE under a zero-centered normal prior with an uncorrelated and equivariant covariance matrix.
-
-Above the Bayes risk of the ridge estimator factorizes with respect to $\sigma^2$ and $\lambda$. Hence, the larger the hyperparameter $\lambda$ the lower the Bayes risk of the ridge estimator. In particular, its Bayes risk converges to zero as $\lambda \rightarrow \infty$. This can be understood as follows. The limit corresponds to an infinite precision of the prior, thus reducing the variance contribution to the MSE. Moreover, as the ridge estimator shrinks towards zero and the prior distribution of $\beta$ has a zero mean, the bias too vanishes as $\lambda \rightarrow \infty$.
-
-The calculation of the Bayes risk above relates the Bayesian and frequentist statements on the MSE of the ridge estimator. For the latter revisit Theorem ref{theo.Theobald2} of Section ref{sect:ridgeMSE}, which states the existence of a $\lambda$ such that the resulting ridge estimator has a superior MSE over that of the ML estimator. This result made no assumption on (the distribution of) $\beta$. In fact, it can be viewed as a statement of the MSE conditional on $\beta$. The Bayesian result integrates out the uncertainty - specified by the prior - in $\beta$ from the (frequentist's) conditional MSE to arrive at the unconditional MSE.
-
-
-
-# The Bayesian result states that the ridge regression estimator minimizes the weighted MSE (over a normal prior on the parameter $\beta$). It thus integrates out the uncertainty in $\beta$ from the (frequentist's) conditional MSE to arrive at the unconditional MSE.
-
-======= Degrees of freedom =======
-
-The degrees of freedom consumed by ridge regression is calculated. The degrees of freedom may be used in combination with an information criterion to decide on the value of the penalty parameter. Recall from ordinary regression that:
-\begin{align*}
-\widehat{\mathbf{Y}} & = \mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} \, \, \, = \, \, \, \mathbf{H} \mathbf{Y},
-\end{align*}
-where $\mathbf{H}$ is the hat matrix. The degrees of freedom used in the regression is then equal to $\mbox{tr}(\mathbf{H})$, the trace of $\mathbf{H}$. In particular, if $ \mathbf{X}$ is of full rank, i.e. $\mbox{rank}(\mathbf{X}) = p$, then $\mbox{tr}(\mathbf{H}) = p$.
-
-By analogy, the ridge-version of the hat matrix is:
-\begin{align*}
-\mathbf{H}(\lambda) & = \mathbf{X} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top}.
-\end{align*}
-Continuing this analogy, the degrees of freedom of ridge regression is given by the trace of the ridge hat matrix $\mathbf{H}(\lambda)$:
-\begin{align*}
-\mbox{tr}[ \mathbf{H}(\lambda)] & = \mbox{tr}[ \mathbf{X} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} ] \, \, \, = \, \, \, \sum_{j=1}^p \frac{d_{jj}^2}{d_{jj}^2 + \lambda}.
-\end{align*}
-The degrees of freedom consumed by ridge regression is monotone decreasing in $\lambda$. In particular:
-\begin{align*}
-\lim_{\lambda \rightarrow \infty} \mbox{tr}[ \mathbf{H}(\lambda)] & = 0.
-\end{align*}
-That is, in the limit no information from $\mathbf{X}$ is used. Indeed, $\beta$ is forced to equal $\mathbf{0}_{p}$ which is not derived from data.
-
-
-
-======= Efficient calculation =======
-
-In the high-dimensional setting the number of covariates $p$ is large compared to the number of samples $n$. In a microarray experiment $p = 40000$ and $n= 100$ is not uncommon. To perform ridge regression in this context, the following expression needs to be evaluated numerically:
-\begin{align*}
-(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}.
-\end{align*}
-For $p=40000$ this requires the inversion of a $40000 \times 40000$ dimensional matrix. This is not feasible on most desktop computers. However, there is a workaround.
-
-Revisit the singular value decomposition of $\mathbf{X} = \mathbf{U}_x \mathbf{D}_x \mathbf{V}_x^{\top}$ and write $\mathbf{R}_x = \mathbf{U}_x \mathbf{D}_x$. As both $\mathbf{U}_x$ and $\mathbf{D}_x$ are $(n \times n)$-dimensional matrices, so is $\mathbf{R}_x$. Consequently, $\mathbf{X}$ is now decomposed as $\mathbf{X} = \mathbf{R}_x \mathbf{V}_x^{\top}$. The ridge estimator can be rewritten in terms of $\mathbf{R}_x$ and $\mathbf{V}_x$:
-\begin{align*}
-\hat{\beta}(\lambda) & = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{R}_x^{\top} \mathbf{R}_x \mathbf{V}_x^{\top} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{V}_x \mathbf{R}_x^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{R}_x^{\top} \mathbf{R}_x \mathbf{V}_x^{\top} + \lambda \mathbf{V}_x \mathbf{V}_x^{\top})^{-1} \mathbf{V}_x \mathbf{R}_x^{\top} \mathbf{Y}
-\\
-& = \mathbf{V}_x (\mathbf{R}_x^{\top} \mathbf{R}_x + \lambda \mathbf{I}_{nn})^{-1} \mathbf{V}_x^{\top} \mathbf{V}_x \mathbf{R}_x^{\top} \mathbf{Y}
-\\
-& = \mathbf{V}_x (\mathbf{R}_x^{\top} \mathbf{R}_x + \lambda \mathbf{I}_{nn})^{-1} \mathbf{R}_x^{\top} \mathbf{Y}.
-\end{align*}
-Hence, the reformulated ridge estimator involves the inversion of an $(n \times n)$-dimensional matrix. With $n= 100$ this is feasible on most standard computers.
-
-cite{Hast2004} point out that the number of computation operations reduces from $\mathcal{O}(p^3)$ to $\mathcal{O}(p n^2)$. In addition, they point out that this computational short-cut can be used in combination with other loss functions, for instance that of standard generalized linear models.
-\\
-\\
-Avoidance of the inversion of the $(p \times p)$-dimensional matrix may be achieved in an other way. Hereto one needs the Woodbury identity. Let $\mathbf{A}$, $\mathbf{U}$ and $\mathbf{V}$ be $(p \times p)$-, $(p \times n)$- and $(n \times p)$-dimensional matrices, respectively.
-The (simplified form of the) Woodbury identity then is:
-\begin{align*}
-(\mathbf{A} + \mathbf{U} \mathbf{V})^{-1} & = \mathbf{A}^{-1} - \mathbf{A}^{-1} \mathbf{U} (\mathbf{I}_{nn} + \mathbf{V} \mathbf{A}^{-1} \mathbf{U})^{-1} \mathbf{V} \mathbf{A}^{-1}.
-\end{align*}
-Application of the Woodbury identity to the matrix inverse in the ridge estimator of the regression parameter gives:
-\begin{align*}
-(\lambda \mathbf{I}_{pp} + \mathbf{X}^{\top} \mathbf{X})^{-1} & = \lambda^{-1} \mathbf{I}_{pp} - \lambda^{-2} \mathbf{X}^{\top} (\mathbf{I}_{nn} + \lambda^{-1} \mathbf{X} \mathbf{X}^{\top})^{-1} \mathbf{X}.
-\end{align*}
-This gives:
-\begin{align*}
-(\lambda \mathbf{I}_{pp} + \mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} & = \lambda^{-1} \mathbf{X}^{\top} \mathbf{Y} - \lambda^{-2} \mathbf{X}^{\top} (\mathbf{I}_{nn} + \lambda^{-1} \mathbf{X} \mathbf{X}^{\top})^{-1} \mathbf{X} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = \lambda^{-1} \mathbf{X}^{\top} \left[ \mathbf{Y} - \lambda^{-1} \mathbf{X}^{\top} (\mathbf{I}_{nn} + \lambda^{-1} \mathbf{X} \mathbf{X}^{\top})^{-1} \mathbf{X} \mathbf{Y} \right].
-\end{align*}
-The inversion of the $(p \times p)$-dimensional matrix $\lambda \mathbf{I}_{pp} + \mathbf{X}^{\top} \mathbf{X}$ is thus replaced by that of the $(n \times n)$-dimensional matrix $\mathbf{I}_{nn} + \lambda^{-1} \mathbf{X} \mathbf{X}^{\top}$. In addition, this expression of the ridge regression estimator avoids the singular value decomposition of $\mathbf{X}$, which may in some cases introduce additional numerical errors (e.g. at the level of machine precision).
-
-
-
-======= Choice of the penalty parameter =======
-
-Throughout the introduction of ridge regression and the subsequent discussion of its properties the penalty parameter is considered known or `given'. In practice, it is unknown and the user needs to make an informed decision on its value. Several strategies to facilitate such a decision are presented.
-
-===== Information criterion =====
-
-A popular strategy is to choose a penalty parameter that yields a good but parsimonious model. Information criteria measure the balance between model fit and model complexity. Here we present the Aikaike's information criterion (AIC), but many other criteria have been presented in the literature (e.g. \citealp{Akai1974}, \citealp{Schw1978}). The AIC measures model fit by the log-likelihood and model complexity is measured by the number of parameters used by the model. The number of model parameters in regular regression simply corresponds to the number of covariates in the model. Or, by the degrees of freedom consumed by the model, which is equivalent to the trace of the hat matrix. For ridge regression it thus seems natural to define model complexity
-analogously by the trace of the ridge hat matrix. This yields the AIC for the linear regression model with ridge estimates:
-\begin{align*}
-\mbox{AIC}(\lambda) & = 2 \, p - 2 \log(\hat{L})
-\\
-& = 2 \, \mbox{tr} [\mathbf{H}(\lambda)] - 2 \log\{L[\hat{\beta}(\lambda), \hat{\sigma}^2(\lambda)]\}
-\\
-& = 2 \, \sum_{j=1}^p \frac{d_{jj}^2}{d_{jj}^2 + \lambda}
-+ 2 n \, \log[\sqrt{2 \, \pi} \, \hat{\sigma}(\lambda)] + \frac{1}{\hat{\sigma}^2(\lambda)} \sum_{i=1}^n [y_i - \mathbf{X}_{i, \ast} \, \hat{\beta}(\lambda)]^2.
-\end{align*}
-The value of $\lambda$ which minimizes $\mbox{AIC}(\lambda)$ corresponds to the `optimal' balance of model complexity and overfitting.
-
-Information criteria guide the decision process when having to decide among various different models. Different models use different sets of explanatory variables to explain the behaviour of the response variable. In that sense, the use of information criteria for the deciding on the ridge penalty parameter may be considered inappropriate: ridge regression uses the same set of explanatory variables irrespective of the value of the penalty parameter. Moreover, often ridge regression is employed to predict a response and not to provide an insightful explanatory model. The latter need not yield the best predictions. Finally, empirically we observe that the AIC often does not show an optimum *inside* the domain of the ridge penalty parameter. Henceforth, we refrain from the use of the AIC (or any of its relatives) in determining the optimal ridge penalty parameter.
-
-
-===== Cross-validation} label{subsect.crossvalidation =====
-Instead of choosing the penalty parameter to balance model fit with model complexity, cross-validation requires it (i.e. the penalty parameter) to yield a model with good prediction performance. Commonly, this performance is evaluated on novel data. Novel data need not be easy to come by and one has to make do with the data at hand. The setting of `original' and novel data is then mimicked by sample splitting: the data set is divided into two (groups of samples). One of these two data sets, called the *training set*, plays the role of `original' data on which the model is built. The second of these data sets, called the *test set*, plays the role of the `novel' data and is used to evaluate the prediction performance (often operationalized as the log-likelihood or the prediction error) of the model built on the training data set. This procedure (model building and prediction evaluation on training and test set, respectively) is done for a collection of possible penalty parameter choices. The penalty parameter that yields the model with the best prediction performance is to be preferred. The thus obtained performance evaluation depends on the actual split of the data set. To remove this dependence the data set is split many times into a training and test set. For each split the model parameters are estimated for all choices of $\lambda$ using the training data and estimated parameters are evaluated on the corresponding test set. The penalty parameter that on average over the test sets performs best (in some sense) is then selected.
-
-When the repetitive splitting of the data set is done randomly, samples may accidently end up in a fast majority of the splits in either training or test set. Such samples may have an unbalanced influence on either model building or prediction evaluation. To avoid this $k$-fold cross-validation structures the data splitting. The samples are divided into $k$ more or less equally sized exhaustive and mutually exclusive subsets. In turn (at each split) one of these subsets plays the role of the test set while the union of the remaining subsets constitutes the training set. Such a splitting warrants a balanced representation of each sample in both training and test set over the splits. Still the division into the $k$ subsets involves a degree of randomness. This may be fully excluded when choosing $k=n$. This particular case is referred to as leave-one-out cross-validation (LOOCV). For illustration purposes the LOOCV procedure is detailed fully below:
-\begin{compactitem}
-\item[0)] Define a range of interest for the penalty parameter.
-
-\item[1)] Divide the data set into training and test set comprising samples $\{1, \ldots, n\} \setminus i$ and $\{ i \}$, respectively.
-
-\item[2)] Fit the linear regression model by means of ridge estimation for each $\lambda$ in the grid using the training set. This yields:
-\begin{align*}
-\hat{\beta}_{-i}(\lambda) & = ( \mathbf{X}_{-i, \ast}^{\top}
-\mathbf{X}_{-i, \ast} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{-i, \ast}^{\top} \mathbf{Y}_{-i}
-\end{align*}
-and the corresponding estimate of the error variance $\hat{\sigma}_{-i}^2(\lambda)$.
-
-\item[3)] Evaluate the prediction performance of these models on the test set by $\log\{L[Y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}$. Or, by the prediction error $|Y_i - \mathbf{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)|$, possibly squared.
-
-\item[4)] Repeat steps 1) to 3) such that each sample plays the role of the test set once.
-
-\item[5)] Average the prediction performances of the test sets at each grid point of the penalty parameter:
-\begin{align*}
-\frac{1}{n} \sum_{i = 1}^n \log\{L[Y_i, \mathbf{X}_{i, \ast}; \hat{\beta}_{-i}(\lambda), \hat{\sigma}_{-i}^2(\lambda)]\}.
-\end{align*}
-The quantity above is called the *cross-validated log-likelihood*. It is an estimate of the prediction performance of the model corresponding to this value of the penalty parameter on novel data.
-
-\item[6)] The value of the penalty parameter that maximizes the cross-validated log-likelihood is the value of choice.
-\end{compactitem}
-The procedure is straightforwardly adopted to $k$-fold cross-validation, a different criterion, and different estimators.
-
-In the LOOCV procedure above resampling can be avoided when the prediction performance is measured by Allen's PRESS (Predicted Residual Error Sum of Squares) statistic \citep{Alle1974}. For then, the LOOCV prediction performance can be expressed analytically in terms of the known quantities derived from the design matrix and response (as pointed out but not detailed in \citealt{Golu1979}). Define the optimal penalty parameter to minimize Allen's PRESS statistic:
-\begin{align*}
-\lambda_{\mbox{{\tiny opt}}} = \arg \min_{\lambda} \frac{1}{n} \sum_{i=1}^n [Y_i - \mathbf{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)]^2.
-\end{align*}
-To derive an analytic expression for the right-hand side first
-rewrite $(\mathbf{X}_{- i, \ast}^{\top} \mathbf{X}_{- i, \ast} + \lambda \mathbf{I}_{pp})^{-1}$ by means of the Woodbury identity as:
-\begin{align*}
-(\mathbf{X}_{- i, \ast}^{\top} \mathbf{X}_{- i, \ast} + \lambda \mathbf{I}_{pp})^{-1} & = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp} - \mathbf{X}_{i, \ast}^{\top} \mathbf{X}_{i, \ast})^{-1}
-\\
-& = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-+ (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top}]^{-1}
-\\
-& & \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\\
-& = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-+ (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\end{align*}
-with $\mathbf{H}_{ii}(\lambda) = \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top}$.
-Furthermore, $\mathbf{X}_{-i}^{\top} \mathbf{Y}_{-i} = \mathbf{X}^{\top} \mathbf{Y} - \mathbf{X}_{i, \ast}^{\top} Y_i$. Substitute both in the leave-one-out ridge regression estimator and manipulate:
-\begin{align*}
-\hat{\beta}_{- i}(\lambda) & = (\mathbf{X}_{- i, \ast}^{\top} \mathbf{X}_{- i, \ast} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{- i, \ast}^{\top} \mathbf{Y}_{- i}
-\\
-& = \{(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-+ (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}\}
-\\
-& & \times (\mathbf{X}^{\top} \mathbf{Y} - \mathbf{X}_{i, \ast}^{\top} Y_i )
-\\
-& = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y} - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} Y_i
-\\
-& & + (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii} (\lambda)]^{-1} \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}^{\top} \mathbf{Y}
-\\
-& & - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1}
-\mathbf{X}_{i, \ast}^{\top} Y_i
-\\
-& = \hat{\beta}(\lambda) - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ 1 - \mathbf{H}_{ii}(\lambda)] Y_i
-\\
-& & + (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \mathbf{X}_{i, \ast} \hat{\beta}(\lambda)
-\\
-& & - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \mathbf{H}_{ii} (\lambda) Y_i
-\\
-& = \hat{\beta}(\lambda) - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \{ [ 1 - \mathbf{H}_{ii}(\lambda)] Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda) + \mathbf{H}_{ii}(\lambda) Y_i \}
-\\
-& = \hat{\beta}(\lambda) - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda) ].
-\end{align*}
-The latter enables the reformulation of the prediction error as:
-\begin{align*}
-Y_i - \mathbf{X}_{i, \ast} \hat{\beta}_{-i}(\lambda) & = Y_i - \mathbf{X}_{i, \ast} \{ \hat{\beta}(\lambda) - (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda)] \}
-\\
-& = Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda) + \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda) ]
-\\
-& = Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda) + \mathbf{H}_{ii} (\lambda)[ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ Y_i - \mathbf{X}_{i, \ast} \hat{\beta}(\lambda) ]
-\\
-& = [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ Y_i - \mathbf{X}_{i, \ast}^{\top} \hat{\beta}(\lambda) ],
-\end{align*}
-which in turn results in the re-expression of Allen's PRESS statistic:
-\begin{align*}
-\lambda_{\mbox{{\tiny opt}}} & = \arg \min_{\lambda} \frac{1}{n} \sum_{i=1}^n [Y_i - \mathbf{X}_{i, \ast} \hat{\beta}_{-i}(\lambda)]^2
-\, \, \, = \, \, \, \arg \min_{\lambda} \tfrac{1}{n} \| \mathbf{B}(\lambda) [\mathbf{I}_{nn} - \mathbf{H}(\lambda)] \mathbf{Y} \|_ F^2,
-\end{align*}
-where $\mathbf{B}(\lambda)$ is diagonal with $[\mathbf{B}(\lambda)]_{ii} = [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1}$. Hence, the prediction performance for a given $\lambda$ can be assessed directly from the ridge hat matrix and the response vector without the recalculation of the $n$ leave-one-out ridge estimators. Computationally, this is a considerable gain.
-
-
-
-======= Simulations =======
-
-Simulations are presented that illustrate properties of the ridge estimator not discussed explicitly in the previous sections of this chapter.
-
-===== Role of the variance of the covariates} label{ridge:covariateVariances =====
-In many applications of high-dimensional data the covariates are standardized prior to the execution of the ridge regression. Before we discuss whether this is appropriate, we first illustrate the effect of ridge penalization on covariates with distinct variances using simulated data.
-
-The simulation involves one response to be (ridge) regressed on fifty covariates. Data (with $n=1000$) for the covariates, denoted $\mathbf{X}$, are drawn from a multivariate normal distribution: $\mathbf{X} \sim \mathcal{N}(\mathbf{0}_{50}, \mathbf{\Sigma})$ with $\mathbf{\Sigma}$ diagonal and $(\mathbf{\Sigma})_{jj} = j / 10$. From this the response is generated through $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$ with $\beta = \mathbf{1}_{50}$ and $\vvarepsilon \sim \mathcal{N}(\mathbf{0}_{50}, \mathbf{I}_{50 \times 50})$.
-
-With the simulated data at hand the ridge regression estimates of $\beta$ are evaluated for a large grid of the penalty parameter $\lambda$. The resulting ridge regularization paths of the regression coefficients are plotted (Figure ref{fig.effectOfRidge_effectOfVariances}). All paths start ($\lambda=0$) close to one and vanish as $\lambda \rightarrow \infty$. However, ridge regularization paths of regression coefficients corresponding to covariates with a large variance dominate those with a low variance.
-
-
-# original latex figure with scale=0.45, angle=0
-
-FIGURE: [covariatesWithDifferentVariance.eps, width=400 frac=1.0] Top panel: Ridge regularization paths for coefficients of the 50 uncorrelated covariates with distinct variances. Color and line type indicated the grouping of the covariates by their variance. Bottom panels: Graphical illustration of the effect of a covariate's variance on the ridge estimator. The grey circle depicts the ridge parameter constraint. The dashed black ellipsoids are the level sets of the least squares loss function. The red dot is the ridge regression estimate. Left and right panels represent the cases with equal and unequal, respectively, variances of the covariates.} label{fig.effectOfRidge_effectOfVariances
-
-
-
-
-
-Ridge regression's preference of covariates with a large variance can intuitively be understood as follows. First note that the ridge regression estimator now can be written as:
-\begin{align*}
-\beta (\lambda) & = [ \mbox{Var}(\mathbf{X}) + \lambda \mathbf{I}_{50 \times 50}]^{-1} \mbox{Cov}(\mathbf{X}, \mathbf{Y})
-\\
-& = ( \SSigma + \lambda \mathbf{I}_{50 \times 50})^{-1} \SSigma [ \mbox{Var}(\mathbf{X}) ]^{-1} \mbox{Cov}(\mathbf{X}, \mathbf{Y})
-\\
-& = ( \SSigma + \lambda \mathbf{I}_{50 \times 50})^{-1} \SSigma \beta.
-\end{align*}
-Plug in the employed parametrization of $\mathbf{\Sigma}$, which gives:
-\begin{align*}
-[\beta (\lambda)]_j & = \frac{j}{j + 50 \lambda} \, (\beta)_j.
-\end{align*}
-Hence, the larger the covariate's variance (corresponding to the larger $j$), the larger its ridge regression coefficient estimate. Ridge regression thus prefers (among a set of covariates with comparable effect sizes) those with larger variances.
-
-
-
-The reformulation of ridge penalized estimation as a constrained estimation problem offers a geometrical interpretation of this phenomenon. Let $p=2$ and the design matrix $\mathbf{X}$ be orthogonal, while both covariates contribute equally to the response. Contrast the cases with $\mbox{Var}(X_1) \approx \mbox{Var}(X_2)$ and $\mbox{Var}(X_1) \gg \mbox{Var}(X_2)$. The level sets of the least squares loss function associated with the former case are circular, while that of the latter are strongly ellipsoidal (see Figure ref{fig.effectOfRidge_effectOfVariances}). The diameters along the principal axes (that - due to the orthogonality of $\mathbf{X}$ - are parallel to that of the $\beta_1$- and $\beta_2$-axes) of both circle and ellipsoid are reciprocals of the variance of the covariates. When the variances of both covariates are equal, the level sets of the loss function expand equally fast along both axis. With the two covariates having the same regression coefficient, the point of these level sets closest to the parameter constraint is to be found on the line $\beta_1 = \beta_2$ (Figure ref{fig.effectOfRidge_effectOfVariances}, left panel). Consequently, the ridge regression estimate satisfies $\hat{\beta}_1 (\lambda) \approx \hat{\beta}_2(\lambda)$. With unequal variances between the covariates, the ellipsoidal level sets of the loss function have diameters of rather different sizes. In particular, along the $\beta_1$-axis it is narrow (as $\mbox{Var}(X_1)$ is large), and - vice versa - wide along the $\beta_2$-axis. Consequently, the point of these level sets closest to the circular parameter constraint will be closer to the $\beta_1$- than to the $\beta_2$-axis (Figure ref{fig.effectOfRidge_effectOfVariances}, left panel). For the ridge estimates of the regression parameter this implies $0 \ll \hat{\beta}_1 (\lambda) < 1$ and $0 < \hat{\beta}_2 (\lambda) \ll 1$. Hence, the covariate with a larger variance yields the larger ridge regression estimate.
-
-
-Should one thus standardize the covariates prior to ridge regression analysis? When dealing with gene expression data from microarrays, the data have been subjected to a series of pre-processing steps (e.g. quality control, background correction, within- and between-normalization). The purpose of these steps is to make the expression levels of genes comparable both within and between hybridizations. The preprocessing should thus be considered an inherent part of the measurement. As such it is to be done independently of whatever down-stream analysis is to follow and further tinkering with the data is preferably to be avoided (as it may mess up the `comparable-ness' of the expression levels as achieved by the preprocessing). For other data types different considerations may apply.
-
-Among the considerations to decide on standardization of the covariates, one should also include the fact that ridge estimates prior and posterior to scaling do not simply differ by a factor. To see this assume that the covariates have been centered. Scaling of the covariates amounts to post-multiplication of the design matrix by a $(p \times p)$-dimensional diagonal matrix $\mathbf{A}$ with the reciprocals of the covariates' scale estimates on its diagonal \citep{Sard2008}. Hence, the ridge estimator (for the rescaled data) is then given by:
-\begin{align*}
-\min_{\beta} \| \mathbf{Y} - \mathbf{X} \mathbf{A} \beta \|_2^2 + \lambda \| \beta \|_ 2^2.
-\end{align*}
-Apply the change-of-variable $\ggamma = \mathbf{A} \beta$ and obtain:
-\begin{align*}
-\min_{\ggamma} \| \mathbf{Y} - \mathbf{X} \gamma \|_2^2 + \lambda \| \mathbf{A}^{-1} \ggamma \|_ 2^2 & = \min_{\beta} \| \mathbf{Y} - \mathbf{X} \gamma \|_2^2 + \sum_{j=1}^p \lambda [(\mathbf{A})_{jj}]^{-2} \gamma_j^2.
-\end{align*}
-Effectively, the scaling is equivalent to covariate-wise penalization. The `scaled' ridge estimator may then be derived along the same lines as before in Section ref{sect.constrainedEstimation}:
-\begin{align*}
-\hat{\beta}^{\mbox{{\tiny (scaled)}}} (\lambda) & = \mathbf{A}^{-1} \hat{\ggamma} (\lambda)
-\, \, \, = \, \, \, \mathbf{A}^{-1} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{A}^{-2})^{-1} \mathbf{X}^{\top} \mathbf{Y}.
-\end{align*}
-In general, this is unequal to the ridge estimator without the rescaling of the columns of the design matrix. Moreover, it should be clear that $\hat{\beta}^{\mbox{{\tiny (scaled)}}} (\lambda) \not= \mathbf{A} \hat{\beta}(\lambda)$.
-
-===== Ridge regression and collinearity} label{sect:collinearCovariates =====
-Initially, ridge regression was motivated as an ad-hoc fix of (super)-collinear covariates in order to obtain a well-defined estimator. We now study the effect of this ad-hoc fix on the regression coefficient estimates of collinear covariates. In particular, their ridge regularization paths are contrasted to those of `non-collinear' covariates.
-
-To this end, we consider a simulation in which one response is regressed on 50 covariates. The data of these covariates, stored in a design matrix denoted $\mathbf{X}$, are sampled from a multivariate normal distribution, with mean zero and a $5 \times 5$ blocked covariance matrix:
-\begin{align*}
-\mathbf{\Sigma} & = \left(
-\begin{array}{ccccc}
-\mathbf{\Sigma}_{11} & \mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-& \mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-\\
-\mathbf{0}_{10 \times 10} & \mathbf{\Sigma}_{22} & \mathbf{0}_{10 \times 10}
-& \mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-\\
-\mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-& \mathbf{\Sigma}_{33} & \mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-\\
-\mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-& \mathbf{0}_{10 \times 10} & \mathbf{\Sigma}_{44} & \mathbf{0}_{10 \times 10}
-\\
-\mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10}
-& \mathbf{0}_{10 \times 10} & \mathbf{0}_{10 \times 10} & \mathbf{\Sigma}_{55}
-\end{array}
-\right)
-\end{align*}
-with
-\begin{align*}
-\mathbf{\Sigma}_{kk} & = \frac{k-1}{5} \, \mathbf{1}_{10 \times 10} + \frac{6-k}{5} \, \mathbf{I}_{10 \times 10}.
-\end{align*}
-The data of the response variable $\mathbf{Y}$ are then obtained through: $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$, with $\vvarepsilon \sim \mathcal{N}( \mathbf{0}_{n}, \mathbf{I}_{nn})$ and $\beta = \mathbf{1}_{50}$. Hence, all covariates contribute equally to the response. Would the columns of $\mathbf{X}$ be orthogonal, little difference in the ridge estimates of the regression coefficients is expected.
-
-The results of this simulation study with sample size $n=1000$ are presented in Figure ref{fig.ridgeEstimates_correlatedCovariates}. All 50 regularization paths start close to one as $\lambda$ is small and converge to zero as $\lambda \rightarrow \infty$. But the paths of covariates of the same block of the covariance matrix $\mathbf{\Sigma}$ quickly group, with those corresponding to a block with larger off-diagonal elements above those with smaller ones. Thus, ridge regression prefers (i.e. shrinks less) coefficient estimates of strongly positively correlated covariates.
-
-# original latex figure with scale=0.45, angle=0
-
-FIGURE: [correlatedCovariates.eps, width=400 frac=1.0] Left panel: Ridge regularization paths for coefficients of the 50 covariates, with various degree of collinearity but equal variance. Color and line type correspond to the five blocks of the covariate matrix $\mathbf{\Sigma}$. Right panel: Graphical illustration of the effect of the collinearity among covariates on the ridge estimator. The solid and dotted grey circles depict the ridge parameter constraint for the collinear and orthogonal cases, respectively. The dashed black ellipsoids are the level sets of the sum-of-squares squares loss function. The red dot and violet diamond are the ridge regression for the positive collinear and orthogonal case, respectively.} label{fig.ridgeEstimates_correlatedCovariates
-# \afterpage{}
-
-
-
-Intuitive understanding of the observed behaviour may be obtained from the $p=2$ case. Let $U$, $V$ and $\varepsilon$ be independent random variables with zero mean. Define $X_1 = U + V$, $X_2 = U - V$, and $Y = \beta_1 X_1 + \beta_2 X_2 + \varepsilon$ with $\beta_1$ and $\beta_2$ constants. Hence, $\mathbb{E}(Y) = 0$. Then:
-\begin{align*}
-# Y & = \beta_1 X_1 + \beta_2 X_2 + \varepsilon
-# \\
-# & = \beta_1 (U + V) + \beta_2 (U - V) + \varepsilon
-# \\
-Y & = (\beta_1 + \beta_2) U + (\beta_1 - \beta_2) V + \varepsilon
-\\
-& = \gamma_u U + \gamma_v V + \varepsilon
-\end{align*}
-and $\mbox{Cor}(X_{1}, X_{2}) = [\mbox{Var}(U) - \mbox{Var}(V)] / [ \mbox{Var}(U) + \mbox{Var}(V) ]$.
-# \begin{align*}
-# \mbox{Cor}(X_{1}, X_{2})
-# & = \frac{ \mbox{Cov}(X_{1}, X_{2}) }{ \sqrt{\mbox{Var}(X_{1})} \sqrt{\mbox{Var}(X_{2})} }
-# \\
-# & = \frac{ \mbox{Cov}(U + V, U - V) }{ \sqrt{\mbox{Var}(U + V)} \sqrt{\mbox{Var}(U - V)} }
-# \\
-# & =
-# \frac{ \mbox{Var}(U) - \mbox{Var}(V) }{ \mbox{Var}(U) + \mbox{Var}(V) }.
-# \end{align*}
-The random variables $X_1$ and $X_2$ are strongly positively correlated if $\mbox{Var}(U) \gg \mbox{Var}(V)$.
-
-The ridge regression estimator associated with regression of $Y$ on $U$ and $V$ is:
-\begin{align*}
-\ggamma(\lambda) & = \left(
-\begin{array}{rr}
-\mbox{Var}(U) + \lambda & 0
-\\
-0 & \mbox{Var}(V) + \lambda
-\end{array}
-\right)^{-1}
-\left(
-\begin{array}{r}
-\mbox{Cov}(U, Y)
-\\
-\mbox{Cov}(V, Y)
-\end{array}
-\right).
-\end{align*}
-For large enough $\lambda$
-\begin{align*}
-\ggamma(\lambda)
-# & & \frac{1}{\lambda} \left( \begin{array}{rr} \mbox{Var}(U) & 0 \\ 0 \mbox{Var}(V) \end{array} \right) \left( \begin{array}{rr} \mbox{Var}(U) & 0
-# \\
-# 0 & \mbox{Var}(V)
-# \end{array}
-# \right)^{-1}
-# \left(
-# \begin{array}{r}
-# \mbox{Cov}(U, Y)
-# \\
-# \mbox{Cov}(V, Y)
-# \end{array}
-# \right)
-# \\
-& \approx \frac{1}{\lambda}
-\left(
-\begin{array}{rr}
-\mbox{Var}(U) & 0
-\\
-0 & \mbox{Var}(V)
-\end{array}
-\right)
-\left(
-\begin{array}{r}
-\beta_1 + \beta_2
-\\
-\beta_1 - \beta_2
-\end{array}
-\right).
-\end{align*}
-When $\mbox{Var}(U) \gg \mbox{Var}(V)$ and $\beta_1 \approx \beta_2$, the ridge estimate of $\gamma_v$ vanishes for large $\lambda$. Hence, ridge regression prefers positively covariates with similar effect sizes.
-
-
-This phenomenon too can be explained geometrically. For the illustration consider ridge estimation with $\lambda=1$ of the linear model $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$ with $\beta = (3, 3)^{\top}$, $\vvarepsilon \sim \mathcal{N}(\mathbf{0}_2, \mathbf{I}_{22})$ and the columns of $\mathbf{X}$ strongly and positively collinear. The level sets of the sum-of-squares loss, $\| \mathbf{Y} - \mathbf{X} \beta \|_2^2$, are plotted in the right panel of Figure ref{fig.ridgeEstimates_correlatedCovariates}. Recall that the ridge estimate is found by looking for the smallest loss level set that hits the ridge contraint. The sought-for estimate is then the point of intersection between this level set and the constraint, and - for the case at hand - is on the $x=y$-line. This is no different from the case with orthogonal $\mathbf{X}$ columns. Yet their estimates differ, even though the same $\lambda$ is applied. The difference is to due to fact that the radius of the ridge constraint depends on $\lambda$, $\mathbf{X}$ and $\mathbf{Y}$. This is immediate from the fact that the radius of the constraint equals $\| \hat{\beta}(\lambda) \|_2^2$ (see Section ref{sect.constrainedEstimation}). To study the effect of $\mathbf{X}$ on the radius, we remove its dependence on $\mathbf{Y}$ by considering its expectation, which is:
-\begin{align*}
-\mathbb{E}[ \| \hat{\beta}(\lambda) \|_2^2 ] & = \mathbb{E} \{ [(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) \, \hat{\beta}]^{\top} \, (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X}) \, \hat{\beta} \} \nonumber % \, \, \, = \, \, \, \mathbb{E} ( \hat{\beta}^{\top} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, \hat{\beta} ) \nonumber
-\\
-# & = \mathbb{E} [ \mathbf{Y}^{\top} \mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{W}_{\lambda}^{\top} \,\mathbf{W}_{\lambda} \, (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} ] \nonumber
-# \\
-# & = \mathbb{E} [ \mathbf{Y}^{\top} \mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} (\mathbf{X}^{\top} \mathbf{X}) (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp}) (\mathbf{X}^{\top} \mathbf{X})(\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} ] \nonumber
-# \\
-& = \mathbb{E} [ \mathbf{Y}^{\top} \mathbf{X}
-(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-2} \mathbf{X}^{\top} \mathbf{Y} ] \nonumber
-\\
-& = \sigma^2 \, \mbox{tr}\big\{ \mathbf{X}
-(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-2} \mathbf{X}^{\top} \big\} + \beta^{\top} \mathbf{X}^{ \top} \mathbf{X}
-(\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-2} \mathbf{X}^{\top} \, \mathbf{X} \beta. % label{form.expectationPenalty}
-\end{align*}
-In the last step we have used $\mathbf{Y} \sim \mathcal{N}( \mathbf{X} \beta, \sigma^2 \mathbf{I}_{pp})$ and the expectation of the quadratic form of a multivariate random variable $\vvarepsilon \sim \mathcal{N}(\mmu_{\varepsilon}, \SSigma_{\varepsilon})$ is $\mathbb{E} ( \vvarepsilon^{\top} \, \LLambda \, \vvarepsilon) = \mbox{tr} ( \LLambda \, \SSigma_{\varepsilon}) + \mmu_{\varepsilon}^{\top} \, \LLambda \, \mmu_{\varepsilon}$ (cf. \citealp{Math1992}). The expression for the expectation of the radius of the ridge constraint can now be evaluated for the orthogonal $\mathbf{X}$ and the strongly, positively collinear $\mathbf{X}$. It turns out that the latter is larger than the former. This results in a larger ridge constraint. For the larger ridge constraint there is a smaller level set that hits it first. The point of intersection, still on the $x=y$-line, is now thus closer to $\beta$ and further from the origin (cf. right panel of Figure ref{fig.ridgeEstimates_correlatedCovariates}). The resulting estimate is thus larger than that from the orthogonal case.
+===== A second-order polynomial with Ridge and Lasso =====
+!bc pycod
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.linear_model import Ridge
+from sklearn.metrics import r2_score
-The above needs some attenuation. Among others it depends on: *i)* the number of covariates in each block, *ii)* the size of the effects, i.e. regression coefficients of each covariate, and
-*iii)* the degree of collinearity. Possibly, there are more factors influencing the behaviour of the ridge estimator presented in this subsection.
+np.random.seed(4155)
-This behaviour of ridge regression is to be understood when using (say) gene expression data to predict a certain clinical outcome. Genes work in concert to fulfil a certain function in the cell. Consequently, one expects their expression levels to be correlated. Indeed, gene expression studies exhibit many co-expressed genes, that is, genes with correlating transcript levels.
+n_samples = 100
+x = np.random.rand(n_samples,1)
+y = 5*x*x + 0.1*np.random.rand(n_samples,1)
+# Centering x and y.
+x_ = x - np.mean(x)
+y_ = y - np.mean(y) # beta_0 = mean(y)
+X = np.c_[np.ones((n_samples,1)), x, x**2]
+X_ = np.c_[x_, x_**2]
-======= Illustration} label{sect.ridgeRegressionDataIllustration =======
-The application of ridge regression to actual data aims to illustrate its use in practice.
-===== MCM7 expression regulation by microRNAs =====
+### 1.
+lmb_values = [1e-4, 1e-3, 1e-2, 10, 1e2, 1e4]
+num_values = len(lmb_values)
-Recently, a new class of RNA was discovered, referred to as microRNA. MicroRNAs are non-coding, single stranded RNAs of approximately 22 nucleotides. Like mRNAs, microRNAs are encoded in and transcribed from the DNA. MicroRNAs play an important role in the regulatory mechanism of the cell. MicroRNAs down-regulate gene expression by either of two post-transcriptional mechanisms: mRNA cleavage or transcriptional repression. This depends on the degree of complementarity between the microRNA and the target. Perfect or nearly perfect complementarity of the mRNA to the microRNA will lead to cleavage and degradation of the target mRNA. Imperfect complementarity will repress the productive translation and reduction in protein levels without affecting the mRNA levels.
-A single microRNA can bind to and regulate many different mRNA targets. Conversely, several microRNAs can bind to and cooperatively control a single mRNA target (\citealp{Bart2004}; \citealp{Esqu2006}; \citealp{Kim2006}).
+## Ridge-regression of centered and not centered data
+beta_ridge = np.zeros((3,num_values))
+beta_ridge_centered = np.zeros((3,num_values))
-In this illustration we wish to confirm the regulation of mRNA expression by microRNAs in an independent data set. We cherry pick an arbitrary finding from literature reported in cite{Ambs2008}, which focusses on the microRNA regulation of the MCM7 gene in prostate cancer. The MCM7 gene is involved in DNA replication \citep{Tye1999}, a cellular process often derailed in cancer. Furthermore, MCM7 interacts with the tumor-suppressor gene RB1 \citep{Ster1998}. Several studies indeed confirm the involvement of MCM7 in prostate cancer \citep{Padm2004}. And recently, it has been reported that in prostate cancer MCM7 may be regulated by microRNAs \citep{Ambs2008}.
+I3 = np.eye(3)
+I2 = np.eye(2)
-We here assess whether the MCM7 down-regulation by microRNAs can be observed in a data set other than the one upon which the microRNA-regulation of MCM7 claim has been based. To this end we download from the Gene Expression Omnibus (GEO) a prostate cancer data set (presented by \citealp{Wang2009GeneNetworks}). This data set (with GEO identifier: GSE20161) has both mRNA and microRNA profiles for all samples available. The preprocessed (as detailed in \citealp{Wang2009GeneNetworks}) data are downloaded and require only minor further manipulations to suit our purpose. These manipulations comprise *i)* averaging of duplicated profiles of several samples, *ii)* gene- and mir-wise zero-centering of the expression data, *iii)* averaging the expression levels of the probes that interrogate MCM7. Eventually, this leaves 90 profiles each comprising of 735 microRNA expression measurements.
+for i,lmb in enumerate(lmb_values):
+ beta_ridge[:,i] = (np.linalg.inv( X.T @ X + lmb*I3) @ X.T @ y).flatten()
+ beta_ridge_centered[1:,i] = (np.linalg.inv( X_.T @ X_ + lmb*I2) @ X_.T @ y_).flatten()
+# sett beta_0 = np.mean(y)
+beta_ridge_centered[0,:] = np.mean(y)
+## OLS (ordinary least squares) solution
+beta_ls = np.linalg.inv( X.T @ X ) @ X.T @ y
-\lstinputlisting{prostate_ridgeAnalysis.r}
+## Evaluate the models
+pred_ls = X @ beta_ls
+pred_ridge = X @ beta_ridge
+pred_ridge_centered = X_ @ beta_ridge_centered[1:] + beta_ridge_centered[0,:]
+## Plot the results
-With this prostate data set at hand we now investigate whether MCM7 is regulated by microRNAs. Hereto we fit a linear regression model regressing the expression levels of MCM7 onto those of the microRNAs. As the number of microRNAs exceeds the number of samples, ordinary least squares fails and we resort to the ridge estimator of the regression coefficients. First, an informed choice of the penalty parameter is made through maximization of the LOOCV log-likelihood, resulting in $\lambda_{\mbox{{\tiny opt}}} = 1812.826$. Having decided on the value of the to-be-employed penalty parameter, the ridge regression estimator can now readily be evaluated. The thus fitted model allows for the evaluation of microRNA-regulation of MCM7. E.g., by the proportion of variation of the MCM7 expression levels by the microRNAs as expressed in coefficient of determination: $R^2 = 0.4492$. Alternatively, but closely related, observed expression levels may be related to the linear predictor of the MCM7 expression levels: $\hat{\mathbf{Y}}(\lambda_{\mbox{{\tiny opt}}}) = \mathbf{X} \hat{\beta} (\lambda_{\mbox{{\tiny opt}}})$. The Spearman correlation of response and predictor equals 0.6295. A visual inspection is provided by the left panel of Figure ref{fig.RidgeProstateExample}. Note the difference in scale of the $x$- and $y$-axes. This is due to the fact that the regression coefficients have been estimated in penalized fashion, consequently shrinking estimates of the regression coefficients towards zero leading to small estimates and in turn compressing the range of the linear prediction. The above suggests there is indeed association between the microRNA expression levels and those of MCM7.
+# Sorting
+sort_ind = np.argsort(x[:,0])
+x_plot = x[sort_ind,0]
+x_centered_plot = x_[sort_ind,0]
+pred_ls_plot = pred_ls[sort_ind,0]
+pred_ridge_plot = pred_ridge[sort_ind,:]
+pred_ridge_centered_plot = pred_ridge_centered[sort_ind,:]
-# original latex figure with scale=0.45, angle=0
+# Plott not centered
+plt.plot(x_plot,pred_ls_plot,label='ls')
-FIGURE: [prostate_ridgeFit.eps, width=400 frac=1.0] Left panel: Observed vs. (ridge) fitted MCM7 expression values. Right panel: Histogram of the ridge regression coefficient estimates.} label{fig.RidgeProstateExample
+for i in range(num_values):
+ plt.plot(x_plot,pred_ridge_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
+plt.plot(x,y,'ro')
-The overall aim of this illustration was to assess whether microRNA-regulation of MCM7 could also be observed in this prostate cancer data set. In this endeavour the dogma (stating this regulation should be negative) has nowhere been used. A first simple assessment of the validity of this dogma studies the signs of the estimated regression coefficients. The ridge regression estimate has 394 out of the 735 microRNA probes with a negative coefficient. Hence, a small majority has a sign in line with the `microRNA $\downarrow$ mRNA' dogma. When, in addition, taking the size of these coefficients into account (Figure ref{fig.RidgeProstateExample}, right panel), the negative regression coefficient estimates do not substantially differ from their positive counterparts (as can be witnessed from their almost symmetrical distribution around zero). Hence, the value of the `microRNA $\downarrow$ mRNA' dogma is not confirmed by this ridge regression analysis of the MCM7-regulation by microRNAs. Nor is it refuted.
+plt.title('linear regression on un-centered data')
+plt.legend()
+# Plott centered
+plt.figure()
-The implementation of ridge regression in the {\tt penalized}-package offers the possibility to fully obey the dogma on negative regulation of mRNA expression by microRNAs. This requires all regression coefficients to be negative. Incorporation of the requirement into the ridge estimation augments the constrained estimation problem with an additional constraint:
-\begin{align*} % label{form.constrEstProblemRidge}
-\hat{\beta}(\lambda) & = \arg \min_{\| \beta \|_2^2 \leq c (\lambda) \atop \beta_j \leq 0 \, \mbox{{\tiny for all $j$}}} \| \mathbf{Y} - \mathbf{X} \, \beta \|^2_2.
-\end{align*}
-With the additional non-positivity constraint on the parameters, there is no explicit solution for the estimator. The ridge estimate of the regression parameters is then found by numerical optimization using e.g. the Newton-Raphson algorithm or a gradient descent approach. The next listing gives the R-code for ridge estimation with the non-positivity constraint of the linear regression model.
+for i in range(num_values):
+ plt.plot(x_centered_plot,pred_ridge_centered_plot[:,i],label='ridge, lmb=%g'%lmb_values[i])
-\lstinputlisting{prostate_ridgeAnalysisWithConstraint.r}
+plt.plot(x_,y,'ro')
-The linear regression model linking MCM7 expression to that of the microRNAs is fitted by ridge regression while simultaneously obeying the `negative regulation of mRNA by microRNA'-dogma to the prostate cancer data. In the resulting model 401 out of 735 microRNA probes have a nonzero (and negative) coefficient. There is a large overlap in microRNAs with a negative coefficient between those from this and the previous fit. The models are also compared in terms of their fit to the data. The Spearman rank correlation coefficient between response and predictor for the model without positive regression coefficients equals 0.679 and its coefficient of determination 0.524 (confer the left panel of ref{fig.RidgeProstateExample_constrainedAnalysis} for a visualization). This is a slight improvement upon the unconstrained ridge estimated model. The improvement may be small but it should be kept in mind that the number of parameters used by both models is 401 (for the model without positive regression coefficients) vs. 735. Hence, with close to half the number of parameters the dogma-obeying model gives a somewhat better description of the data. This may suggest that there is some value in the dogma as inclusion of this prior information leads to a more parsimonious model without any loss in fit.
+plt.title('linear regression on centered data')
+plt.legend()
+# 2.
-# original latex figure with scale=0.45, angle=0
+pred_ridge_scikit = np.zeros((n_samples,num_values))
+for i,lmb in enumerate(lmb_values):
+ pred_ridge_scikit[:,i] = (Ridge(alpha=lmb,fit_intercept=False).fit(X,y).predict(X)).flatten() # fit_intercept=False fordi bias er allerede i X
-FIGURE: [prostate_ridgeFitConstrained.eps, width=400 frac=1.0] Left panel: Observed vs. (ridge) fitted MCM7 expression values (with the non-positive constraint on the parameters in place). Right panel: Histogram of the ridge regression coefficient estimates (from the non-positivity constrained analysis).} label{fig.RidgeProstateExample_constrainedAnalysis
+plt.figure()
+plt.plot(x_plot,pred_ls_plot,label='ls')
+for i in range(num_values):
+ plt.plot(x_plot,pred_ridge_scikit[sort_ind,i],label='scikit-ridge, lmb=%g'%lmb_values[i])
-The dogma-obeying model selects 401 microRNAs that aid in the explanation of the variation in the gene expression levels of MCM7. There is an active field of research, called *target prediction*, trying to identify which microRNAs target the mRNA of which genes. Within {\tt R} there is a collection of packages that provide the target prediction of known microRNAs. The packages differ on the method (e.g. experimental or sequence comparison) that has been used to arrive at the prediction. These target predictions may be used to evaluate the value of the found 401 microRNAs. Ideally, there would be a substantial amount of overlap. The {\tt R}-script that loads the target predictions and does the comparison is below.
+plt.plot(x,y,'ro')
+plt.legend()
+plt.title('linear regression using scikit')
-\lstinputlisting{prostate_link2predictionTools.r}
+plt.show()
+### R2-score of the results
+for i in range(num_values):
+ print('lambda = %g'%lmb_values[i])
+ print('r2 for scikit: %g'%r2_score(y,pred_ridge_scikit[:,i]))
+ print('r2 for own code, not centered: %g'%r2_score(y,pred_ridge[:,i]))
+ print('r2 for own, centered: %g\n'%r2_score(y,pred_ridge_centered[:,i]))
-|-----------------------------------------------------------------|
-| # #if FORMAT in ("latex", "pdflatex")
-\vspace{-7pt}
-# #endif | | |
-|---------r--------------------r--------------------r-------------|
-| # #if FORMAT in ("latex", "pdflatex")
-\vspace{3pt}
-# #endif | $\hat{\beta}_j = 0$ | $\hat{\beta}_j < 0$ |
-| # #if FORMAT in ("latex", "pdflatex")
-\vspace{-4pt}
-# #endif | | |
-| microRNA not target | 323 | 390 |
-| microRNA target | 11 | 11 |
-| # #if FORMAT in ("latex", "pdflatex")
-\vspace{-9pt}
-# #endif | | |
-|-----------------------------------------------------------------|
+!ec
-\caption{Cross-tabulation of the microRNAs being a potential target of MCM7 vs. the value of its regression coefficient in the dogma-obeying model.} label{table.mir2mcm7prediction}
+!split
+===== Fitting vs. predicting when data is in the model class =====
-With knowledge available on each microRNA whether it is predicted (by at least one target prediction package) to be a potential target of MCM7, it may be cross-tabulated against its corresponding regression coefficient estimate in the dogma-obeying model being equal to zero or not. Table ref{table.mir2mcm7prediction} contains the result. Somewhat superfluous considering the data, we may test whether the targets of MCM7 are overrepresented in the group of strictly negatively estimated regression coefficients. The corresponding chi-squared test (with Yates' continuity correction) yields the test statistic $\chi^2 = 0.0478$ with a $p$-value equal to 0.827. Hence, there is no enrichment among the 401 microRNAS of those that have been predicted to target MCM7. This may seem worrisome. However, the microRNAs have been selected for their predictive power of the expression levels of MCM7. Variable selection has not been a criterion (although the sign constraint implies selection). Moreover, criticism on the value of the microRNA target prediction has been accumulating in recent years.
+We start by considering the case
+$f(x)=2x$.
+
+Then the data is clearly generated by a model that is contained within
+all three model classes we are using to make predictions (linear
+models, third order polynomials, and tenth order polynomials).
+Run the code for the following cases:
-======= Conclusion =======
+o For $f(x)=2x$ , $Ntrain=10$ and $\sigma =0$ (noiseless case), train the three classes of models (linear, third-order polynomial, and tenth order polynomial) for a training set when $x \in [0,1]$ . Make graphs comparing fits for different order of polynomials. Which model fits the data the best?
+o Do you think that the data that has the least error on the training set will also make the best predictions? Why or why not? Can you try to discuss and formalize your intuition? What can go right and what can go wrong?
+o Check your answer by seeing how well your fits predict newly generated test data (including on data outside the range you fit on, for example $x \in [0,1.2]$ ) using the code below. How well do you do on points in the range of x where you trained the model? How about points outside the original training data set?
+o Repeat the above for $f(x)=2x$ , $Ntrain=10$ , and $\sigma=1$ . What changes?
+Repeat the exercises above for $f(x)=2x$ , $Ntrain=100$ , and $\sigma=1$ . What changes?
+Summarize what you have learned about the relationship between model complexity (number of parameters), goodness of fit on training data, and the ability to predict well.
-We discussed ridge regression as a modification of linear regression to overcome the empirical non-identifiability of the latter when confronted with high-dimensional data. The means to this end was the addition of a (ridge) penalty to the sum-of-squares loss function of the linear regression model, which turned out to be equivalent to constraining the parameter domain. This warranted the identification of the regression coefficients, but came at the cost of introducing bias in the estimates. Several properties of ridge regression like moments, MSE, and its Bayesian interpretation have been reviewed. Finally, its behaviour and use have been illustrated in simulation and omics data.
+!split
+===== Fitting versus predicting when data is not in the model class =====
-======= Exercises =======
-\begin{question}[^footnote2][^footnote2]: This exercise is freely rendered from cite{Drap1998} \mbox{ }
-\\
-Find the ridge regression solution for the data below for a general value of $\lambda$ and for the straight line model $Y = \beta_0 + \beta_1 X + \varepsilon$ (only apply the ridge penalty to the slope parameter, not to the intercept). Show that when $\lambda$ is chosen as 0.4, the ridge solution fit is $\hat{Y} = 40 + 1.75 X$. Data: $\mathbf{X}^{\top} = (X_1, X_2, \ldots, X_{8})^{\top} = (-2, -1, -1, -1, 0, 1, 2, 2)^{\top}$, and $\mathbf{Y}^{\top} = (Y_1, Y_2, \ldots, Y_{8})^{\top} = (35, 40, 36, 38, 40, 43, 45, 43)^{\top}$.
-\end{question}
+Thus far, we have considered the case where the data is generated using a model contained in the model class. Now consider $f(x)=2x-10x^5+15x^{10}$ . Notice that the for linear and third-order polynomial the true model $f(x)$ is not contained in model class.
-\begin{question}[^footnote3][^footnote3]: This exercise is freely rendered from cite{Hast2009, but can be found in many other places. The original source is unknown to the author.} label{question:ridgeAugmentation} \mbox{ }
-\\
-Show that the ridge regression estimates can be obtained by ordinary least squares regression on an augmented data set. We augment the centered matrix $\mathbf{X}$ with $p$ additional row $\sqrt{\lambda} \mathbf{I}$, and augment $\mathbf{y}$ with $p$ zeros.
-\end{question}
+o Do better fits lead to better predictions?
+o What is the relationship between the true model for generating the data and the model class that has the most predictive power? How is this related to the model complexity? How does this depend on the number of data points $Ntrain$ and $\sigma$?
+Summarize what you think you learned about the relationship of knowing the true model class and predictive power.
-\begin{question} label{question.ridgeResidualsProjection} \mbox{ }
-\\
-The coefficients $\beta$ of a linear regression model, $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$, are estimated by $\hat{\beta} = (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}$. The associated fitted values then given by $\widehat{\mathbf{Y}} = \mathbf{X} \, \hat{\beta} = \mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y} = \mathbf{H} \mathbf{Y}$, where $\mathbf{H} =\mathbf{X} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top}$ referred to as the hat matrix. The matrix $\mathbf{P}$ is a projection matrix and satisfies $\mathbf{H} = \mathbf{H}^ 2$. Hence, linear regression projects the response $\mathbf{Y}$ onto the vector space spanned by the columns of $\mathbf{Y}$. Consequently, the residuals $\hat{\vvarepsilon}$ and $\hat{\mathbf{Y}}$ are orthogonal. Now consider the ridge estimator of the regression coefficients: $\hat{\beta}(\lambda) = (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top} \mathbf{Y}$. Let $\hat{\mathbf{Y}}(\lambda) = \mathbf{X} \hat{\beta}(\lambda)$ be the vector of associated fitted values.
+!split
+===== The code =====
-\begin{compactitem}
-\item[*a)*] Show that the matrix $\mathbf{H}(\lambda) = \mathbf{X} (\mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}^{\top}$, associated with ridge regression, is not a projection matrix (for any $\lambda > 0$).
+!bc pycod
+import numpy as np
+import sklearn as sk
+from sklearn import datasets, linear_model
+from sklearn.preprocessing import PolynomialFeatures
-\item[*b)*] Show that the `ridge fit' $\widehat{\mathbf{Y}}(\lambda)$ is not orthogonal to the associated `ridge residuals' $\hat{\vvarepsilon}(\lambda)$ (for any $\lambda > 0$).
+import matplotlib as mpl
+from matplotlib import pyplot as plt
-\item[*c)*] Derive the distribution of the `ridge residuals'.
-\end{compactitem}
-\end{question}
+%matplotlib notebook
+# The Training Data
-\begin{question} label{question.MSEofLinearPredictor} \mbox{ }
-\\
-Recall that there exists $\lambda > 0$ such that $MSE(\hat{\beta}) > MSE[\hat{\beta}(\lambda)]$. Verify that this carries over to the linear predictor. That is, there exists a $\lambda > 0$ such that$MSE(\widehat{\mathbf{Y}}) = MSE(\mathbf{X} \hat{\beta}) > MSE[\mathbf{X}\hat{\beta}(\lambda)]$.
-\end{question}
+N_train=100
-\begin{question} \mbox{ }
-\\
-Consider a 3-gene pathway. Expression levels of these three genes have been measured in an observational study involving hundred individuals. In order to assess how the expression levels of gene A are affect by that of genes B and C, a medical researcher fits the
-\begin{align*}
-Y_{i}^{(A)} &= \beta_b Y_{i}^{(B)} + \beta_c Y_{i}^{(C)} +
-\varepsilon_{i},
-\end{align*}
-with $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$. This model fitted by means of ridge regression, but with a separate penalty parameter, $\lambda_{2,b}$ and $\lambda_{2,c}$, for the two regression coefficient, $\beta_b$ and $\beta_c$, respectively.
+sigma_train=1;
-\begin{compactitem}
-\item[*a)*] Write down the ridge penalized loss function employed by the researcher.
+# Train on integers
+x=np.linspace(0.05,0.95,N_train)
+# Draw random noise
+s = sigma_train*np.random.randn(N_train)
-\item[*b)*] Does a different choice of penalty parameter for the second regression coefficient affect the estimation of the first regression coefficient? Motivate your answer.
+#linear
+y=2*x+s
-\item[*c)*] The researcher decides that the second covariate $Y_i^{(C)}$ is irrelevant. Instead of removing the covariate from model, the researcher decides to set $\lambda_{2,c} =
-\infty$. Show that this results in the same ridge estimate for $\beta_b$ as when fitting (again by means of ridge regression) the model without the second covariate.
-\end{compactitem}
-\end{question}
+#Tenth Order
+#y=2*x-10*x**5+15*x**10+s
-\begin{question} \mbox{ }
-\\
-The expression levels of the $j$-the gene are explained by a linear regression model in terms of those of all other genes. Consider the following two ridge regression estimators of the regression parameter of this model, defined as:
-\begin{align*}
-\arg \max_\beta \sum_{i=1}^n (Y_{i,j} - \mathbf{Y}_{i,\setminus j} \beta_j)^2 + \lambda \| \beta_j \|_ 2^2
-\quad
-\mbox{and} \quad
-\arg \max_\beta \sum_{i=1}^n (Y_{i,j} - \mathbf{Y}_{i,\setminus j} \beta_j)^2 + n \lambda \| \beta_j \|_ 2^2.
-\end{align*}
-Which do you prefer? Motivate.
-\end{question}
+p1=plt.plot(x,y, "o",ms=15, label='Training')
+#Linear Regression
+# Create linear regression object
+clf = linear_model.LinearRegression()
+# Train the model using the training sets
+clf.fit(x[:, np.newaxis], y)
+# The coefficients
+xplot=np.linspace(0.02,0.98,200)
+linear_plot=plt.plot(xplot, clf.predict(xplot[:, np.newaxis]),label='Linear')
-========= Generalizing ridge regression ========= label{chap:genRidge}
-# ======= Generalized ridge regression =======
+#Polynomial Regression
-The expos\'{e} on ridge regression may be generalized in many ways. Among others different generalized linear models may be considered (confer Section ref{sect.ridgeLogistic}). In this section we stick to the linear regression model $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$ with the usual assumptions, but fit it in weighted fashion and generalize the common, spherical penalty. The loss function corresponding to this scenario is:
-!bt
-
-!bt
-\begin{align} label{form:generalizedRidgeLoss}
-(\mathbf{Y} - \mathbf{X} \beta)^{\top} \mathbf{W} (\mathbf{Y} - \mathbf{X} \beta) + (\beta - \beta_0)^{\top} \mathbf{\Delta}
-(\beta - \beta_0),
-\end{align}
-!et
-!et
-which comprises a weighted least squares criterion and a generalized ridge penalty. In this $\mathbf{W}$ is a $(n \times n)$-dimensional, diagonal matrix with $(\mathbf{W})_{ii} \in [0,1]$ representing the weight of the $i$-th observation. The penalty is now a quadratic form with penalty parameter $\mathbf{\Delta}$, a $(p \times p)$-dimensional, positive definite, symmetric matrix. When $\mathbf{\Delta} = \lambda \mathbf{I}_{pp}$, one regains the spherical penalty of `regular ridge regression'. This penalty shrinks each element of the regression parameter $\beta$ equally along the unit vectors $\mathbf{e}_j$. Generalizing $\mathbf{\Delta}$ to the class of symmetric, positive definite matrices $\mathcal{S}_{++}$ allows for *i)* different penalization per regression parameter, and *ii)* joint (or correlated) shrinkage among the elements of $\beta$. The penalty parameter $\mathbf{\Delta}$ determines the speed and direction of shrinkage. The $p$-dimensional column vector $\beta_0$ is a user-specified, non-random target towards which $\beta$ is shrunken as the penalty parameter increases. When recasting generalized ridge estimation as a constrained estimation problem, the implications of the penalty may be visualized (Figure ref{fig:generalizedRidgeIllustration}, left panel). The generalized ridge penalty is a quadratic form centered around $\beta_0$. In Figure ref{fig:generalizedRidgeIllustration} the parameter constraint clearly is ellipsoidal (and not spherical). Moreover, the center of this ellipsoid is not at zero.
+poly3 = PolynomialFeatures(degree=3)
+X = poly3.fit_transform(x[:,np.newaxis])
+clf3 = linear_model.LinearRegression()
+clf3.fit(X,y)
-# original latex figure with scale=0.40, angle=0
-FIGURE: [generalizedRidgeIllustration_penalty.eps, width=400 frac=1.0] Left panel: the contours of the likelihood (grey solid ellipsoids) and the parameter constraint implied by the generalized penalty (black dashed ellipsoid. Right panel: generalized (fat coloured lines) and `regular' (thin coloured lines) regularization paths of four regression coefficients. The dotted grey (straight) lines indicated the targets towards the generalized ridge penalty shrinks regression coefficient estimates.} label{fig:generalizedRidgeIllustration
+Xplot=poly3.fit_transform(xplot[:,np.newaxis])
+poly3_plot=plt.plot(xplot, clf3.predict(Xplot), label='Poly 3')
-The addition of the generalized ridge penalty to the sum-of-squares ensures the existence of a unique regression estimator in the face of super-collinearity. The generalized penalty is a non-degenerated quadratic form in $\beta$ due to the positive definiteness of the matrix $\mathbf{\Delta}$. As it is non-degenerate, it is strictly convex. Consequently, the generalized ridge regression loss function (ref{form:generalizedRidgeLoss}), being the sum of a convex and strictly convex function, is also strictly convex. This warrants the existence of a unique global minimum and, thereby, a unique estimator.
+#poly5 = PolynomialFeatures(degree=5)
+#X = poly5.fit_transform(x[:,np.newaxis])
+#clf5 = linear_model.LinearRegression()
+#clf5.fit(X,y)
-Like for the `regular' ridge loss function (ref{form.ridgeLossFunction}), there is an explicit expression for the optimum of the generalized ridge loss function (ref{form:generalizedRidgeLoss}). To see this, obtain the estimating equation of $\beta$ through equating its derivative with respect to $\beta$ to zero:
-\begin{align*}
-2 \mathbf{X}^{\top} \mathbf{W} \mathbf{Y} - 2 \mathbf{X}^{\top} \mathbf{W} \mathbf{X} \beta - 2 \mathbf{\Delta} \beta + 2 \mathbf{\Delta} \beta_0 & = \mathbf{0}_{p}.
-\end{align*}
-This is solved by:
-
-!bt
-
-!bt
-\begin{align}
-\hat{\beta}(\mathbf{\Delta}) & = (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-1} (\mathbf{X}^{\top} \mathbf{W} \mathbf{Y} + \mathbf{\Delta} \beta_0).
-\end{align}
-!et
-!et
-Clearly, this reduces to the `regular' ridge estimator by setting $\mathbf{W} = \mathbf{I}_{nn}$, $\beta_0 = \mathbf{0}_{p}$, and $\mathbf{\Delta} = \lambda \mathbf{I}_{pp}$. The effects of the generalized ridge penalty on the estimates can be seen in the regularization paths of the estimates. Figure ref{fig:generalizedRidgeIllustration} (right panel) contains an example of the regularization paths for coefficients of a linear regression model with four explanatory variables. Most striking is the limiting behaviour of the estimates of $\beta_3$ and $\beta_4$ for large values of the penalty parameter $\lambda$: they convergence to a non-zero value (as was specified by a nonzero $\beta_0$). More subtle is the (temporary) convergence of the regularization paths of the estimates of $\beta_2$ and $\beta_3$. That of $\beta_2$ is pulled away from zero (its true value and approximately its unpenalized estimate) towards the estimate of $\beta_3$. In the regularization path of $\beta_3$ this can be observed in a delayed convergence to its nonzero target value (for comparison consider that of $\beta_4$). For reference the corresponding regularization paths of the `regular' ridge estimates (as thinner lines of the same colour) are included in Figure ref{fig:generalizedRidgeIllustration}.
-
-
-
-\begin{example} *Fused ridge estimation* label{example.gRidge2orthonormalDesign}
-\\
-An example of a generalized ridge penalty is the *fused ridge penalty*. Consider the standard linear model $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$. The fused ridge estimator of $\beta$ then minimizes:
-
-!bt
-
-!bt
-\begin{align} label{form:fusedRidgeLoss}
-\| \mathbf{Y} - \mathbf{X} \beta \|_2^2 + \lambda \sum_{j=2}^p \| \beta_{j} - \beta_{j-1} \|_2^2.
-\end{align}
-!et
-!et
-The penalty in the loss function above can be written as a generalized ridge penalty:
-\begin{align*}
-\lambda \sum_{j=2}^p \| \beta_{j} - \beta_{j-1} \|_2^2 & = \beta^{\top}
-\left(
-\begin{array}{rrrrrr}
-\lambda & -\lambda & 0 & \ldots & \ldots & 0
-# #if FORMAT in ("latex", "pdflatex")
-\vspace{-4pt}
-# #endif
-\\
--\lambda & 2 \lambda & -\lambda & \ddots & & \vdots
-# #if FORMAT in ("latex", "pdflatex")
-\vspace{-4pt}
-# #endif
-\\
-0 & -\lambda & 2 \lambda & \ddots & \ddots & \vdots
-# #if FORMAT in ("latex", "pdflatex")
-\vspace{-4pt}
-# #endif
-\\
-\vdots & \ddots & \ddots & \ddots & \ddots & 0
-# #if FORMAT in ("latex", "pdflatex")
-\vspace{-4pt}
-# #endif
-\\
-\vdots & & \ddots & \ddots & \ddots & -\lambda
-# # #if FORMAT in ("latex", "pdflatex")
-\vspace{-4pt}
-# #endif
-\\
-0 & \ldots & \ldots & 0 & -\lambda & \lambda
-\end{array}\right)
-\beta.
-\end{align*}
-The matrix $\mathbf{\Delta}$ employed above is semi-positive definite and therefore the loss function (ref{form:fusedRidgeLoss}) is not strictly convex. Hence, often a regular ridge penalty $\| \beta \|_2^2$ is added (with its own penalty parameter).
-
-To illustrate the effect of the fused ridge penalty on the estimation of the linear regression model $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$, let $\beta_j = \phi_{0,1}(z_j)$ with $z_j =-30 + \tfrac{6}{50} j$ for $j=1, \ldots, 500$. Sample the elements of the design matrix $\mathbf{X}$ and those of the error vector $\vvarepsilon$ from the standard normal distribution, then form the response $\mathbf{Y}$ from the linear model. The regression parameter is estimated through fused ridge loss minimization with $\lambda=1000$. The estimate is shown in Figure ref{fig:fusedRidgeIllustration} (red line). For reference the figure includes the true $\beta$ (black line) and the `regular ridge' estimate with $\lambda=1$ (blue line). Clearly, the fused ridge estimate yields a nice smooth vector of $\beta$ estimates
-\end{example}
-
-# original latex figure with scale=0.40, angle=0
-
-FIGURE: [fusedRidgeIllustration_simulation.eps, width=400 frac=1.0] Left panel: illustration of the fused ridge estimator (in simulation). The true parameter $\beta$ and its ridge and fused ridge estimates against their spatial order. Right panel: Ridge vs. fused ridge estimates of the DNA copy effect on KRAS expression levels. The dashed, grey vertical bar indicates the location of the KRAS gene. } label{fig:fusedRidgeIllustration
-
-
-
-
-======= Moments =======
-
-The expectation and variance of $\hat{\beta}(\mathbf{\Delta})$ are obtained through application of the same matrix algebra and expectation and covariance rules used in the derivation of their counterparts of the `regular' ridge regression estimator. This leads to:
-\begin{align*}
-\mathbb{E}[\hat{\beta}(\mathbf{\Delta})] & = (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-1} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} \beta + \mathbf{\Delta} \beta_0),
-\\
-\mbox{Var}[\hat{\beta}(\mathbf{\Delta})] & = \sigma^2 (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-1} \mathbf{X}^{\top} \mathbf{W}^2 \mathbf{X} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-1}.
-\end{align*}
-From these expressions similar limiting behaviour as for the `regular' ridge regression case can be deduced. To this end let $\mathbf{V}_{\delta} \mathbf{D}_{\delta} \mathbf{V}_{\delta}^{\top}$ be the eigendecomposition of $\mathbf{\Delta}$ and $d_{\delta,j} = (\mathbf{D}_{\delta})_{jj}$. Furthermore, define (with some abuse of notation) $\lim_{\mathbf{\Delta} \rightarrow \infty}$ as the limit of all $d_{\delta,j}$ simultaneously tending to infinity. Then, $\lim_{\mathbf{\Delta} \rightarrow \infty} \mathbb{E}[\hat{\beta}(\mathbf{\Delta})] = \beta_0$ and $\lim_{\mathbf{\Delta} \rightarrow \infty} \mbox{Var}[\hat{\beta}(\mathbf{\Delta})] = \mathbf{0}_{pp}$.
-
-
-\begin{example} \mbox{ } label{example.gRidge2orthonormalDesign}
-\\
-Let $\mathbf{X}$ be an $(n \times p)$-dimensional, orthonormal design matrix. Contrast the regular and generalized ridge regression estimator, the latter with $\mathbf{W} = \mathbf{I}_{pp}$, $\beta_0 = \mathbf{0}_p$ and $\mathbf{\Delta} = \lambda \mathbf{R}$ where $\mathbf{R} = (1-\rho) \mathbf{I}_{pp} + \rho \mathbf{1}_{pp}$ for $\rho \in (-(p-1)^{-1}, 1)$. For $\rho =0$ the two estimators coincide. The variance of the generalized ridge regression estimator then is $\mbox{Var}[ \hat{\beta}(\mathbf{\Delta})] = (\mathbf{I}_{pp} + \mathbf{\Delta})^{-2}$. The efficiency of this estimator, measured by its generalized variance, is:
-\begin{align*}
-\det \{ \mbox{Var}[ \hat{\beta}(\mathbf{\Delta})] \} & = \{ [1 + \lambda + (p-1) \rho] (1 + \lambda-\rho)^{p-1} \}^{-2}.
-\end{align*}
-This efficiency attains its minimum at $\rho = 0$. In the present case, the regular ridge regression estimator is thus more efficient than its generalized counterpart.
-\end{example}
-
-
-\begin{example} *(MSE with perfect target)*
-\\
-Set $\beta_0 = \beta$, i.e. the target is equal to the true value of the regression parameter. Then:
-\begin{align*}
-\mathbb{E}[\hat{\beta}(\Delta)] & = (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \Delta)^{-1} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} \beta + \Delta \beta) \, \, \, = \, \, \, \beta.
-\end{align*}
-Hence, irrespective of the choice of $\Delta$, the generalized ridge is then unbiased. Thus:
-\begin{align*}
-\mbox{MSE}[\hat{\beta}(\Delta)] & = \mbox{tr} \{
-\mbox{Var}[\hat{\beta}(\Delta)] \}
-\\
-& = \mbox{tr}[ \sigma^{2} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-1} \mathbf{X}^{\top} \mathbf{W}^2 \mathbf{X} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-1}]
-\\
-& = \sigma^2 \mbox{tr}[ \mathbf{X}^{\top} \mathbf{W}^2 \mathbf{X} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \mathbf{\Delta})^{-2}].
-\end{align*}
-When $\mathbf{\Delta} = \lambda \mathbf{I}_{pp}$, this MSE is smaller than that of the ML regression estimator, irrespective of the choice of $\lambda$.
-\end{example}
-
-
-======= The Bayesian connection} label{sect:genRidgeBayes =======
-This generalized ridge estimator can, like the regular ridge estimator, be viewed as a Bayesian estimator. It requires to replace the conjugate prior on $\beta$ by a more general normal law, $\beta \sim \mathcal{N}(\beta_0, \sigma^2 \mathbf{\Delta}^{-1})$, but retains the gamma prior on $\sigma^2$. The joint posterior distribution of $\beta$ and $\sigma^2$ is then obtained analogously (the details are left as Exercise ref{question.generalizedRidgeAndBayes}) to Section ref{sect:Bayes}:
-\begin{align*}
-f_{\beta, \sigma^2} (\beta, \sigma^2 \, | \, \mathbf{Y}, \mathbf{X}) & = f_Y (\mathbf{Y} \, | \, \mathbf{X}, \beta, \sigma^2) \, f_{\beta}(\beta | \sigma^2) \, f_{\sigma}(\sigma^2)
-\\
-& \propto & g_{\beta} (\beta \, | \, \sigma^2, \mathbf{Y}, \mathbf{X}) \,
-g_{\sigma^2} (\sigma^2 \, | \, \mathbf{Y}, \mathbf{X})
-\end{align*}
-with
-\begin{align*}
-g_{\beta} (\beta \, | \, \sigma^2, \mathbf{Y}, \mathbf{X})
-& \propto & \exp \Big\{ - \frac{1}{2\sigma^2} \big[ \beta - \hat{\beta}(\mathbf{\Delta}) \big]^{\top} (\mathbf{X}^{\top} \mathbf{X} + \mathbf{\Delta}) \big[ \beta - \hat{\beta}(\mathbf{\Delta}) \big] \Big\}.
-\end{align*}
-This implies $\mathbb{E}(\beta \, | \, \sigma^2, \mathbf{Y}, \mathbf{X}) = \hat{\beta}(\mathbf{\Delta})$. Hence, the generalized ridge regression estimator too can be viewed as the Bayesian posterior mean estimator of $\beta$ when imposing a multivariate Gaussian prior on the regression parameter.
-
-
-
-
-
-======= Application =======
-
-An illustration involving omics data can be found in the explanation of a gene's expression levels in terms of its DNA copy number. The latter is simply the number of gene copies encoded in the DNA. For instance, for most genes on the autosomal chromosomes the DNA copy number is two, as there is a single gene copy on each chromosome and autosomal chromosomes come in pairs. Alternatively, in males the copy number is one for genes that map to the X or Y chromosome, while in females it is zero for genes on the Y chromosome. In cancer the DNA replication process has often been compromised leading to a (partially) reshuffled and aberrated DNA. Consequently, the cancer cell may exhibit gene copy numbers well over a hundred for classic oncogenes. A faulted replication process does - of course - not nicely follow the boundaries of gene encoding regions. This causes contiguous genes to commonly share aberrated copy numbers. With genes being transcribed from the DNA and a higher DNA copy number implying an enlarged availability of the gene's template, the latter is expected to lead to elevated expression levels. Intuitively, one expects this effect to be localized (a so-called *cis*-effect), but some suggest that aberrations elsewhere in the DNA may directly affect the expression levels of distant genes (referred to as a *trans*-effect).
-
-The *cis*- and *trans*-effects of DNA copy aberrations on the expression levels of the KRAS oncogene in colorectal cancer are investigated. Data of both molecular levels from the TCGA (The Cancer Genome Atlas) repository are downloaded \citep{TCGA2012colon}. The gene expression data are limited to that of KRAS, while for the DNA copy number data only that of chromosome 12, which harbors KRAS, is retained. This leaves genomic profiles of 195 samples comprising 927 aberrations. Both molecular data types are zero centered feature-wise. Moreover, the data are limited to ten - conveniently chosen? - samples. The KRAS expression levels are explained by the DNA copy number aberrations through the linear regression model. The model is fitted by means of ridge regression, with $\lambda \mathbf{\Delta}$ and $\mathbf{\Delta} = \mathbf{I}_{pp}$ and a single-banded $\mathbf{\Delta}$ with unit diagonal and the elements of the first off-diagonal equal to the arbitrary value of $-0.4$. The latter choice appeals to the spatial structure of the genome and encourages similar regression estimates for contiguous DNA copy numbers. The penalty parameter is chosen by means of leave-one-out cross-validation using the squared error loss.
-
-\lstinputlisting{fusedRidge_KRASexample.r}
-
-The right panel of Figure ref{fig:fusedRidgeIllustration} shows the ridge regression estimate with both choices of $\mathbf{\Delta}$ and optimal penalty parameters plotted against the chromosomal order. The location of KRAS is indicated by a vertical dashed bar. The ordinary ridge regression estimates show a minor peak at the location of KRAS but is otherwise more or less flat. In the generalized ridge estimates the peak at KRAS is emphasized. Moreover, the region close to KRAS exhibits clearly elevated estimates, suggesting co-abberated DNA. For the remainder the generalized ridge estimates portray a flat surface, with the exception of a single downward spike away from KRAS. Such negative effects are biologically nonsensible (more gene templates leading to reduced expression levels?). On the whole the generalized ridge estimates point towards the *cis*-effect as the dominant genomic regulation mechanism of KRAS expression. The isolated spike may suggest the presence of a *trans*-effect, but its sign is biological nonsensible and the spike is fully absent in the ordinary ridge estimates. This leads us to ignore the possibility of a genomic *trans*-effect on KRAS expression levels in colorectal cancer.
-
-The sample selection demands justification. It yields a clear illustrate-able difference between the ordinary and ridge estimates. When all samples are left in, the *cis*-effect is clearly present, discernable from both estimates that yield a virtually similar profile.
-
-
-======= Generalized ridge regression =======
-
-What is generally referred to as `generalized ridge regression' (cf. \citealp{Hoer1970, Hemm1975}) is the particular case of loss function (ref{form:generalizedRidgeLoss}) in which $\mathbf{W} = \mathbf{I}_{nn}$, $\beta_0 = \mathbf{0}_{p}$, and $\mathbf{\Delta} = \mathbf{V}_{x} \mathbf{\Lambda} \mathbf{V}_x^{\top}$, where $\mathbf{V}_x$ is obtained from the singular value decomposition of $\mathbf{X}$ (i.e., $\mathbf{X} = \mathbf{U}_{x} \mathbf{D}_x \mathbf{V}_x^{\top}$ with its constituents endowed with the usual interpretation) and $\mathbf{\Lambda}$ a positive definite diagonal matrix. This gives the estimator:
-\begin{align*}
-\hat{\beta}(\mathbf{\Lambda}) & = (\mathbf{X}^{\top} \mathbf{X} + \mathbf{\Delta})^{-1} \mathbf{X}^{\top} \mathbf{Y}
-\\
-& = (\mathbf{V}_x \mathbf{D}_x \mathbf{U}_x^{\top} \mathbf{U}_x \mathbf{D}_x \mathbf{V}_x^{\top} + \mathbf{V}_x \mathbf{\Lambda} \mathbf{V}_x^{\top})^{-1} \mathbf{V}_x \mathbf{D}_x \mathbf{U}_x \mathbf{Y}
-\\
-& = \mathbf{V}_x (\mathbf{D}_x^2 + \mathbf{\Lambda})^{-1} \mathbf{D}_x \mathbf{U}_x \mathbf{Y}.
-\end{align*}
-From this last expression it becomes clear how this estimator generalizes the `regular ridge estimator'. The latter shrinks all eigenvalues, irrespectively of their size, in the same manner through a common penalty parameter. The `generalized ridge estimator', through differing penalty parameters (i.e. the diagonal elements of $\mathbf{\Lambda}$), shrinks them individually.
-
-The generalized ridge estimator coincides with the Bayesian linear regression estimator with the normal prior $\mathcal{N}[\mathbf{0}_p, (\mathbf{V}_x \mathbf{\Lambda} \mathbf{V}_x^{\top})^{-1}]$ on the regression parameter $\beta$ (and preserving the inverse gamma prior on the error variance). Assume $\mathbf{X}$ to be of full column rank and choose $\mathbf{\Lambda} = g^{-1} \mathbf{D}_x^2$ with $g$ a positive scalar. The prior on $\beta$ then - assuming $(\mathbf{X}^{\top} \mathbf{X})^{-1}$ exits - reduces to Zellner's $g$-prior: $\beta \sim \mathcal{N}[\mathbf{0}_p, g (\mathbf{X}^{\top} \mathbf{X})^{-1}]$ \citep{Zell1986}. The corresponding estimator of the regression coefficient is: $\hat{\beta}(g) = g (1+g)^{-1} (\mathbf{X}^{\top} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{Y}$, which is proportional to the unpenalized ordinary least squares estimator of $\beta$.
-
-For convenience of notation in the analysis of the generalized ridge estimator the linear regression model is usually rewritten as:
-\begin{align*}
-\mathbf{Y} & = \mathbf{X} \beta + \vvarepsilon \, \, \, = \, \, \,
-\mathbf{X} \mathbf{V}_x \mathbf{V}_x^{\top} \beta + \vvarepsilon \, \, \, = \, \, \, \tilde{\mathbf{X}} \aalpha + \vvarepsilon,
-\end{align*}
-with $\tilde{\mathbf{X}} = \mathbf{X} \mathbf{V}_x = \mathbf{U}_x
-\mathbf{D}_x$ (and thus $\tilde{\mathbf{X}}^{\top} \tilde{\mathbf{X}} = \mathbf{D}_x^ 2$) and $\aalpha = \mathbf{V}_x^{\top} \beta$ with loss function $(\mathbf{Y} - \tilde{\mathbf{X}} \aalpha)^{\top} (\mathbf{Y} - \tilde{\mathbf{X}} \aalpha) + \aalpha^{\top} \mathbf{\Lambda} \aalpha$. In the notation above the generalized ridge estimator is then:
-\begin{align*}
-\hat{\aalpha}(\mathbf{\Lambda}) & = (\tilde{\mathbf{X}}^{\top} \tilde{\mathbf{X}} + \mathbf{\Lambda})^ {-1} \tilde{\mathbf{X}}^{\top} \mathbf{Y} = (\mathbf{D}_x^ 2 + \mathbf{\Lambda})^{-1} \tilde{\mathbf{X}}^{\top} \mathbf{Y},
-\end{align*}
-from which one obtains $\hat{\beta}(\mathbf{\Lambda}) = \mathbf{V}_x \hat{\aalpha}(\mathbf{\Lambda})$. Using $\mathbb{E}[\hat{\aalpha}(\mathbf{\Lambda})] = (\mathbf{D}_x^2 + \mathbf{\Lambda})^{-1} \mathbf{D}_x^2 \aalpha$ and $\mbox{Var}[\hat{\aalpha}(\mathbf{\Lambda})] = \sigma^2 (\mathbf{D}_x^2 + \mathbf{\Lambda})^{-1} \mathbf{D}_x^2 (\mathbf{D}_x^2 + \mathbf{\Lambda})^{-1}$, the MSE for the generalized ridge estimator can be written as:
-\begin{align*}
-\mbox{MSE}[\hat{\aalpha}(\mathbf{\Lambda})] & = \sum_{j=1}^p ( \sigma^2 d_{x,j}^2 + \alpha_j^2 \lambda_{j}^2 ) (d_{x,j}^2 + \lambda_{j} )^{-2},
-\end{align*}
-where $d_{x,j} = (\mathbf{D}_x)_{jj}$ and $\lambda_j = (\mathbf{\Lambda})_{jj}$. Having $\aalpha$ and $\sigma^ 2$ available, it is easily seen (equate the derivative w.r.t. $\lambda_j$ to zero and solve) that the MSE of $\hat{\aalpha}(\mathbf{\Lambda})$ is minimized by $\lambda_j = \sigma^2 / \alpha_j^2$ for all $j$. With $\aalpha$ and $\sigma^2$ unknown, cite{Hoer1970} suggest an iterative procedure to estimate the $\lambda_j$'s. Initiate the procedure with the OLS estimates of $\aalpha$ and $\sigma^2$, followed by sequentially updating the $\lambda_j$'s and
-the estimates of $\aalpha$ and $\sigma^2$. An analytic expression of the limit of this procedure exists (\citealp{Hemm1975}). This limit, however, still depends on the observed $\mathbf{Y}$ and as such it does not necessarily yield the minimal attainable value of the MSE. This limit may nonetheless still yield a potential gain in MSE. This is investigated in cite{Lawl1981}. Under a variety of cases it seems to indeed outperform the OLS estimator, but there are exceptions.
-
-======= Conclusion =======
-
-To conclude: a note of caution. The generalized ridge penalty is extremely flexible. It can incorporate any prior knowledge on the parameter values (through specification of $\beta_0$) and the relations among these parameters (via $\mathbf{\Delta}$). While a pilot study or literature may provide a suggestion for $\beta_0$, it is less obvious how to choose an informative $\mathbf{\Delta}$ (although a spatial structure is a nice exception). In general, exact knowledge on the parameters should not be incorporated implicitly via the penalty (read: prior) but preferably be used explicitly in the model - the likelihood - itself. Though this may be the viewpoint of a prudent frequentist and a subjective Bayesian might disagree.
-
-
-======= Exercises =======
-\begin{question} \mbox{ }
-\\
-Consider the linear regression model $Y_i = \beta_1 X_{i,1} + \beta_2 X_{i,2} + \varepsilon_i$ for $i=1, \ldots, n$.
-Suppose estimates of the regression parameters $(\beta_1, \beta_2)$ of this model are obtained through the minimization of the sum-of-squares augmented with a ridge-type penalty:
-\begin{align*}
-\Big[ \sum_{i=1}^n (Y_i - \beta_1 X_{i,1} - \beta_2 X_{i,2})^2 \Big] + \lambda (\beta_1^2 + \beta_2^2 + 2 \nu \beta_1 \beta_2),
-\end{align*}
-with penalty parameters $\lambda \in \mathbb{R}_{> 0}$ and $\nu \in (-1, 1)$.
-
-\begin{compactitem}
-\item[*a)*] Sketch (for both $\nu=0$ and $\nu=0.9$) the shape of the parameter constraint induced by the penalty above and describe in words the qualitative difference between both shapes.
-
-\item[*b)*] When $\nu = -1$ and $\lambda \rightarrow \infty$ the estimates of $\beta_1$ and $\beta_2$ (resulting from minimization of the penalized loss function above) converge towards each other:
-$\lim_{\lambda \rightarrow \infty} \hat{\beta}_1(\lambda, -1) = \lim_{\lambda \rightarrow \infty} \hat{\beta}_2(\lambda, -1)$. Motivated by this observation a data scientists incorporates the equality constraint $\beta_1 = \beta = \beta_2$ explicitly into the model, and s/he estimates the `joint regression parameter' $\beta$ through the minimization (with respect to $\beta$) of:
-\begin{align*}
-\Big[ \sum_{i=1}^n (Y_i - \beta X_{i,1} - \beta X_{i,2})^2 \Big] + \delta \beta^2,
-\end{align*}
-with penalty parameter $\delta \in \mathbb{R}_{> 0}$. The data scientist is surprised to find that resulting estimate $\hat{\beta}(\delta)$ does not have the same limiting (in the penalty parameter) behavior as the $\hat{\beta}_1(\lambda, -1)$, i.e. $\lim_{\delta \rightarrow \infty} \hat{\beta} (\delta) \not= \lim_{\lambda \rightarrow \infty} \hat{\beta}_1(\lambda, -1)$. Explain the misconception of the data scientist.
-
-\item[*c)*] Assume that *i)* $n \gg 2$, *ii)* the unpenalized estimates $(\hat{\beta}_1(0, 0), \hat{\beta}_2(0, 0))$ equal $(-2,2)$, and *iii)* that the two covariates $X_1$ and $X_2$ are zero-centered, have equal variance, and are strongly negatively correlated. Consider $(\hat{\beta}_1(\lambda, \nu), \hat{\beta}_2(\lambda, \nu))$ for both $\nu=-0.9$ and $\nu=0.9$. For which value of $\nu$ do you expect the sum of the absolute value of the estimates to be largest? *Hint:* Distinguish between small and large values of $\lambda$ and think geometrically!
-\end{compactitem}
-\end{question}
-
-\begin{question} label{question.generalizedRidgeAndBayes} \mbox{ }
-\\
-Consider the linear regression model $\mathbf{Y} = \mathbf{X} \beta + \vvarepsilon$ with $\vvarepsilon \sim \mathcal{N}(\mathbf{0}_p, \sigma^2 \mathbf{I}_{pp})$. Assume $\beta \sim \mathcal{N}(\beta_0, \sigma^2 \mathbf{\Delta}^{-1})$ with $\beta_0 \in \mathbb{R}^p$ and $\mathbf{\Delta} \succ 0$ and a gamma prior on the error variance. Verify (i.e., work out the details of the derivation) that the posterior mean coincides with the generalized ridge estimator defined as:
-\begin{align*}
-\hat{\beta} & = (\mathbf{X}^{\top} \mathbf{X} + \mathbf{\Delta})^{-1} (\mathbf{X}^{\top} \mathbf{Y} + \mathbf{\Delta} \beta_0).
-\end{align*}
-\end{question}
-
-
-
-\begin{question} label{question.ridgeAR1penalty} \mbox{ }
-\\
-The ridge penalty may be interpreted as a multivariate normal prior on the regression coefficients: $\beta \sim \mathcal{N}(\mathbf{0}, \lambda^{-1} \mathbf{I}_{pp})$. Different priors may be considered. In case the covariates are spatially related in some sense (e.g. genomically), it may of interest to assume a first-order autoregressive prior: $\beta \sim \mathcal{N}(\mathbf{0}, \lambda^{-1} \SSigma_A)$, in which $\SSigma_A$ is a $p \times p$-correlation matrix with $(\SSigma_A)_{j_1, j_2} = \rho^{ | j_1 - j_2 | } $ for some correlation coefficient $\rho \in [0, 1)$. Hence,
-\begin{align*}
-\SSigma_A \, \, \, = \, \, \,
-\left(
-\begin{array}{cccc}
-1 & \rho & \ldots & \rho^{p-1}
-\\
-\rho & 1 & \ldots & \rho^{p-2}
-\\
-\vdots & \vdots & \ddots & \vdots
-\\
-\rho^{p-1} & \rho^{p-2} & \ldots & 1
-\end{array}
-\right).
-\end{align*}
-
-\begin{compactitem}
-\item[*a)*] The penalized loss function associated with this AR(1) prior is:
-\begin{align*}
-\mathcal{L}(\beta; \lambda, \SSigma_A) & = \| \mathbf{Y} - \mathbf{X} \beta \|_2^2 + \lambda \beta^{\top} \SSigma_A^{-1} \beta.
-\end{align*}
-Find the minimizer of this loss function.
-
-\item[*b)*] What is the effect of $\rho$ on the ridge estimates? Contrast this to the effect of $\lambda$. Illustrate this on (simulated) data.
-
-\item[*c)*] Instead of an AR(1) prior assume a prior with a uniform correlation between the elements of $\beta$. That is, replace $\SSigma_A$ by $\SSigma_U$, given by:
-\begin{align*}
-\SSigma_U \, \, \, = \, \, \, \left(
-\begin{array}{ccc}
-1 & \rho & \rho
-\\
-\rho & 1 & \rho
-\\
-\rho & \rho & 1
-\end{array}
-\right)
-\end{align*}
-\\
-Investigate (again on data) the effect of changing from the AR(1) to the uniform prior on the ridge regression estimates.
-\end{compactitem}
-\end{question}
-
-
-
-
-========= Ridge logistic regression ========= label{sect.ridgeLogistic}
-# ======= Ridge logistic regression =======
-
-Ridge penalized estimation is not limited to the standard linear regression model, but may be used to estimate (virtually) any model. Here we illustrate how it may be used to fit the logistic regression model. To this end we first recap this model and the (unpenalized) maximum likelihood estimation of its parameters. After which the model is estimated by means of ridge penalized maximum likelihood, which will turn out to be a relatively straightforward modification of unpenalized estimation.
-
-======= Logistic regression =======
-
-The logistic regression model explains a binary response variable (through some transformation) by a linear combination of a set of covariates (as in the linear regression model). Denote this response of the $i$-th sample by $Y_i$ with $Y_i \in \{ 0, 1 \}$ for $i=1, \ldots, n$. The $n$-dimensional column vector $\mathbf{Y}$ stacks these $n$ responses. For each sample information on the $p$ explanatory variables $X_{i,1}, \ldots, X_{i,p}$ is available. In row vector form this information is denoted $\mathbf{X}_{i,\ast} = (X_{i,1}, \ldots, X_{i,p})$. Or, in short, $\mathbf{X}_i$ when the context tolerates no confusion. The $(n \times p)$-dimensional matrix $\mathbf{X}$ aggregates these vectors, such that $\mathbf{X}_i$ is the $i$-th row vector.
-
-The binary response cannot be modelled as in the linear model like $Y_i = \mathbf{X}_i \beta + \varepsilon_i$. With each element of $\mathbf{X}_i$ and $\beta$ assuming a value in $\mathbb{R}$, the linear predictor is not restricted to the domain of the response. This is resolved by modeling $p_i = P(Y_i = 1)$ instead. Still the linear predictor may exceed the domain of the response ($p_i \in [0,1]$). Hence, a transformation is applied to map $p_i$ to $\mathbb{R}$, the range of the linear predictor.
-# original latex figure with scale=0.40, angle=0
-
-FIGURE: [logistic_effectOfBeta0.eps, width=400 frac=1.0] Top row, left panel: the response curve for various choices of the intercept $\beta_0$. Top row, right panel: the response curve for various choices of the regression coefficent $\beta_1$. Bottom row, left panel: the responce curve for various choices of the link function. Bottom panel, right panel: observations, fits and their deviations. } label{fig.logisticRidge_modelAndFitIllustration
-The transformation associated with the logistic regression model is the logarithm of the odds, with the odds defined as: $\mbox{*odds*} = P(\mbox{succes}) / P(\mbox{failure}) = p_i/ (1-p_i)$. The logistic model is then written as $\log[ p_i / (1-p_i)] = \mathbf{X}_i \beta$ for all $i$. Or, expressed in terms of the response:
-\begin{align*}
-p_i & = P(Y_i = 1) \, \, \, = \, \, \, g^{-1}(\mathbf{X}_i; \beta) \, \, \, = \, \, \,\frac{\exp(\mathbf{X}_i \beta) }{1 + \exp(\mathbf{X}_i \beta) }.
-\end{align*}
-The function $g(\cdot; \cdot)$ is called the *link function*. It links the response to the explanatory variables. The one above is called the logistic link function. Or short, logit. The regression parameters have tangible interpretations. When the first covariate represents the intercept, i.e. $X_{i,j} = 1$ for all $i$, then $\beta_1$ determines where the link function equals a half when all other covariates fail to contribute to the linear predictor (i.e. where $P (Y_i = 1 \, | \, \mathbf{X}_{i}) = 0.5$ when $\mathbf{X}_{i} \beta = \beta_1$). This is illustrated in the top-left panel of Figure ref{fig.logisticRidge_modelAndFitIllustration} for various choices of the intercept. On the other hand, the regression parameters are directly related to the odds ratio: $\mbox{*odds ratio*} = \mbox{odds}(X_{i,j}+1) / \mbox{odds}(X_{i,j}) = \exp(\beta_j)$. Hence, the effect of a unit change in the $j$-th covariate on the odds ratio is $\exp(\beta_j)$ (see Figure ref{fig.logisticRidge_modelAndFitIllustration}, top-right panel). Other link functions (depicted in Figure ref{fig.logisticRidge_modelAndFitIllustration}, bottom-left panel) are common, e.g. the *probit*: $p_i = \Phi_{0,1}(\mathbf{X}_i \beta)$; the *cloglog*: $p_i = \frac{1}{\pi} \arctan(\mathbf{X}_i \beta) + \frac{1}{2}$; the *Cauchit*: $p_i = \exp[ - \exp(\mathbf{X}_i \beta)]$. All these link function are invertible. Irrespective of the choice of the link function, the binary data are thus modelled as $Y_i \sim \mathcal{B}[g^{-1}(\mathbf{X}_i; \beta), 1]$. That is, as a single draw from the Binomial distribution with success probability $g^{-1}(\mathbf{X}_i; \beta)$.
-
-
-Let us now estimate the parameter of the logistic regression model by means of the maximum likelihood method. The likelihood of the experiment is then:
-\begin{align*}
-L(\mathbf{Y} \, | \, \mathbf{X}; \beta) & = \prod_{i=1}^n \big[ P(Y_i = 1 \, | \, \mathbf{X}_i) \big]^{Y_i} \big[ P(Y_i = 0 \, | \, \mathbf{X}_i) \big]^{1-Y_i}.
-\end{align*}
-After taking the logarithm and some ready algebra, the log-likelihood is found to be:
-\begin{align*}
-\mathcal{L}(\mathbf{Y} \, | \, \mathbf{X}; \beta) & = \sum_{i=1}^n \big\{ Y_i \mathbf{X}_i \beta - \log [ 1 + \exp(\mathbf{X}_i \beta) ] \big\}.
-\end{align*}
-Differentiate the log-likelihood with respect to $\beta$, equate it zero, and obtain the estimating equation for $\beta$:
-
-!bt
-
-!bt
-\begin{align} label{form:logisticRidge_estimatingEquationOfBeta}
-\frac{\partial \mathcal{L}}{\partial \beta } & = \sum_{i=1}^n \Big[ Y_i - \frac{\exp(\mathbf{X}_i \beta)}{ 1 + \exp(\mathbf{X}_i \beta)} \Big] \mathbf{X}_i^{\top} \, \, \, = \, \, \, \mathbf{0}_p.
-\end{align}
-!et
-!et
-The ML estimate of $\beta$ strikes a (weighted by the $\mathbf{X}_i$) balance between observation and model. Put differently (and illustrated in the bottom-right panel of Figure ref{fig.logisticRidge_modelAndFitIllustration}), a curve is fit through data by minimizing the distance between them: at the ML estimate of $\beta$ a weighted average of their deviations is zero.
-
-The maximum likelihood estimate of $\beta$ is evaluated by solving Equation (ref{form:logisticRidge_estimatingEquationOfBeta}) with respect to $\beta$ by means of the Newton-Raphson algorithm. The Newton-Raphson algorithm iteratively finds the zeros of a smooth enough function $f(\cdot)$. Let $x_0$ denote an initial guess of the zero. Then, approximate $f(\cdot)$ around $x_0$ by means of a first order Taylor series: $f(x) \approx x_0 + (x - x_0) \, (d f / d x) |_{x=x_0}$. Solve this for $x$ and obtain: $x = x_0 - [ (d f / d x) |_{x=x_0} ]^{-1} f(x_0)$. Let $x_1$ be the solution for $x$, use this as the new guess and repeat the above until convergence. When the function $f(\cdot)$ has multiple arguments, is vector-valued and denoted by $\vec{\mathbf{f}}$, and the Taylor approximation becomes: $\vec{\mathbf{f}}(\mathbf{x}) \approx \mathbf{x}_0 + J \vec{\mathbf{f}} \big|_{\mathbf{x}=\mathbf{x}_0} (\mathbf{x} - \mathbf{x}_0)$ with
-\begin{align*}
-J \vec{\mathbf{f}} = \left(
-\begin{array}{llll}
-\frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} & \ldots & \frac{\partial f_1}{\partial x_p}
-\\
-\frac{\partial f_1}{\partial x_1} & \frac{\partial f_2}{\partial x_2} & \ldots & \frac{\partial f_2}{\partial x_p}
-\\
-\vdots & \vdots & \ddots & \vdots
-\\
-\frac{\partial f_q}{\partial x_1} & \frac{\partial f_q}{\partial x_2} & \ldots & \frac{\partial f_q}{\partial x_p}
-\end{array}
-\right),
-\end{align*}
-the Jacobi matrix. An update of $x_0$ is now readily constructed by solving (the approximation for) $\vec{\mathbf{f}}(\mathbf{x}) = \mathbf{0}$ for $\mathbf{x}$.
-
-When applied here to the maximum likelihood estimation of the regression parameter $\beta$ of the logistic regression model, the Newton-Raphson update is:
-\begin{align*}
-\hat{\beta}^{\mbox{{\scriptsize new}}} & = \hat{\beta}^{\mbox{{\scriptsize old}}} - \Big( \frac{\partial^2 \mathcal{L}}{\partial \beta \partial \beta^{\top}} \Big)^{-1} \Big|_{\beta = \hat{\beta}^{\mbox{{\tiny old}}} } \, \, \frac{\partial \mathcal{L}}{\partial \beta } \Big|_{\beta = \hat{\beta}^{\mbox{{\tiny old}}} }
-\end{align*}
-where the Hessian of the log-likelihood equals:
-\begin{align*}
-\frac{\partial^2 \mathcal{L}}{\partial \beta \partial \beta^{\top}} & = - \sum_{i=1}^n \frac{\exp(\mathbf{X}_i \beta)}{ [1 + \exp(\mathbf{X}_i \beta)]^2} \mathbf{X}_i^{\top} \mathbf{X}_i.
-\end{align*}
-Iterative application of this updating formula converges to the ML estimate of $\beta$.
-
-The Newton-Raphson algorithm is often reformulated as an iteratively re-weighted least squares algorithm. Hereto, first write the gradient and Hessian in matrix notation:
-\begin{align*}
-\frac{\partial \mathcal{L}}{\partial \beta } \, \, \, = \, \, \, \mathbf{X}^{\top} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta)]
-& \mbox{ and } &
-\frac{\partial^2 \mathcal{L}}{\partial \beta \partial \beta^{\top}} \, \, \, = \, \, \, - \mathbf{X}^{\top} \mathbf{W} \mathbf{X},
-\end{align*}
-where $\vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta) = [g^{-1}( \mathbf{X}_{1, \ast}; \beta), \ldots, g^{-1}( \mathbf{X}_{n, \ast}; \beta)]^{\top}$ with $g^{-1}(\cdot; \cdot) = \exp(\cdot; \cdot) / [1 + \exp(\cdot; \cdot)]$ and $\mathbf{W}$ diagonal with $(\mathbf{W})_{ii} = \exp(\mathbf{X}_i \hat{\beta}^{\mbox{{\scriptsize old}}} ) [ 1 + \exp(\mathbf{X}_i \hat{\beta}^{\mbox{{\scriptsize old}}} ) ]^{-2}$. The updating formula of the estimate then becomes:
-\begin{align*}
-\hat{\beta}^{\mbox{{\scriptsize new}}} & = \hat{\beta}^{\mbox{{\scriptsize old}}} + (\mathbf{X}^{\top} \mathbf{W} \mathbf{X})^{-1} \mathbf{X}^{\top} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}})]
-\\
-& = (\mathbf{X}^{\top} \mathbf{W} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{W} \{ \mathbf{X} \hat{\beta}^{\mbox{{\scriptsize old}}} + \mathbf{W}^{-1} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}})] \}
-\\
-& = (\mathbf{X}^{\top} \mathbf{W} \mathbf{X})^{-1} \mathbf{X}^{\top} \mathbf{W} \mathbf{Z},
-\end{align*}
-where $\mathbf{Z} = \{ \mathbf{X} \hat{\beta}^{\mbox{{\scriptsize old}}} + \mathbf{W}^{-1} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}})] \}$. The Newton-Raphson update is thus the solution to the following weighted least squares problem:
-\begin{align*}
-\hat{\beta}^{\mbox{{\scriptsize new}}} & = \arg \min_{\beta} (\mathbf{Z} - \mathbf{X} \beta)^{\top} \mathbf{W} (\mathbf{Z} - \mathbf{X} \beta).
-\end{align*}
-Effectively, at each iteration the *adjusted response* $\mathbf{Z}$ is regressed on the covariates that comprise $\mathbf{X}$. For more on logistic regression confer the monograph of cite{Hosm2013}.
-
-
-======= Ridge estimation =======
-
-High-dimensionally, the linear predictor $\mathbf{X} \beta$ may be uniquely defined, but the maximum likelihood estimate of the logistic regression parameter is not. Assume $p > n$ and an estimate $\hat{\beta}$ available. Due to the high-dimensionality, the null space of $\mathbf{X}$ is non-trivial. Hence, let $\ggamma \in \mbox{null}(\mbox{span}(\mathbf{X}))$. Then: $\mathbf{X} \hat{\beta} = \mathbf{X} \hat{\beta} + \mathbf{X} \ggamma = \mathbf{X} (\hat{\beta} + \ggamma)$. As the null space is a $p-n$-dimensional subspace, $\ggamma$ need not equal zero. Hence, an infinite number of estimates of the logistic regression parameter exists that yield the same log-likelihood. Augmentation of the loss function with a ridge penalty resolves the matter, as their sum is strictly concave in $\beta$ (not convex as a maximum rather than a minimum is sought here) and thereby has a unique maximum.
-
-# original latex figure with scale=0.40, angle=0
-
-FIGURE: [logisticPenLL2ridgePenalty.eps, width=400 frac=1.0] Top row, left panel: contour plot of the penalized log-likelihood of a logistic regression model with the ridge constraint (red line). Top row, right panel: the regularization paths of the ridge estimator of the logistic regression parameter. Bottom row, left panel: variance of the ridge estimator of the logistic regression parameter against the logarithm of the penalty parameter. Bottom panel, right panel: the predicted success probability versus the linear predictor for various choices of the penalty parameter. } label{fig.logisticRidge_effectOfPenalization
-\mbox{ }
-
-
-Ridge maximum likelihood estimates of the logistic model parameters are found by the maximization of the ridge penalized loglikelihood (cf. \citealt{Scha1984,LeCe1992}):
-\begin{align*}
-\mathcal{L}^{\mbox{{\tiny pen}}}(\mathbf{Y}, \mathbf{X}; \beta, \lambda) & = \mathcal{L} (\mathbf{Y}, \mathbf{X}; \beta) - \tfrac{1}{2} \lambda \| \beta \|_2^2
-\\
-& = \sum_{i=1}^n \big\{ Y_i \mathbf{X}_i \beta - \log [ 1 + \exp(\mathbf{X}_i \beta) ] \big\} - \tfrac{1}{2} \lambda \beta^{\top} \beta,
-\end{align*}
-where the second summand is the ridge penalty (the sum of the square of the elements of $\beta$) with $\lambda$ the penalty parameter. Note that as in Section ref{sect.constrainedEstimation} maximization of this penalized loss function can be reformulated as a constrained estimation problem. This is illustrated by the top left panel of Figure ref{fig.logisticRidge_effectOfPenalization}, which depicts the contours (black lines) of the log-likelihood and the spherical domain of the parameter (red line). The optimization of the above loss function proceeds, due to the differentiability of the penalty, fully analogous to the unpenalized case and uses the Newton-Raphson algorithm for solving the (penalized) estimating equation. Hence, the unpenalized ML estimation procedure is modified straightforwardly by replacing gradient and Hessian by their `penalized' counterparts:
-\begin{align*}
-\frac{\partial \mathcal{L}^{\mbox{{\tiny pen}}}}{\partial \beta } \, \, \, = \, \, \, \frac{\partial \mathcal{L}}{\partial \beta } - \lambda \beta & \mbox{ and } & \frac{\partial^2 \mathcal{L}^{\mbox{{\tiny pen}}}}{\partial \beta \partial \beta^{\top}} \, \, \, = \, \, \, \frac{\partial^2 \mathcal{L}}{\partial \beta \partial \beta^{\top}} - \lambda \mathbf{I}_{pp}.
-\end{align*}
-With these at hand, the Newton-Raphson algorithm is (again) reformulated as an iteratively re-weighted least squares algorithm with the updating step changes accordingly to:
-\begin{align*}
-\hat{\beta}^{\mbox{{\scriptsize new}}} & = \hat{\beta}^{\mbox{{\scriptsize old}}} + \mathbf{V}^{-1} \{ \mathbf{X}^{\top} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}}) ] - \lambda \beta^{\mbox{{\scriptsize old}}} \}
-\\
-& = \mathbf{V}^{-1} \mathbf{V} \hat{\beta}^{\mbox{{\scriptsize old}}} - \lambda \mathbf{V}^{-1} \hat{\beta}^{\mbox{{\scriptsize old}}} + \mathbf{V}^{-1} \mathbf{X}^{\top} \mathbf{W} \mathbf{W}^{-1} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}})]
-\\
-& = \mathbf{V}^{-1} \mathbf{X}^{\top} \mathbf{W} \{ \mathbf{X} \hat{\beta}^{\mbox{{\scriptsize old}}} + \mathbf{W}^{-1} [\mathbf{Y} - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}})] \}
-\\
-& = [\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{W} \mathbf{Z},
-\end{align*}
-where $\mathbf{V} = \mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp}$ and $\mathbf{W}$ and $\mathbf{Z}$ as before. Hence, use this to update the estimate of $\beta$ until convergence, which yields the desired ridge ML estimate.
-
-Obviously, the ridge estimate of the logistic regression parameter tends to zero as $\lambda \rightarrow \infty$. Now consider a linear predictor with an intercept that is left unpenalized. When $\lambda$ tends to infinity, all regression coefficients but the intercept vanish. The intercept is left to model the success probability. Hence, in this case $\lim_{\lambda \rightarrow \infty} \hat{\beta}_0 (\lambda) = \log [ \tfrac{1}{n} \sum_{i=1}^n Y_i / \tfrac{1}{n} \sum_{i=1}^n (1-Y_i)]$.
-
-
-# original latex figure with scale=0.40, angle=0
-
-FIGURE: [logisticRidge_effectOnPrediction_lambda0.eps, width=400 frac=1.0] The realized design as scatter plot ($X_1$ vs $X_2$ overlayed by the success (\textcolor{red}{RED}) and failure regions (\textcolor{green}{GREEN}) for various choices of the penalty parameter: $\lambda = 0$ (top row, left panel), $\lambda = 10$ (top row, right panel) $\lambda = 40$ (bottom row, left panel), $\lambda = 100$ (bottom row, right panel).} label{fig.logisticRidge_effectOnPrediction
-
-
-
-The effect of the ridge penalty on parameter estimates propagates to the predictor $\hat{p}_i$. The linear predictor of the linear regression model involving the ridge estimator $\mathbf{X}_i \hat{\beta}(\lambda)$ shrinks towards a common value for each $i$, leading to a scale difference between observation and predictor (as seen before in Section ref{sect.ridgeRegressionDataIllustration}). This behaviour transfers to the ridge logistic regression predictor, as is illustrated on simulated data. The dimension and sample size of these data are $p=2$ and $n=200$, respectively. The covariate data are drawn from the standard normal, while that of the response is sampled from a Bernoulli distribution with success probability $P(Y_i=1) = \exp(2 X_{i,1} - 2 X_{i,2}) / [ 1 + \exp(2 X_{i,1} - 2 X_{i,2})]$. The logistic regression model is estimated from these data by means of ridge penalized likelihood maximization with various choices of the penalty parameter. The bottom right plot in Figure ref{fig.logisticRidge_effectOfPenalization} shows the predicted success probability versus the linear predictor for various choices of the penalty parameter. Larger values of the penalty parameter $\lambda$ flatten the slope of this curve. Consequently, for larger $\lambda$ more excessive values of the covariates are needed to achieve the same predicted success probability as those obtained with smaller $\lambda$ at more moderate covariate values. The implications for the resulting classification may become clearer when studying the effect of the penalty parameter on the `failure' and `success regions' respectively defined by:
-\\
-\indent $\{(x_1, x_2) : P({\color{green}{\mathbf{Y=0}}} \, | \, X_1=x_1, X_2=x_2, \hat{\beta}(\lambda)) > 0.75 \}$,
-\\
-\indent $\{(x_1, x_2) : P({\color{red}{\mathbf{Y=1}}} \, | \, X_1=x_1, X_2=x_2, \hat{\beta}(\lambda)) > 0.75 \}$.
-\\
-This separates the design space in a light red (`failure') and light green (`success') domain. The white bar between them is the domain where samples cannot be classified with high enough certainty. As $\lambda$ grows, so does the white area that separates the failure and success regions. Hence, as stronger penalization shrinks the logistic regression parameter estimate towards zero, it produces a predictor that is less outspoken in its class assignments.
-
-
-
-
-======= Moments =======
-
-The $1^{\mbox{{\tiny st}}}$ and $2^{\mbox{{\tiny nd}}}$ order moment of the ridge ML parameter of the logistic model may be approximated by the final update of the Newton-Raphson estimate. Assume the one-to-last update $\hat{\beta}^{\mbox{{\scriptsize old}}}$ to be non-random and proceed as for the ridge estimator of the linear regression model parameter to arrive at:
-# \begin{align*}
-# & & \hat{\beta}^{\mbox{{\scriptsize new}}} = [\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + 2 \lambda \mathbf{I}_{p \times p} ]^{-1} \mathbf{X}^{\top} \mathbf{W} \mathbf{z}
-# \end{align*}
-\begin{align*}
-\mathbb{E} \big( \hat{\beta}^{\mbox{{\scriptsize new}}} \big) & = [\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{W} \mathbb{E}( \mathbf{Z}),
-\\
-\mbox{Var} \big( \hat{\beta}^{\mbox{{\scriptsize new}}} \big) & = [\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1} \mathbf{X}^{\top} \mathbf{W} \big[ \mbox{Var} ( \mathbf{Z} ) \big] \mathbf{W} \mathbf{X} [\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp} ]^{-1},
-\end{align*}
-with
-\begin{align*}
-\mathbb{E}(\mathbf{Z}) & = \{ \mathbf{X} \hat{\beta}^{\mbox{{\scriptsize old}}} + \mathbf{W}^{-1} [\mathbb{E}(\mathbf{Y}) - \vec{\mathbf{g}}^{-1}( \mathbf{X}; \beta^{\mbox{{\scriptsize old}}})] \},
-\\
-\mbox{Var}(\mathbf{Z}) & = \mathbf{W}^{-1} \mbox{Var}(\mathbf{Y}) \mathbf{W}^{-1} = \mathbf{W}^{-1},
-\end{align*}
-where the identity $\mbox{Var}(\mathbf{Y}) = \mathbf{W}$ follows from the variance of a Binomial distributed random variable. From these expressions similar properties as for the ridge ML estimate of the regression parameter of the linear model may be deduced. For instance, the ridge ML estimate of the logistic regression parameter converges to zero as the penalty parameter tends to infinity (confer the top right panel of Figure ref{fig.logisticRidge_effectOfPenalization}). Similarly, their variances vanish as $\lambda \rightarrow \infty$ (illustrated in the bottom left panel of Figure ref{fig.logisticRidge_effectOfPenalization}).
-
-======= The Bayesian connection =======
-
-All penalized estimators can be formulated as Bayesian estimators, including the ridge logistic estimator. In particular, ridge estimators correspond to Bayesian estimators with a multivariate normal prior on the regression coefficients. Thus, assume $\beta \sim \mathcal{N}(\mathbf{0}_p, \mathbf{\Delta}^{-1})$. The posterior distribution of $\beta$ then is:
-\begin{align*}
-f_{\beta}(\beta \, | \, \mathbf{Y}, \mathbf{X}) & \propto & \Big\{ \prod_{i=1}^n \big[ P(Y_i = 1 \, | \, \mathbf{X}_i) \big]^{Y_i} \big[ P(Y_i = 0 \, | \, \mathbf{X}_i) \big]^{1-Y_i} \Big\} \exp( - \tfrac{1}{2} \beta \mathbf{\Delta} \beta).
-\end{align*}
-This does not coincide with any standard distribution. But, under appropriate conditions, the posterior distribution is asymptotically normal. This invites a (multivariate) normal approximation to the posterior distribution above. The Laplace's method provides (cf. \citealp{Bish2006}).
-
-# original latex figure with scale=0.40, angle=0
-
-FIGURE: [logisticRidge_laplace2posterior.eps, width=400 frac=1.0] Right panel: Laplace approximation to the posterior density of the Bayesian logistic regression parameter.} label{fig.logisticRidge_MSEandLaplace2posterior
-
-Laplace's method *i)* centers the normal approximation at the mode of the posterior, and *ii)* chooses the covariance to match the curvature of the posterior at the mode. The posterior mode is the location of the maximum of the posterior distribution. The location of this maximum coincides with that of the logarithm of the posterior. The latter is the log-likelihood augmented with a ridge penalty. Hence, the posterior mode, which is taken as the mean of the approximating Gaussian, coincides
-with the ridge logistic estimator. For the covariance of the approximating Gaussian, the logarithm of the posterior is approximated by a second order Taylor series around the posterior mode and limited to second order terms:
-\begin{align*}
-\log[f_{\beta}(\beta \, | \, \mathbf{Y}, \mathbf{X})] & \propto &
-\left. \log[f_{\beta}(\beta \, | \, \mathbf{Y}, \mathbf{X})]
-\right|_{\beta = \hat{\beta}_{\mbox{{\tiny MAP}}}}
-\\
-& & + \tfrac{1}{2} (\beta - \hat{\beta}_{\mbox{{\tiny MAP}}})^{\top} \left. \frac{\partial^2}{\partial \beta \partial \beta^{\top}}
-\log[f_{\beta}(\beta \, | \, \mathbf{Y}, \mathbf{X})]
- \right|_{\beta = \hat{\beta}_{\mbox{{\tiny MAP}}}} (\beta - \hat{\beta}_{\mbox{{\tiny MAP}}})^{\top},
-\end{align*}
-in which the first order term cancels as the derivative of $f_{\beta}(\beta \, | \, \mathbf{Y}, \mathbf{X})$ with respect to $\beta$ vanishes at the posterior mode - its maximum. Take the exponential of this approximation and match its arguments to that of a multivariate Gaussian $\exp[-\tfrac{1}{2} (\beta - \mmu_{\beta})^{\top} \mathbf{\Sigma}_{\beta}^{-1} (\beta - \mmu_{\beta})]$. The covariance of the sought Gaussian approximation is thus the inverse of the Hessian of the negative penalized log-likelihood. Put together the posterior is approximated by:
-\begin{align*}
-\beta \, | \, \mathbf{Y}, \mathbf{X} \sim \mathcal{N} \Big( \hat{\beta}_{\mbox{{\tiny MAP}}}, \Big\{ \mathbf{\Delta} + \sum_{i=1}^n \frac{\exp(\mathbf{X}_i \beta)}{ [1+ \exp(\mathbf{X}_i \beta) ]^2 } \mathbf{X}_i \mathbf{X}_i^{\top} \Big\}^{-1} \Big).
-\end{align*}
-The Gaussian approximation is convenient but need not be good. Fortunately, the Bernstein-Von Mises Theorem \citep{VdVa2000} tells it is very accurate when the model is regular, the prior smooth, and the sample size sufficiently large. The quality of the approximation for an artificial example data set is shown in Figure ref{fig.logisticRidge_MSEandLaplace2posterior}.
+#Xplot=poly5.fit_transform(xplot[:,np.newaxis])
+#plt.plot(xplot, clf5.predict(Xplot), 'r--',linewidth=1)
-======= Penalty parameter selection =======
-
-As before the penalty parameter may be chosen through $K$-fold cross-validation. For the $K=n$ case cite{Meij2013} describe a computationally efficient approximation of the leave-one-out cross-validated loglikelihood. It is based on the exact evaluation of the LOOCV loss, discussed in Section ref{subsect.crossvalidation}, that avoided resampling. The approach of cite{Meij2013} hinges upon the first-order Taylor expansion of the left-out penalized loglikelihood of the left-out estimate $\hat{\beta}_{-i} (\lambda)$ around $\hat{\beta} (\lambda)$, which yields an approximation of the former:
-\begin{align*}
-\hat{\beta}_{-i} (\lambda) & \approx \hat{\beta} (\lambda) -
-\left( \left. \frac{\partial^2 \mathcal{L}_{-i}^{\mbox{{\tiny pen}}}}{\partial \beta \partial \beta^{\top}} \right|_{\beta = \hat{\beta}(\lambda)} \right)^{-1} \left. \frac{\partial \mathcal{L}_{-i}^{\mbox{{\tiny pen}}}}{\partial \beta } \right|_{\beta = \hat{\beta}(\lambda)}
-\\
-& = \hat{\beta} (\lambda) + (\mathbf{X}_{- i, \ast}^{\top} \mathbf{W}_{-i, -i} \mathbf{X}_{- i, \ast} + \lambda \mathbf{I}_{pp})^{-1} \{ \mathbf{X}_{- i, \ast}^{\top} [\mathbf{Y}_{-i} - \vec{\mathbf{g}}^{-1}(\mathbf{X}_{-i, \ast}; \hat{\beta}(\lambda))] - \lambda \hat{\beta}(\lambda) \}.
-\end{align*}
-This approximation involves the inverse of a $p \times p$ dimensional matrix, which amounts to the evaluation of $n$ such inverses for the LOOCV loss. As in Section ref{subsect.crossvalidation} this may be avoided. Rewrite both the gradient and the Hessian of the left-out loglikelihood in the approximation of the preceding display:
-\begin{align*}
-\mathbf{X}_{-i, \ast}^{\top} \{ \mathbf{Y}_{-i} - \vec{\mathbf{g}}^{-1}(\mathbf{X}_{-i, \ast}; \hat{\beta}(\lambda)]\} - \lambda \hat{\beta}(\lambda)
-& = \mathbf{X}^{\top} \{ \mathbf{Y} - \vec{\mathbf{g}}^{-1}[\mathbf{X}; \hat{\beta}(\lambda)]\} - \lambda \hat{\beta}(\lambda)
-- \mathbf{X}_{i, \ast}^{\top} \{Y_{i} - g^{-1}[\mathbf{X}_{i, \ast}; \hat{\beta}(\lambda)]\}
-\\
-& = - \mathbf{X}_{i, \ast}^{\top} \{Y_{i} - g^{-1}[\mathbf{X}_{i, \ast}; \hat{\beta}(\lambda)]\}
-\end{align*}
-and
-\begin{align*}
-(\mathbf{X}_{- i, \ast}^{\top} \mathbf{W}_{-i, -i} \mathbf{X}_{- i, \ast} + \lambda \mathbf{I}_{pp})^{-1} & = (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} + \mathbf{W}_{ii} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top}
-\\
-& & \qquad \qquad \qquad \qquad \qquad \quad [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} \mathbf{X}_{i, \ast} (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1},
-\end{align*}
-where the Woodbury identity has been used and now $\mathbf{H}_{ii}(\lambda) = \mathbf{W}_{ii} \mathbf{X}_{i, \ast}(\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top}$. Substitute both in the approximation of the left-out ridge logistic regression estimator and manipulate as in Section ref{subsect.crossvalidation} to obtain:
-\begin{align*}
-\hat{\beta}_{- i}(\lambda) & \approx \hat{\beta}(\lambda) - (\mathbf{X}^{\top} \mathbf{W} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} \mathbf{X}_{i, \ast}^{\top} [ 1 - \mathbf{H}_{ii}(\lambda)]^{-1} [ Y_i - g^{-1}(\mathbf{X}_{i, \ast}; \hat{\beta}(\lambda)) ].
-\end{align*}
-Hence, the leave-one-out cross-validated loglikelihood $\sum_{i=1}^n \mathcal{L} [Y_i \, | \, \mathbf{X}_{i, \ast}, \hat{\beta}_{-i}(\lambda)]$ can now be evaluated by means of a single inverse of a $p \times p$ dimensional matrix and some matrix multiplications. For the performance of this approximation in terms of accuracy and speed confer cite{Meij2013}.
+poly10 = PolynomialFeatures(degree=10)
+X = poly10.fit_transform(x[:,np.newaxis])
+clf10 = linear_model.LinearRegression()
+clf10.fit(X,y)
+Xplot=poly10.fit_transform(xplot[:,np.newaxis])
+poly10_plot=plt.plot(xplot, clf10.predict(Xplot), label='Poly 10')
+axes = plt.gca()
+axes.set_ylim([-7,7])
+handles, labels=axes.get_legend_handles_labels()
+plt.legend(handles,labels, loc='lower center')
+plt.xlabel("$x$")
+plt.ylabel("$y$")
+Title="$N=$"+str(N_train)+", $\sigma=$"+str(sigma_train)
+plt.title(Title+" (train)")
+plt.tight_layout()
+plt.show()
+!ec
+!split
+===== Generating test data =====
+!bc pycod
+# Generate Test Data
+#Number of test data
+N_test=20
+sigma_test=sigma_train
+max_x=1.2
+x_test=max_x*np.random.random(N_test)
+# Draw random noise
+s_test = sigma_test*np.random.randn(N_test)
+#Linear
+y_test=2*x_test+s_test
+#Tenth order
+#y_test=2*x_test-10*x_test**5+15*x_test**10+s_test
+
+#Make design matrices for prediction
+x_plot=np.linspace(0,max_x, 200)
+X3 = poly3.fit_transform(x_plot[:,np.newaxis])
+X10 = poly10.fit_transform(x_plot[:,np.newaxis])
+
+%matplotlib notebook
+
+fig = plt.figure()
+p1=plt.plot(x_test,y_test.transpose(), 'o', ms=12, label='data')
+p2=plt.plot(x_plot,clf.predict(x_plot[:,np.newaxis]), label='linear')
+p3=plt.plot(x_plot,clf3.predict(X3), label='3rd order')
+p10=plt.plot(x_plot,clf10.predict(X10), label='10th order')
+
+
+plt.legend(loc=2)
+plt.xlabel('$x$')
+plt.ylabel('$y$')
+plt.legend(loc='best')
+plt.title(Title+" (pred.)")
+plt.tight_layout()
+plt.show()
+
+#Linear Filename
+#filename_test=Title+"pred-linear.pdf"
+#Tenth Order Filename
+#filename_test=Title+"pred-o10.pdf"
+#plt.savefig(filename_test)
+#plt.ylim((-6,12))
+
+!ec
+
+!split
+===== Lasso regression =====
+
+
+!split
+===== Logistic regression =====