update week 36

doc/src/week36/week36.do.txt

@@ -1853,3 +1853,100 @@ C(\bm{\beta}=\frac{\vert\vert (\bm{y}-\bm{X}\bm{\beta})\vert\vert_2^2}{2\sigma^2
 !et
 which is our Lasso cost function!  
 
+
+
+===== Exercise: mean values and variances in linear regression  =====
+
+
+
+This exercise deals with various mean values ad variances in  linear regression method (here it may be useful to look up chapter 3, equation (3.8) of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570").
+
+The assumption we have made is 
+that there exists a function $f(\bm{x})$ and  a normal distributed error $\bm{\varepsilon}\sim \mathcal{N}(0, \sigma^2)$
+which describes our data
+!bt
+\[
+\bm{y} = f(\bm{x})+\bm{\varepsilon}
+\]
+!et
+
+We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our
+function $f$ is approximated by $\bm{\tilde{y}}$ where we minimized  $(\bm{y}-\bm{\tilde{y}})^2$, with
+!bt
+\[
+\bm{\tilde{y}} = \bm{X}\bm{\beta}.
+\]
+!et
+The matrix $\bm{X}$ is the so-called design matrix. 
+
+!bsubex
+Show that  the expectation value of $\bm{y}$ for a given element $i$ 
+!bt
+\begin{align*} 
+\mathbb{E}(y_i) & =\mathbf{X}_{i, \ast} \, \beta, 
+\end{align*} 
+!et
+and that
+its variance is 
+!bt
+\begin{align*} \mbox{Var}(y_i) & = \sigma^2.  
+\end{align*}
+!et
+Hence, $y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \bm{\beta}, \sigma^2)$, that is $\bm{y}$ follows a normal distribution with 
+mean value $\bm{X}\bm{\beta}$ and variance $\sigma^2$.
+
+!esubex
+
+!bsubex
+With the OLS expressions for the parameters $\bm{\beta}$ show that
+!bt
+\[
+\mathbb{E}(\bm{\beta}) = \bm{\beta}.
+\]
+!et
+!esubex
+
+!bsubex
+Show finally that the variance of $\bm{\beta}$ is
+!bt
+\begin{eqnarray*}
+\mbox{Var}(\bm{\beta}) & = & \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
+\end{eqnarray*}
+!et
+
+!esubex
+
+
+===== Exercise: Adding Ridge and Lasso Regression  =====
+
+
+This exercise is a continuation of the exercises from week 35.
+
+We will
+use the same function to generate our data set, still staying with a
+simple function $y(x)$ which we want to fit using linear regression,
+but now extending the analysis to include the Ridge and the Lasso
+regression methods. 
+
+We will thus again generate our own dataset for a function $y(x)$ where 
+$x \in [0,1]$ and defined by random numbers computed with the uniform
+distribution. The function $y$ is a quadratic polynomial in $x$ with
+added stochastic noise according to the normal distribution $\cal{N}(0,1)$.
+
+The following simple Python instructions define our $x$ and $y$ values (with 100 data points).
+!bc pycod
+x = np.random.rand(100)
+y = 2.0+5*x*x+0.1*np.random.randn(100)
+!ec
+
+!bsubex
+Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\lambda$. Study the dependence on $\lambda$ while also varying the strength of the noise in your expression for $y(x)$. 
+
+!esubex
+
+!bsubex
+Our next step is to study the variance of the parameters $\beta_1$ and $\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression.  You can now opt for either writing your own function or using _Scikit-Learn_ to find the parameters $\beta$. From your results calculate the variance of these parameters (recall that this is equal to the diagonal elements of the matrix $(\hat{X}^T\hat{X})+\lambda\hat{I})^{-1}$). Discuss the results of these variances as functions of $\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figures 3.10 and  3.11. _Scikit-Learn_ may not provide the variance of the parameters $\beta$. This needs to be checked. With your own code you can however do so.
+!esubex
+
+
+