Update exercisesweek36.do.txt

doc/src/week36/exercisesweek36.do.txt

@@ -88,19 +88,24 @@ The ordinary least squares result is
 Use the singular value decomposition of an $m\times n$ matrix $\bm{X}$ (our design matrix)
 !bt
 \[
+\bm{X}=\bm{U}\bm{\Sigma}\bm{V}^T,
+\]
+!et
 
+where $\bm{U}$ and $\bm{V}$ are orthogonal matrices of dimensions
+$m\times m$ and $n\times n$, respectively, and $\bm{\Sigma}$ is an
+$m\times n$ matrix which contains the ingular values only. This material was discussed during the lectures of week 35.
 
-discussed in the lecture notes from week 35 to sh
-
-We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
+Show that you can write the 
+OLS solutions in terms of the eigenvectors (the columns) of the orthogonal matrix  $\bm{U}$ as
 !bt
 \[
-\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta}  =\bm{U}\bm{U}^T\bm{y}.
+\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta}  = \sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}.
 \]
 !et
 
 
-For Ridge regression this becomes
+For Ridge regression, show that the corresponding equation is
 
 !bt
 \[
@@ -110,30 +115,7 @@ For Ridge regression this becomes
 
 with the vectors $\bm{u}_j$ being the columns of $\bm{U}$ from the SVD of the matrix $\bm{X}$. 
 
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
+Give an interpretation of the results.  Section 3.4 of Hastie et al's textbook gives a good discussion of the results.
 
 
 
@@ -146,42 +128,33 @@ generate our data set, still staying with a simple function $y(x)$
 which we want to fit using linear regression, but now extending the
 analysis to include the Ridge regression method.
 
+In this exercise you need to include the same elements from last week
+o Scale your data by subtracting the mean value from each column in the design matrix.
+o Perform a split of the data in training and test set.
 
+The addition to the analysis this time is the introduction of the hyperparameter $\lambda$ when introducing Ridge regression.
 
-Write your own code for the Ridge method (see chapter 3.4 of Hastie
-*et al.*, equations (3.43) and (3.44)) and compute the parametrization
-for different values of $\lambda$. Compare and analyze your results
-with those from exercise 3. Study the dependence on $\lambda$ while
-also varying the strength of the noise in your expression for $y(x)$.
-
-
-Repeat the above but using the functionality of
-_Scikit-Learn_. Compare your code with the results from
-_Scikit-Learn_. Remember to run with the same random numbers for
-generating $x$ and $y$.  Observe also that when you compare with _Scikit-Learn_, you need to pay attention to how the intercept is dealt with.
-
-
-
-Finally, using _Scikit-Learn_ or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as
-!bt 
-\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n}
-\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, 
-\] 
-!et 
-and the $R^2$ score function.
-If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as
+Extend the code from exercise 3 from week 35 to include Ridge regression with the hyperparameter $\lambda$. The optimal parameters $\hat{\beta}$ for Ridge regression can be obtained by matrix inversion in a similar way as done for ordinary least squares. You need to add to your code the following equations
 !bt
 \[
-R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
+\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
 \]
 !et
-where we have defined the mean value  of $\hat{y}$ as
+
+The ordinary least squares result you encoded last week is given by
 !bt
 \[
-\bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
+\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
 \]
 !et
-Discuss these quantities as functions of the variable $\lambda$ in Ridge regression.
+
+Use these results to compute the mean squared error for ordinary least
+squares and Ridge regression first for a polynomial of degree five
+with $n=100$ data points and five selected values of
+$\lambda=[0.0001,0.001, 0.01,0.1,1.0]$. Compute thereafter the mean
+squared error for the same values of $\lambda$ for polynomials of degree ten
+and $15$. Discuss your results for the training MSE and test MSE with
+Ridge regression and ordinary least squares.