From e69df0de9c6ce98c7d14db9de03656292236dfa0 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sun, 3 Sep 2023 13:22:11 +0200 Subject: [PATCH] Update exercisesweek36.do.txt --- doc/src/week36/exercisesweek36.do.txt | 83 +++++++++------------------ 1 file changed, 28 insertions(+), 55 deletions(-) diff --git a/doc/src/week36/exercisesweek36.do.txt b/doc/src/week36/exercisesweek36.do.txt index 9def12dac..0a0361d8e 100644 --- a/doc/src/week36/exercisesweek36.do.txt +++ b/doc/src/week36/exercisesweek36.do.txt @@ -88,19 +88,24 @@ The ordinary least squares result is Use the singular value decomposition of an $m\times n$ matrix $\bm{X}$ (our design matrix) !bt \[ +\bm{X}=\bm{U}\bm{\Sigma}\bm{V}^T, +\] +!et +where $\bm{U}$ and $\bm{V}$ are orthogonal matrices of dimensions +$m\times m$ and $n\times n$, respectively, and $\bm{\Sigma}$ is an +$m\times n$ matrix which contains the ingular values only. This material was discussed during the lectures of week 35. -discussed in the lecture notes from week 35 to sh - -We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as +Show that you can write the +OLS solutions in terms of the eigenvectors (the columns) of the orthogonal matrix $\bm{U}$ as !bt \[ -\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} =\bm{U}\bm{U}^T\bm{y}. +\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} = \sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}. \] !et -For Ridge regression this becomes +For Ridge regression, show that the corresponding equation is !bt \[ @@ -110,30 +115,7 @@ For Ridge regression this becomes with the vectors $\bm{u}_j$ being the columns of $\bm{U}$ from the SVD of the matrix $\bm{X}$. - - - - - - - - - - - - - - - - - - - - - - - - +Give an interpretation of the results. Section 3.4 of Hastie et al's textbook gives a good discussion of the results. @@ -146,42 +128,33 @@ generate our data set, still staying with a simple function $y(x)$ which we want to fit using linear regression, but now extending the analysis to include the Ridge regression method. +In this exercise you need to include the same elements from last week +o Scale your data by subtracting the mean value from each column in the design matrix. +o Perform a split of the data in training and test set. +The addition to the analysis this time is the introduction of the hyperparameter $\lambda$ when introducing Ridge regression. -Write your own code for the Ridge method (see chapter 3.4 of Hastie -*et al.*, equations (3.43) and (3.44)) and compute the parametrization -for different values of $\lambda$. Compare and analyze your results -with those from exercise 3. Study the dependence on $\lambda$ while -also varying the strength of the noise in your expression for $y(x)$. - - -Repeat the above but using the functionality of -_Scikit-Learn_. Compare your code with the results from -_Scikit-Learn_. Remember to run with the same random numbers for -generating $x$ and $y$. Observe also that when you compare with _Scikit-Learn_, you need to pay attention to how the intercept is dealt with. - - - -Finally, using _Scikit-Learn_ or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as -!bt -\[ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} -\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, -\] -!et -and the $R^2$ score function. -If $\tilde{\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as +Extend the code from exercise 3 from week 35 to include Ridge regression with the hyperparameter $\lambda$. The optimal parameters $\hat{\beta}$ for Ridge regression can be obtained by matrix inversion in a similar way as done for ordinary least squares. You need to add to your code the following equations !bt \[ -R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, +\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}. \] !et -where we have defined the mean value of $\hat{y}$ as + +The ordinary least squares result you encoded last week is given by !bt \[ -\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. +\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}, \] !et -Discuss these quantities as functions of the variable $\lambda$ in Ridge regression. + +Use these results to compute the mean squared error for ordinary least +squares and Ridge regression first for a polynomial of degree five +with $n=100$ data points and five selected values of +$\lambda=[0.0001,0.001, 0.01,0.1,1.0]$. Compute thereafter the mean +squared error for the same values of $\lambda$ for polynomials of degree ten +and $15$. Discuss your results for the training MSE and test MSE with +Ridge regression and ordinary least squares.