From f090ebe38ea225fc1acb13a9ed810d52319aad1c Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sun, 8 Sep 2024 21:20:58 +0200 Subject: [PATCH] added exercises --- doc/LectureNotes/exercisesweek37.ipynb | 268 +++++++++++++++++++++++++ doc/src/week37/exercisesweek37.do.txt | 4 +- 2 files changed, 270 insertions(+), 2 deletions(-) create mode 100644 doc/LectureNotes/exercisesweek37.ipynb diff --git a/doc/LectureNotes/exercisesweek37.ipynb b/doc/LectureNotes/exercisesweek37.ipynb new file mode 100644 index 000000000..5528002c1 --- /dev/null +++ b/doc/LectureNotes/exercisesweek37.ipynb @@ -0,0 +1,268 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2a3463de", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "442e0844", + "metadata": { + "editable": true + }, + "source": [ + "# Exercises week 37\n", + "**September 9-13, 2024**\n", + "\n", + "Date: **Deadline is Friday September 13 at midnight**" + ] + }, + { + "cell_type": "markdown", + "id": "0c0df373", + "metadata": { + "editable": true + }, + "source": [ + "## Overarching aims of the exercises this week\n", + "\n", + "This exercise deals with various mean values and variances in linear\n", + "regression method (here it may be useful to look up chapter 3,\n", + "equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome\n", + "H. Friedman, The Elements of Statistical Learning,\n", + "Springer](https://www.springer.com/gp/book/9780387848570)). The\n", + "exercise is also a part of project 1 and can be reused in the theory\n", + "part of the project.\n", + "\n", + "For more discussions on Ridge regression and calculation of\n", + "expectation values, [Wessel van\n", + "Wieringen's](https://arxiv.org/abs/1509.09169) article is highly\n", + "recommended.\n", + "\n", + "The assumption we have made is that there exists a continuous function\n", + "$f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim N(0,\n", + "\\sigma^2)$ which describes our data" + ] + }, + { + "cell_type": "markdown", + "id": "a1ac8666", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "ee6ce4be", + "metadata": { + "editable": true + }, + "source": [ + "We then approximate this function $f(\\boldsymbol{x})$ with our model $\\boldsymbol{\\tilde{y}}$ from the solution of the linear regression equations (ordinary least squares OLS), that is our\n", + "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we minimized $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, with" + ] + }, + { + "cell_type": "markdown", + "id": "2d50b2e4", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3645cccd", + "metadata": { + "editable": true + }, + "source": [ + "The matrix $\\boldsymbol{X}$ is the so-called design or feature matrix." + ] + }, + { + "cell_type": "markdown", + "id": "6a378434", + "metadata": { + "editable": true + }, + "source": [ + "## Exercise 1: Expectation values for ordinary least squares expressions\n", + "\n", + "Show that the expectation value of $\\boldsymbol{y}$ for a given element $i$" + ] + }, + { + "cell_type": "markdown", + "id": "ce9b87a9", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathbb{E}(y_i) =\\sum_{j}x_{ij} \\beta_j=\\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "a6435e39", + "metadata": { + "editable": true + }, + "source": [ + "and that\n", + "its variance is" + ] + }, + { + "cell_type": "markdown", + "id": "75d89e64", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mbox{Var}(y_i) = \\sigma^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b5aef4d3", + "metadata": { + "editable": true + }, + "source": [ + "Hence, $y_i \\sim N( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", + "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$.\n", + "\n", + "With the OLS expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}}$ show that" + ] + }, + { + "cell_type": "markdown", + "id": "17012dce", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\boldsymbol{\\beta}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "4da3a821", + "metadata": { + "editable": true + }, + "source": [ + "Show finally that the variance of $\\boldsymbol{\\boldsymbol{\\beta}}$ is" + ] + }, + { + "cell_type": "markdown", + "id": "0ba77c7b", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mbox{Var}(\\boldsymbol{\\hat{\\beta}}) = \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "5c66de05", + "metadata": { + "editable": true + }, + "source": [ + "We can use the last expression when we define a [so-called confidence interval](https://en.wikipedia.org/wiki/Confidence_interval) for the parameters $\\beta$. \n", + "A given parameter $\\beta_j$ is given by the diagonal matrix element of the above matrix." + ] + }, + { + "cell_type": "markdown", + "id": "43b92138", + "metadata": { + "editable": true + }, + "source": [ + "## Exercise 2: Expectation values for Ridge regression\n", + "\n", + "Show that" + ] + }, + { + "cell_type": "markdown", + "id": "afea98ca", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "13d7fd96", + "metadata": { + "editable": true + }, + "source": [ + "We see clearly that\n", + "$\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n", + "\n", + "Show also that the variance is" + ] + }, + { + "cell_type": "markdown", + "id": "77d1d47e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mbox{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T}\\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2fa6bdfd", + "metadata": { + "editable": true + }, + "source": [ + "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/src/week37/exercisesweek37.do.txt b/doc/src/week37/exercisesweek37.do.txt index 8051429ac..9950aa9d0 100644 --- a/doc/src/week37/exercisesweek37.do.txt +++ b/doc/src/week37/exercisesweek37.do.txt @@ -1,6 +1,6 @@ TITLE: Exercises week 37 -AUTHOR: September 11-15, 2023 -DATE: Deadline is Sunday September 17 at midnight +AUTHOR: September 9-13, 2024 +DATE: Deadline is Friday September 13 at midnight ===== Overarching aims of the exercises this week =====