{ "cells": [ { "cell_type": "markdown", "id": "1da77599", "metadata": {}, "source": [ "# Exercises week 38\n", "\n", "## September 15-19\n", "\n", "## Resampling and the Bias-Variance Trade-off\n" ] }, { "cell_type": "markdown", "id": "1b63f3b9", "metadata": {}, "source": [ "**Python Code can be found at https://github.uio.no/larsbog/FYS-STK4155**" ] }, { "cell_type": "markdown", "id": "e9f27b0e", "metadata": {}, "source": [ "### Learning goals\n", "\n", "After completing these exercises, you will know how to\n", "\n", "- Derive expectation and variances values related to linear regression\n", "- Compute expectation and variances values related to linear regression\n", "- Compute and evaluate the trade-off between bias and variance of a model\n", "\n", "### Deliverables\n", "\n", "Complete the following exercises while working in a jupyter notebook. Then, in canvas, include\n", "\n", "- The jupyter notebook with the exercises completed\n", "- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)\n" ] }, { "cell_type": "markdown", "id": "984af8e3", "metadata": {}, "source": [ "## Use the books!\n", "\n", "This week deals with various mean values and variances in linear regression methods (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)).\n", "\n", "For more discussions on Ridge regression and calculation of expectation values, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", "\n", "The exercises this week are also a part of project 1 and can be reused in the theory part of the project.\n", "\n", "### Definitions\n", "\n", "We assume that there exists a continuous function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim N(0, \\sigma^2)$ which describes our data\n" ] }, { "cell_type": "markdown", "id": "c16f7d0e", "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", "$$\n" ] }, { "cell_type": "markdown", "id": "9fcf981a", "metadata": { "vscode": { "languageId": "plaintext" } }, "source": [ "We further assume that this continous function can be modeled with a linear model $\\mathbf{\\tilde{y}}$ of some features $\\mathbf{X}$.\n" ] }, { "cell_type": "markdown", "id": "d4189366", "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = \\boldsymbol{\\tilde{y}} + \\boldsymbol{\\varepsilon} = \\boldsymbol{X}\\boldsymbol{\\beta} +\\boldsymbol{\\varepsilon}\n", "$$\n" ] }, { "cell_type": "markdown", "id": "f4fca21b", "metadata": {}, "source": [ "We therefore get that our data $\\boldsymbol{y}$ has an expectation value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$, that is $\\boldsymbol{y}$ follows a normal distribution with mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$.\n" ] }, { "cell_type": "markdown", "id": "5de0c7e6", "metadata": {}, "source": [ "## Exercise 1: Expectation values for ordinary least squares expressions\n" ] }, { "cell_type": "markdown", "id": "d878c699", "metadata": {}, "source": [ "**a)** With the expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ show that\n" ] }, { "cell_type": "markdown", "id": "08b7007d", "metadata": {}, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}_{OLS}}) = \\boldsymbol{\\beta}.\n", "$$\n" ] }, { "cell_type": "markdown", "id": "3a6e76d0", "metadata": {}, "source": [ "