FYS-STK4155/doc/LectureNotes/exercisesweek38.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "78bc86fe",
   "metadata": {
    "editable": true
   },
   "source": [
    "<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
    "doconce format html exercisesweek38.do.txt  -->\n",
    "<!-- dom:TITLE: Exercises week 38 -->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfac1a23",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Exercises week 38\n",
    "**September 18-22, 2023**\n",
    "\n",
    "Date: **Deadline is Sunday September 24 at midnight**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "248903ce",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Overarching aims of the exercises this week\n",
    "\n",
    "The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well.\n",
    "\n",
    "Consider a\n",
    "dataset $\\mathcal{L}$ consisting of the data\n",
    "$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n",
    "\n",
    "We assume that the true data is generated from a noisy model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d08c4671",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56f9ca3e",
   "metadata": {
    "editable": true
   },
   "source": [
    "Here $\\epsilon$ is normally distributed with mean zero and standard\n",
    "deviation $\\sigma^2$.\n",
    "\n",
    "In our derivation of the ordinary least squares method we defined \n",
    "an approximation to the function $f$ in terms of the parameters\n",
    "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
    "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n",
    "\n",
    "The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the mean\n",
    "squared error via the so-called cost function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae36b494",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be3eadf1",
   "metadata": {
    "editable": true
   },
   "source": [
    "Here the expected value $\\mathbb{E}$ is the sample value. \n",
    "\n",
    "Show that you can rewrite  this in terms of a term which contains the variance of the model itself (the so-called variance term), a\n",
    "term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.\n",
    "That is, show that"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae9ebea0",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=(\\mathrm{Bias}[\\tilde{y}])^2+\\mathrm{var}[\\tilde{f}]+\\sigma^2,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1cbae1b",
   "metadata": {
    "editable": true
   },
   "source": [
    "with"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2e1f899",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "(\\mathrm{Bias}[\\tilde{y}])^2=\\left(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2,\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0486221c",
   "metadata": {
    "editable": true
   },
   "source": [
    "and"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86746df2",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "\\mathrm{var}[\\tilde{f}]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aa6d3dc",
   "metadata": {
    "editable": true
   },
   "source": [
    "Explain what the terms mean and discuss their interpretations.\n",
    "\n",
    "Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice)  function by\n",
    "studying the MSE value as function of the complexity of your model. Use ordinary least squares only.\n",
    "\n",
    "Discuss the bias and variance trade-off as function\n",
    "of your model complexity (the degree of the polynomial) and the number\n",
    "of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n",
    "You can follow the code example in the jupyter-book at <https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff>.\n",
    "\n",
    "See also the whiteboard notes from week 37 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf>"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 5
}