diff --git a/doc/LectureNotes/exercisesweek38.ipynb b/doc/LectureNotes/exercisesweek38.ipynb new file mode 100644 index 000000000..3cdd38af9 --- /dev/null +++ b/doc/LectureNotes/exercisesweek38.ipynb @@ -0,0 +1,183 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "78bc86fe", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "bfac1a23", + "metadata": { + "editable": true + }, + "source": [ + "# Exercises week 38\n", + "**September 18-22, 2023**\n", + "\n", + "Date: **Deadline is Sunday September 24 at midnight**" + ] + }, + { + "cell_type": "markdown", + "id": "248903ce", + "metadata": { + "editable": true + }, + "source": [ + "## Overarching aims of the exercises this week\n", + "\n", + "The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well.\n", + "\n", + "Consider a\n", + "dataset $\\mathcal{L}$ consisting of the data\n", + "$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n", + "\n", + "We assume that the true data is generated from a noisy model" + ] + }, + { + "cell_type": "markdown", + "id": "d08c4671", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "56f9ca3e", + "metadata": { + "editable": true + }, + "source": [ + "Here $\\epsilon$ is normally distributed with mean zero and standard\n", + "deviation $\\sigma^2$.\n", + "\n", + "In our derivation of the ordinary least squares method we defined \n", + "an approximation to the function $f$ in terms of the parameters\n", + "$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n", + "that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n", + "\n", + "The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the mean\n", + "squared error via the so-called cost function" + ] + }, + { + "cell_type": "markdown", + "id": "ae36b494", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "be3eadf1", + "metadata": { + "editable": true + }, + "source": [ + "Here the expected value $\\mathbb{E}$ is the sample value. \n", + "\n", + "Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a\n", + "term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.\n", + "That is, show that" + ] + }, + { + "cell_type": "markdown", + "id": "ae9ebea0", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=(\\mathrm{Bias}[\\tilde{y}])^2+\\mathrm{var}[\\tilde{f}]+\\sigma^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "d1cbae1b", + "metadata": { + "editable": true + }, + "source": [ + "with" + ] + }, + { + "cell_type": "markdown", + "id": "d2e1f899", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "(\\mathrm{Bias}[\\tilde{y}])^2=\\left(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "0486221c", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "86746df2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathrm{var}[\\tilde{f}]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9aa6d3dc", + "metadata": { + "editable": true + }, + "source": [ + "Explain what the terms mean and discuss their interpretations.\n", + "\n", + "Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice) function by\n", + "studying the MSE value as function of the complexity of your model. Use ordinary least squares only.\n", + "\n", + "Discuss the bias and variance trade-off as function\n", + "of your model complexity (the degree of the polynomial) and the number\n", + "of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n", + "You can follow the code example in the jupyter-book at .\n", + "\n", + "See also the whiteboard notes from week 37 at " + ] + } + ], + "metadata": {}, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/doc/src/week38/exercisesweek38.do.txt b/doc/src/week38/exercisesweek38.do.txt index 06ca7f869..b3c8ea27a 100644 --- a/doc/src/week38/exercisesweek38.do.txt +++ b/doc/src/week38/exercisesweek38.do.txt @@ -5,7 +5,68 @@ DATE: Deadline is Sunday September 24 at midnight ===== Overarching aims of the exercises this week ===== +The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well. + +Consider a +dataset $\mathcal{L}$ consisting of the data +$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$. + +We assume that the true data is generated from a noisy model + +!bt +\[ +\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}. +\] +!et + +Here $\epsilon$ is normally distributed with mean zero and standard +deviation $\sigma^2$. + +In our derivation of the ordinary least squares method we defined +an approximation to the function $f$ in terms of the parameters +$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model, +that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$. + +The parameters $\bm{\beta}$ are in turn found by optimizing the mean +squared error via the so-called cost function + +!bt +\[ +C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]. +\] +!et +Here the expected value $\mathbb{E}$ is the sample value. + +Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a +term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise. +That is, show that +!bt +\[ +\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2, +\] +!et +with +!bt +\[ +(\mathrm{Bias}[\tilde{y}])^2=\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2, +\] +!et +and +!bt +\[ +\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2. +\] +!et + +Explain what the terms mean and discuss their interpretations. + +Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice) function by +studying the MSE value as function of the complexity of your model. Use ordinary least squares only. + +Discuss the bias and variance trade-off as function +of your model complexity (the degree of the polynomial) and the number +of data points, and possibly also your training and test data using the _bootstrap_ resampling method. +You can follow the code example in the jupyter-book at URL:"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff". - - +See also the whiteboard notes from week 37 at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf"