lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added exercise

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-09-18 07:25:16 +02:00
parent 5f0e6fde1c
commit 59c7d669ff
2 changed files with 246 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/exercisesweek38.ipynb
+183
View File
@@ -0,0 +1,183 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "78bc86fe",
"metadata": {
"editable": true
},
"source": [
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
"doconce format html exercisesweek38.do.txt -->\n",
"<!-- dom:TITLE: Exercises week 38 -->"
]
},
{
"cell_type": "markdown",
"id": "bfac1a23",
"metadata": {
"editable": true
},
"source": [
"# Exercises week 38\n",
"**September 18-22, 2023**\n",
"\n",
"Date: **Deadline is Sunday September 24 at midnight**"
]
},
{
"cell_type": "markdown",
"id": "248903ce",
"metadata": {
"editable": true
},
"source": [
"## Overarching aims of the exercises this week\n",
"\n",
"The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well.\n",
"\n",
"Consider a\n",
"dataset $\\mathcal{L}$ consisting of the data\n",
"$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n",
"\n",
"We assume that the true data is generated from a noisy model"
]
},
{
"cell_type": "markdown",
"id": "d08c4671",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "56f9ca3e",
"metadata": {
"editable": true
},
"source": [
"Here $\\epsilon$ is normally distributed with mean zero and standard\n",
"deviation $\\sigma^2$.\n",
"\n",
"In our derivation of the ordinary least squares method we defined \n",
"an approximation to the function $f$ in terms of the parameters\n",
"$\\boldsymbol{\\beta}$ and the design matrix $\\boldsymbol{X}$ which embody our model,\n",
"that is $\\boldsymbol{\\tilde{y}}=\\boldsymbol{X}\\boldsymbol{\\beta}$.\n",
"\n",
"The parameters $\\boldsymbol{\\beta}$ are in turn found by optimizing the mean\n",
"squared error via the so-called cost function"
]
},
{
"cell_type": "markdown",
"id": "ae36b494",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "be3eadf1",
"metadata": {
"editable": true
},
"source": [
"Here the expected value $\\mathbb{E}$ is the sample value. \n",
"\n",
"Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a\n",
"term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.\n",
"That is, show that"
]
},
{
"cell_type": "markdown",
"id": "ae9ebea0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=(\\mathrm{Bias}[\\tilde{y}])^2+\\mathrm{var}[\\tilde{f}]+\\sigma^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "d1cbae1b",
"metadata": {
"editable": true
},
"source": [
"with"
]
},
{
"cell_type": "markdown",
"id": "d2e1f899",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"(\\mathrm{Bias}[\\tilde{y}])^2=\\left(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "0486221c",
"metadata": {
"editable": true
},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"id": "86746df2",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathrm{var}[\\tilde{f}]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "9aa6d3dc",
"metadata": {
"editable": true
},
"source": [
"Explain what the terms mean and discuss their interpretations.\n",
"\n",
"Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice) function by\n",
"studying the MSE value as function of the complexity of your model. Use ordinary least squares only.\n",
"\n",
"Discuss the bias and variance trade-off as function\n",
"of your model complexity (the degree of the polynomial) and the number\n",
"of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n",
"You can follow the code example in the jupyter-book at <https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff>.\n",
"\n",
"See also the whiteboard notes from week 37 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf>"
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
doc/src/week38/exercisesweek38.do.txt
+63 -2
View File
@@ -5,7 +5,68 @@ DATE: Deadline is Sunday September 24 at midnight
===== Overarching aims of the exercises this week =====
The aim of the exercises this week is to derive the equations for the bias-variance tradeoff to be used in project 1 as well as testing this for a simpler function using the bootstrap method. The exercises here can be reused in project 1 as well.
Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
We assume that the true data is generated from a noisy model
!bt
\[
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
\]
!et
Here $\epsilon$ is normally distributed with mean zero and standard
deviation $\sigma^2$.
In our derivation of the ordinary least squares method we defined
an approximation to the function $f$ in terms of the parameters
$\bm{\beta}$ and the design matrix $\bm{X}$ which embody our model,
that is $\bm{\tilde{y}}=\bm{X}\bm{\beta}$.
The parameters $\bm{\beta}$ are in turn found by optimizing the mean
squared error via the so-called cost function
!bt
\[
C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right].
\]
!et
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
!bt
\[
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
\]
!et
with
!bt
\[
(\mathrm{Bias}[\tilde{y}])^2=\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2,
\]
!et
and
!bt
\[
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2.
\]
!et
Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of a simple one-dimensional (or other models of your choice) function by
studying the MSE value as function of the complexity of your model. Use ordinary least squares only.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the _bootstrap_ resampling method.
You can follow the code example in the jupyter-book at URL:"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff".
See also the whiteboard notes from week 37 at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep14.pdf"