This part should be included in your theory description of the report.
-This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
The assumption we have made is @@ -349,7 +347,7 @@ $$ \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}. $$ -
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). +
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). . A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
Let us assume that the true data is generated from a noisy model
+As in part a), we assume that the true data is generated from a noisy model
$$ \boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}. @@ -462,14 +460,26 @@ $$Here the expected value \( \mathbb{E} \) is the sample value.
-Show that you can rewrite this as
+Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a +term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise. +That is, show that +
$$ -\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2, $$ -The answer to this exercise can be included in the theory part of the report. -Explain what the terms mean, which one is the bias and which one is -the variance and discuss their interpretations. +
with
+$$ +(\mathrm{Bias}[\tilde{y}])^2=\left(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2, +$$ + +and
+$$ +\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2. +$$ + +The answer to this exercise should be included in the theory part of the report. +Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by @@ -479,6 +489,7 @@ studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method. +You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
diff --git a/doc/Projects/2022/Project1/html/Project1-bs.html b/doc/Projects/2022/Project1/html/Project1-bs.html index 2a2707c2d..229495544 100644 --- a/doc/Projects/2022/Project1/html/Project1-bs.html +++ b/doc/Projects/2022/Project1/html/Project1-bs.html @@ -302,8 +302,6 @@ plt.show()This part should be included in your theory description of the report.
-This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
The assumption we have made is @@ -349,7 +347,7 @@ $$ \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}. $$ -
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). +
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). . A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
Let us assume that the true data is generated from a noisy model
+As in part a), we assume that the true data is generated from a noisy model
$$ \boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}. @@ -462,14 +460,26 @@ $$Here the expected value \( \mathbb{E} \) is the sample value.
-Show that you can rewrite this as
+Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a +term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise. +That is, show that +
$$ -\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2, $$ -The answer to this exercise can be included in the theory part of the report. -Explain what the terms mean, which one is the bias and which one is -the variance and discuss their interpretations. +
with
+$$ +(\mathrm{Bias}[\tilde{y}])^2=\left(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2, +$$ + +and
+$$ +\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2. +$$ + +The answer to this exercise should be included in the theory part of the report. +Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by @@ -479,6 +489,7 @@ studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method. +You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
diff --git a/doc/Projects/2022/Project1/html/Project1.html b/doc/Projects/2022/Project1/html/Project1.html index 5873ccbfd..2034ad644 100644 --- a/doc/Projects/2022/Project1/html/Project1.html +++ b/doc/Projects/2022/Project1/html/Project1.html @@ -338,8 +338,6 @@ plt.show()This part should be included in your theory description of the report.
-This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
The assumption we have made is @@ -385,7 +383,7 @@ $$ \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}. $$ -
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). +
We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). . A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
Let us assume that the true data is generated from a noisy model
+As in part a), we assume that the true data is generated from a noisy model
$$ \boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}. @@ -498,14 +496,26 @@ $$Here the expected value \( \mathbb{E} \) is the sample value.
-Show that you can rewrite this as
+Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a +term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise. +That is, show that +
$$ -\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2. +\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2, $$ -The answer to this exercise can be included in the theory part of the report. -Explain what the terms mean, which one is the bias and which one is -the variance and discuss their interpretations. +
with
+$$ +(\mathrm{Bias}[\tilde{y}])^2=\left(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2, +$$ + +and
+$$ +\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2. +$$ + +The answer to this exercise should be included in the theory part of the report. +Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by @@ -515,6 +525,7 @@ studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function of your model complexity (the degree of the polynomial) and the number of data points, and possibly also your training and test data using the bootstrap resampling method. +You can follow the code example in the jupyter-book at https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff.
Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).
diff --git a/doc/Projects/2022/Project1/ipynb/Project1.ipynb b/doc/Projects/2022/Project1/ipynb/Project1.ipynb index 4758c2997..28018807d 100644 --- a/doc/Projects/2022/Project1/ipynb/Project1.ipynb +++ b/doc/Projects/2022/Project1/ipynb/Project1.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "98e2a2bd", + "id": "2f0d0303", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "1951a8ff", + "id": "383fa1e8", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "755dd202", + "id": "00edcc37", "metadata": { "editable": true }, @@ -63,7 +63,7 @@ }, { "cell_type": "markdown", - "id": "3b39d52d", + "id": "b8f880fc", "metadata": { "editable": true }, @@ -85,7 +85,7 @@ }, { "cell_type": "markdown", - "id": "92521420", + "id": "1e667978", "metadata": { "editable": true }, @@ -100,7 +100,7 @@ }, { "cell_type": "markdown", - "id": "5034b29f", + "id": "3ce1ab3a", "metadata": { "editable": true }, @@ -129,7 +129,7 @@ { "cell_type": "code", "execution_count": 1, - "id": "9ee70500", + "id": "965d1f48", "metadata": { "collapsed": false, "editable": true @@ -181,15 +181,13 @@ }, { "cell_type": "markdown", - "id": "94aa10c7", + "id": "6171ab22", "metadata": { "editable": true }, "source": [ "### Part a): Paper and pencil part (also as weekly exercise for week 36)\n", "\n", - "This part should be included in your theory description of the report.\n", - "\n", "This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)).\n", "\n", "The assumption we have made is \n", @@ -199,7 +197,7 @@ }, { "cell_type": "markdown", - "id": "1c6e721e", + "id": "969dd687", "metadata": { "editable": true }, @@ -211,7 +209,7 @@ }, { "cell_type": "markdown", - "id": "d3b0bc88", + "id": "9194b1fc", "metadata": { "editable": true }, @@ -222,7 +220,7 @@ }, { "cell_type": "markdown", - "id": "76226748", + "id": "cb6e0c5a", "metadata": { "editable": true }, @@ -234,7 +232,7 @@ }, { "cell_type": "markdown", - "id": "735571e2", + "id": "ef0c8ce0", "metadata": { "editable": true }, @@ -246,7 +244,7 @@ }, { "cell_type": "markdown", - "id": "7b93fc13", + "id": "c4e904d3", "metadata": { "editable": true }, @@ -258,7 +256,7 @@ }, { "cell_type": "markdown", - "id": "80c44548", + "id": "8ce55ba5", "metadata": { "editable": true }, @@ -269,7 +267,7 @@ }, { "cell_type": "markdown", - "id": "1e70c3f1", + "id": "3aa94ac5", "metadata": { "editable": true }, @@ -281,7 +279,7 @@ }, { "cell_type": "markdown", - "id": "31f5d975", + "id": "8a15da0c", "metadata": { "editable": true }, @@ -294,7 +292,7 @@ }, { "cell_type": "markdown", - "id": "a564fee7", + "id": "9761270b", "metadata": { "editable": true }, @@ -306,7 +304,7 @@ }, { "cell_type": "markdown", - "id": "05fc2911", + "id": "12f47ebc", "metadata": { "editable": true }, @@ -316,7 +314,7 @@ }, { "cell_type": "markdown", - "id": "ca40d503", + "id": "be6e6c6e", "metadata": { "editable": true }, @@ -328,18 +326,18 @@ }, { "cell_type": "markdown", - "id": "039784c1", + "id": "734989a3", "metadata": { "editable": true }, "source": [ - "We can use the last expression when we define a so-called confidence interval for the parameters $\\beta$.\n", + "We can use the last expression when we define a so-called confidence interval for the parameters $\\beta$. .\n", "A given parameter $\\beta_j$ is given by the diagonal matrix element of the above matrix." ] }, { "cell_type": "markdown", - "id": "df1c03c6", + "id": "7e01db05", "metadata": { "editable": true }, @@ -361,7 +359,7 @@ }, { "cell_type": "markdown", - "id": "7e229e98", + "id": "6431c5d5", "metadata": { "editable": true }, @@ -374,7 +372,7 @@ }, { "cell_type": "markdown", - "id": "00b00673", + "id": "e4eae5de", "metadata": { "editable": true }, @@ -386,7 +384,7 @@ }, { "cell_type": "markdown", - "id": "506c2d83", + "id": "6b7f7088", "metadata": { "editable": true }, @@ -398,7 +396,7 @@ }, { "cell_type": "markdown", - "id": "2cfcf061", + "id": "57e680b6", "metadata": { "editable": true }, @@ -408,7 +406,7 @@ }, { "cell_type": "markdown", - "id": "84225fa8", + "id": "bbee460a", "metadata": { "editable": true }, @@ -420,7 +418,7 @@ }, { "cell_type": "markdown", - "id": "7f22646c", + "id": "ab772d43", "metadata": { "editable": true }, @@ -449,7 +447,7 @@ }, { "cell_type": "markdown", - "id": "05fd370a", + "id": "022b31d9", "metadata": { "editable": true }, @@ -476,12 +474,12 @@ "dataset $\\mathcal{L}$ consisting of the data\n", "$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n", "\n", - "Let us assume that the true data is generated from a noisy model" + "As in part a), we assume that the true data is generated from a noisy model" ] }, { "cell_type": "markdown", - "id": "ac09204e", + "id": "38c61204", "metadata": { "editable": true }, @@ -493,7 +491,7 @@ }, { "cell_type": "markdown", - "id": "09bceede", + "id": "aa00d7dd", "metadata": { "editable": true }, @@ -512,7 +510,7 @@ }, { "cell_type": "markdown", - "id": "f5bae3fa", + "id": "6240984f", "metadata": { "editable": true }, @@ -524,38 +522,83 @@ }, { "cell_type": "markdown", - "id": "a8afd8c5", + "id": "0f01484d", "metadata": { "editable": true }, "source": [ "Here the expected value $\\mathbb{E}$ is the sample value. \n", "\n", - "Show that you can rewrite this as" + "Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a\n", + "term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.\n", + "That is, show that" ] }, { "cell_type": "markdown", - "id": "66c12c5c", + "id": "37f1b3f8", "metadata": { "editable": true }, "source": [ "$$\n", - "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\frac{1}{n}\\sum_i(f_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\sigma^2.\n", + "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=(\\mathrm{Bias}[\\tilde{y}])^2+\\mathrm{var}[\\tilde{f}]+\\sigma^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "80d3e8a0", + "id": "0673dacf", "metadata": { "editable": true }, "source": [ - "The answer to this exercise can be included in the theory part of the report.\n", - "Explain what the terms mean, which one is the bias and which one is\n", - "the variance and discuss their interpretations.\n", + "with" + ] + }, + { + "cell_type": "markdown", + "id": "281ae919", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "(\\mathrm{Bias}[\\tilde{y}])^2=\\left(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3604e435", + "metadata": { + "editable": true + }, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "id": "cb6e58c2", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\mathrm{var}[\\tilde{f}]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "65111a57", + "metadata": { + "editable": true + }, + "source": [ + "The answer to this exercise should be included in the theory part of the report.\n", + "Explain what the terms mean and discuss their interpretations.\n", "\n", "Perform then a bias-variance analysis of the Franke function by\n", "studying the MSE value as function of the complexity of your model.\n", @@ -563,13 +606,14 @@ "Discuss the bias and variance trade-off as function\n", "of your model complexity (the degree of the polynomial) and the number\n", "of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n", + "You can follow the code example in the jupyter-book at