lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-09-05 13:03:33 +02:00
parent d0c2d4993f
commit 3d4b9f7f4a
9 changed files with 214 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/Projects/2022/Project1/html/._Project1-bs000.html
+20 -9
View File
@@ -302,8 +302,6 @@ plt<span style="color: #666666">.</span>show()
</div>
<h3 id="part-a-paper-and-pencil-part-also-as-weekly-exercise-for-week-36" class="anchor">Part a): Paper and pencil part (also as weekly exercise for week 36) </h3>
<p>This part should be included in your theory description of the report.</p>
<p>This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>).</p>
<p>The assumption we have made is
@@ -349,7 +347,7 @@ $$
\mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
$$
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \).
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
</p>
<h3 id="part-b-ordinary-least-square-ols-on-the-franke-function" class="anchor">Part b) : Ordinary Least Square (OLS) on the Franke function </h3>
@@ -436,7 +434,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
</p>
<p>Let us assume that the true data is generated from a noisy model</p>
<p>As in part a), we assume that the true data is generated from a noisy model</p>
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -462,14 +460,26 @@ $$
<p>Here the expected value \( \mathbb{E} \) is the sample value. </p>
<p>Show that you can rewrite this as</p>
<p>Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
</p>
$$
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
$$
<p>The answer to this exercise can be included in the theory part of the report.
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>with </p>
$$
(\mathrm{Bias}[\tilde{y}])^2=\left(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2,
$$
<p>and </p>
$$
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2.
$$
<p>The answer to this exercise should be included in the theory part of the report.
Explain what the terms mean and discuss their interpretations.
</p>
<p>Perform then a bias-variance analysis of the Franke function by
@@ -479,6 +489,7 @@ studying the MSE value as function of the complexity of your model.
<p>Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
You can follow the code example in the jupyter-book at <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff" target="_self"><tt>https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff</tt></a>.
</p>
<p>Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).</p>
doc/Projects/2022/Project1/html/Project1-bs.html
+20 -9
View File
@@ -302,8 +302,6 @@ plt<span style="color: #666666">.</span>show()
</div>
<h3 id="part-a-paper-and-pencil-part-also-as-weekly-exercise-for-week-36" class="anchor">Part a): Paper and pencil part (also as weekly exercise for week 36) </h3>
<p>This part should be included in your theory description of the report.</p>
<p>This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of <a href="https://www.springer.com/gp/book/9780387848570" target="_self">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>).</p>
<p>The assumption we have made is
@@ -349,7 +347,7 @@ $$
\mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
$$
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \).
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
</p>
<h3 id="part-b-ordinary-least-square-ols-on-the-franke-function" class="anchor">Part b) : Ordinary Least Square (OLS) on the Franke function </h3>
@@ -436,7 +434,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
</p>
<p>Let us assume that the true data is generated from a noisy model</p>
<p>As in part a), we assume that the true data is generated from a noisy model</p>
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -462,14 +460,26 @@ $$
<p>Here the expected value \( \mathbb{E} \) is the sample value. </p>
<p>Show that you can rewrite this as</p>
<p>Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
</p>
$$
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
$$
<p>The answer to this exercise can be included in the theory part of the report.
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>with </p>
$$
(\mathrm{Bias}[\tilde{y}])^2=\left(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2,
$$
<p>and </p>
$$
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2.
$$
<p>The answer to this exercise should be included in the theory part of the report.
Explain what the terms mean and discuss their interpretations.
</p>
<p>Perform then a bias-variance analysis of the Franke function by
@@ -479,6 +489,7 @@ studying the MSE value as function of the complexity of your model.
<p>Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
You can follow the code example in the jupyter-book at <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff" target="_self"><tt>https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff</tt></a>.
</p>
<p>Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).</p>
doc/Projects/2022/Project1/html/Project1.html
+20 -9
View File
@@ -338,8 +338,6 @@ plt<span style="color: #666666">.</span>show()
</div>
<h3 id="part-a-paper-and-pencil-part-also-as-weekly-exercise-for-week-36">Part a): Paper and pencil part (also as weekly exercise for week 36) </h3>
<p>This part should be included in your theory description of the report.</p>
<p>This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of <a href="https://www.springer.com/gp/book/9780387848570" target="_blank">Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer</a>).</p>
<p>The assumption we have made is
@@ -385,7 +383,7 @@ $$
\mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
$$
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \).
<p>We can use the last expression when we define a so-called confidence interval for the parameters \( \beta \). .
A given parameter \( \beta_j \) is given by the diagonal matrix element of the above matrix.
</p>
<h3 id="part-b-ordinary-least-square-ols-on-the-franke-function">Part b) : Ordinary Least Square (OLS) on the Franke function </h3>
@@ -472,7 +470,7 @@ dataset \( \mathcal{L} \) consisting of the data
\( \mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\} \).
</p>
<p>Let us assume that the true data is generated from a noisy model</p>
<p>As in part a), we assume that the true data is generated from a noisy model</p>
$$
\boldsymbol{y}=f(\boldsymbol{x}) + \boldsymbol{\epsilon}.
@@ -498,14 +496,26 @@ $$
<p>Here the expected value \( \mathbb{E} \) is the sample value. </p>
<p>Show that you can rewrite this as</p>
<p>Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
</p>
$$
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
$$
<p>The answer to this exercise can be included in the theory part of the report.
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
<p>with </p>
$$
(\mathrm{Bias}[\tilde{y}])^2=\left(\boldsymbol{y}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2,
$$
<p>and </p>
$$
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2.
$$
<p>The answer to this exercise should be included in the theory part of the report.
Explain what the terms mean and discuss their interpretations.
</p>
<p>Perform then a bias-variance analysis of the Franke function by
@@ -515,6 +525,7 @@ studying the MSE value as function of the complexity of your model.
<p>Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the <b>bootstrap</b> resampling method.
You can follow the code example in the jupyter-book at <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff" target="_blank"><tt>https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff</tt></a>.
</p>
<p>Note also that when you calculate the bias, in all applications you don't know the function values \( f_i \). You would hence replace them with the actual data points \( y_i \).</p>
doc/Projects/2022/Project1/ipynb/Project1.ipynb
+99 -55
View File
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "98e2a2bd",
"id": "2f0d0303",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
"id": "1951a8ff",
"id": "383fa1e8",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
"id": "755dd202",
"id": "00edcc37",
"metadata": {
"editable": true
},
@@ -63,7 +63,7 @@
},
{
"cell_type": "markdown",
"id": "3b39d52d",
"id": "b8f880fc",
"metadata": {
"editable": true
},
@@ -85,7 +85,7 @@
},
{
"cell_type": "markdown",
"id": "92521420",
"id": "1e667978",
"metadata": {
"editable": true
},
@@ -100,7 +100,7 @@
},
{
"cell_type": "markdown",
"id": "5034b29f",
"id": "3ce1ab3a",
"metadata": {
"editable": true
},
@@ -129,7 +129,7 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "9ee70500",
"id": "965d1f48",
"metadata": {
"collapsed": false,
"editable": true
@@ -181,15 +181,13 @@
},
{
"cell_type": "markdown",
"id": "94aa10c7",
"id": "6171ab22",
"metadata": {
"editable": true
},
"source": [
"### Part a): Paper and pencil part (also as weekly exercise for week 36)\n",
"\n",
"This part should be included in your theory description of the report.\n",
"\n",
"This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)).\n",
"\n",
"The assumption we have made is \n",
@@ -199,7 +197,7 @@
},
{
"cell_type": "markdown",
"id": "1c6e721e",
"id": "969dd687",
"metadata": {
"editable": true
},
@@ -211,7 +209,7 @@
},
{
"cell_type": "markdown",
"id": "d3b0bc88",
"id": "9194b1fc",
"metadata": {
"editable": true
},
@@ -222,7 +220,7 @@
},
{
"cell_type": "markdown",
"id": "76226748",
"id": "cb6e0c5a",
"metadata": {
"editable": true
},
@@ -234,7 +232,7 @@
},
{
"cell_type": "markdown",
"id": "735571e2",
"id": "ef0c8ce0",
"metadata": {
"editable": true
},
@@ -246,7 +244,7 @@
},
{
"cell_type": "markdown",
"id": "7b93fc13",
"id": "c4e904d3",
"metadata": {
"editable": true
},
@@ -258,7 +256,7 @@
},
{
"cell_type": "markdown",
"id": "80c44548",
"id": "8ce55ba5",
"metadata": {
"editable": true
},
@@ -269,7 +267,7 @@
},
{
"cell_type": "markdown",
"id": "1e70c3f1",
"id": "3aa94ac5",
"metadata": {
"editable": true
},
@@ -281,7 +279,7 @@
},
{
"cell_type": "markdown",
"id": "31f5d975",
"id": "8a15da0c",
"metadata": {
"editable": true
},
@@ -294,7 +292,7 @@
},
{
"cell_type": "markdown",
"id": "a564fee7",
"id": "9761270b",
"metadata": {
"editable": true
},
@@ -306,7 +304,7 @@
},
{
"cell_type": "markdown",
"id": "05fc2911",
"id": "12f47ebc",
"metadata": {
"editable": true
},
@@ -316,7 +314,7 @@
},
{
"cell_type": "markdown",
"id": "ca40d503",
"id": "be6e6c6e",
"metadata": {
"editable": true
},
@@ -328,18 +326,18 @@
},
{
"cell_type": "markdown",
"id": "039784c1",
"id": "734989a3",
"metadata": {
"editable": true
},
"source": [
"We can use the last expression when we define a so-called confidence interval for the parameters $\\beta$.\n",
"We can use the last expression when we define a so-called confidence interval for the parameters $\\beta$. .\n",
"A given parameter $\\beta_j$ is given by the diagonal matrix element of the above matrix."
]
},
{
"cell_type": "markdown",
"id": "df1c03c6",
"id": "7e01db05",
"metadata": {
"editable": true
},
@@ -361,7 +359,7 @@
},
{
"cell_type": "markdown",
"id": "7e229e98",
"id": "6431c5d5",
"metadata": {
"editable": true
},
@@ -374,7 +372,7 @@
},
{
"cell_type": "markdown",
"id": "00b00673",
"id": "e4eae5de",
"metadata": {
"editable": true
},
@@ -386,7 +384,7 @@
},
{
"cell_type": "markdown",
"id": "506c2d83",
"id": "6b7f7088",
"metadata": {
"editable": true
},
@@ -398,7 +396,7 @@
},
{
"cell_type": "markdown",
"id": "2cfcf061",
"id": "57e680b6",
"metadata": {
"editable": true
},
@@ -408,7 +406,7 @@
},
{
"cell_type": "markdown",
"id": "84225fa8",
"id": "bbee460a",
"metadata": {
"editable": true
},
@@ -420,7 +418,7 @@
},
{
"cell_type": "markdown",
"id": "7f22646c",
"id": "ab772d43",
"metadata": {
"editable": true
},
@@ -449,7 +447,7 @@
},
{
"cell_type": "markdown",
"id": "05fd370a",
"id": "022b31d9",
"metadata": {
"editable": true
},
@@ -476,12 +474,12 @@
"dataset $\\mathcal{L}$ consisting of the data\n",
"$\\mathbf{X}_\\mathcal{L}=\\{(y_j, \\boldsymbol{x}_j), j=0\\ldots n-1\\}$.\n",
"\n",
"Let us assume that the true data is generated from a noisy model"
"As in part a), we assume that the true data is generated from a noisy model"
]
},
{
"cell_type": "markdown",
"id": "ac09204e",
"id": "38c61204",
"metadata": {
"editable": true
},
@@ -493,7 +491,7 @@
},
{
"cell_type": "markdown",
"id": "09bceede",
"id": "aa00d7dd",
"metadata": {
"editable": true
},
@@ -512,7 +510,7 @@
},
{
"cell_type": "markdown",
"id": "f5bae3fa",
"id": "6240984f",
"metadata": {
"editable": true
},
@@ -524,38 +522,83 @@
},
{
"cell_type": "markdown",
"id": "a8afd8c5",
"id": "0f01484d",
"metadata": {
"editable": true
},
"source": [
"Here the expected value $\\mathbb{E}$ is the sample value. \n",
"\n",
"Show that you can rewrite this as"
"Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a\n",
"term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.\n",
"That is, show that"
]
},
{
"cell_type": "markdown",
"id": "66c12c5c",
"id": "37f1b3f8",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\frac{1}{n}\\sum_i(f_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\sigma^2.\n",
"\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=(\\mathrm{Bias}[\\tilde{y}])^2+\\mathrm{var}[\\tilde{f}]+\\sigma^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "80d3e8a0",
"id": "0673dacf",
"metadata": {
"editable": true
},
"source": [
"The answer to this exercise can be included in the theory part of the report.\n",
"Explain what the terms mean, which one is the bias and which one is\n",
"the variance and discuss their interpretations.\n",
"with"
]
},
{
"cell_type": "markdown",
"id": "281ae919",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"(\\mathrm{Bias}[\\tilde{y}])^2=\\left(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "3604e435",
"metadata": {
"editable": true
},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"id": "cb6e58c2",
"metadata": {
"editable": true
},
"source": [
"$$\n",
"\\mathrm{var}[\\tilde{f}]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"id": "65111a57",
"metadata": {
"editable": true
},
"source": [
"The answer to this exercise should be included in the theory part of the report.\n",
"Explain what the terms mean and discuss their interpretations.\n",
"\n",
"Perform then a bias-variance analysis of the Franke function by\n",
"studying the MSE value as function of the complexity of your model.\n",
@@ -563,13 +606,14 @@
"Discuss the bias and variance trade-off as function\n",
"of your model complexity (the degree of the polynomial) and the number\n",
"of data points, and possibly also your training and test data using the **bootstrap** resampling method.\n",
"You can follow the code example in the jupyter-book at <https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff>.\n",
"\n",
"Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$."
]
},
{
"cell_type": "markdown",
"id": "a28b6e71",
"id": "54283cc7",
"metadata": {
"editable": true
},
@@ -594,7 +638,7 @@
},
{
"cell_type": "markdown",
"id": "d9cfcca9",
"id": "47078a1a",
"metadata": {
"editable": true
},
@@ -614,7 +658,7 @@
},
{
"cell_type": "markdown",
"id": "66f94cf5",
"id": "97849c2c",
"metadata": {
"editable": true
},
@@ -631,7 +675,7 @@
},
{
"cell_type": "markdown",
"id": "c0fbed56",
"id": "c37b1811",
"metadata": {
"editable": true
},
@@ -659,7 +703,7 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "62fe430e",
"id": "646b70df",
"metadata": {
"collapsed": false,
"editable": true
@@ -671,7 +715,7 @@
},
{
"cell_type": "markdown",
"id": "46a403c0",
"id": "5107f3b6",
"metadata": {
"editable": true
},
@@ -683,7 +727,7 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "e3f11b71",
"id": "24ae3ba0",
"metadata": {
"collapsed": false,
"editable": true
@@ -709,7 +753,7 @@
},
{
"cell_type": "markdown",
"id": "9a69144e",
"id": "1f239875",
"metadata": {
"editable": true
},
@@ -734,7 +778,7 @@
},
{
"cell_type": "markdown",
"id": "ad7c0e60",
"id": "ea729b77",
"metadata": {
"editable": true
},
@@ -748,7 +792,7 @@
},
{
"cell_type": "markdown",
"id": "abc7b07b",
"id": "61fd5713",
"metadata": {
"editable": true
},
@@ -778,7 +822,7 @@
},
{
"cell_type": "markdown",
"id": "a85b7b4b",
"id": "9579c5f3",
"metadata": {
"editable": true
},
@@ -800,7 +844,7 @@
},
{
"cell_type": "markdown",
"id": "10af239b",
"id": "ecf2eb04",
"metadata": {
"editable": true
},
doc/Projects/2022/Project1/ipynb/ipynb-Project1-src.tar.gz
BIN
View File
Binary file not shown.
doc/Projects/2022/Project1/pdf/Project1.p.tex
+17 -9
View File
@@ -290,8 +290,6 @@ plt.show()
\paragraph{Part a): Paper and pencil part (also as weekly exercise for week 36).}
This part should be included in your theory description of the report.
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
The assumption we have made is
@@ -329,7 +327,7 @@ Show finally that the variance of $\bm{\beta}$ is
\mbox{Var}(\bm{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
\]
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$.
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
\paragraph{Part b) : Ordinary Least Square (OLS) on the Franke function.}
@@ -405,7 +403,7 @@ Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
Let us assume that the true data is generated from a noisy model
As in part a), we assume that the true data is generated from a noisy model
\[
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
@@ -427,13 +425,22 @@ C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\
\]
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this as
Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
\[
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2.
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
\]
The answer to this exercise can be included in the theory part of the report.
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
with
\[
(\mathrm{Bias}[\tilde{y}])^2=\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2,
\]
and
\[
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2.
\]
The answer to this exercise should be included in the theory part of the report.
Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
@@ -441,6 +448,7 @@ studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method.
You can follow the code example in the jupyter-book at \href{{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff}}{\nolinkurl{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html\#the-bias-variance-tradeoff}}.
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
doc/Projects/2022/Project1/pdf/Project1.pdf
BIN
View File
Binary file not shown.
doc/Projects/2022/Project1/pdf/Project1.tex
+17 -9
View File
@@ -260,8 +260,6 @@ plt.show()
\paragraph{Part a): Paper and pencil part (also as weekly exercise for week 36).}
This part should be included in your theory description of the report.
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of \href{{https://www.springer.com/gp/book/9780387848570}}{Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer}).
The assumption we have made is
@@ -299,7 +297,7 @@ Show finally that the variance of $\bm{\beta}$ is
\mbox{Var}(\bm{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}.
\]
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$.
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
\paragraph{Part b) : Ordinary Least Square (OLS) on the Franke function.}
@@ -375,7 +373,7 @@ Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
Let us assume that the true data is generated from a noisy model
As in part a), we assume that the true data is generated from a noisy model
\[
\bm{y}=f(\boldsymbol{x}) + \bm{\epsilon}.
@@ -397,13 +395,22 @@ C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\
\]
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this as
Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
\[
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2.
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
\]
The answer to this exercise can be included in the theory part of the report.
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
with
\[
(\mathrm{Bias}[\tilde{y}])^2=\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2,
\]
and
\[
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2.
\]
The answer to this exercise should be included in the theory part of the report.
Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
@@ -411,6 +418,7 @@ studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the \textbf{bootstrap} resampling method.
You can follow the code example in the jupyter-book at \href{{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff}}{\nolinkurl{https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html\#the-bias-variance-tradeoff}}.
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.
doc/src/Projects/2022/Project1/Project1.do.txt
+21 -8
View File
@@ -123,7 +123,6 @@ plt.show()
=== Part a): Paper and pencil part (also as weekly exercise for week 36) ===
This part should be included in your theory description of the report.
This exercise deals with various mean values and variances in linear regression method (here it may be useful to look up chapter 3, equation (3.8) of "Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer":"https://www.springer.com/gp/book/9780387848570").
@@ -176,7 +175,7 @@ Show finally that the variance of $\bm{\beta}$ is
!et
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$.
We can use the last expression when we define a so-called confidence interval for the parameters $\beta$. .
A given parameter $\beta_j$ is given by the diagonal matrix element of the above matrix.
=== Part b) : Ordinary Least Square (OLS) on the Franke function ===
@@ -264,7 +263,7 @@ Consider a
dataset $\mathcal{L}$ consisting of the data
$\mathbf{X}_\mathcal{L}=\{(y_j, \boldsymbol{x}_j), j=0\ldots n-1\}$.
Let us assume that the true data is generated from a noisy model
As in part a), we assume that the true data is generated from a noisy model
!bt
\[
@@ -290,15 +289,28 @@ C(\bm{X},\bm{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\
!et
Here the expected value $\mathbb{E}$ is the sample value.
Show that you can rewrite this as
Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise.
That is, show that
!bt
\[
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2+\sigma^2.
\mathbb{E}\left[(\bm{y}-\bm{\tilde{y}})^2\right]=(\mathrm{Bias}[\tilde{y}])^2+\mathrm{var}[\tilde{f}]+\sigma^2,
\]
!et
The answer to this exercise can be included in the theory part of the report.
Explain what the terms mean, which one is the bias and which one is
the variance and discuss their interpretations.
with
!bt
\[
(\mathrm{Bias}[\tilde{y}])^2=\left(\bm{y}-\mathbb{E}\left[\bm{\tilde{y}}\right]\right)^2,
\]
!et
and
!bt
\[
\mathrm{var}[\tilde{f}]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\bm{\tilde{y}}\right])^2.
\]
!et
The answer to this exercise should be included in the theory part of the report.
Explain what the terms mean and discuss their interpretations.
Perform then a bias-variance analysis of the Franke function by
studying the MSE value as function of the complexity of your model.
@@ -306,6 +318,7 @@ studying the MSE value as function of the complexity of your model.
Discuss the bias and variance trade-off as function
of your model complexity (the degree of the polynomial) and the number
of data points, and possibly also your training and test data using the _bootstrap_ resampling method.
You can follow the code example in the jupyter-book at URL:"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#the-bias-variance-tradeoff".
Note also that when you calculate the bias, in all applications you don't know the function values $f_i$. You would hence replace them with the actual data points $y_i$.