From 9f547ac70d410cc5b48b162e8d4a22bd48955112 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 16 Sep 2025 03:56:04 +0200 Subject: [PATCH] update --- .../exercisesweek38-checkpoint.ipynb | 172 ++++++++++++------ doc/LectureNotes/exercisesweek38.ipynb | 13 +- 2 files changed, 116 insertions(+), 69 deletions(-) diff --git a/doc/LectureNotes/.ipynb_checkpoints/exercisesweek38-checkpoint.ipynb b/doc/LectureNotes/.ipynb_checkpoints/exercisesweek38-checkpoint.ipynb index 065cbdf95..4ffd81af5 100644 --- a/doc/LectureNotes/.ipynb_checkpoints/exercisesweek38-checkpoint.ipynb +++ b/doc/LectureNotes/.ipynb_checkpoints/exercisesweek38-checkpoint.ipynb @@ -6,9 +6,10 @@ "metadata": {}, "source": [ "# Exercises week 38\n", + "\n", "## September 15-19\n", "\n", - "## Resampling and the Bias-Variance Trade-off" + "## Resampling and the Bias-Variance Trade-off\n" ] }, { @@ -16,12 +17,20 @@ "id": "e9f27b0e", "metadata": {}, "source": [ - "## Learning goals\n", + "### Learning goals\n", "\n", "After completing these exercises, you will know how to\n", + "\n", "- Derive expectation and variances values related to linear regression\n", "- Compute expectation and variances values related to linear regression\n", - "- Compute and evaluate the trade-off between bias and variance of a model" + "- Compute and evaluate the trade-off between bias and variance of a model\n", + "\n", + "### Deliverables\n", + "\n", + "Complete the following exercises while working in a jupyter notebook. Then, in canvas, include\n", + "\n", + "- The jupyter notebook with the exercises completed\n", + "- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)\n" ] }, { @@ -29,11 +38,17 @@ "id": "984af8e3", "metadata": {}, "source": [ - "This week deals with various mean values and variances in linear regression methods (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)). The exercises are also a part of project 1 and can be reused in the theory part of the project.\n", + "## Use the books!\n", + "\n", + "This week deals with various mean values and variances in linear regression methods (here it may be useful to look up chapter 3, equation (3.8) of [Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer](https://www.springer.com/gp/book/9780387848570)).\n", "\n", "For more discussions on Ridge regression and calculation of expectation values, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", "\n", - "We assume that there exists a continuous function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim N(0, \\sigma^2)$ which describes our data" + "The exercises this week are also a part of project 1 and can be reused in the theory part of the project.\n", + "\n", + "### Definitions\n", + "\n", + "We assume that there exists a continuous function $f(\\boldsymbol{x})$ and a normal distributed error $\\boldsymbol{\\varepsilon}\\sim N(0, \\sigma^2)$ which describes our data\n" ] }, { @@ -43,7 +58,7 @@ "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", - "$$" + "$$\n" ] }, { @@ -55,7 +70,7 @@ } }, "source": [ - "We further assume that this continous function can be modeled with a linear model $\\mathbf{\\tilde{y}}$ of some features $\\mathbf{X}$." + "We further assume that this continous function can be modeled with a linear model $\\mathbf{\\tilde{y}}$ of some features $\\mathbf{X}$.\n" ] }, { @@ -65,7 +80,7 @@ "source": [ "$$\n", "\\boldsymbol{y} = \\boldsymbol{\\tilde{y}} + \\boldsymbol{\\varepsilon} = \\boldsymbol{X}\\boldsymbol{\\beta} +\\boldsymbol{\\varepsilon}\n", - "$$" + "$$\n" ] }, { @@ -73,7 +88,7 @@ "id": "f4fca21b", "metadata": {}, "source": [ - "We therefore get that our data $\\boldsymbol{y}$ has an expectation value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$, that is $\\boldsymbol{y}$ follows a normal distribution with mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$." + "We therefore get that our data $\\boldsymbol{y}$ has an expectation value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$, that is $\\boldsymbol{y}$ follows a normal distribution with mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$.\n" ] }, { @@ -81,7 +96,7 @@ "id": "5de0c7e6", "metadata": {}, "source": [ - "## Exercise 1: Expectation values for ordinary least squares expressions" + "## Exercise 1: Expectation values for ordinary least squares expressions\n" ] }, { @@ -89,7 +104,7 @@ "id": "d878c699", "metadata": {}, "source": [ - "**a)** With the expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ show that" + "**a)** With the expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ show that\n" ] }, { @@ -99,7 +114,7 @@ "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\hat{\\beta}_{OLS}}) = \\boldsymbol{\\beta}.\n", - "$$" + "$$\n" ] }, { @@ -107,7 +122,7 @@ "id": "46e93394", "metadata": {}, "source": [ - "**b)** Show that the variance of $\\boldsymbol{\\hat{\\beta}_{OLS}}$ is" + "**b)** Show that the variance of $\\boldsymbol{\\hat{\\beta}_{OLS}}$ is\n" ] }, { @@ -117,7 +132,7 @@ "source": [ "$$\n", "\\mathbf{Var}(\\boldsymbol{\\hat{\\beta}_{OLS}}) = \\sigma^2 \\, (\\mathbf{X}^{T} \\mathbf{X})^{-1}.\n", - "$$" + "$$\n" ] }, { @@ -125,8 +140,8 @@ "id": "d2143684", "metadata": {}, "source": [ - "We can use the last expression when we define a [confidence interval](https://en.wikipedia.org/wiki/Confidence_interval) for the parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$. \n", - "A given parameter ${\\boldsymbol{\\hat{\\beta}_{OLS}}}_j$ is given by the diagonal matrix element of the above matrix." + "We can use the last expression when we define a [confidence interval](https://en.wikipedia.org/wiki/Confidence_interval) for the parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$.\n", + "A given parameter ${\\boldsymbol{\\hat{\\beta}_{OLS}}}_j$ is given by the diagonal matrix element of the above matrix.\n" ] }, { @@ -134,7 +149,7 @@ "id": "f5c2dc22", "metadata": {}, "source": [ - "## Exercise 2: Expectation values for Ridge regression" + "## Exercise 2: Expectation values for Ridge regression\n" ] }, { @@ -142,7 +157,7 @@ "id": "3893e3e7", "metadata": {}, "source": [ - "**a)** With the expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}_{Ridge}}$ show that" + "**a)** With the expressions for the optimal parameters $\\boldsymbol{\\hat{\\beta}_{Ridge}}$ show that\n" ] }, { @@ -152,7 +167,7 @@ "source": [ "$$\n", "\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}\n", - "$$" + "$$\n" ] }, { @@ -160,15 +175,7 @@ "id": "028209a1", "metadata": {}, "source": [ - "We see that $\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$." - ] - }, - { - "cell_type": "markdown", - "id": "65f6f914", - "metadata": {}, - "source": [ - "**b)** Why do we say that Ridge regression gives a biased estimate? Is this a problem?" + "We see that $\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n" ] }, { @@ -176,7 +183,7 @@ "id": "b4e721fc", "metadata": {}, "source": [ - "**c)** Show that the variance is" + "**b)** Show that the variance is\n" ] }, { @@ -186,7 +193,7 @@ "source": [ "$$\n", "\\mathbf{Var}[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T}\\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}\n", - "$$" + "$$\n" ] }, { @@ -194,7 +201,7 @@ "id": "6b8e8697", "metadata": {}, "source": [ - "We see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero." + "We see that if the parameter $\\lambda$ goes to infinity then the variance of the Ridge parameters $\\boldsymbol{\\beta}$ goes to zero.\n" ] }, { @@ -202,7 +209,7 @@ "id": "74bc300b", "metadata": {}, "source": [ - "## Exercise 3: Deriving the expression for the Bias-Variance Trade-off" + "## Exercise 3: Deriving the expression for the Bias-Variance Trade-off\n" ] }, { @@ -212,7 +219,7 @@ "source": [ "The aim of this exercise is to derive the equations for the bias-variance tradeoff to be used in project 1.\n", "\n", - "The parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ are found by optimizing the mean squared error via the so-called cost function" + "The parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ are found by optimizing the mean squared error via the so-called cost function\n" ] }, { @@ -222,7 +229,7 @@ "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]\n", - "$$" + "$$\n" ] }, { @@ -230,9 +237,13 @@ "id": "831db06c", "metadata": {}, "source": [ - "**a)** Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a\n", - "term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise Note that in order to be able to evaluate the bias them, you will need to approximate the function $f$ with the model ${\\bf y}$.\n", - "show that" + "**a)** Show that you can rewrite this into an expression which contains\n", + "\n", + "- the variance of the model (the variance term)\n", + "- the expected deviation of the mean of the model from the true data (the bias term)\n", + "- the variance of the noise\n", + "\n", + "In other words, show that:\n" ] }, { @@ -242,7 +253,7 @@ "source": [ "$$\n", "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathrm{Bias}[\\tilde{y}]+\\mathrm{var}[\\tilde{y}]+\\sigma^2,\n", - "$$" + "$$\n" ] }, { @@ -250,7 +261,7 @@ "id": "8cb50416", "metadata": {}, "source": [ - "with" + "with\n" ] }, { @@ -260,7 +271,7 @@ "source": [ "$$\n", "\\mathrm{Bias}[\\tilde{y}]=\\mathbb{E}\\left[\\left(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2\\right],\n", - "$$" + "$$\n" ] }, { @@ -268,7 +279,7 @@ "id": "eca5554a", "metadata": {}, "source": [ - "and" + "and\n" ] }, { @@ -278,7 +289,8 @@ "source": [ "$$\n", "\\mathrm{var}[\\tilde{y}]=\\mathbb{E}\\left[\\left(\\tilde{\\boldsymbol{y}}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2\\right]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n", - "$$" + "$$\n", + "In order to arrive at the last equation, we have to approximate the unknown function $f$ with the output/target values $y$." ] }, { @@ -286,7 +298,7 @@ "id": "70fbfcd7", "metadata": {}, "source": [ - "**b)** Explain what the terms mean and discuss their interpretations." + "**b)** Explain what the terms mean and discuss their interpretations.\n" ] }, { @@ -294,7 +306,7 @@ "id": "b8f8b9d1", "metadata": {}, "source": [ - "## Exercise 4: Computing the Bias and Variance" + "## Exercise 4: Computing the Bias and Variance\n" ] }, { @@ -304,7 +316,7 @@ "source": [ "Before you compute the bias and variance of a real model for different complexities, let's for now assume that you have sampled predictions and targets for a single model complexity using bootstrap resampling.\n", "\n", - "**a)** Using the expression above, compute the mean squared error, bias and variance of the given data. Check that the sum of the bias and variance correctly gives (approximately) the mean squared error." + "**a)** Using the expression above, compute the mean squared error, bias and variance of the given data. Check that the sum of the bias and variance correctly gives (approximately) the mean squared error.\n" ] }, { @@ -334,7 +346,7 @@ "source": [ "**b)** Change the prediction values in some way to increase the bias while decreasing the variance.\n", "\n", - "**c)** Change the prediction values in some way to increase the variance while decreasing the bias." + "**c)** Change the prediction values in some way to increase the variance while decreasing the bias.\n" ] }, { @@ -342,19 +354,21 @@ "id": "8da63362", "metadata": {}, "source": [ - "**d)** Perform a bias-variance analysis of a polynomial OLS model fit to a one-dimensional function by computing and plotting the bias and variances values as a function of the polynomial degree of your model." + "**d)** Perform a bias-variance analysis of a polynomial OLS model fit to a one-dimensional function by computing and plotting the bias and variances values as a function of the polynomial degree of your model.\n" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": null, "id": "dd5855e4", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", - "from sklearn.preprocessing import PolynomialFeatures # use the fit_transform method of the created object!\n", + "from sklearn.preprocessing import (\n", + " PolynomialFeatures,\n", + ") # use the fit_transform method of the created object!\n", "from sklearn.linear_model import LinearRegression\n", "from sklearn.metrics import mean_squared_error\n", "from sklearn.model_selection import train_test_split\n", @@ -372,23 +386,27 @@ "bootstraps = 1000\n", "\n", "x = np.linspace(-3, 3, n)\n", - "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)\n", + "y = np.exp(-(x**2)) + 1.5 * np.exp(-((x - 2) ** 2)) + np.random.normal(0, 0.1)\n", "\n", "biases = []\n", "variances = []\n", "mses = []\n", "\n", - "#for p in range(1, 5):\n", + "# for p in range(1, 5):\n", "# predictions = ...\n", "# targets = ...\n", - "# for b in range(bootstraps):\n", - "# x_sample, y_sample = ...\n", - "# X = ...\n", - "# X_train, X_test, y_train, y_test = ...\n", "#\n", - "# predictions[b, :] = \n", - "# targets[b, :] = \n", - "# \n", + "# X = ...\n", + "# X_train, X_test, y_train, y_test = ...\n", + "# for b in range(bootstraps):\n", + "# X_train_re, y_train_re = ...\n", + "#\n", + "# # fit your model on the sampled data\n", + "#\n", + "# # make predictions on the test data\n", + "# predictions[b, :] =\n", + "# targets[b, :] =\n", + "#\n", "# biases.append(...)\n", "# variances.append(...)\n", "# mses.append(...)" @@ -401,7 +419,43 @@ "source": [ "**e)** Discuss the bias-variance trade-off as function of your model complexity (the degree of the polynomial).\n", "\n", - "**f)** Compute and discuss the bias and variance as function of the number of data points (choose a suitable polynomial degree to show something interesiting)." + "**f)** Compute and discuss the bias and variance as function of the number of data points (choose a suitable polynomial degree to show something interesting).\n" + ] + }, + { + "cell_type": "markdown", + "id": "46250fbc", + "metadata": {}, + "source": [ + "## Exercise 5: Interpretation of scaling and metrics\n" + ] + }, + { + "cell_type": "markdown", + "id": "5af53055", + "metadata": {}, + "source": [ + "In this course, we often ask you to scale data and compute various metrics. Although these practices are \"standard\" in the field, we will require you to demonstrate an understanding of _why_ you need to scale data and use these metrics. Both so that you can make better arguements about your results, and so that you will hopefully make fewer mistakes.\n", + "\n", + "First, a few reminders: In this course you should always scale the columns of the feature matrix, and sometimes scale the target data, when it is worth the effort. By scaling, we mean subtracting the mean and dividing by the standard deviation, though there are many other ways to scale data. When scaling either the feature matrix or the target data, the intercept becomes a bit harder to implement and understand, so take care.\n", + "\n", + "Briefly answer the following:\n", + "\n", + "**a)** Why do we scale data?\n", + "\n", + "**b)** Why does the OLS method give practically equivelent models on scaled and unscaled data?\n", + "\n", + "**c)** Why does the Ridge method **not** give practically equivelent models on scaled and unscaled data? Why do we only consider the model on scaled data correct?\n", + "\n", + "**d)** Why do we say that the Ridge method gives a biased model?\n", + "\n", + "**e)** Is the MSE of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?\n", + "\n", + "**f)** Read about the R2 score, a metric we will ask you to use a lot later in the course. Is the R2 score of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?\n", + "\n", + "**g)** Give interpretations of the following R2 scores: 0, 0.5, 1.\n", + "\n", + "**h)** What is an advantage of the R2 score over the MSE?\n" ] } ], diff --git a/doc/LectureNotes/exercisesweek38.ipynb b/doc/LectureNotes/exercisesweek38.ipynb index c26fbccf9..4ffd81af5 100644 --- a/doc/LectureNotes/exercisesweek38.ipynb +++ b/doc/LectureNotes/exercisesweek38.ipynb @@ -178,20 +178,12 @@ "We see that $\\mathbb{E} \\big[ \\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}} \\big] \\not= \\mathbb{E} \\big[\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}\\big ]$ for any $\\lambda > 0$.\n" ] }, - { - "cell_type": "markdown", - "id": "65f6f914", - "metadata": {}, - "source": [ - "**b)** Why do we say that Ridge regression gives a biased estimate? Is this a problem?\n" - ] - }, { "cell_type": "markdown", "id": "b4e721fc", "metadata": {}, "source": [ - "**c)** Show that the variance is\n" + "**b)** Show that the variance is\n" ] }, { @@ -297,7 +289,8 @@ "source": [ "$$\n", "\\mathrm{var}[\\tilde{y}]=\\mathbb{E}\\left[\\left(\\tilde{\\boldsymbol{y}}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]\\right)^2\\right]=\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2.\n", - "$$\n" + "$$\n", + "In order to arrive at the last equation, we have to approximate the unknown function $f$ with the output/target values $y$." ] }, {