We assume that there exists a continuous function $f(\boldsymbol{x})$ and a normal distributed error $\boldsymbol{\varepsilon}\sim N(0, \sigma^2)$ which describes our data
We therefore get that our data $\boldsymbol{y}$ has an expectation value $\boldsymbol{X}\boldsymbol{\beta}$ and variance $\sigma^2$, that is $\boldsymbol{y}$ follows a normal distribution with mean value $\boldsymbol{X}\boldsymbol{\beta}$ and variance $\sigma^2$.
+
+
+
+
+
+
+
+
+
+
+
Exercise 1: Expectation values for ordinary least squares expressions¶
+
+
+
+
+
+
+
+
+
+
+
a) With the expressions for the optimal parameters $\boldsymbol{\hat{\beta}_{OLS}}$ show that
$$
+\begin{aligned}
+\mathbf{Var}(\hat \beta_{OLS}) &= \mathbb E (\beta_{OLS} - \mathbb{E}(\hat{\beta}_{OLS}))^2 \\ &= \mathbb E (\beta_{OLS} - \beta)^2\\ &= \mathbb E (\beta + (X^T X)^{-1} X^T \epsilon - \beta)^2\\ &= \mathbb E ((X^T X)^{-1} X^T \epsilon)^2\\ &= \sigma^2 \mathbb E ((X^T X)^{-1} X^T X ((X^T X)^{-1})^T)\\ &= \sigma^2 ((X^T X)^{-1})^T\\ &= \sigma^2 (X^T X)^{-1}
+\end{aligned}
+$$
+
+
+
+
+
+
+
+
+
+
+
+
We can use the last expression when we define a confidence interval for the parameters $\boldsymbol{\hat{\beta}_{OLS}}$.
+A given parameter ${\boldsymbol{\hat{\beta}_{OLS}}}_j$ is given by the diagonal matrix element of the above matrix.
+
+
+
+
+
+
+
+
+
+
+
Exercise 2: Expectation values for Ridge regression¶
+
+
+
+
+
+
+
+
+
+
+
a) With the expressions for the optimal parameters $\boldsymbol{\hat{\beta}_{Ridge}}$ show that
We see that $\mathbb{E} [ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}} ] \not= \mathbb{E} [\hat{\boldsymbol{\beta}}^{\mathrm{OLS}} ]$ for any $\lambda > 0$.
+
+
+
+
+
+
+
+
+
+
+
+
$$
+\mathbb E \hat \beta_\mathrm{Ridge} = \mathbb E \left((X^TX + \lambda I)^{-1} X^T \tilde y\right) + \mathbb E \left((X^T X + \lambda I)^{-1} X^T \epsilon\right) = \mathbb E \left((X^TX + \lambda I)^{-1} X^T \tilde y\right) = (X^TX + \lambda I)^{-1} X^T X \beta
+$$
$$
+\mathrm{var}[\tilde{y}]=\mathbb{E}\left[\left(\tilde{\boldsymbol{y}}-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right]\right)^2\right]=\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2.
+$$
+In order to arrive at the equation for the bias, we have to approximate the unknown function $f$ with the output/target values $y$.
Before you compute the bias and variance of a real model for different complexities, let's for now assume that you have sampled predictions and targets for a single model complexity using bootstrap resampling.
+
a) Using the expression above, compute the mean squared error, bias and variance of the given data. Check that the sum of the bias and variance correctly gives (approximately) the mean squared error.
d) Perform a bias-variance analysis of a polynomial OLS model fit to a one-dimensional function by computing and plotting the bias and variances values as a function of the polynomial degree of your model.
+
+
+
+
+
+
+
+
+
In [4]:
+
+
+
importnumpyasnp
+importmatplotlib.pyplotasplt
+fromsklearn.preprocessingimport(
+ PolynomialFeatures,
+)# use the fit_transform method of the created object!
+fromsklearn.linear_modelimportLinearRegression
+fromsklearn.metricsimportmean_squared_error
+fromsklearn.model_selectionimporttrain_test_split
+fromsklearn.utilsimportresample
+
e) Discuss the bias-variance trade-off as function of your model complexity (the degree of the polynomial).
+
+
+
+
+
+
+
+
+
+
+
+ The assymmetry of bias and variance reaches a minimum at the same point where the total model error reaches a minimum wrt to the model complexity. At a polynomial degree of 3 we have the best model complexity.
+
+
+
+
+
+
+
+
+
+
+
+
f) Compute and discuss the bias and variance as function of the number of data points (choose a suitable polynomial degree to show something interesting).
Exercise 5: Interpretation of scaling and metrics¶
+
+
+
+
+
+
+
+
+
+
+
In this course, we often ask you to scale data and compute various metrics. Although these practices are "standard" in the field, we will require you to demonstrate an understanding of why you need to scale data and use these metrics. Both so that you can make better arguements about your results, and so that you will hopefully make fewer mistakes.
+
First, a few reminders: In this course you should always scale the columns of the feature matrix, and sometimes scale the target data, when it is worth the effort. By scaling, we mean subtracting the mean and dividing by the standard deviation, though there are many other ways to scale data. When scaling either the feature matrix or the target data, the intercept becomes a bit harder to implement and understand, so take care.
+
Briefly answer the following:
+
a) Why do we scale data?
+
+
+
+
+
+
+
+
+
+
+
+Because it allows for
+
+
+
You can get a feeling for parameter values
+
Similar Hyperparameters are usable over multiple different problem sets
+
No Numerical Inaccuracies due to float limitations
+
No Bias of Features in regualrized regression (see next questions)
+
There's no major disadvantage
+
+
+
+
+
+
+
+
+
+
+
+
+
b) Why does the OLS method give practically equivelent models on scaled and unscaled data?
+
+
+
+
+
+
+
+
+
+
+
+Because the cost function depends only on the deviation between output and target variables. There is no influence of parameter values on the cost value. Thus the minimum of the cost function is invariant wrt to scaling of the features.
+
+
+
+
+
+
+
+
+
+
+
+
c) Why does the Ridge method not give practically equivelent models on scaled and unscaled data? Why do we only consider the model on scaled data correct?
+
+
+
+
+
+
+
+
+
+
+
+ Because the cost function directly depends on the parameter values. But if for example we scale feature 1 by an factor of 3, the parameter of feature 1 needs to be multiplied by 1/3 for an optimal result. But because the cost function also depends on the parameter values, this may not be the minimum of the cost function anymore.
+
+
+
+
+
+
+
+
+
+
+
+
d) Why do we say that the Ridge method gives a biased model?
+
+
+
+
+
+
+
+
+
+
+
+
The expectation value for the ridge parameters only approaches the true value $\beta$ in the limit $\lambda \to 0$, this is why we call it biased. As independent of number of training samples our estimate will always differ from the true value given $\lambda \neq 0$ (in which case it would be OLS). The larger the parameter value (i.e. parameter not equal to zero), the larger our cost value. Even though it may be needed that the paramter is bigger for our targets and predictions to align perfectly. This may be a problem if the goal of our analysis is the closest possible parameter estimates given near infinite number of training samples (because in this case the deviation from the true value approaches 0).
+
+
+
+
+
+
+
+
+
+
+
+
e) Is the MSE of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?
+
+
+
+
+
+
+
+
+
+
+
+
It is only affected by scaling of the target data. The MSE is in units of $[y]^2$. So if our scaled unit is $[y_{scaled}] = 10[y]$, then our MSE will be scaled by a factor of 100.
+
+
+
+
+
+
+
+
+
+
+
+
f) Read about the R2 score, a metric we will ask you to use a lot later in the course. Is the R2 score of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?
+
+
+
+
+
+
+
+
+
+
+
+ The R2 score is invariant under scaling of the feature and target data.
+
+
+
+
+
+
+
+
+
+
+
+
g) Give interpretations of the following R2 scores: 0, 0.5, 1.
+
+
+
+
+
+
+
+
+
+
+
+
+
0: There is no correlation between the prediction and the target values. The target values can be equally good described by just giving the mean of the values.
+
0.5: Half of the variation in the target values can be explained by the models predictions.
+
1: The prediction perfectly aligns with the target values.
+
+
+
+
+
+
+
+
+
+
+
+
+
h) What is an advantage of the R2 score over the MSE?
+
+
+
+
+
+
+
+
+
+
+
+
+
It is invariant under scaling of the target values
+
It has a limited range of values with a particular meaning.
+
+
+
+
+
+
+
+
+
diff --git a/doc/LectureNotes/exercisesweek38.ipynb b/doc/LectureNotes/exercisesweek38.ipynb
index 48df7c2b8..7ddf0a32a 100644
--- a/doc/LectureNotes/exercisesweek38.ipynb
+++ b/doc/LectureNotes/exercisesweek38.ipynb
@@ -223,26 +223,6 @@
""
]
},
- {
- "cell_type": "markdown",
- "id": "65f6f914",
- "metadata": {},
- "source": [
- "**b)** Why do we say that Ridge regression gives a biased estimate? Is this a problem?\n"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "241e8533",
- "metadata": {},
- "source": [
- "
\n",
- "\n",
- "The expectation value for the ridge parameters only approaches the true value $\\beta$ in the limit $\\lambda \\to 0$, this is why we call it biased. As independent of number of training samples our estimate will always differ from the true value given $\\lambda \\neq 0$ (in which case it would be OLS). This may be a problem if the goal of our analysis is the closest possible parameter estimates given near infinite number of training samples (because in this case the deviation from the true value approaches 0).\n",
- "\n",
- "
\n",
+ " The assymmetry of bias and variance reaches a minimum at the same point where the total model error reaches a minimum wrt to the model complexity. At a polynomial degree of 3 we have the best model complexity.\n",
+ "
\n",
+ "Because it allows for\n",
"\n",
- "**b)** Why does the OLS method give practically equivelent models on scaled and unscaled data?\n",
+ "- You can get a feeling for parameter values\n",
+ "- Similar Hyperparameters are usable over multiple different problem sets\n",
+ "- No Numerical Inaccuracies due to float limitations\n",
+ "- No Bias of Features in regualrized regression (see next questions)\n",
+ "- There's no major disadvantage\n",
+ "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fc04426b",
+ "metadata": {},
+ "source": [
"\n",
- "**c)** Why does the Ridge method **not** give practically equivelent models on scaled and unscaled data? Why do we only consider the model on scaled data correct?\n",
"\n",
- "**d)** Why do we say that the Ridge method gives a biased model?\n",
+ "**b)** Why does the OLS method give practically equivelent models on scaled and unscaled data?\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2028408",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "Because the cost function depends only on the deviation between output and target variables. There is no influence of parameter values on the cost value. Thus the minimum of the cost function is invariant wrt to scaling of the features.\n",
+ "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "48c44d22",
+ "metadata": {},
+ "source": [
"\n",
- "**e)** Is the MSE of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?\n",
+ "**c)** Why does the Ridge method **not** give practically equivelent models on scaled and unscaled data? Why do we only consider the model on scaled data correct?\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2b19fbb4",
+ "metadata": {},
+ "source": [
+ "
\n",
+ " Because the cost function directly depends on the parameter values. But if for example we scale feature 1 by an factor of 3, the parameter of feature 1 needs to be multiplied by 1/3 for an optimal result. But because the cost function also depends on the parameter values, this may not be the minimum of the cost function anymore.\n",
+ "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d9979876",
+ "metadata": {},
+ "source": [
"\n",
- "**f)** Read about the R2 score, a metric we will ask you to use a lot later in the course. Is the R2 score of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?\n",
+ "**d)** Why do we say that the Ridge method gives a biased model?\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c9265015",
+ "metadata": {},
+ "source": [
+ "
\n",
"\n",
- "**g)** Give interpretations of the following R2 scores: 0, 0.5, 1.\n",
+ "The expectation value for the ridge parameters only approaches the true value $\\beta$ in the limit $\\lambda \\to 0$, this is why we call it biased. As independent of number of training samples our estimate will always differ from the true value given $\\lambda \\neq 0$ (in which case it would be OLS). The larger the parameter value (i.e. parameter not equal to zero), the larger our cost value. Even though it may be needed that the paramter is bigger for our targets and predictions to align perfectly. This may be a problem if the goal of our analysis is the closest possible parameter estimates given near infinite number of training samples (because in this case the deviation from the true value approaches 0).\n",
+ "\n",
+ "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8287f760",
+ "metadata": {},
+ "source": [
+ "\n",
+ "**e)** Is the MSE of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "edcdff21",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ " It is only affected by scaling of the target data. The MSE is in units of $[y]^2$. So if our scaled unit is $[y_{scaled}] = 10[y]$, then our MSE will be scaled by a factor of 100.\n",
+ "\n",
+ "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5145de2d",
+ "metadata": {},
+ "source": [
+ "\n",
+ "**f)** Read about the R2 score, a metric we will ask you to use a lot later in the course. Is the R2 score of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a552db5a",
+ "metadata": {},
+ "source": [
+ "
\n",
+ " The R2 score is invariant under scaling of the feature and target data. \n",
+ "
\n",
+ " \n",
+ "- 0: There is no correlation between the prediction and the target values. The target values can be equally good described by just giving the mean of the values.\n",
+ "- 0.5: Half of the variation in the target values can be explained by the models predictions.\n",
+ "- 1: The prediction perfectly aligns with the target values.\n",
+ "\n",
+ "
\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5938cb96",
+ "metadata": {},
+ "source": [
"\n",
"**h)** What is an advantage of the R2 score over the MSE?\n"
]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a73b6cd3",
+ "metadata": {},
+ "source": [
+ "
\n",
+ "\n",
+ "- It is invariant under scaling of the target values\n",
+ "- It has a limited range of values with a particular meaning.\n",
+ "\n",
+ "
+
+