b) Compute the mean square error for the line model and for the second degree polynomial model.
import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures # use the fit_transform method of the created object!
-from sklearn.linear_model import LinearRegression
-from sklearn.metrics import mean_squared_error
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.preprocessing import PolynomialFeatures # use the fit_transform method of the created object!
+from sklearn.linear_model import LinearRegression
+from sklearn.metrics import mean_squared_error
Hopefully your model fit the data quite well, but to know how well the model actually generalizes to unseen data, which is most often what we care about, we need to split our data into training and testing data.
from sklearn.model_selection import train_test_split
+from sklearn.model_selection import train_test_split
We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter \(\theta_0\) from the OLS method into the optimal intercept value (which will be \(\theta_0\)). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.
import numpy as np
+import numpy as np
b) Use the expression from 3d) to find the optimal parameters \(\boldsymbol{\hat{\beta}_{OLS}}\) for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot.
def OLS_parameters(X, y):
+def OLS_parameters(X, y):
return ...
#beta = OLS_parameters(X, y)
@@ -567,7 +567,7 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
a) Create a feature matrix \(\boldsymbol{X}\) for the features \(x, x^2, x^3, x^4, x^5\), including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks.
-def polynomial_features(x, p):
+def polynomial_features(x, p):
n = len(x)
X = np.zeros((n, p + 1))
#X[:, 0] = ...
@@ -591,7 +591,7 @@ f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
c) Like in exercise 4 last week, split your feature matrix and target data into a training split and test split.
-from sklearn.model_selection import train_test_split
+from sklearn.model_selection import train_test_split
#X_train, X_test, y_train, y_test = ...
diff --git a/doc/LectureNotes/_build/html/exercisesweek36.html b/doc/LectureNotes/_build/html/exercisesweek36.html
index 751e6cabd..533249336 100644
--- a/doc/LectureNotes/_build/html/exercisesweek36.html
+++ b/doc/LectureNotes/_build/html/exercisesweek36.html
@@ -28,7 +28,7 @@
-
+
@@ -465,10 +465,10 @@ defining a new cost function to be optimized, that is
Exercise 3 - Scaling data#
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn.preprocessing import StandardScaler
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn.preprocessing import StandardScaler
@@ -485,7 +485,7 @@ defining a new cost function to be optimized, that is
a) Adapt your function from last week to only include the intercept column if the boolean argument intercept is set to true.
-def polynomial_features(x, p, intercept=False):
+def polynomial_features(x, p, intercept=False):
n = len(x)
X = np.zeros((n, p + 1))
#X[:, 0] = ...
@@ -498,7 +498,7 @@ defining a new cost function to be optimized, that is
-def polynomial_features(x, p, intercept=False):
+def polynomial_features(x, p, intercept=False):
n = len(x)
X = np.zeros((n, p))
X[:, 0] = x[:]
@@ -544,7 +544,7 @@ defining a new cost function to be optimized, that is
a) Implement a function for computing the optimal Ridge parameters using the expression from 2a).
-def Ridge_parameters(X, y):
+def Ridge_parameters(X, y):
# Assumes X is scaled and has no intercept column
return np.linalg.inv(X.T @ X) @ X.T @ y
diff --git a/doc/LectureNotes/_build/html/exercisesweek37.html b/doc/LectureNotes/_build/html/exercisesweek37.html
index 541dd9132..57fd8df99 100644
--- a/doc/LectureNotes/_build/html/exercisesweek37.html
+++ b/doc/LectureNotes/_build/html/exercisesweek37.html
@@ -28,7 +28,7 @@
-
+
diff --git a/doc/LectureNotes/_build/html/exercisesweek38.html b/doc/LectureNotes/_build/html/exercisesweek38.html
index d6dbb2009..4afe575f5 100644
--- a/doc/LectureNotes/_build/html/exercisesweek38.html
+++ b/doc/LectureNotes/_build/html/exercisesweek38.html
@@ -28,7 +28,7 @@
-
+
@@ -385,12 +385,20 @@ document.write(`
@@ -409,17 +417,31 @@ document.write(`
Resampling and the Bias-Variance Trade-off#
-
-Learning goals#
+Learning goals#
After completing these exercises, you will know how to
Derive expectation and variances values related to linear regression
Compute expectation and variances values related to linear regression
Compute and evaluate the trade-off between bias and variance of a model
-This week deals with various mean values and variances in linear regression methods (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer). The exercises are also a part of project 1 and can be reused in the theory part of the project.
+
+
+Deliverables#
+Complete the following exercises while working in a jupyter notebook. Then, in canvas, include
+
+The jupyter notebook with the exercises completed
An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)
+
+
+
+Use the books!#
+This week deals with various mean values and variances in linear regression methods (here it may be useful to look up chapter 3, equation (3.8) of Trevor Hastie, Robert Tibshirani, Jerome H. Friedman, The Elements of Statistical Learning, Springer).
For more discussions on Ridge regression and calculation of expectation values, Wessel van Wieringen’s article is highly recommended.
+The exercises this week are also a part of project 1 and can be reused in the theory part of the project.
+
+Definitions#
We assume that there exists a continuous function \(f(\boldsymbol{x})\) and a normal distributed error \(\boldsymbol{\varepsilon}\sim N(0, \sigma^2)\) which describes our data
\[
@@ -432,6 +454,7 @@ document.write(`
\]
We therefore get that our data \(\boldsymbol{y}\) has an expectation value \(\boldsymbol{X}\boldsymbol{\beta}\) and variance \(\sigma^2\), that is \(\boldsymbol{y}\) follows a normal distribution with mean value \(\boldsymbol{X}\boldsymbol{\beta}\) and variance \(\sigma^2\).
+
Exercise 1: Expectation values for ordinary least squares expressions#
a) With the expressions for the optimal parameters \(\boldsymbol{\hat{\beta}_{OLS}}\) show that
@@ -471,7 +494,7 @@ A given parameter \({\boldsymbol{\hat
\[
C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2=\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]
\]
-a) Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
+
a) Show that you can rewrite this in terms of a term which contains the variance of the model itself (the so-called variance term), a
term which measures the deviation from the true data and the mean value of the model (the bias term) and finally the variance of the noise Note that in order to be able to evaluate the bias them, you will need to approximate the function \(f\) with the model \({\bf y}\).
show that
@@ -496,7 +519,7 @@ show that
a) Using the expression above, compute the mean squared error, bias and variance of the given data. Check that the sum of the bias and variance correctly gives (approximately) the mean squared error.
-import numpy as np
+import numpy as np
n = 100
bootstraps = 1000
@@ -516,13 +539,15 @@ show that
d) Perform a bias-variance analysis of a polynomial OLS model fit to a one-dimensional function by computing and plotting the bias and variances values as a function of the polynomial degree of your model.
-import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.preprocessing import PolynomialFeatures # use the fit_transform method of the created object!
-from sklearn.linear_model import LinearRegression
-from sklearn.metrics import mean_squared_error
-from sklearn.model_selection import train_test_split
-from sklearn.utils import resample
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.preprocessing import (
+ PolynomialFeatures,
+) # use the fit_transform method of the created object!
+from sklearn.linear_model import LinearRegression
+from sklearn.metrics import mean_squared_error
+from sklearn.model_selection import train_test_split
+from sklearn.utils import resample
@@ -533,13 +558,13 @@ show that
bootstraps = 1000
x = np.linspace(-3, 3, n)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)
+y = np.exp(-(x**2)) + 1.5 * np.exp(-((x - 2) ** 2)) + np.random.normal(0, 0.1)
biases = []
variances = []
mses = []
-#for p in range(1, 5):
+# for p in range(1, 5):
# predictions = ...
# targets = ...
# for b in range(bootstraps):
@@ -547,9 +572,9 @@ show that
# X = ...
# X_train, X_test, y_train, y_test = ...
#
-# predictions[b, :] =
-# targets[b, :] =
-#
+# predictions[b, :] =
+# targets[b, :] =
+#
# biases.append(...)
# variances.append(...)
# mses.append(...)
@@ -560,6 +585,20 @@ show that
e) Discuss the bias-variance trade-off as function of your model complexity (the degree of the polynomial).
f) Compute and discuss the bias and variance as function of the number of data points (choose a suitable polynomial degree to show something interesiting).
+
+Exercise 5: Interpretation of scaling and metrics#
+In this course, we often ask you to scale data and compute various metrics. Although these practices are “standard” in the field, we will require you to demonstrate an understanding of why you need to scale data and use these metrics. Both so that you can make better arguements about your results, and so that you will hopefully make fewer mistakes.
+First, a few reminders: In this course you should always scale the columns of the feature matrix, and sometimes scale the target data, when it is worth the effort. By scaling, we mean subtracting the mean and dividing by the standard deviation, though there are many other ways to scale data. When scaling either the feature matrix or the target data, the intercept becomes a bit harder to implement and understand, so take care.
+Briefly answer the following:
+a) Why do we scale data?
+b) Why does the OLS method give practically equivelent models on scaled and unscaled data?
+c) Why does the Ridge method not give practically equivelent models on scaled and unscaled data? Why do we only consider the model on scaled data correct?
+d) Why do we say that the Ridge method gives a biased model?
+e) Is the MSE of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?
+f) Read about the R2 score, a metric we will ask you to use a lot later in the course. Is the R2 score of the OLS method affected by scaling of the feature matrix? Is it affected by scaling of the target data?
+g) Give interpretations of the following R2 scores: 0, 0.5, 1.
+h) What is an advantage of the R2 score over the MSE?
+