This week you will be deriving the analytical expressions for linear regression, building up the model from scratch. This will include taking several derivatives of products of vectors and matrices. Such derivatives are central to the optimization of many machine learning models. Although we will often use automatic differentiation in actual calculations, to be able to have analytical expressions is extremely helpful in case we have simpler derivatives as well as when we analyze various properties (like second derivatives) of the chosen cost functions.
Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. You will find useful the notes from week 35 on derivatives of vectors and matrices. See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf
Learning goals¶
After completing these exercises, you will know how to
- Take the derivatives of simple products between vectors and matrices
- Implement OLS using the analytical expressions
- Create a feature matrix from a set of data
- Create a feature matrix for a polynomial model
- Evaluate the MSE score of various model on training and test data, and comparing their performance
Deliverables¶
Complete the following exercises while working in a jupyter notebook. Then, in canvas, include
- The jupyter notebook with the exercises completed
- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)
How to take derivatives of Matrix-Vector expressions¶
In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.
As an example, consider the function
$$ f(\boldsymbol{x}) =\boldsymbol{A}\boldsymbol{x}, $$
which reads for a specific component $f_i$ (we define the matrix $\boldsymbol{A}$ to have dimension $n\times n$ and the vector $\boldsymbol{x}$ to have length $n$)
$$ f_i =\sum_{j=0}^{n-1}a_{ij}x_j, $$
which leads to
$$ \frac{\partial f_i}{\partial x_j}= a_{ij}, $$
and written out in terms of the vector $\boldsymbol{x}$ we have
$$ \frac{\partial f(\boldsymbol{x})}{\partial \boldsymbol{x}}= \boldsymbol{A}. $$
Exercise 1 - Finding the derivative of Matrix-Vector expressions¶
a) Consider the expression
$$ \frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}}, $$
Where $\boldsymbol{a}$ and $\boldsymbol{x}$ are column-vectors with length $n$.
What is the shape of the expression we are taking the derivative of?
What is the shape of the thing we are taking the derivative with respect to?
What is the shape of the result of the expression?
- We're taking the derivative of a expression of shape $1\times1$, as the product of row-vector $a^T$ and column-vector $x$ is of shape $1\times 1$.
- We're taking the derivative with respect to a column vector of length n.
- The resulting shape is a row vector of length n.
b) Show that
$$ \frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T, $$
We define $\alpha \equiv a^T x = \sum_j a_j x_j$. Then
$$ \frac{\partial \alpha}{\partial x} = \frac{\partial}{\partial x_j} (a_j x_j) = a_j = a^T $$
c) Show that
$$ \frac{\partial (\boldsymbol{a}^T\boldsymbol{A}\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{a}^T(\boldsymbol{A}+\boldsymbol{A}^T), $$
If we define $\phi \equiv a^T A a$ which is evidently a scalar (thus $\phi = \phi^T$), we can write it as: $$ \phi = \sum_{i, j} a_i A_{ij} a_j $$ Using the scalar property we can rewrite the problem as $$ \frac{\partial \phi}{\partial a} = \frac{\partial}{\partial a} a^T A a = \frac{\partial \phi^T}{\partial a} = \frac{\partial}{\partial a}a^T A^T a $$ Now evaluating $$ \frac{\partial \phi}{\partial a_k} = \sum_{i,j} (\frac{\partial a_i}{\partial a_k} A_{ij} a_j + a_j A_{ij} \frac{\partial a_j}{\partial a_k}) = \sum_{i,j} (\delta_{ik} A_{ij} a_j + a_j A_{ij} \delta_{jk}) = \sum_{j} A_{kj} a_j + \sum_{i} A_{ik} a_i = a^T (A + A^T) $$ with the Kronecker-Delta $\delta_{xy} = \begin{cases} 0 & x \neq y \\ 1 & x = y \end{cases}$. As the derivative of two components can be written as $\frac{\partial a_x}{\partial a_y} = \delta_{xy}$.
Exercise 2 - Deriving the expression for OLS¶
The ordinary least squares method finds the parameters $\boldsymbol{\theta}$ which minimizes the squared error between our model $\boldsymbol{X\theta}$ and the true values $\boldsymbol{y}$.
To find the parameters $\boldsymbol{\theta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\boldsymbol{\theta}$, and set it equal to 0.
a) Very briefly explain why the approach above finds the parameters $\boldsymbol{\theta}$ which minimizes this error.
$$ \vert\vert\boldsymbol{y} - \boldsymbol{X\theta}\vert\vert^2 $$
which we can rewrite in matrix-vector form as
$$ \left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) $$. If we take the derivative and set it to 0 we find extrema in the squared error term. Since the squared error is positivly definit this point will be the minimum. We minimized the squared error.
b) If $\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\boldsymbol{\theta}$? (Hint: Don't compute any derivatives, but solve $\boldsymbol{X\theta}=\boldsymbol{y}$ for $\boldsymbol{\theta}$)
$$ \theta = X^{-1} y $$ since then $$X \theta = X X^{-1} y = y$$
c) Show that
$$ \frac{\partial \left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)}{\partial \boldsymbol{s}} = -2\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\boldsymbol{A}, $$
$$ \begin{aligned} \frac{\partial}{\partial s} (x-As)^T(x-As) &= \frac{\partial (x-As)^T}{\partial s} (x-As) + (x-As)^T \frac{\partial (x-As)}{\partial s} \\ &= - A^T (x-As) - (x-As)^T A \\ &= -2 (x-As)^T A \end{aligned} $$ We can make the last step due to the fact, that $(x-As)^T(x-As) \equiv \gamma$ is a scalar and thus $\gamma = \gamma^T$. Therefore the term for the derivative of $\gamma^T$ has to be equivalent and we can derive the last equivalence.
d) Using the expression from c), but substituting back in $\boldsymbol{\theta}$, $\boldsymbol{y}$ and $\boldsymbol{X}$, find the expression for the optimal parameters $\boldsymbol{\theta}$ in the case that $\boldsymbol{X}$ is not invertible, but $\boldsymbol{X^T X}$ is, which is most often the case.
$$ \boldsymbol{\hat{\theta}_{OLS}} = (X^T X)^{-1} X^T y $$
since then $X\theta = X (X^TX)^{-1} X^T y = (X X^{-1}) ((X^T)^{-1} X^T) y = y$
Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression¶
With the expression for $\boldsymbol{\hat{\theta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\boldsymbol{X}$.
In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, with the montly income and the number of children as columns.
We typically also include an intercept in our models. The intercept is a value that is added to our prediction regardless of the value of the other features. The intercept tries to account for constant effects in our data that are not dependant on anything else. In our current example, the intercept could account for living expenses which are typical regardless of income or childcare expenses.
We calculate the optimal intercept by including a feature with the constant value of 1 in our model, which is then multplied by some parameter $\theta_0$ from the OLS method into the optimal intercept value (which will be $\theta_0$). In practice, we include the intercept in our model by adding a column of ones to the start of our feature matrix.
import numpy as np
n = 20
income = np.array([116., 161., 167., 118., 172., 163., 179., 173., 162., 116., 101., 176., 178., 172., 143., 135., 160., 101., 149., 125.])
children = np.array([5, 3, 0, 4, 5, 3, 0, 4, 4, 3, 3, 5, 1, 0, 2, 3, 2, 1, 5, 4])
spending = np.array([152., 141., 102., 136., 161., 129., 99., 159., 160., 107., 98., 164., 121., 93., 112., 127., 117., 69., 156., 131.])
a) Create a feature matrix $\boldsymbol{X}$ for the features income and children, including an intercept column of ones at the start.
X = np.stack((np.ones(n), income, children)).T
display(X)
array([[ 1., 116., 5.],
[ 1., 161., 3.],
[ 1., 167., 0.],
[ 1., 118., 4.],
[ 1., 172., 5.],
[ 1., 163., 3.],
[ 1., 179., 0.],
[ 1., 173., 4.],
[ 1., 162., 4.],
[ 1., 116., 3.],
[ 1., 101., 3.],
[ 1., 176., 5.],
[ 1., 178., 1.],
[ 1., 172., 0.],
[ 1., 143., 2.],
[ 1., 135., 3.],
[ 1., 160., 2.],
[ 1., 101., 1.],
[ 1., 149., 5.],
[ 1., 125., 4.]])
b) Use the expression from 3d) to find the optimal parameters $\boldsymbol{\hat{\beta}_{OLS}}$ for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot.
def OLS_parameters(X, y):
return np.linalg.inv(X.T @ X) @ X.T @ y
beta = OLS_parameters(X, spending)
display(beta)
array([ 9.12808583, 0.5119025 , 14.60743095])
Exercise 4 - Fitting a polynomial¶
In this course, we typically do linear regression using polynomials, though in real world applications it is also very common to make linear models based on measured features like you did in the previous exercise.
When fitting a polynomial with linear regression, we make each polynomial degree($x, x^2, x^3, ..., x^p$) its own feature.
n = 100
x = np.linspace(-3, 3, n)
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)
a) Create a feature matrix $\boldsymbol{X}$ for the features $x, x^2, x^3, x^4, x^5$, including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks.
def polynomial_features(x, p):
n = len(x)
X = np.power(x[:, np.newaxis], np.arange(p + 1))
return X
X = polynomial_features(x, 5)
display(X)
array([[ 1.00000000e+00, -3.00000000e+00, 9.00000000e+00,
-2.70000000e+01, 8.10000000e+01, -2.43000000e+02],
[ 1.00000000e+00, -2.93939394e+00, 8.64003673e+00,
-2.53964716e+01, 7.46502347e+01, -2.19426447e+02],
[ 1.00000000e+00, -2.87878788e+00, 8.28741965e+00,
-2.38577232e+01, 6.86813245e+01, -1.97718964e+02],
[ 1.00000000e+00, -2.81818182e+00, 7.94214876e+00,
-2.23824192e+01, 6.30777269e+01, -1.77764503e+02],
[ 1.00000000e+00, -2.75757576e+00, 7.60422406e+00,
-2.09692239e+01, 5.78242235e+01, -1.59454677e+02],
[ 1.00000000e+00, -2.69696970e+00, 7.27364555e+00,
-1.96168016e+01, 5.29059195e+01, -1.42685662e+02],
[ 1.00000000e+00, -2.63636364e+00, 6.95041322e+00,
-1.83238167e+01, 4.83082440e+01, -1.27358098e+02],
[ 1.00000000e+00, -2.57575758e+00, 6.63452709e+00,
-1.70889334e+01, 4.40169497e+01, -1.13376992e+02],
[ 1.00000000e+00, -2.51515152e+00, 6.32598714e+00,
-1.59108162e+01, 4.00181133e+01, -1.00651618e+02],
[ 1.00000000e+00, -2.45454545e+00, 6.02479339e+00,
-1.47881292e+01, 3.62981354e+01, -8.90954232e+01],
[ 1.00000000e+00, -2.39393939e+00, 5.73094582e+00,
-1.37195370e+01, 3.28437400e+01, -7.86259231e+01],
[ 1.00000000e+00, -2.33333333e+00, 5.44444444e+00,
-1.27037037e+01, 2.96419753e+01, -6.91646091e+01],
[ 1.00000000e+00, -2.27272727e+00, 5.16528926e+00,
-1.17392938e+01, 2.66802131e+01, -6.06368480e+01],
[ 1.00000000e+00, -2.21212121e+00, 4.89348026e+00,
-1.08249715e+01, 2.39461490e+01, -5.29717842e+01],
[ 1.00000000e+00, -2.15151515e+00, 4.62901745e+00,
-9.95940117e+00, 2.14278025e+01, -4.61022418e+01],
[ 1.00000000e+00, -2.09090909e+00, 4.37190083e+00,
-9.14124718e+00, 1.91135168e+01, -3.99646261e+01],
[ 1.00000000e+00, -2.03030303e+00, 4.12213039e+00,
-8.36917383e+00, 1.69919590e+01, -3.44988258e+01],
[ 1.00000000e+00, -1.96969697e+00, 3.87970615e+00,
-7.64184545e+00, 1.50521198e+01, -2.96481148e+01],
[ 1.00000000e+00, -1.90909091e+00, 3.64462810e+00,
-6.95792637e+00, 1.32833140e+01, -2.53590540e+01],
[ 1.00000000e+00, -1.84848485e+00, 3.41689624e+00,
-6.31608092e+00, 1.16751799e+01, -2.15813931e+01],
[ 1.00000000e+00, -1.78787879e+00, 3.19651056e+00,
-5.71497343e+00, 1.02176798e+01, -1.82679729e+01],
[ 1.00000000e+00, -1.72727273e+00, 2.98347107e+00,
-5.15326822e+00, 8.90109965e+00, -1.53746267e+01],
[ 1.00000000e+00, -1.66666667e+00, 2.77777778e+00,
-4.62962963e+00, 7.71604938e+00, -1.28600823e+01],
[ 1.00000000e+00, -1.60606061e+00, 2.57943067e+00,
-4.14272199e+00, 6.65346258e+00, -1.06858641e+01],
[ 1.00000000e+00, -1.54545455e+00, 2.38842975e+00,
-3.69120962e+00, 5.70459668e+00, -8.81619487e+00],
[ 1.00000000e+00, -1.48484848e+00, 2.20477502e+00,
-3.27375685e+00, 4.86103290e+00, -7.21789734e+00],
[ 1.00000000e+00, -1.42424242e+00, 2.02846648e+00,
-2.88902802e+00, 4.11467627e+00, -5.86029651e+00],
[ 1.00000000e+00, -1.36363636e+00, 1.85950413e+00,
-2.53568745e+00, 3.45775562e+00, -4.71512130e+00],
[ 1.00000000e+00, -1.30303030e+00, 1.69788797e+00,
-2.21239948e+00, 2.88282356e+00, -3.75640646e+00],
[ 1.00000000e+00, -1.24242424e+00, 1.54361800e+00,
-1.91782842e+00, 2.38275652e+00, -2.96039447e+00],
[ 1.00000000e+00, -1.18181818e+00, 1.39669421e+00,
-1.65063862e+00, 1.95075473e+00, -2.30543741e+00],
[ 1.00000000e+00, -1.12121212e+00, 1.25711662e+00,
-1.40949439e+00, 1.58034220e+00, -1.77189883e+00],
[ 1.00000000e+00, -1.06060606e+00, 1.12488522e+00,
-1.19306008e+00, 1.26536675e+00, -1.34205564e+00],
[ 1.00000000e+00, -1.00000000e+00, 1.00000000e+00,
-1.00000000e+00, 1.00000000e+00, -1.00000000e+00],
[ 1.00000000e+00, -9.39393939e-01, 8.82460973e-01,
-8.28978490e-01, 7.78737370e-01, -7.31541165e-01],
[ 1.00000000e+00, -8.78787879e-01, 7.72268136e-01,
-6.78659877e-01, 5.96398074e-01, -5.24107398e-01],
[ 1.00000000e+00, -8.18181818e-01, 6.69421488e-01,
-5.47708490e-01, 4.48125128e-01, -3.66647832e-01],
[ 1.00000000e+00, -7.57575758e-01, 5.73921028e-01,
-4.34788658e-01, 3.29385347e-01, -2.49534354e-01],
[ 1.00000000e+00, -6.96969697e-01, 4.85766758e-01,
-3.38564710e-01, 2.35969344e-01, -1.64463482e-01],
[ 1.00000000e+00, -6.36363636e-01, 4.04958678e-01,
-2.57700977e-01, 1.63991531e-01, -1.04358247e-01],
[ 1.00000000e+00, -5.75757576e-01, 3.31496786e-01,
-1.90861786e-01, 1.09890119e-01, -6.32700686e-02],
[ 1.00000000e+00, -5.15151515e-01, 2.65381084e-01,
-1.36711467e-01, 7.04271195e-02, -3.62806373e-02],
[ 1.00000000e+00, -4.54545455e-01, 2.06611570e-01,
-9.39143501e-02, 4.26883410e-02, -1.94037913e-02],
[ 1.00000000e+00, -3.93939394e-01, 1.55188246e-01,
-6.11347636e-02, 2.40833917e-02, -9.48739674e-03],
[ 1.00000000e+00, -3.33333333e-01, 1.11111111e-01,
-3.70370370e-02, 1.23456790e-02, -4.11522634e-03],
[ 1.00000000e+00, -2.72727273e-01, 7.43801653e-02,
-2.02854996e-02, 5.53240899e-03, -1.50883882e-03],
[ 1.00000000e+00, -2.12121212e-01, 4.49954086e-02,
-9.54448062e-03, 2.02458680e-03, -4.29457806e-04],
[ 1.00000000e+00, -1.51515152e-01, 2.29568411e-02,
-3.47830926e-03, 5.27016555e-04, -7.98509932e-05],
[ 1.00000000e+00, -9.09090909e-02, 8.26446281e-03,
-7.51314801e-04, 6.83013455e-05, -6.20921323e-06],
[ 1.00000000e+00, -3.03030303e-02, 9.18273646e-04,
-2.78264741e-05, 8.43226488e-07, -2.55523178e-08],
[ 1.00000000e+00, 3.03030303e-02, 9.18273646e-04,
2.78264741e-05, 8.43226488e-07, 2.55523178e-08],
[ 1.00000000e+00, 9.09090909e-02, 8.26446281e-03,
7.51314801e-04, 6.83013455e-05, 6.20921323e-06],
[ 1.00000000e+00, 1.51515152e-01, 2.29568411e-02,
3.47830926e-03, 5.27016555e-04, 7.98509932e-05],
[ 1.00000000e+00, 2.12121212e-01, 4.49954086e-02,
9.54448062e-03, 2.02458680e-03, 4.29457806e-04],
[ 1.00000000e+00, 2.72727273e-01, 7.43801653e-02,
2.02854996e-02, 5.53240899e-03, 1.50883882e-03],
[ 1.00000000e+00, 3.33333333e-01, 1.11111111e-01,
3.70370370e-02, 1.23456790e-02, 4.11522634e-03],
[ 1.00000000e+00, 3.93939394e-01, 1.55188246e-01,
6.11347636e-02, 2.40833917e-02, 9.48739674e-03],
[ 1.00000000e+00, 4.54545455e-01, 2.06611570e-01,
9.39143501e-02, 4.26883410e-02, 1.94037913e-02],
[ 1.00000000e+00, 5.15151515e-01, 2.65381084e-01,
1.36711467e-01, 7.04271195e-02, 3.62806373e-02],
[ 1.00000000e+00, 5.75757576e-01, 3.31496786e-01,
1.90861786e-01, 1.09890119e-01, 6.32700686e-02],
[ 1.00000000e+00, 6.36363636e-01, 4.04958678e-01,
2.57700977e-01, 1.63991531e-01, 1.04358247e-01],
[ 1.00000000e+00, 6.96969697e-01, 4.85766758e-01,
3.38564710e-01, 2.35969344e-01, 1.64463482e-01],
[ 1.00000000e+00, 7.57575758e-01, 5.73921028e-01,
4.34788658e-01, 3.29385347e-01, 2.49534354e-01],
[ 1.00000000e+00, 8.18181818e-01, 6.69421488e-01,
5.47708490e-01, 4.48125128e-01, 3.66647832e-01],
[ 1.00000000e+00, 8.78787879e-01, 7.72268136e-01,
6.78659877e-01, 5.96398074e-01, 5.24107398e-01],
[ 1.00000000e+00, 9.39393939e-01, 8.82460973e-01,
8.28978490e-01, 7.78737370e-01, 7.31541165e-01],
[ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00,
1.00000000e+00, 1.00000000e+00, 1.00000000e+00],
[ 1.00000000e+00, 1.06060606e+00, 1.12488522e+00,
1.19306008e+00, 1.26536675e+00, 1.34205564e+00],
[ 1.00000000e+00, 1.12121212e+00, 1.25711662e+00,
1.40949439e+00, 1.58034220e+00, 1.77189883e+00],
[ 1.00000000e+00, 1.18181818e+00, 1.39669421e+00,
1.65063862e+00, 1.95075473e+00, 2.30543741e+00],
[ 1.00000000e+00, 1.24242424e+00, 1.54361800e+00,
1.91782842e+00, 2.38275652e+00, 2.96039447e+00],
[ 1.00000000e+00, 1.30303030e+00, 1.69788797e+00,
2.21239948e+00, 2.88282356e+00, 3.75640646e+00],
[ 1.00000000e+00, 1.36363636e+00, 1.85950413e+00,
2.53568745e+00, 3.45775562e+00, 4.71512130e+00],
[ 1.00000000e+00, 1.42424242e+00, 2.02846648e+00,
2.88902802e+00, 4.11467627e+00, 5.86029651e+00],
[ 1.00000000e+00, 1.48484848e+00, 2.20477502e+00,
3.27375685e+00, 4.86103290e+00, 7.21789734e+00],
[ 1.00000000e+00, 1.54545455e+00, 2.38842975e+00,
3.69120962e+00, 5.70459668e+00, 8.81619487e+00],
[ 1.00000000e+00, 1.60606061e+00, 2.57943067e+00,
4.14272199e+00, 6.65346258e+00, 1.06858641e+01],
[ 1.00000000e+00, 1.66666667e+00, 2.77777778e+00,
4.62962963e+00, 7.71604938e+00, 1.28600823e+01],
[ 1.00000000e+00, 1.72727273e+00, 2.98347107e+00,
5.15326822e+00, 8.90109965e+00, 1.53746267e+01],
[ 1.00000000e+00, 1.78787879e+00, 3.19651056e+00,
5.71497343e+00, 1.02176798e+01, 1.82679729e+01],
[ 1.00000000e+00, 1.84848485e+00, 3.41689624e+00,
6.31608092e+00, 1.16751799e+01, 2.15813931e+01],
[ 1.00000000e+00, 1.90909091e+00, 3.64462810e+00,
6.95792637e+00, 1.32833140e+01, 2.53590540e+01],
[ 1.00000000e+00, 1.96969697e+00, 3.87970615e+00,
7.64184545e+00, 1.50521198e+01, 2.96481148e+01],
[ 1.00000000e+00, 2.03030303e+00, 4.12213039e+00,
8.36917383e+00, 1.69919590e+01, 3.44988258e+01],
[ 1.00000000e+00, 2.09090909e+00, 4.37190083e+00,
9.14124718e+00, 1.91135168e+01, 3.99646261e+01],
[ 1.00000000e+00, 2.15151515e+00, 4.62901745e+00,
9.95940117e+00, 2.14278025e+01, 4.61022418e+01],
[ 1.00000000e+00, 2.21212121e+00, 4.89348026e+00,
1.08249715e+01, 2.39461490e+01, 5.29717842e+01],
[ 1.00000000e+00, 2.27272727e+00, 5.16528926e+00,
1.17392938e+01, 2.66802131e+01, 6.06368480e+01],
[ 1.00000000e+00, 2.33333333e+00, 5.44444444e+00,
1.27037037e+01, 2.96419753e+01, 6.91646091e+01],
[ 1.00000000e+00, 2.39393939e+00, 5.73094582e+00,
1.37195370e+01, 3.28437400e+01, 7.86259231e+01],
[ 1.00000000e+00, 2.45454545e+00, 6.02479339e+00,
1.47881292e+01, 3.62981354e+01, 8.90954232e+01],
[ 1.00000000e+00, 2.51515152e+00, 6.32598714e+00,
1.59108162e+01, 4.00181133e+01, 1.00651618e+02],
[ 1.00000000e+00, 2.57575758e+00, 6.63452709e+00,
1.70889334e+01, 4.40169497e+01, 1.13376992e+02],
[ 1.00000000e+00, 2.63636364e+00, 6.95041322e+00,
1.83238167e+01, 4.83082440e+01, 1.27358098e+02],
[ 1.00000000e+00, 2.69696970e+00, 7.27364555e+00,
1.96168016e+01, 5.29059195e+01, 1.42685662e+02],
[ 1.00000000e+00, 2.75757576e+00, 7.60422406e+00,
2.09692239e+01, 5.78242235e+01, 1.59454677e+02],
[ 1.00000000e+00, 2.81818182e+00, 7.94214876e+00,
2.23824192e+01, 6.30777269e+01, 1.77764503e+02],
[ 1.00000000e+00, 2.87878788e+00, 8.28741965e+00,
2.38577232e+01, 6.86813245e+01, 1.97718964e+02],
[ 1.00000000e+00, 2.93939394e+00, 8.64003673e+00,
2.53964716e+01, 7.46502347e+01, 2.19426447e+02],
[ 1.00000000e+00, 3.00000000e+00, 9.00000000e+00,
2.70000000e+01, 8.10000000e+01, 2.43000000e+02]])
b) Use the expression from 3d) to find the optimal parameters $\boldsymbol{\hat{\beta}_{OLS}}$ for predicting $\boldsymbol{y}$ based on these features. If you have done everything right so far, this code will not need changing.
beta = OLS_parameters(X, y)
display(beta)
array([ 0.92452576, 0.27464654, -0.02326439, 0.05342623, -0.0034652 ,
-0.0087781 ])
c) Like in exercise 4 last week, split your feature matrix and target data into a training split and test split.
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)
d) Train your model on the training data(find the parameters which best fit) and compute the MSE on both the training and test data.
beta = OLS_parameters(polynomial_features(x_train, 5), y_train)
from sklearn.metrics import mean_squared_error
def evaluate_model(beta, X, y):
y_pred = X @ beta
mse = mean_squared_error(y, y_pred)
return mse
mse_train = evaluate_model(beta, polynomial_features(x_train, 5), y_train)
mse_test = evaluate_model(beta, polynomial_features(x_test, 5), y_test)
print(f"Training MSE: {mse_train:.4f}")
print(f"Testing MSE: {mse_test:.4f}")
Training MSE: 0.0133 Testing MSE: 0.0162
e) Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of Hastie et al. Feel free to read the discussions leading to figure 2.11 of Hastie et al.
import pandas as pd
results = []
for degree in range(2, 11):
beta = OLS_parameters(polynomial_features(x_train, degree), y_train)
mse_train = evaluate_model(beta, polynomial_features(x_train, degree), y_train)
mse_test = evaluate_model(beta, polynomial_features(x_test, degree), y_test)
results.append({"degree": degree, "mse_train": mse_train, "mse_test": mse_test})
df_results = pd.DataFrame(results)
display(df_results)
| degree | mse_train | mse_test | |
|---|---|---|---|
| 0 | 2 | 0.051641 | 0.016256 |
| 1 | 3 | 0.022066 | 0.020394 |
| 2 | 4 | 0.020904 | 0.023035 |
| 3 | 5 | 0.013347 | 0.016155 |
| 4 | 6 | 0.009560 | 0.008152 |
| 5 | 7 | 0.005748 | 0.006163 |
| 6 | 8 | 0.001083 | 0.000885 |
| 7 | 9 | 0.000958 | 0.001134 |
| 8 | 10 | 0.000079 | 0.000099 |
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.plot(df_results["degree"], df_results["mse_train"], label="Training MSE")
ax.plot(df_results["degree"], df_results["mse_test"], label="Testing MSE")
ax.set_xlabel("Polynomial Degree (Model Complexity)")
ax.set_ylabel("Mean Squared Error w.r.t. True Values")
ax.legend()
<matplotlib.legend.Legend at 0x7fa9522bfe00>
f) Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?
Exercise 5 - Comparing your code with sklearn¶
When implementing different algorithms for the first time, it can be helpful to double check your results with established implementations before you go on to add more complexity.
a) Make sure your polynomial_features function creates the same feature matrix as sklearns PolynomialFeatures.
(https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)
from sklearn.preprocessing import PolynomialFeatures
poly_features_sklearn = PolynomialFeatures(degree=5, include_bias=True).fit_transform(x.reshape(-1, 1))
poly_features_own = polynomial_features(x, 5)
np.allclose(poly_features_sklearn, poly_features_own) # If this is true, our output is identical (within numerical precision)
True
b) Make sure your OLS_parameters function computes the same parameters as sklearns LinearRegression with fit_intercept set to False, since the intercept is included in the feature matrix. Use your_model_object.coef_ to extract the computed parameters.
(https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)
from sklearn.linear_model import LinearRegression
model = LinearRegression(fit_intercept=False)
model.fit(poly_features_sklearn, y)
beta_sklearn = model.coef_
beta_own = OLS_parameters(poly_features_own, y)
np.allclose(beta_sklearn, beta_own) # If this is true, our coefficients are identical (within numerical precision)
True