adding first two revised weekly assignments
This commit is contained in:
KarlHenrik
File diff suppressed because one or more lines are too long
@@ -0,0 +1,535 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b4005770",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Exercises week 35\n",
|
||||
"\n",
|
||||
"## Deriving and Implementing Ordinary Least Squares"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ca1b589",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This week you will be deriving the analytical expressions for linear regression, building up the model from scratch. This will include taking several derivatives of products of vectors and matrices. Such derivatives are central to the optimization of many machine learning models. Although we will often use automatic differentiation in actual calculations, to be able to have analytical expressions is extremely helpful in case we have simpler derivatives as well as when we analyze various properties (like second derivatives) of the chosen cost functions.\n",
|
||||
"\n",
|
||||
"Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. You will find useful the notes from week 35 on derivatives of vectors and matrices. See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "41e92bf9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Learning goals\n",
|
||||
"\n",
|
||||
"After completing these exercises, you will know how to\n",
|
||||
"- Take the derivatives of simple products between vectors and matrices\n",
|
||||
"- Implement OLS using the analytical expressions\n",
|
||||
"- Create a feature matrix from a set of data\n",
|
||||
"- Create a feature matrix for a polynomial model\n",
|
||||
"- Evaluate the MSE score of various model on training and test data, and comparing their performance\n",
|
||||
"\n",
|
||||
"### Deliverables\n",
|
||||
"\n",
|
||||
"Complete the following exercises while working in a jupyter notebook. Then, in canvas, include\n",
|
||||
"- The jupyter notebook with the exercises completed\n",
|
||||
"- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7a9209d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How to take derivatives of Matrix-Vector expressions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "45f3712e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.\n",
|
||||
"\n",
|
||||
"As an example, consider the function\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f(\\boldsymbol{x}) =\\boldsymbol{A}\\boldsymbol{x},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which reads for a specific component $f_i$ (we define the matrix $\\boldsymbol{A}$ to have dimension $n\\times n$ and the vector $\\boldsymbol{x}$ to have length $n$)\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f_i =\\sum_{j=0}^{n-1}a_{ij}x_j,\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which leads to\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial f_i}{\\partial x_j}= a_{ij},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"and written out in terms of the vector $\\boldsymbol{x}$ we have\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial f(\\boldsymbol{x})}{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5fa8a4e6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 1 - Finding the derivative of Matrix-Vector expressions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "df7a2270",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Consider the expression\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"Where $\\boldsymbol{a}$ and $\\boldsymbol{x}$ are column-vectors with length $n$.\n",
|
||||
"\n",
|
||||
"What is the *shape* of the expression we are taking the derivative of?\n",
|
||||
"\n",
|
||||
"What is the *shape* of the thing we are taking the derivative with respect to?\n",
|
||||
"\n",
|
||||
"What is the *shape* of the result of the expression?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c0396734",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dc39d541",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{A}\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{a}^T(\\boldsymbol{A}+\\boldsymbol{A}^T),\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "498d13ec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 2 - Deriving the expression for OLS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f3f771de",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n",
|
||||
"\n",
|
||||
"To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "49690237",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7cccc9d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We typically write the squared error as\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which we can rewrite in matrix-vector form as\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8fbecf74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f37af8f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial \\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)}{\\partial \\boldsymbol{s}} = -2\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\boldsymbol{A},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "869fca4d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "57ca3d74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5dc179f7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
|
||||
"\n",
|
||||
"In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, and in each column put a 1 for the intercept, the montly income and the number of children."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 59,
|
||||
"id": "e5ff2a69",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 60,
|
||||
"id": "a3cf2792",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 20\n",
|
||||
"income = np.array([116., 161., 167., 118., 172., 163., 179., 173., 162., 116., 101., 176., 178., 172., 143., 135., 160., 101., 149., 125.])\n",
|
||||
"children = np.array([5, 3, 0, 4, 5, 3, 0, 4, 4, 3, 3, 5, 1, 0, 2, 3, 2, 1, 5, 4])\n",
|
||||
"spending = np.array([152., 141., 102., 136., 161., 129., 99., 159., 160., 107., 98., 164., 121., 93., 112., 127., 117., 69., 156., 131.])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5da61481",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Create a feature matrix $\\boldsymbol{X}$ for the features income and children, including an intercept column of ones at the start."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 65,
|
||||
"id": "5ad87a65",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = np.zeros((n, 3))\n",
|
||||
"#X[:, 0] = ...\n",
|
||||
"#X[:, 1] = ...\n",
|
||||
"#X[:, 2] = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0ddfac2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 37,
|
||||
"id": "8f3f68aa",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def OLS_parameters(X, y):\n",
|
||||
" return ...\n",
|
||||
"\n",
|
||||
"#beta = OLS_parameters(X, y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cb6da80",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 4 - Fitting a polynomial"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "71015064",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this course, we typically do linear regression using polynomials, though in real world applications it is also very common to make linear models based on measured features like you did in the previous exercise.\n",
|
||||
"\n",
|
||||
"When fitting a polynomial with linear regression, we make each polynomial degree($x, x^2, x^3, ..., x^p$) its own feature."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 38,
|
||||
"id": "d7476c84",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 100\n",
|
||||
"x = np.linspace(-3, 3, n)\n",
|
||||
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8321451b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Create a feature matrix $\\boldsymbol{X}$ for the features $x, x^2, x^3, x^4, x^5$, including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 66,
|
||||
"id": "91496e40",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def polynomial_features(x, p):\n",
|
||||
" n = len(x)\n",
|
||||
" X = np.zeros((n, p + 1))\n",
|
||||
" #X[:, 0] = ...\n",
|
||||
" #X[:, 1] = ...\n",
|
||||
" #X[:, 2] = ...\n",
|
||||
" # could this be a loop?\n",
|
||||
"\n",
|
||||
"#X = polynomial_features(x, 5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b84b1e31",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting $\\boldsymbol{y}$ based on these features. If you have done everything right so far, this code will not need changing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"id": "034f502c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#beta = OLS_parameters(X, y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d703f788",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Like in exercise 4 last week, split your feature matrix and target data into a training split and test split."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 68,
|
||||
"id": "29171358",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"#X_train, X_test, y_train, y_test = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a0e3509f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**d)** Train your model on the training data(find the parameters which best fit) and compute the MSE on both the training and test data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 42,
|
||||
"id": "1e346f4c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Ellipsis"
|
||||
]
|
||||
},
|
||||
"execution_count": 42,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e431889",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**e)** Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf). Feel free to read the discussions leading to figure 2.11 of Hastie et al. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 43,
|
||||
"id": "ceb57457",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Ellipsis"
|
||||
]
|
||||
},
|
||||
"execution_count": 43,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5e5b5954",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**f)** Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ad2acfb9",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5994f0c5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 5 - Comparing your code with sklearn"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f595b7a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When implementing different algorithms for the first time, it can be helpful to double check your results with established implementations before you go on to add more complexity."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8ab310c1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Make sure your `polynomial_features` function creates the same feature matrix as sklearns PolynomialFeatures.\n",
|
||||
"\n",
|
||||
"(https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "85b964d1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73c32c52",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Make sure your `OLS_parameters` function computes the same parameters as sklearns LinearRegression with fit_intercept set to False, since the intercept is included in the feature matrix. Use `your_model_object.coef_` to extract the computed parameters.\n",
|
||||
"\n",
|
||||
"(https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "35b04126",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.13.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+137
-124
@@ -57,9 +57,7 @@
|
||||
<script async="async" src="_static/sphinx-thebe.js?v=c100c467"></script>
|
||||
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
|
||||
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
|
||||
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
|
||||
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||
<script>DOCUMENTATION_OPTIONS.pagename = 'exercisesweek34';</script>
|
||||
<script>DOCUMENTATION_OPTIONS.pagename = 'E1';</script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="Week 34: Introduction to the course, Logistics and Practicalities" href="week34.html" />
|
||||
@@ -225,6 +223,7 @@
|
||||
<ul class="current nav bd-sidenav">
|
||||
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
@@ -279,7 +278,7 @@
|
||||
|
||||
|
||||
|
||||
<li><a href="_sources/exercisesweek34.ipynb" target="_blank"
|
||||
<li><a href="_sources/E1.ipynb" target="_blank"
|
||||
class="btn btn-sm btn-download-source-button dropdown-item"
|
||||
title="Download source file"
|
||||
data-bs-placement="left" data-bs-toggle="tooltip"
|
||||
@@ -373,10 +372,16 @@ document.write(`
|
||||
</div>
|
||||
<nav aria-label="Page">
|
||||
<ul class="visible nav section-nav flex-column">
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercises">Exercises</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-setting-up-various-python-environments">Exercise 1: Setting up various Python environments</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn">Exercise 2: making your own data and exploring scikit-learn</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-split-data-in-test-and-training-data">Exercise 3: Split data in test and training data</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#coding-setup-and-linear-regression">Coding Setup and Linear Regression</a><ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-goals">Learning goals</a></li>
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deliverables">Deliverables</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-github-setup">Exercise 1 - Github Setup</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-setting-up-a-github-repository">Exercise 2 - Setting up a Github repository</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-setting-up-a-python-virtual-environment">Exercise 3 - Setting up a Python virtual environment</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-fitting-an-ols-model-to-data">Exercise 3 - Fitting an OLS model to data</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-the-train-test-split">Exercise 4 - The train-test split</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
</div>
|
||||
@@ -388,135 +393,137 @@ document.write(`
|
||||
<div id="searchbox"></div>
|
||||
<article class="bd-article">
|
||||
|
||||
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
|
||||
doconce format html exercisesweek34.do.txt -->
|
||||
<!-- dom:TITLE: Exercises week 34 --><section class="tex2jax_ignore mathjax_ignore" id="exercises-week-34">
|
||||
<section class="tex2jax_ignore mathjax_ignore" id="exercises-week-34">
|
||||
<h1>Exercises week 34<a class="headerlink" href="#exercises-week-34" title="Link to this heading">#</a></h1>
|
||||
<p><strong>FYS-STK3155/4155</strong></p>
|
||||
<p>Date: <strong>August 19-23, 2024</strong></p>
|
||||
<section id="exercises">
|
||||
<h2>Exercises<a class="headerlink" href="#exercises" title="Link to this heading">#</a></h2>
|
||||
<p>Here are three possible exercises for week 34</p>
|
||||
<section id="coding-setup-and-linear-regression">
|
||||
<h2>Coding Setup and Linear Regression<a class="headerlink" href="#coding-setup-and-linear-regression" title="Link to this heading">#</a></h2>
|
||||
<p>Welcome to FYS-STK3155/4155!</p>
|
||||
<p>In this first week will focus on getting you set up with the programs you are going to be using throughout this course. We expect that many of you will encounter some trouble with setting these programs up, as they can be extremely finnicky and prone to not working the same on all machines, so we strongly encourage you to not get discouraged, and to show up to the group-sessions where we can help you along. The group sessions are also the best place to find group partners for the projects and to be challenged on your understanding of the material, which are both essential to doing well in this course.</p>
|
||||
<p>If you are unable to complete this weeks exercises, don’t worry, this will likely be the most frustrating week for many of you. You have time to get back on track next week, especially if you come to the group-sessions!</p>
|
||||
<section id="learning-goals">
|
||||
<h3>Learning goals<a class="headerlink" href="#learning-goals" title="Link to this heading">#</a></h3>
|
||||
<p>After completing these exercises, you will know how to</p>
|
||||
<ul class="simple">
|
||||
<li><p>Create and use a Github repository</p></li>
|
||||
<li><p>Set up and use a virtual environment in Python</p></li>
|
||||
<li><p>Fit an OLS model to data using scikit-learn</p></li>
|
||||
<li><p>Fit a model on training data and evaluate it on test data</p></li>
|
||||
</ul>
|
||||
</section>
|
||||
<section id="exercise-1-setting-up-various-python-environments">
|
||||
<h2>Exercise 1: Setting up various Python environments<a class="headerlink" href="#exercise-1-setting-up-various-python-environments" title="Link to this heading">#</a></h2>
|
||||
<p>The first exercise here is of a mere technical art. We want you to have</p>
|
||||
<section id="deliverables">
|
||||
<h3>Deliverables<a class="headerlink" href="#deliverables" title="Link to this heading">#</a></h3>
|
||||
<p>Complete the following exercises while working in a jupyter notebook. Exercises 1,2 and 3 require no writing in the notebook. Then, in canvas, include</p>
|
||||
<ul class="simple">
|
||||
<li><p>git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo <a class="reference external" href="https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html">GitHub facilities</a>.</p></li>
|
||||
<li><p>Install various Python packages</p></li>
|
||||
<li><p>The jupyter notebook with the exercises completed</p></li>
|
||||
<li><p>An exported PDF of the notebook (<a class="reference external" href="https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook">https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook</a>)</p></li>
|
||||
<li><p>Optional: A link to your github repository, which must be set to public, include the notebook file, a README file, requirements file and gitignore file.</p></li>
|
||||
</ul>
|
||||
<p>We will make extensive use of Python as programming language and its
|
||||
myriad of available libraries. You will find
|
||||
IPython/Jupyter notebooks invaluable in your work. You can run <strong>R</strong>
|
||||
codes in the Jupyter/IPython notebooks, with the immediate benefit of
|
||||
visualizing your data. You can also use compiled languages like C++,
|
||||
Rust, Fortran etc if you prefer. The focus in these lectures will be
|
||||
on Python.</p>
|
||||
<p>If you have Python installed (we recommend Python3) and you feel
|
||||
pretty familiar with installing different packages, we recommend that
|
||||
you install the following Python packages via <strong>pip</strong> as</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow</p></li>
|
||||
</ol>
|
||||
<p>For <strong>Tensorflow</strong>, we recommend following the instructions in the text of
|
||||
<a class="reference external" href="http://shop.oreilly.com/product/0636920052289.do">Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O’Reilly</a></p>
|
||||
<p>We will come back to <strong>tensorflow</strong> later.</p>
|
||||
<p>For Python3, replace <strong>pip</strong> with <strong>pip3</strong>.</p>
|
||||
<p>For OSX users we recommend, after having installed Xcode, to
|
||||
install <strong>brew</strong>. Brew allows for a seamless installation of additional
|
||||
software via for example</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>brew install python3</p></li>
|
||||
</ol>
|
||||
<p>For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,
|
||||
you can use <strong>pip</strong> as well and simply install Python as</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p>sudo apt-get install python3 (or python for Python2.7)</p></li>
|
||||
</ol>
|
||||
<p>If you don’t want to perform these operations separately and venture
|
||||
into the hassle of exploring how to set up dependencies and paths, we
|
||||
recommend two widely used distrubutions which set up all relevant
|
||||
dependencies for Python, namely</p>
|
||||
<ul class="simple">
|
||||
<li><p><a class="reference external" href="https://docs.anaconda.com/">Anaconda</a>,</p></li>
|
||||
</ul>
|
||||
<p>which is an open source
|
||||
distribution of the Python and R programming languages for large-scale
|
||||
data processing, predictive analytics, and scientific computing, that
|
||||
aims to simplify package management and deployment. Package versions
|
||||
are managed by the package management system <strong>conda</strong>.</p>
|
||||
<ul class="simple">
|
||||
<li><p><a class="reference external" href="https://www.enthought.com/product/canopy/">Enthought canopy</a></p></li>
|
||||
</ul>
|
||||
<p>is a Python
|
||||
distribution for scientific and analytic computing distribution and
|
||||
analysis environment, available for free and under a commercial
|
||||
license.</p>
|
||||
<p>We recommend using <strong>Anaconda</strong> if you are not too familiar with setting paths in a terminal environment.</p>
|
||||
<p>We require you to deliver a jupyter notebook so that we can evaluate the results of your code without needing to download and run the code of every student, as well as to teach you to use this useful tool.</p>
|
||||
</section>
|
||||
<section id="exercise-2-making-your-own-data-and-exploring-scikit-learn">
|
||||
<h2>Exercise 2: making your own data and exploring scikit-learn<a class="headerlink" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn" title="Link to this heading">#</a></h2>
|
||||
<p>We will generate our own dataset for a function <span class="math notranslate nohighlight">\(y(x)\)</span> where <span class="math notranslate nohighlight">\(x \in [0,1]\)</span> and defined by random numbers computed with the uniform distribution. The function <span class="math notranslate nohighlight">\(y\)</span> is a quadratic polynomial in <span class="math notranslate nohighlight">\(x\)</span> with added stochastic noise according to the normal distribution <span class="math notranslate nohighlight">\(\cal {N}(0,1)\)</span>.
|
||||
The following simple Python instructions define our <span class="math notranslate nohighlight">\(x\)</span> and <span class="math notranslate nohighlight">\(y\)</span> values (with 100 data points).</p>
|
||||
</section>
|
||||
<section id="exercise-1-github-setup">
|
||||
<h2>Exercise 1 - Github Setup<a class="headerlink" href="#exercise-1-github-setup" title="Link to this heading">#</a></h2>
|
||||
<p>In this course, we require you to pay extra mind to the reproducibility of your results and the shareability of your code. The first step toward these goals is using a version control system like git and online repository like Github.</p>
|
||||
<p><strong>a)</strong> Download git if you don’t already have it on your machine, check with the terminal command ´git –version´ (<a class="reference external" href="https://git-scm.com/downloads">https://git-scm.com/downloads</a>).</p>
|
||||
<p><strong>b)</strong> Create a Github account(<a class="github reference external" href="https://github.com/"></a>), or log in to github with your UiO account (<a class="reference external" href="https://github.uio.no/login">https://github.uio.no/login</a>).</p>
|
||||
<p><strong>c)</strong> Learn the basics of opening the terminal and navigating folders on your operating system. Things to learn: Opening a terminal, opening a terminal in a specific folder, listing the contents of the current folder, navigating into a folder, navigating out of a folder.</p>
|
||||
<p><strong>d)</strong> Download the Github CLI tool and run ´gh auth login´ in your terminal to authenticate your local machine for some of the later steps. (<a class="github reference external" href="https://github.com/cli/cli#installation">cli/cli</a>). You might need to change file permissions to make it work, ask us or ChatGPT for help with these issues.</p>
|
||||
</section>
|
||||
<section id="exercise-2-setting-up-a-github-repository">
|
||||
<h2>Exercise 2 - Setting up a Github repository<a class="headerlink" href="#exercise-2-setting-up-a-github-repository" title="Link to this heading">#</a></h2>
|
||||
<p><strong>a)</strong> Create an empty repository for your coursework in this course in your browser at <a class="reference external" href="http://github.com">github.com</a> (or uio github).</p>
|
||||
<p><strong>b)</strong> Open a terminal in the location you want to create your local folder for this repository, like your desktop.</p>
|
||||
<p><strong>c)</strong> Clone the repository to your laptop using the terminal command ´gh repo clone username/repository-name´. This creates a folder with the same name as the repository. Moving it or renaming it might require some extra steps.</p>
|
||||
<p><strong>d)</strong> Download this jupyter notebook. Add the notebook to the local folder.</p>
|
||||
<p><strong>e)</strong> Run the ´git add .´ command command in a terminal opened in the local folder to stage the current changes in the folder to be commited to the version control history. Run ´git status´ to see the staged files.</p>
|
||||
<p><strong>f)</strong> Run the ´git commit -m “Adding first weekly assignment file”´ command to commit the staged changes to the version control history. Run ´git status´ to see that no files are staged.</p>
|
||||
<p><strong>g)</strong> Run the ´git push” command to upload the commited changes to the remote repository on Github.</p>
|
||||
<p><strong>h)</strong> Add a file called README.txt to the repository at <a class="reference external" href="http://Github.com">Github.com</a>. Don’t do this in your local folder. Add a suitable title for your repository and some inforomation to the file.</p>
|
||||
<p><strong>i)</strong> Run the ´git fetch origin´ command to fetch the latest remote changes to your repository.</p>
|
||||
<p><strong>j)</strong> Run the ´git pull´ command to download and update files to match the remote changes.</p>
|
||||
</section>
|
||||
<section id="exercise-3-setting-up-a-python-virtual-environment">
|
||||
<h2>Exercise 3 - Setting up a Python virtual environment<a class="headerlink" href="#exercise-3-setting-up-a-python-virtual-environment" title="Link to this heading">#</a></h2>
|
||||
<p>Following the theme of the previous exercises, another way of improving the reproducibility of your results and shareability of your code is having a good handle on which python packages you are using.</p>
|
||||
<p>There are many ways to manage your packages in Python, and you are free to use any approach you want, but in this course we encourage you to use something called a virtual environment. A virtual environemnt is a folder in your project which contains a Python runtime executable as well as all the packages you are using in the current project. In this way, each of your projects has its required set of packages installed in the same folder, so that if anything goes wrong while managing your packages it only affects the one project, and if multiple projects require different versions of the same package, you don’t need to worry about messing up old projects. Also, it’s easy to just delete the folder and start over if anything goes wrong.</p>
|
||||
<p>Virtual environments are typically created, activated, managed and updated using terminal commands, but for now we recommend that you let VS Code handle it for you to make the coding experience much easier.</p>
|
||||
<p><strong>a)</strong> Open this notebook in VS Code (<a class="reference external" href="https://code.visualstudio.com/Download">https://code.visualstudio.com/Download</a>). Download the Python and Jupyter extensions.</p>
|
||||
<p><strong>b)</strong> Press ´Cmd + Shift + P´, then search and run ´Python: Create Environment…´</p>
|
||||
<p><strong>c)</strong> Select ´Venv´</p>
|
||||
<p><strong>d)</strong> Choose the most up-to-date version of Python your have installed.</p>
|
||||
<p><strong>e)</strong> Press ´Cmd + Shift + P´, then search and run ´Python: Select Interpreter´</p>
|
||||
<p><strong>f)</strong> Selevet the (.venv) option you just created.</p>
|
||||
<p><strong>g)</strong> Open a terminal in VS Code, the venv name should be visible at the beginning of the line. Run <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">list</span></code> to see that there are no packages install in the environment.</p>
|
||||
<p><strong>h)</strong> In this terminal, run <code class="docutils literal notranslate"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">matplotlib</span> <span class="pre">numpy</span> <span class="pre">scikit-learn</span></code>. This will install the listed packages.</p>
|
||||
<p><strong>i)</strong> To make these installations reproducible, which is important for reproducing results and sharing your code, run ´pip freeze > requirements.txt´ to create the file requirements.txt with all your dependencies.</p>
|
||||
<p>Now, anyone who wants to recreate your package setup can download your requirements.txt file and run ´pip install -r requirements.txt´ to install the correct packages and versions. To keep the requirements.txt file up to date with your environment, you will need to re-run the freeze command whenever you install a new package.</p>
|
||||
<p><strong>j)</strong> Create a .gitignore file at the root of your project folder, and add the line ´.venv´ to it. This way, you won’t try to upload a copy of all your python packages when you regularly push your changes to Github. Ignored files should not show up when you run ´git status´, and are not staged when running ´git add .´, try it!</p>
|
||||
</section>
|
||||
<section id="exercise-3-fitting-an-ols-model-to-data">
|
||||
<h2>Exercise 3 - Fitting an OLS model to data<a class="headerlink" href="#exercise-3-fitting-an-ols-model-to-data" title="Link to this heading">#</a></h2>
|
||||
<p>Great job on getting through all of that! Now it’s time to do some actual machine learning!</p>
|
||||
<p><strong>a)</strong> Complete the code below so that you fit a second order polynomial to the data. You will need to look up some skelarn documentation online (look at the imported functions for hints).</p>
|
||||
<p><strong>b)</strong> Compute the mean square error for the line model and for the second degree polynomial model.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>import numpy as np
|
||||
x = np.random.rand(100,1)
|
||||
y = 2.0+5*x*x+0.1*np.random.randn(100,1)
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">numpy</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">np</span>
|
||||
<span class="kn">import</span><span class="w"> </span><span class="nn">matplotlib.pyplot</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">plt</span>
|
||||
<span class="kn">from</span><span class="w"> </span><span class="nn">sklearn.preprocessing</span><span class="w"> </span><span class="kn">import</span> <span class="n">PolynomialFeatures</span> <span class="c1"># use the fit_transform method of the created object!</span>
|
||||
<span class="kn">from</span><span class="w"> </span><span class="nn">sklearn.linear_model</span><span class="w"> </span><span class="kn">import</span> <span class="n">LinearRegression</span>
|
||||
<span class="kn">from</span><span class="w"> </span><span class="nn">sklearn.metrics</span><span class="w"> </span><span class="kn">import</span> <span class="n">mean_squared_error</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<ol class="arabic simple">
|
||||
<li><p>Write your own code (following the examples under the <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter1.html">regression notes</a>) for computing the parametrization of the data set fitting a second-order polynomial.</p></li>
|
||||
<li><p>Use thereafter <strong>scikit-learn</strong> (see again the examples in the regression slides) and compare with your own code.</p></li>
|
||||
<li><p>Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as</p></li>
|
||||
</ol>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
MSE(\boldsymbol{y},\boldsymbol{\tilde{y}}) = \frac{1}{n}
|
||||
\sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2,
|
||||
\]</div>
|
||||
<p>and the <span class="math notranslate nohighlight">\(R^2\)</span> score function.
|
||||
If <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}_i\)</span> is the predicted value of the <span class="math notranslate nohighlight">\(i-th\)</span> sample and <span class="math notranslate nohighlight">\(y_i\)</span> is the corresponding true value, then the score <span class="math notranslate nohighlight">\(R^2\)</span> is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
R^2(\boldsymbol{y}, \tilde{\boldsymbol{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2},
|
||||
\]</div>
|
||||
<p>where we have defined the mean value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i.
|
||||
\]</div>
|
||||
<p>You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
|
||||
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
|
||||
</section>
|
||||
<section id="exercise-3-split-data-in-test-and-training-data">
|
||||
<h2>Exercise 3: Split data in test and training data<a class="headerlink" href="#exercise-3-split-data-in-test-and-training-data" title="Link to this heading">#</a></h2>
|
||||
<p>In this exercise we want you to to compute the MSE for the training
|
||||
data and the test data as function of the complexity of a polynomial,
|
||||
that is the degree of a given polynomial.</p>
|
||||
<p>The aim is to reproduce Figure 2.11 of <a class="reference external" href="https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf">Hastie et al</a>.</p>
|
||||
<p>Our data is defined by <span class="math notranslate nohighlight">\(x\in [-3,3]\)</span> with a total of for example <span class="math notranslate nohighlight">\(n=100\)</span> data points. You should try to vary the number of data points <span class="math notranslate nohighlight">\(n\)</span> in your analysis.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>np.random.seed()
|
||||
n = 100
|
||||
# Make data set.
|
||||
x = np.linspace(-3, 3, n).reshape(-1, 1)
|
||||
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span> <span class="o">+</span> <span class="mi">5</span> <span class="o">*</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span> <span class="o">+</span> <span class="mf">0.1</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
|
||||
|
||||
<span class="n">line_model</span> <span class="o">=</span> <span class="n">LinearRegression</span><span class="p">()</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">)</span>
|
||||
<span class="n">line_predict</span> <span class="o">=</span> <span class="n">line_model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
|
||||
<span class="c1">#poly_features = ...</span>
|
||||
<span class="c1">#poly_model = LinearRegression().fit(..., y)</span>
|
||||
<span class="c1">#poly_predict = ...</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s2">"Data"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">scatter</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">line_predict</span><span class="p">,</span> <span class="n">label</span> <span class="o">=</span> <span class="s2">"Line model"</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">legend</span><span class="p">()</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/fbd74da3fcd21613c4c425622594644bbfe45b6a7fd50ac62da4e3136a737128.png" src="_images/fbd74da3fcd21613c4c425622594644bbfe45b6a7fd50ac62da4e3136a737128.png" />
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="exercise-4-the-train-test-split">
|
||||
<h2>Exercise 4 - The train-test split<a class="headerlink" href="#exercise-4-the-train-test-split" title="Link to this heading">#</a></h2>
|
||||
<p>Hopefully your model fit the data quite well, but to know how well the model actually generalizes to unseen data, which is most often what we care about, we need to split our data into training and testing data.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">sklearn.model_selection</span><span class="w"> </span><span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>a)</strong> Complete the code below so that the polynomial features and the targets y get split into training and test data.</p>
|
||||
<p><strong>b)</strong> What is the shape of X_test?</p>
|
||||
<p><strong>c)</strong> Fit your model to X_train</p>
|
||||
<p><strong>d)</strong> Compute the MSE when your model predicts on the training data and on the testing data, using y_train and y_test as targets for the two cases.</p>
|
||||
<p><strong>e)</strong> Why do we not fit the model to X_test?</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">polynomial_features</span> <span class="o">=</span> <span class="o">...</span>
|
||||
|
||||
<span class="c1">#X_train, X_test, y_train, y_test = train_test_split(polynomial_features, y, test_size=0.2)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>where <span class="math notranslate nohighlight">\(y\)</span> is the function we want to fit with a given polynomial.</p>
|
||||
<p><strong>a)</strong>
|
||||
Write a first code which sets up a design matrix <span class="math notranslate nohighlight">\(X\)</span> defined by a fifth-order polynomial and split your data set in training and test data.</p>
|
||||
<p><strong>b)</strong>
|
||||
Write thereafter (using either <strong>scikit-learn</strong> or your matrix inversion code using for example <strong>numpy</strong>)
|
||||
and perform an ordinary least squares fitting and compute the mean squared error for the training data and the test data. These calculations should apply to a model given by a fifth-order polynomial.</p>
|
||||
<p><strong>c)</strong>
|
||||
Add now a model which allows you to make polynomials up to degree <span class="math notranslate nohighlight">\(15\)</span>. Perform a standard OLS fitting of the training data and compute the MSE for the training and test data and plot both test and training data MSE as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?</p>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
@@ -584,10 +591,16 @@ Add now a model which allows you to make polynomials up to degree <span class="m
|
||||
</div>
|
||||
<nav class="bd-toc-nav page-toc">
|
||||
<ul class="visible nav section-nav flex-column">
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercises">Exercises</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-setting-up-various-python-environments">Exercise 1: Setting up various Python environments</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-making-your-own-data-and-exploring-scikit-learn">Exercise 2: making your own data and exploring scikit-learn</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-split-data-in-test-and-training-data">Exercise 3: Split data in test and training data</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#coding-setup-and-linear-regression">Coding Setup and Linear Regression</a><ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-goals">Learning goals</a></li>
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deliverables">Deliverables</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-github-setup">Exercise 1 - Github Setup</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-setting-up-a-github-repository">Exercise 2 - Setting up a Github repository</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-setting-up-a-python-virtual-environment">Exercise 3 - Setting up a Python virtual environment</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-fitting-an-ols-model-to-data">Exercise 3 - Fitting an OLS model to data</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-the-train-test-split">Exercise 4 - The train-test split</a></li>
|
||||
</ul>
|
||||
</nav></div>
|
||||
|
||||
@@ -0,0 +1,746 @@
|
||||
|
||||
<!DOCTYPE html>
|
||||
|
||||
|
||||
<html lang="en" data-content_root="./" >
|
||||
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
|
||||
<title>Exercises week 35 — Applied Data Analysis and Machine Learning</title>
|
||||
|
||||
|
||||
|
||||
<script data-cfasync="false">
|
||||
document.documentElement.dataset.mode = localStorage.getItem("mode") || "";
|
||||
document.documentElement.dataset.theme = localStorage.getItem("theme") || "";
|
||||
</script>
|
||||
|
||||
<!-- Loaded before other Sphinx assets -->
|
||||
<link href="_static/styles/theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
|
||||
<link href="_static/styles/bootstrap.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
|
||||
<link href="_static/styles/pydata-sphinx-theme.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
|
||||
|
||||
|
||||
<link href="_static/vendor/fontawesome/6.5.2/css/all.min.css?digest=dfe6caa3a7d634c4db9b" rel="stylesheet" />
|
||||
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-solid-900.woff2" />
|
||||
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-brands-400.woff2" />
|
||||
<link rel="preload" as="font" type="font/woff2" crossorigin href="_static/vendor/fontawesome/6.5.2/webfonts/fa-regular-400.woff2" />
|
||||
|
||||
<link rel="stylesheet" type="text/css" href="_static/pygments.css?v=03e43079" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/styles/sphinx-book-theme.css?v=eba8b062" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/togglebutton.css?v=13237357" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/copybutton.css?v=76b2166b" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/mystnb.8ecb98da25f57f5357bf6f572d296f466b2cfe2517ffebfabe82451661e28f02.css" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/sphinx-thebe.css?v=4fa983c6" />
|
||||
<link rel="stylesheet" type="text/css" href="_static/sphinx-design.min.css?v=95c83b7e" />
|
||||
|
||||
<!-- Pre-loaded scripts that we'll load fully later -->
|
||||
<link rel="preload" as="script" href="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b" />
|
||||
<link rel="preload" as="script" href="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b" />
|
||||
<script src="_static/vendor/fontawesome/6.5.2/js/all.min.js?digest=dfe6caa3a7d634c4db9b"></script>
|
||||
|
||||
<script src="_static/documentation_options.js?v=9eb32ce0"></script>
|
||||
<script src="_static/doctools.js?v=9a2dae69"></script>
|
||||
<script src="_static/sphinx_highlight.js?v=dc90522c"></script>
|
||||
<script src="_static/clipboard.min.js?v=a7894cd8"></script>
|
||||
<script src="_static/copybutton.js?v=f281be69"></script>
|
||||
<script src="_static/scripts/sphinx-book-theme.js?v=887ef09a"></script>
|
||||
<script>let toggleHintShow = 'Click to show';</script>
|
||||
<script>let toggleHintHide = 'Click to hide';</script>
|
||||
<script>let toggleOpenOnPrint = 'true';</script>
|
||||
<script src="_static/togglebutton.js?v=4a39c7ea"></script>
|
||||
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
|
||||
<script src="_static/design-tabs.js?v=f930bc37"></script>
|
||||
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
|
||||
<script async="async" src="_static/sphinx-thebe.js?v=c100c467"></script>
|
||||
<script>var togglebuttonSelector = '.toggle, .admonition.dropdown';</script>
|
||||
<script>const THEBE_JS_URL = "https://unpkg.com/thebe@0.8.2/lib/index.js"; const thebe_selector = ".thebe,.cell"; const thebe_selector_input = "pre"; const thebe_selector_output = ".output, .cell_output"</script>
|
||||
<script>window.MathJax = {"options": {"processHtmlClass": "tex2jax_process|mathjax_process|math|output_area"}}</script>
|
||||
<script defer="defer" src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script>
|
||||
<script>DOCUMENTATION_OPTIONS.pagename = 'E2';</script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="prev" title="Week 34: Introduction to the course, Logistics and Practicalities" href="week34.html" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<meta name="docsearch:language" content="en"/>
|
||||
</head>
|
||||
|
||||
|
||||
<body data-bs-spy="scroll" data-bs-target=".bd-toc-nav" data-offset="180" data-bs-root-margin="0px 0px -60%" data-default-mode="">
|
||||
|
||||
|
||||
|
||||
<div id="pst-skip-link" class="skip-link d-print-none"><a href="#main-content">Skip to main content</a></div>
|
||||
|
||||
<div id="pst-scroll-pixel-helper"></div>
|
||||
|
||||
<button type="button" class="btn rounded-pill" id="pst-back-to-top">
|
||||
<i class="fa-solid fa-arrow-up"></i>Back to top</button>
|
||||
|
||||
|
||||
<input type="checkbox"
|
||||
class="sidebar-toggle"
|
||||
id="pst-primary-sidebar-checkbox"/>
|
||||
<label class="overlay overlay-primary" for="pst-primary-sidebar-checkbox"></label>
|
||||
|
||||
<input type="checkbox"
|
||||
class="sidebar-toggle"
|
||||
id="pst-secondary-sidebar-checkbox"/>
|
||||
<label class="overlay overlay-secondary" for="pst-secondary-sidebar-checkbox"></label>
|
||||
|
||||
<div class="search-button__wrapper">
|
||||
<div class="search-button__overlay"></div>
|
||||
<div class="search-button__search-container">
|
||||
<form class="bd-search d-flex align-items-center"
|
||||
action="search.html"
|
||||
method="get">
|
||||
<i class="fa-solid fa-magnifying-glass"></i>
|
||||
<input type="search"
|
||||
class="form-control"
|
||||
name="q"
|
||||
id="search-input"
|
||||
placeholder="Search this book..."
|
||||
aria-label="Search this book..."
|
||||
autocomplete="off"
|
||||
autocorrect="off"
|
||||
autocapitalize="off"
|
||||
spellcheck="false"/>
|
||||
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd>K</kbd></span>
|
||||
</form></div>
|
||||
</div>
|
||||
|
||||
<div class="pst-async-banner-revealer d-none">
|
||||
<aside id="bd-header-version-warning" class="d-none d-print-none" aria-label="Version warning"></aside>
|
||||
</div>
|
||||
|
||||
|
||||
<header class="bd-header navbar navbar-expand-lg bd-navbar d-print-none">
|
||||
</header>
|
||||
|
||||
|
||||
<div class="bd-container">
|
||||
<div class="bd-container__inner bd-page-width">
|
||||
|
||||
|
||||
|
||||
<div class="bd-sidebar-primary bd-sidebar">
|
||||
|
||||
|
||||
|
||||
<div class="sidebar-header-items sidebar-primary__section">
|
||||
|
||||
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<div class="sidebar-primary-items__start sidebar-primary__section">
|
||||
<div class="sidebar-primary-item">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<a class="navbar-brand logo" href="intro.html">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<img src="_static/logo.png" class="logo__image only-light" alt="Applied Data Analysis and Machine Learning - Home"/>
|
||||
<script>document.write(`<img src="_static/logo.png" class="logo__image only-dark" alt="Applied Data Analysis and Machine Learning - Home"/>`);</script>
|
||||
|
||||
|
||||
</a></div>
|
||||
<div class="sidebar-primary-item">
|
||||
|
||||
<script>
|
||||
document.write(`
|
||||
<button class="btn search-button-field search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
|
||||
<i class="fa-solid fa-magnifying-glass"></i>
|
||||
<span class="search-button__default-text">Search</span>
|
||||
<span class="search-button__kbd-shortcut"><kbd class="kbd-shortcut__modifier">Ctrl</kbd>+<kbd class="kbd-shortcut__modifier">K</kbd></span>
|
||||
</button>
|
||||
`);
|
||||
</script></div>
|
||||
<div class="sidebar-primary-item"><nav class="bd-links bd-docs-nav" aria-label="Main">
|
||||
<div class="bd-toc-item navbar-nav active">
|
||||
|
||||
<ul class="nav bd-sidenav bd-sidenav__home-link">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="intro.html">
|
||||
Applied Data Analysis and Machine Learning
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">About the course</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="schedule.html">Course setting</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="teachers.html">Teachers and Grading</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="textbooks.html">Textbooks</a></li>
|
||||
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Review of Statistics with Resampling Techniques and Linear Algebra</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="statistics.html">1. Elements of Probability Theory and Statistical Data Analysis</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="linalg.html">2. Linear Algebra, Handling of Arrays and more Python Features</a></li>
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">From Regression to Support Vector Machines</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter1.html">3. Linear Regression</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter2.html">4. Ridge and Lasso Regression</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter3.html">5. Resampling Methods</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter4.html">6. Logistic Regression</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapteroptimization.html">7. Optimization, the central part of any Machine Learning algortithm</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter5.html">8. Support Vector Machines, overarching aims</a></li>
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Decision Trees, Ensemble Methods and Boosting</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter6.html">9. Decision trees, overarching aims</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter7.html">10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a></li>
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Dimensionality Reduction</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter8.html">11. Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="clustering.html">12. Clustering and Unsupervised Learning</a></li>
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Deep Learning Methods</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter9.html">13. Neural networks</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter10.html">14. Building a Feed Forward Neural Network</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter11.html">15. Solving Differential Equations with Deep Learning</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter12.html">16. Convolutional Neural Networks</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="chapter13.html">17. Recurrent neural networks: Overarching view</a></li>
|
||||
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="current nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
</nav></div>
|
||||
</div>
|
||||
|
||||
|
||||
<div class="sidebar-primary-items__end sidebar-primary__section">
|
||||
</div>
|
||||
|
||||
<div id="rtd-footer-container"></div>
|
||||
|
||||
|
||||
</div>
|
||||
|
||||
<main id="main-content" class="bd-main" role="main">
|
||||
|
||||
|
||||
|
||||
<div class="sbt-scroll-pixel-helper"></div>
|
||||
|
||||
<div class="bd-content">
|
||||
<div class="bd-article-container">
|
||||
|
||||
<div class="bd-header-article d-print-none">
|
||||
<div class="header-article-items header-article__inner">
|
||||
|
||||
<div class="header-article-items__start">
|
||||
|
||||
<div class="header-article-item"><button class="sidebar-toggle primary-toggle btn btn-sm" title="Toggle primary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
|
||||
<span class="fa-solid fa-bars"></span>
|
||||
</button></div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
<div class="header-article-items__end">
|
||||
|
||||
<div class="header-article-item">
|
||||
|
||||
<div class="article-header-buttons">
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="dropdown dropdown-download-buttons">
|
||||
<button class="btn dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Download this page">
|
||||
<i class="fas fa-download"></i>
|
||||
</button>
|
||||
<ul class="dropdown-menu">
|
||||
|
||||
|
||||
|
||||
<li><a href="_sources/E2.ipynb" target="_blank"
|
||||
class="btn btn-sm btn-download-source-button dropdown-item"
|
||||
title="Download source file"
|
||||
data-bs-placement="left" data-bs-toggle="tooltip"
|
||||
>
|
||||
|
||||
|
||||
<span class="btn__icon-container">
|
||||
<i class="fas fa-file"></i>
|
||||
</span>
|
||||
<span class="btn__text-container">.ipynb</span>
|
||||
</a>
|
||||
</li>
|
||||
|
||||
|
||||
|
||||
|
||||
<li>
|
||||
<button onclick="window.print()"
|
||||
class="btn btn-sm btn-download-pdf-button dropdown-item"
|
||||
title="Print to PDF"
|
||||
data-bs-placement="left" data-bs-toggle="tooltip"
|
||||
>
|
||||
|
||||
|
||||
<span class="btn__icon-container">
|
||||
<i class="fas fa-file-pdf"></i>
|
||||
</span>
|
||||
<span class="btn__text-container">.pdf</span>
|
||||
</button>
|
||||
</li>
|
||||
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
|
||||
<button onclick="toggleFullScreen()"
|
||||
class="btn btn-sm btn-fullscreen-button"
|
||||
title="Fullscreen mode"
|
||||
data-bs-placement="bottom" data-bs-toggle="tooltip"
|
||||
>
|
||||
|
||||
|
||||
<span class="btn__icon-container">
|
||||
<i class="fas fa-expand"></i>
|
||||
</span>
|
||||
|
||||
</button>
|
||||
|
||||
|
||||
|
||||
<script>
|
||||
document.write(`
|
||||
<button class="btn btn-sm nav-link pst-navbar-icon theme-switch-button" title="light/dark" aria-label="light/dark" data-bs-placement="bottom" data-bs-toggle="tooltip">
|
||||
<i class="theme-switch fa-solid fa-sun fa-lg" data-mode="light"></i>
|
||||
<i class="theme-switch fa-solid fa-moon fa-lg" data-mode="dark"></i>
|
||||
<i class="theme-switch fa-solid fa-circle-half-stroke fa-lg" data-mode="auto"></i>
|
||||
</button>
|
||||
`);
|
||||
</script>
|
||||
|
||||
|
||||
<script>
|
||||
document.write(`
|
||||
<button class="btn btn-sm pst-navbar-icon search-button search-button__button" title="Search" aria-label="Search" data-bs-placement="bottom" data-bs-toggle="tooltip">
|
||||
<i class="fa-solid fa-magnifying-glass fa-lg"></i>
|
||||
</button>
|
||||
`);
|
||||
</script>
|
||||
<button class="sidebar-toggle secondary-toggle btn btn-sm" title="Toggle secondary sidebar" data-bs-placement="bottom" data-bs-toggle="tooltip">
|
||||
<span class="fa-solid fa-list"></span>
|
||||
</button>
|
||||
</div></div>
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<div id="jb-print-docs-body" class="onlyprint">
|
||||
<h1>Exercises week 35</h1>
|
||||
<!-- Table of contents -->
|
||||
<div id="print-main-content">
|
||||
<div id="jb-print-toc">
|
||||
|
||||
<div>
|
||||
<h2> Contents </h2>
|
||||
</div>
|
||||
<nav aria-label="Page">
|
||||
<ul class="visible nav section-nav flex-column">
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#deriving-and-implementing-ordinary-least-squares">Deriving and Implementing Ordinary Least Squares</a><ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-goals">Learning goals</a></li>
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deliverables">Deliverables</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#how-to-take-derivatives-of-matrix-vector-expressions">How to take derivatives of Matrix-Vector expressions</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-finding-the-derivative-of-matrix-vector-expressions">Exercise 1 - Finding the derivative of Matrix-Vector expressions</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-deriving-the-expression-for-ols">Exercise 2 - Deriving the expression for OLS</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression">Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-fitting-a-polynomial">Exercise 4 - Fitting a polynomial</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-comparing-your-code-with-sklearn">Exercise 5 - Comparing your code with sklearn</a></li>
|
||||
</ul>
|
||||
</nav>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<div id="searchbox"></div>
|
||||
<article class="bd-article">
|
||||
|
||||
<section class="tex2jax_ignore mathjax_ignore" id="exercises-week-35">
|
||||
<h1>Exercises week 35<a class="headerlink" href="#exercises-week-35" title="Link to this heading">#</a></h1>
|
||||
<section id="deriving-and-implementing-ordinary-least-squares">
|
||||
<h2>Deriving and Implementing Ordinary Least Squares<a class="headerlink" href="#deriving-and-implementing-ordinary-least-squares" title="Link to this heading">#</a></h2>
|
||||
<p>This week you will be deriving the analytical expressions for linear regression, building up the model from scratch. This will include taking several derivatives of products of vectors and matrices. Such derivatives are central to the optimization of many machine learning models. Although we will often use automatic differentiation in actual calculations, to be able to have analytical expressions is extremely helpful in case we have simpler derivatives as well as when we analyze various properties (like second derivatives) of the chosen cost functions.</p>
|
||||
<p>Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. You will find useful the notes from week 35 on derivatives of vectors and matrices. See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <a class="github reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf">CompPhysics/MachineLearning</a></p>
|
||||
<section id="learning-goals">
|
||||
<h3>Learning goals<a class="headerlink" href="#learning-goals" title="Link to this heading">#</a></h3>
|
||||
<p>After completing these exercises, you will know how to</p>
|
||||
<ul class="simple">
|
||||
<li><p>Take the derivatives of simple products between vectors and matrices</p></li>
|
||||
<li><p>Implement OLS using the analytical expressions</p></li>
|
||||
<li><p>Create a feature matrix from a set of data</p></li>
|
||||
<li><p>Create a feature matrix for a polynomial model</p></li>
|
||||
<li><p>Evaluate the MSE score of various model on training and test data, and comparing their performance</p></li>
|
||||
</ul>
|
||||
</section>
|
||||
<section id="deliverables">
|
||||
<h3>Deliverables<a class="headerlink" href="#deliverables" title="Link to this heading">#</a></h3>
|
||||
<p>Complete the following exercises while working in a jupyter notebook. Then, in canvas, include</p>
|
||||
<ul class="simple">
|
||||
<li><p>The jupyter notebook with the exercises completed</p></li>
|
||||
<li><p>An exported PDF of the notebook (<a class="reference external" href="https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook">https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook</a>)</p></li>
|
||||
</ul>
|
||||
</section>
|
||||
</section>
|
||||
<section id="how-to-take-derivatives-of-matrix-vector-expressions">
|
||||
<h2>How to take derivatives of Matrix-Vector expressions<a class="headerlink" href="#how-to-take-derivatives-of-matrix-vector-expressions" title="Link to this heading">#</a></h2>
|
||||
<p>In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.</p>
|
||||
<p>As an example, consider the function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
f(\boldsymbol{x}) =\boldsymbol{A}\boldsymbol{x},
|
||||
\]</div>
|
||||
<p>which reads for a specific component <span class="math notranslate nohighlight">\(f_i\)</span> (we define the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> to have dimension <span class="math notranslate nohighlight">\(n\times n\)</span> and the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> to have length <span class="math notranslate nohighlight">\(n\)</span>)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
f_i =\sum_{j=0}^{n-1}a_{ij}x_j,
|
||||
\]</div>
|
||||
<p>which leads to</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial f_i}{\partial x_j}= a_{ij},
|
||||
\]</div>
|
||||
<p>and written out in terms of the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial f(\boldsymbol{x})}{\partial \boldsymbol{x}}= \boldsymbol{A}.
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="exercise-1-finding-the-derivative-of-matrix-vector-expressions">
|
||||
<h2>Exercise 1 - Finding the derivative of Matrix-Vector expressions<a class="headerlink" href="#exercise-1-finding-the-derivative-of-matrix-vector-expressions" title="Link to this heading">#</a></h2>
|
||||
<p><strong>a)</strong> Consider the expression</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}},
|
||||
\]</div>
|
||||
<p>Where <span class="math notranslate nohighlight">\(\boldsymbol{a}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> are column-vectors with length <span class="math notranslate nohighlight">\(n\)</span>.</p>
|
||||
<p>What is the <em>shape</em> of the expression we are taking the derivative of?</p>
|
||||
<p>What is the <em>shape</em> of the thing we are taking the derivative with respect to?</p>
|
||||
<p>What is the <em>shape</em> of the result of the expression?</p>
|
||||
<p><strong>b)</strong> Show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial (\boldsymbol{a}^T\boldsymbol{x})}{\partial \boldsymbol{x}} = \boldsymbol{a}^T,
|
||||
\]</div>
|
||||
<p><strong>c)</strong> Show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial (\boldsymbol{a}^T\boldsymbol{A}\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{a}^T(\boldsymbol{A}+\boldsymbol{A}^T),
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="exercise-2-deriving-the-expression-for-ols">
|
||||
<h2>Exercise 2 - Deriving the expression for OLS<a class="headerlink" href="#exercise-2-deriving-the-expression-for-ols" title="Link to this heading">#</a></h2>
|
||||
<p>The ordinary least squares method finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes the squared error between our model <span class="math notranslate nohighlight">\(\boldsymbol{X\beta}\)</span> and the true values <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>.</p>
|
||||
<p>To find the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes this error, we take the derivative of the squared error expression with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, and set it equal to 0.</p>
|
||||
<p><strong>a)</strong> Very briefly explain why the approach above finds the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> which minimizes this error.</p>
|
||||
<p>We typically write the squared error as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\vert\vert\boldsymbol{y} - \boldsymbol{X\beta}\vert\vert^2
|
||||
\]</div>
|
||||
<p>which we can rewrite in matrix-vector form as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)
|
||||
\]</div>
|
||||
<p><strong>b)</strong> If <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is invertible, what is the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>? (<strong>Hint:</strong> Don’t compute any derivatives, but solve <span class="math notranslate nohighlight">\(\boldsymbol{X\beta}=\boldsymbol{y}\)</span> for <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>)</p>
|
||||
<p><strong>c)</strong> Show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial \left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)}{\partial \boldsymbol{s}} = -2\left(\boldsymbol{x}-\boldsymbol{A}\boldsymbol{s}\right)^T\boldsymbol{A},
|
||||
\]</div>
|
||||
<p><strong>d)</strong> Using the expression from <strong>c)</strong>, but substituting back in <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>, find the expression for the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> in the case that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is not invertible, but <span class="math notranslate nohighlight">\(\boldsymbol{X^T X}\)</span> is, which is most often the case.</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{\hat{\beta}_{OLS}} = ...
|
||||
\]</div>
|
||||
</section>
|
||||
<section id="exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression">
|
||||
<h2>Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression<a class="headerlink" href="#exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression" title="Link to this heading">#</a></h2>
|
||||
<p>With the expression for <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}_{OLS}}\)</span>, you now have what you need to implement OLS regression with your input data and target data <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span>. But before you can do that, you need to set up you input data as a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.</p>
|
||||
<p>In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, and in each column put a 1 for the intercept, the montly income and the number of children.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span><span class="w"> </span><span class="nn">numpy</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">np</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">n</span> <span class="o">=</span> <span class="mi">20</span>
|
||||
<span class="n">income</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">116.</span><span class="p">,</span> <span class="mf">161.</span><span class="p">,</span> <span class="mf">167.</span><span class="p">,</span> <span class="mf">118.</span><span class="p">,</span> <span class="mf">172.</span><span class="p">,</span> <span class="mf">163.</span><span class="p">,</span> <span class="mf">179.</span><span class="p">,</span> <span class="mf">173.</span><span class="p">,</span> <span class="mf">162.</span><span class="p">,</span> <span class="mf">116.</span><span class="p">,</span> <span class="mf">101.</span><span class="p">,</span> <span class="mf">176.</span><span class="p">,</span> <span class="mf">178.</span><span class="p">,</span> <span class="mf">172.</span><span class="p">,</span> <span class="mf">143.</span><span class="p">,</span> <span class="mf">135.</span><span class="p">,</span> <span class="mf">160.</span><span class="p">,</span> <span class="mf">101.</span><span class="p">,</span> <span class="mf">149.</span><span class="p">,</span> <span class="mf">125.</span><span class="p">])</span>
|
||||
<span class="n">children</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">4</span><span class="p">])</span>
|
||||
<span class="n">spending</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mf">152.</span><span class="p">,</span> <span class="mf">141.</span><span class="p">,</span> <span class="mf">102.</span><span class="p">,</span> <span class="mf">136.</span><span class="p">,</span> <span class="mf">161.</span><span class="p">,</span> <span class="mf">129.</span><span class="p">,</span> <span class="mf">99.</span><span class="p">,</span> <span class="mf">159.</span><span class="p">,</span> <span class="mf">160.</span><span class="p">,</span> <span class="mf">107.</span><span class="p">,</span> <span class="mf">98.</span><span class="p">,</span> <span class="mf">164.</span><span class="p">,</span> <span class="mf">121.</span><span class="p">,</span> <span class="mf">93.</span><span class="p">,</span> <span class="mf">112.</span><span class="p">,</span> <span class="mf">127.</span><span class="p">,</span> <span class="mf">117.</span><span class="p">,</span> <span class="mf">69.</span><span class="p">,</span> <span class="mf">156.</span><span class="p">,</span> <span class="mf">131.</span><span class="p">])</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>a)</strong> Create a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> for the features income and children, including an intercept column of ones at the start.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
|
||||
<span class="c1">#X[:, 0] = ...</span>
|
||||
<span class="c1">#X[:, 1] = ...</span>
|
||||
<span class="c1">#X[:, 2] = ...</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>b)</strong> Use the expression from <strong>3d)</strong> to find the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}_{OLS}}\)</span> for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">OLS_parameters</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="o">...</span>
|
||||
|
||||
<span class="c1">#beta = OLS_parameters(X, y)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
<section id="exercise-4-fitting-a-polynomial">
|
||||
<h2>Exercise 4 - Fitting a polynomial<a class="headerlink" href="#exercise-4-fitting-a-polynomial" title="Link to this heading">#</a></h2>
|
||||
<p>In this course, we typically do linear regression using polynomials, though in real world applications it is also very common to make linear models based on measured features like you did in the previous exercise.</p>
|
||||
<p>When fitting a polynomial with linear regression, we make each polynomial degree(<span class="math notranslate nohighlight">\(x, x^2, x^3, ..., x^p\)</span>) its own feature.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">n</span> <span class="o">=</span> <span class="mi">100</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linspace</span><span class="p">(</span><span class="o">-</span><span class="mi">3</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="mf">1.5</span> <span class="o">*</span> <span class="n">np</span><span class="o">.</span><span class="n">exp</span><span class="p">(</span><span class="o">-</span><span class="p">(</span><span class="n">x</span><span class="o">-</span><span class="mi">2</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span> <span class="o">+</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">normal</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>a)</strong> Create a feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> for the features <span class="math notranslate nohighlight">\(x, x^2, x^3, x^4, x^5\)</span>, including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="k">def</span><span class="w"> </span><span class="nf">polynomial_features</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">p</span><span class="p">):</span>
|
||||
<span class="n">n</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
|
||||
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="n">n</span><span class="p">,</span> <span class="n">p</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span>
|
||||
<span class="c1">#X[:, 0] = ...</span>
|
||||
<span class="c1">#X[:, 1] = ...</span>
|
||||
<span class="c1">#X[:, 2] = ...</span>
|
||||
<span class="c1"># could this be a loop?</span>
|
||||
|
||||
<span class="c1">#X = polynomial_features(x, 5)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>b)</strong> Use the expression from <strong>3d)</strong> to find the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\hat{\beta}_{OLS}}\)</span> for predicting <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> based on these features. If you have done everything right so far, this code will not need changing.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1">#beta = OLS_parameters(X, y)</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>c)</strong> Like in exercise 4 last week, split your feature matrix and target data into a training split and test split.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">sklearn.model_selection</span><span class="w"> </span><span class="kn">import</span> <span class="n">train_test_split</span>
|
||||
|
||||
<span class="c1">#X_train, X_test, y_train, y_test = ...</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>d)</strong> Train your model on the training data(find the parameters which best fit) and compute the MSE on both the training and test data.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">...</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Ellipsis
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>e)</strong> Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of <a class="reference external" href="https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf">Hastie et al</a>. Feel free to read the discussions leading to figure 2.11 of Hastie et al.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">...</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Ellipsis
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p><strong>f)</strong> Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?</p>
|
||||
</section>
|
||||
<section id="exercise-5-comparing-your-code-with-sklearn">
|
||||
<h2>Exercise 5 - Comparing your code with sklearn<a class="headerlink" href="#exercise-5-comparing-your-code-with-sklearn" title="Link to this heading">#</a></h2>
|
||||
<p>When implementing different algorithms for the first time, it can be helpful to double check your results with established implementations before you go on to add more complexity.</p>
|
||||
<p><strong>a)</strong> Make sure your <code class="docutils literal notranslate"><span class="pre">polynomial_features</span></code> function creates the same feature matrix as sklearns PolynomialFeatures.</p>
|
||||
<p>(<a class="reference external" href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html">https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html</a>)</p>
|
||||
<p><strong>b)</strong> Make sure your <code class="docutils literal notranslate"><span class="pre">OLS_parameters</span></code> function computes the same parameters as sklearns LinearRegression with fit_intercept set to False, since the intercept is included in the feature matrix. Use <code class="docutils literal notranslate"><span class="pre">your_model_object.coef_</span></code> to extract the computed parameters.</p>
|
||||
<p>(<a class="reference external" href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html">https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html</a>)</p>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<script type="text/x-thebe-config">
|
||||
{
|
||||
requestKernel: true,
|
||||
binderOptions: {
|
||||
repo: "binder-examples/jupyter-stacks-datascience",
|
||||
ref: "master",
|
||||
},
|
||||
codeMirrorConfig: {
|
||||
theme: "abcdef",
|
||||
mode: "python"
|
||||
},
|
||||
kernelOptions: {
|
||||
name: "python3",
|
||||
path: "./."
|
||||
},
|
||||
predefinedOutput: true
|
||||
}
|
||||
</script>
|
||||
<script>kernelName = 'python3'</script>
|
||||
|
||||
</article>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<footer class="prev-next-footer d-print-none">
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev"
|
||||
href="week34.html"
|
||||
title="previous page">
|
||||
<i class="fa-solid fa-angle-left"></i>
|
||||
<div class="prev-next-info">
|
||||
<p class="prev-next-subtitle">previous</p>
|
||||
<p class="prev-next-title">Week 34: Introduction to the course, Logistics and Practicalities</p>
|
||||
</div>
|
||||
</a>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<div class="bd-sidebar-secondary bd-toc"><div class="sidebar-secondary-items sidebar-secondary__inner">
|
||||
|
||||
|
||||
<div class="sidebar-secondary-item">
|
||||
<div class="page-toc tocsection onthispage">
|
||||
<i class="fa-solid fa-list"></i> Contents
|
||||
</div>
|
||||
<nav class="bd-toc-nav page-toc">
|
||||
<ul class="visible nav section-nav flex-column">
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#deriving-and-implementing-ordinary-least-squares">Deriving and Implementing Ordinary Least Squares</a><ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#learning-goals">Learning goals</a></li>
|
||||
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link" href="#deliverables">Deliverables</a></li>
|
||||
</ul>
|
||||
</li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#how-to-take-derivatives-of-matrix-vector-expressions">How to take derivatives of Matrix-Vector expressions</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-1-finding-the-derivative-of-matrix-vector-expressions">Exercise 1 - Finding the derivative of Matrix-Vector expressions</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-2-deriving-the-expression-for-ols">Exercise 2 - Deriving the expression for OLS</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-3-creating-feature-matrix-and-implementing-ols-using-the-analytical-expression">Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-4-fitting-a-polynomial">Exercise 4 - Fitting a polynomial</a></li>
|
||||
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link" href="#exercise-5-comparing-your-code-with-sklearn">Exercise 5 - Comparing your code with sklearn</a></li>
|
||||
</ul>
|
||||
</nav></div>
|
||||
|
||||
</div></div>
|
||||
|
||||
|
||||
</div>
|
||||
<footer class="bd-footer-content">
|
||||
|
||||
<div class="bd-footer-content__inner container">
|
||||
|
||||
<div class="footer-item">
|
||||
|
||||
<p class="component-author">
|
||||
By Morten Hjorth-Jensen
|
||||
</p>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer-item">
|
||||
|
||||
|
||||
<p class="copyright">
|
||||
|
||||
© Copyright 2023.
|
||||
<br/>
|
||||
|
||||
</p>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer-item">
|
||||
|
||||
</div>
|
||||
|
||||
<div class="footer-item">
|
||||
|
||||
</div>
|
||||
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Scripts loaded after <body> so the DOM is not blocked -->
|
||||
<script src="_static/scripts/bootstrap.js?digest=dfe6caa3a7d634c4db9b"></script>
|
||||
<script src="_static/scripts/pydata-sphinx-theme.js?digest=dfe6caa3a7d634c4db9b"></script>
|
||||
|
||||
<footer class="bd-footer">
|
||||
</footer>
|
||||
</body>
|
||||
</html>
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
File diff suppressed because one or more lines are too long
@@ -0,0 +1,535 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b4005770",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Exercises week 35\n",
|
||||
"\n",
|
||||
"## Deriving and Implementing Ordinary Least Squares"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ca1b589",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This week you will be deriving the analytical expressions for linear regression, building up the model from scratch. This will include taking several derivatives of products of vectors and matrices. Such derivatives are central to the optimization of many machine learning models. Although we will often use automatic differentiation in actual calculations, to be able to have analytical expressions is extremely helpful in case we have simpler derivatives as well as when we analyze various properties (like second derivatives) of the chosen cost functions.\n",
|
||||
"\n",
|
||||
"Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. You will find useful the notes from week 35 on derivatives of vectors and matrices. See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "41e92bf9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Learning goals\n",
|
||||
"\n",
|
||||
"After completing these exercises, you will know how to\n",
|
||||
"- Take the derivatives of simple products between vectors and matrices\n",
|
||||
"- Implement OLS using the analytical expressions\n",
|
||||
"- Create a feature matrix from a set of data\n",
|
||||
"- Create a feature matrix for a polynomial model\n",
|
||||
"- Evaluate the MSE score of various model on training and test data, and comparing their performance\n",
|
||||
"\n",
|
||||
"### Deliverables\n",
|
||||
"\n",
|
||||
"Complete the following exercises while working in a jupyter notebook. Then, in canvas, include\n",
|
||||
"- The jupyter notebook with the exercises completed\n",
|
||||
"- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7a9209d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How to take derivatives of Matrix-Vector expressions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "45f3712e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.\n",
|
||||
"\n",
|
||||
"As an example, consider the function\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f(\\boldsymbol{x}) =\\boldsymbol{A}\\boldsymbol{x},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which reads for a specific component $f_i$ (we define the matrix $\\boldsymbol{A}$ to have dimension $n\\times n$ and the vector $\\boldsymbol{x}$ to have length $n$)\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f_i =\\sum_{j=0}^{n-1}a_{ij}x_j,\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which leads to\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial f_i}{\\partial x_j}= a_{ij},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"and written out in terms of the vector $\\boldsymbol{x}$ we have\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial f(\\boldsymbol{x})}{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5fa8a4e6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 1 - Finding the derivative of Matrix-Vector expressions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "df7a2270",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Consider the expression\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"Where $\\boldsymbol{a}$ and $\\boldsymbol{x}$ are column-vectors with length $n$.\n",
|
||||
"\n",
|
||||
"What is the *shape* of the expression we are taking the derivative of?\n",
|
||||
"\n",
|
||||
"What is the *shape* of the thing we are taking the derivative with respect to?\n",
|
||||
"\n",
|
||||
"What is the *shape* of the result of the expression?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c0396734",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dc39d541",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{A}\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{a}^T(\\boldsymbol{A}+\\boldsymbol{A}^T),\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "498d13ec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 2 - Deriving the expression for OLS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f3f771de",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n",
|
||||
"\n",
|
||||
"To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "49690237",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7cccc9d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We typically write the squared error as\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which we can rewrite in matrix-vector form as\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8fbecf74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f37af8f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial \\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)}{\\partial \\boldsymbol{s}} = -2\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\boldsymbol{A},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "869fca4d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "57ca3d74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5dc179f7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
|
||||
"\n",
|
||||
"In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, and in each column put a 1 for the intercept, the montly income and the number of children."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 59,
|
||||
"id": "e5ff2a69",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 60,
|
||||
"id": "a3cf2792",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 20\n",
|
||||
"income = np.array([116., 161., 167., 118., 172., 163., 179., 173., 162., 116., 101., 176., 178., 172., 143., 135., 160., 101., 149., 125.])\n",
|
||||
"children = np.array([5, 3, 0, 4, 5, 3, 0, 4, 4, 3, 3, 5, 1, 0, 2, 3, 2, 1, 5, 4])\n",
|
||||
"spending = np.array([152., 141., 102., 136., 161., 129., 99., 159., 160., 107., 98., 164., 121., 93., 112., 127., 117., 69., 156., 131.])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5da61481",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Create a feature matrix $\\boldsymbol{X}$ for the features income and children, including an intercept column of ones at the start."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 65,
|
||||
"id": "5ad87a65",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = np.zeros((n, 3))\n",
|
||||
"#X[:, 0] = ...\n",
|
||||
"#X[:, 1] = ...\n",
|
||||
"#X[:, 2] = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0ddfac2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 37,
|
||||
"id": "8f3f68aa",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def OLS_parameters(X, y):\n",
|
||||
" return ...\n",
|
||||
"\n",
|
||||
"#beta = OLS_parameters(X, y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cb6da80",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 4 - Fitting a polynomial"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "71015064",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this course, we typically do linear regression using polynomials, though in real world applications it is also very common to make linear models based on measured features like you did in the previous exercise.\n",
|
||||
"\n",
|
||||
"When fitting a polynomial with linear regression, we make each polynomial degree($x, x^2, x^3, ..., x^p$) its own feature."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 38,
|
||||
"id": "d7476c84",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 100\n",
|
||||
"x = np.linspace(-3, 3, n)\n",
|
||||
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8321451b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Create a feature matrix $\\boldsymbol{X}$ for the features $x, x^2, x^3, x^4, x^5$, including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 66,
|
||||
"id": "91496e40",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def polynomial_features(x, p):\n",
|
||||
" n = len(x)\n",
|
||||
" X = np.zeros((n, p + 1))\n",
|
||||
" #X[:, 0] = ...\n",
|
||||
" #X[:, 1] = ...\n",
|
||||
" #X[:, 2] = ...\n",
|
||||
" # could this be a loop?\n",
|
||||
"\n",
|
||||
"#X = polynomial_features(x, 5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b84b1e31",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting $\\boldsymbol{y}$ based on these features. If you have done everything right so far, this code will not need changing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"id": "034f502c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#beta = OLS_parameters(X, y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d703f788",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Like in exercise 4 last week, split your feature matrix and target data into a training split and test split."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 68,
|
||||
"id": "29171358",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"#X_train, X_test, y_train, y_test = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a0e3509f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**d)** Train your model on the training data(find the parameters which best fit) and compute the MSE on both the training and test data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 42,
|
||||
"id": "1e346f4c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Ellipsis"
|
||||
]
|
||||
},
|
||||
"execution_count": 42,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e431889",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**e)** Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf). Feel free to read the discussions leading to figure 2.11 of Hastie et al. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 43,
|
||||
"id": "ceb57457",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Ellipsis"
|
||||
]
|
||||
},
|
||||
"execution_count": 43,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5e5b5954",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**f)** Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ad2acfb9",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5994f0c5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 5 - Comparing your code with sklearn"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f595b7a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When implementing different algorithms for the first time, it can be helpful to double check your results with established implementations before you go on to add more complexity."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8ab310c1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Make sure your `polynomial_features` function creates the same feature matrix as sklearns PolynomialFeatures.\n",
|
||||
"\n",
|
||||
"(https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "85b964d1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73c32c52",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Make sure your `OLS_parameters` function computes the same parameters as sklearns LinearRegression with fit_intercept set to False, since the intercept is included in the feature matrix. Use `your_model_object.coef_` to extract the computed parameters.\n",
|
||||
"\n",
|
||||
"(https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "35b04126",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.13.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1,303 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7791881e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
||||
"doconce format html exercisesweek34.do.txt -->\n",
|
||||
"<!-- dom:TITLE: Exercises week 34 -->"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd145f2c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"# Exercises week 34\n",
|
||||
"**FYS-STK3155/4155**\n",
|
||||
"\n",
|
||||
"Date: **August 19-23, 2024**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5543700e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercises\n",
|
||||
"\n",
|
||||
"Here are three possible exercises for week 34"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e490470",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 1: Setting up various Python environments\n",
|
||||
"\n",
|
||||
"The first exercise here is of a mere technical art. We want you to have \n",
|
||||
"* git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo [GitHub facilities](https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html). \n",
|
||||
"\n",
|
||||
"* Install various Python packages\n",
|
||||
"\n",
|
||||
"We will make extensive use of Python as programming language and its\n",
|
||||
"myriad of available libraries. You will find\n",
|
||||
"IPython/Jupyter notebooks invaluable in your work. You can run **R**\n",
|
||||
"codes in the Jupyter/IPython notebooks, with the immediate benefit of\n",
|
||||
"visualizing your data. You can also use compiled languages like C++,\n",
|
||||
"Rust, Fortran etc if you prefer. The focus in these lectures will be\n",
|
||||
"on Python.\n",
|
||||
"\n",
|
||||
"If you have Python installed (we recommend Python3) and you feel\n",
|
||||
"pretty familiar with installing different packages, we recommend that\n",
|
||||
"you install the following Python packages via **pip** as \n",
|
||||
"\n",
|
||||
"1. pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow \n",
|
||||
"\n",
|
||||
"For **Tensorflow**, we recommend following the instructions in the text of \n",
|
||||
"[Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly](http://shop.oreilly.com/product/0636920052289.do)\n",
|
||||
"\n",
|
||||
"We will come back to **tensorflow** later. \n",
|
||||
"\n",
|
||||
"For Python3, replace **pip** with **pip3**.\n",
|
||||
"\n",
|
||||
"For OSX users we recommend, after having installed Xcode, to\n",
|
||||
"install **brew**. Brew allows for a seamless installation of additional\n",
|
||||
"software via for example \n",
|
||||
"\n",
|
||||
"1. brew install python3\n",
|
||||
"\n",
|
||||
"For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,\n",
|
||||
"you can use **pip** as well and simply install Python as \n",
|
||||
"\n",
|
||||
"1. sudo apt-get install python3 (or python for Python2.7)\n",
|
||||
"\n",
|
||||
"If you don't want to perform these operations separately and venture\n",
|
||||
"into the hassle of exploring how to set up dependencies and paths, we\n",
|
||||
"recommend two widely used distrubutions which set up all relevant\n",
|
||||
"dependencies for Python, namely \n",
|
||||
"\n",
|
||||
"* [Anaconda](https://docs.anaconda.com/), \n",
|
||||
"\n",
|
||||
"which is an open source\n",
|
||||
"distribution of the Python and R programming languages for large-scale\n",
|
||||
"data processing, predictive analytics, and scientific computing, that\n",
|
||||
"aims to simplify package management and deployment. Package versions\n",
|
||||
"are managed by the package management system **conda**. \n",
|
||||
"\n",
|
||||
"* [Enthought canopy](https://www.enthought.com/product/canopy/) \n",
|
||||
"\n",
|
||||
"is a Python\n",
|
||||
"distribution for scientific and analytic computing distribution and\n",
|
||||
"analysis environment, available for free and under a commercial\n",
|
||||
"license.\n",
|
||||
"\n",
|
||||
"We recommend using **Anaconda** if you are not too familiar with setting paths in a terminal environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5b1c397f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 2: making your own data and exploring scikit-learn\n",
|
||||
"\n",
|
||||
"We will generate our own dataset for a function $y(x)$ where $x \\in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\\cal {N}(0,1)$.\n",
|
||||
"The following simple Python instructions define our $x$ and $y$ values (with 100 data points)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "eb74f73c",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"x = np.random.rand(100,1)\n",
|
||||
"y = 2.0+5*x*x+0.1*np.random.randn(100,1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "49257b1a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"1. Write your own code (following the examples under the [regression notes](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter1.html)) for computing the parametrization of the data set fitting a second-order polynomial. \n",
|
||||
"\n",
|
||||
"2. Use thereafter **scikit-learn** (see again the examples in the regression slides) and compare with your own code. \n",
|
||||
"\n",
|
||||
"3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3bfa5d7b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n",
|
||||
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5df6889b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"and the $R^2$ score function.\n",
|
||||
"If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2f021f71",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7b91a34b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"where we have defined the mean value of $\\boldsymbol{y}$ as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "58da66a3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2d3dc22b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions. \n",
|
||||
"Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4527a3e2",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 3: Split data in test and training data\n",
|
||||
"\n",
|
||||
"In this exercise we want you to to compute the MSE for the training\n",
|
||||
"data and the test data as function of the complexity of a polynomial,\n",
|
||||
"that is the degree of a given polynomial.\n",
|
||||
"\n",
|
||||
"The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf).\n",
|
||||
"\n",
|
||||
"Our data is defined by $x\\in [-3,3]$ with a total of for example $n=100$ data points. You should try to vary the number of data points $n$ in your analysis."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "87ac310b",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"np.random.seed()\n",
|
||||
"n = 100\n",
|
||||
"# Make data set.\n",
|
||||
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
|
||||
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2b923c6f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"where $y$ is the function we want to fit with a given polynomial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e62af227",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**a)**\n",
|
||||
"Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial and split your data set in training and test data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8df2cfd3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**b)**\n",
|
||||
"Write thereafter (using either **scikit-learn** or your matrix inversion code using for example **numpy**)\n",
|
||||
"and perform an ordinary least squares fitting and compute the mean squared error for the training data and the test data. These calculations should apply to a model given by a fifth-order polynomial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "01a227d5",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**c)**\n",
|
||||
"Add now a model which allows you to make polynomials up to degree $15$. Perform a standard OLS fitting of the training data and compute the MSE for the training and test data and plot both test and training data MSE as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -62,7 +62,7 @@
|
||||
<script>DOCUMENTATION_OPTIONS.pagename = 'chapter13';</script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="next" title="Exercises week 34" href="exercisesweek34.html" />
|
||||
<link rel="next" title="Exercises week 34" href="E1.html" />
|
||||
<link rel="prev" title="16. Convolutional Neural Networks" href="chapter12.html" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<meta name="docsearch:language" content="en"/>
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
@@ -1729,7 +1730,7 @@ plot_results(results, plot_number)
|
||||
</div>
|
||||
</a>
|
||||
<a class="right-next"
|
||||
href="exercisesweek34.html"
|
||||
href="E1.html"
|
||||
title="next page">
|
||||
<div class="prev-next-info">
|
||||
<p class="prev-next-subtitle">next</p>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -220,8 +220,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -224,8 +224,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
Binary file not shown.
@@ -221,8 +221,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -222,8 +222,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -223,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -221,8 +221,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -221,8 +221,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="week34.html">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
@@ -62,7 +62,8 @@
|
||||
<script>DOCUMENTATION_OPTIONS.pagename = 'week34';</script>
|
||||
<link rel="index" title="Index" href="genindex.html" />
|
||||
<link rel="search" title="Search" href="search.html" />
|
||||
<link rel="prev" title="Exercises week 34" href="exercisesweek34.html" />
|
||||
<link rel="next" title="Exercises week 35" href="E2.html" />
|
||||
<link rel="prev" title="Exercises week 34" href="E1.html" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
||||
<meta name="docsearch:language" content="en"/>
|
||||
</head>
|
||||
@@ -222,8 +223,9 @@
|
||||
</ul>
|
||||
<p aria-level="2" class="caption" role="heading"><span class="caption-text">Weekly material, notes and exercises</span></p>
|
||||
<ul class="current nav bd-sidenav">
|
||||
<li class="toctree-l1"><a class="reference internal" href="exercisesweek34.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E1.html">Exercises week 34</a></li>
|
||||
<li class="toctree-l1 current active"><a class="current reference internal" href="#">Week 34: Introduction to the course, Logistics and Practicalities</a></li>
|
||||
<li class="toctree-l1"><a class="reference internal" href="E2.html">Exercises week 35</a></li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
@@ -2801,7 +2803,7 @@ Add now a model which allows you to make polynomials up to degree <span class="m
|
||||
|
||||
<div class="prev-next-area">
|
||||
<a class="left-prev"
|
||||
href="exercisesweek34.html"
|
||||
href="E1.html"
|
||||
title="previous page">
|
||||
<i class="fa-solid fa-angle-left"></i>
|
||||
<div class="prev-next-info">
|
||||
@@ -2809,6 +2811,15 @@ Add now a model which allows you to make polynomials up to degree <span class="m
|
||||
<p class="prev-next-title">Exercises week 34</p>
|
||||
</div>
|
||||
</a>
|
||||
<a class="right-next"
|
||||
href="E2.html"
|
||||
title="next page">
|
||||
<div class="prev-next-info">
|
||||
<p class="prev-next-subtitle">next</p>
|
||||
<p class="prev-next-title">Exercises week 35</p>
|
||||
</div>
|
||||
<i class="fa-solid fa-angle-right"></i>
|
||||
</a>
|
||||
</div>
|
||||
</footer>
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,535 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b4005770",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Exercises week 35\n",
|
||||
"\n",
|
||||
"## Deriving and Implementing Ordinary Least Squares"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2ca1b589",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This week you will be deriving the analytical expressions for linear regression, building up the model from scratch. This will include taking several derivatives of products of vectors and matrices. Such derivatives are central to the optimization of many machine learning models. Although we will often use automatic differentiation in actual calculations, to be able to have analytical expressions is extremely helpful in case we have simpler derivatives as well as when we analyze various properties (like second derivatives) of the chosen cost functions.\n",
|
||||
"\n",
|
||||
"Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. You will find useful the notes from week 35 on derivatives of vectors and matrices. See also the textbook of Faisal at al, chapter 5 and in particular sections 5.3-5.5 at <https://github.com/CompPhysics/MachineLearning/blob/master/doc/Textbooks/MathMLbook.pdf>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "41e92bf9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Learning goals\n",
|
||||
"\n",
|
||||
"After completing these exercises, you will know how to\n",
|
||||
"- Take the derivatives of simple products between vectors and matrices\n",
|
||||
"- Implement OLS using the analytical expressions\n",
|
||||
"- Create a feature matrix from a set of data\n",
|
||||
"- Create a feature matrix for a polynomial model\n",
|
||||
"- Evaluate the MSE score of various model on training and test data, and comparing their performance\n",
|
||||
"\n",
|
||||
"### Deliverables\n",
|
||||
"\n",
|
||||
"Complete the following exercises while working in a jupyter notebook. Then, in canvas, include\n",
|
||||
"- The jupyter notebook with the exercises completed\n",
|
||||
"- An exported PDF of the notebook (https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_export-your-jupyter-notebook)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f7a9209d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How to take derivatives of Matrix-Vector expressions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "45f3712e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In these exercises it is always useful to write out with summation indices the various quantities. Take also a look at the weekly slides from week 35 and the various examples included there.\n",
|
||||
"\n",
|
||||
"As an example, consider the function\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f(\\boldsymbol{x}) =\\boldsymbol{A}\\boldsymbol{x},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which reads for a specific component $f_i$ (we define the matrix $\\boldsymbol{A}$ to have dimension $n\\times n$ and the vector $\\boldsymbol{x}$ to have length $n$)\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"f_i =\\sum_{j=0}^{n-1}a_{ij}x_j,\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which leads to\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial f_i}{\\partial x_j}= a_{ij},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"and written out in terms of the vector $\\boldsymbol{x}$ we have\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial f(\\boldsymbol{x})}{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5fa8a4e6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 1 - Finding the derivative of Matrix-Vector expressions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "df7a2270",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Consider the expression\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}},\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"Where $\\boldsymbol{a}$ and $\\boldsymbol{x}$ are column-vectors with length $n$.\n",
|
||||
"\n",
|
||||
"What is the *shape* of the expression we are taking the derivative of?\n",
|
||||
"\n",
|
||||
"What is the *shape* of the thing we are taking the derivative with respect to?\n",
|
||||
"\n",
|
||||
"What is the *shape* of the result of the expression?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "c0396734",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dc39d541",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{A}\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{a}^T(\\boldsymbol{A}+\\boldsymbol{A}^T),\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "498d13ec",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 2 - Deriving the expression for OLS"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f3f771de",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The ordinary least squares method finds the parameters $\\boldsymbol{\\beta}$ which minimizes the squared error between our model $\\boldsymbol{X\\beta}$ and the true values $\\boldsymbol{y}$.\n",
|
||||
"\n",
|
||||
"To find the parameters $\\boldsymbol{\\beta}$ which minimizes this error, we take the derivative of the squared error expression with respect to $\\boldsymbol{\\beta}$, and set it equal to 0."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "49690237",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Very briefly explain why the approach above finds the parameters $\\boldsymbol{\\beta}$ which minimizes this error."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b7cccc9d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We typically write the squared error as\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\vert\\vert\\boldsymbol{y} - \\boldsymbol{X\\beta}\\vert\\vert^2\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"which we can rewrite in matrix-vector form as\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8fbecf74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** If $\\boldsymbol{X}$ is invertible, what is the expression for the optimal parameters $\\boldsymbol{\\beta}$? (**Hint:** Don't compute any derivatives, but solve $\\boldsymbol{X\\beta}=\\boldsymbol{y}$ for $\\boldsymbol{\\beta}$)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f37af8f0",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Show that\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial \\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)}{\\partial \\boldsymbol{s}} = -2\\left(\\boldsymbol{x}-\\boldsymbol{A}\\boldsymbol{s}\\right)^T\\boldsymbol{A},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "869fca4d",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**d)** Using the expression from **c)**, but substituting back in $\\boldsymbol{\\beta}$, $\\boldsymbol{y}$ and $\\boldsymbol{X}$, find the expression for the optimal parameters $\\boldsymbol{\\beta}$ in the case that $\\boldsymbol{X}$ is not invertible, but $\\boldsymbol{X^T X}$ is, which is most often the case.\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\boldsymbol{\\hat{\\beta}_{OLS}} = ...\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "57ca3d74",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 3 - Creating feature matrix and implementing OLS using the analytical expression"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5dc179f7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"With the expression for $\\boldsymbol{\\hat{\\beta}_{OLS}}$, you now have what you need to implement OLS regression with your input data and target data $\\boldsymbol{y}$. But before you can do that, you need to set up you input data as a feature matrix $\\boldsymbol{X}$.\n",
|
||||
"\n",
|
||||
"In a feature matrix, each row is a datapoint and each column is a feature of that data. If you want to predict someones spending based on their income and number of children, for instance, you would create a row for each person in your dataset, and in each column put a 1 for the intercept, the montly income and the number of children."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 59,
|
||||
"id": "e5ff2a69",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 60,
|
||||
"id": "a3cf2792",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 20\n",
|
||||
"income = np.array([116., 161., 167., 118., 172., 163., 179., 173., 162., 116., 101., 176., 178., 172., 143., 135., 160., 101., 149., 125.])\n",
|
||||
"children = np.array([5, 3, 0, 4, 5, 3, 0, 4, 4, 3, 3, 5, 1, 0, 2, 3, 2, 1, 5, 4])\n",
|
||||
"spending = np.array([152., 141., 102., 136., 161., 129., 99., 159., 160., 107., 98., 164., 121., 93., 112., 127., 117., 69., 156., 131.])\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5da61481",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Create a feature matrix $\\boldsymbol{X}$ for the features income and children, including an intercept column of ones at the start."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 65,
|
||||
"id": "5ad87a65",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"X = np.zeros((n, 3))\n",
|
||||
"#X[:, 0] = ...\n",
|
||||
"#X[:, 1] = ...\n",
|
||||
"#X[:, 2] = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e0ddfac2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting spending based on these features. Create a function for this operation, as you are going to need to use it a lot."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 37,
|
||||
"id": "8f3f68aa",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def OLS_parameters(X, y):\n",
|
||||
" return ...\n",
|
||||
"\n",
|
||||
"#beta = OLS_parameters(X, y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0cb6da80",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 4 - Fitting a polynomial"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "71015064",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"In this course, we typically do linear regression using polynomials, though in real world applications it is also very common to make linear models based on measured features like you did in the previous exercise.\n",
|
||||
"\n",
|
||||
"When fitting a polynomial with linear regression, we make each polynomial degree($x, x^2, x^3, ..., x^p$) its own feature."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 38,
|
||||
"id": "d7476c84",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"n = 100\n",
|
||||
"x = np.linspace(-3, 3, n)\n",
|
||||
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2) + np.random.normal(0, 0.1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8321451b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Create a feature matrix $\\boldsymbol{X}$ for the features $x, x^2, x^3, x^4, x^5$, including an intercept column of ones at the start. Make this into a function, as you will do this a lot over the next weeks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 66,
|
||||
"id": "91496e40",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def polynomial_features(x, p):\n",
|
||||
" n = len(x)\n",
|
||||
" X = np.zeros((n, p + 1))\n",
|
||||
" #X[:, 0] = ...\n",
|
||||
" #X[:, 1] = ...\n",
|
||||
" #X[:, 2] = ...\n",
|
||||
" # could this be a loop?\n",
|
||||
"\n",
|
||||
"#X = polynomial_features(x, 5)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "b84b1e31",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Use the expression from **3d)** to find the optimal parameters $\\boldsymbol{\\hat{\\beta}_{OLS}}$ for predicting $\\boldsymbol{y}$ based on these features. If you have done everything right so far, this code will not need changing."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 67,
|
||||
"id": "034f502c",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#beta = OLS_parameters(X, y)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d703f788",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**c)** Like in exercise 4 last week, split your feature matrix and target data into a training split and test split."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 68,
|
||||
"id": "29171358",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"\n",
|
||||
"#X_train, X_test, y_train, y_test = ..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "a0e3509f",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**d)** Train your model on the training data(find the parameters which best fit) and compute the MSE on both the training and test data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 42,
|
||||
"id": "1e346f4c",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Ellipsis"
|
||||
]
|
||||
},
|
||||
"execution_count": 42,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7e431889",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**e)** Do the same for each polynomial degree from 2 to 10, and plot the MSE on both the training and test data as a function of polynomial degree. The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf). Feel free to read the discussions leading to figure 2.11 of Hastie et al. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 43,
|
||||
"id": "ceb57457",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Ellipsis"
|
||||
]
|
||||
},
|
||||
"execution_count": 43,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"..."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5e5b5954",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**f)** Interpret the graph. Why do the lines move as they do? What does it tell us about model performance and generalizability?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ad2acfb9",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5994f0c5",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Exercise 5 - Comparing your code with sklearn"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8f595b7a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"When implementing different algorithms for the first time, it can be helpful to double check your results with established implementations before you go on to add more complexity."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8ab310c1",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**a)** Make sure your `polynomial_features` function creates the same feature matrix as sklearns PolynomialFeatures.\n",
|
||||
"\n",
|
||||
"(https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "85b964d1",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "73c32c52",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**b)** Make sure your `OLS_parameters` function computes the same parameters as sklearns LinearRegression with fit_intercept set to False, since the intercept is included in the feature matrix. Use `your_model_object.coef_` to extract the computed parameters.\n",
|
||||
"\n",
|
||||
"(https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "35b04126",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": ".venv",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.13.0"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -1,303 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7791881e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
||||
"doconce format html exercisesweek34.do.txt -->\n",
|
||||
"<!-- dom:TITLE: Exercises week 34 -->"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "fd145f2c",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"# Exercises week 34\n",
|
||||
"**FYS-STK3155/4155**\n",
|
||||
"\n",
|
||||
"Date: **August 19-23, 2024**"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5543700e",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercises\n",
|
||||
"\n",
|
||||
"Here are three possible exercises for week 34"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "0e490470",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 1: Setting up various Python environments\n",
|
||||
"\n",
|
||||
"The first exercise here is of a mere technical art. We want you to have \n",
|
||||
"* git as a version control software and to establish a user account on a provider like GitHub. Other providers like GitLab etc are equally fine. You can also use the University of Oslo [GitHub facilities](https://www.uio.no/tjenester/it/maskin/filer/versjonskontroll/github.html). \n",
|
||||
"\n",
|
||||
"* Install various Python packages\n",
|
||||
"\n",
|
||||
"We will make extensive use of Python as programming language and its\n",
|
||||
"myriad of available libraries. You will find\n",
|
||||
"IPython/Jupyter notebooks invaluable in your work. You can run **R**\n",
|
||||
"codes in the Jupyter/IPython notebooks, with the immediate benefit of\n",
|
||||
"visualizing your data. You can also use compiled languages like C++,\n",
|
||||
"Rust, Fortran etc if you prefer. The focus in these lectures will be\n",
|
||||
"on Python.\n",
|
||||
"\n",
|
||||
"If you have Python installed (we recommend Python3) and you feel\n",
|
||||
"pretty familiar with installing different packages, we recommend that\n",
|
||||
"you install the following Python packages via **pip** as \n",
|
||||
"\n",
|
||||
"1. pip install numpy scipy matplotlib ipython scikit-learn sympy pandas pillow \n",
|
||||
"\n",
|
||||
"For **Tensorflow**, we recommend following the instructions in the text of \n",
|
||||
"[Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow, O'Reilly](http://shop.oreilly.com/product/0636920052289.do)\n",
|
||||
"\n",
|
||||
"We will come back to **tensorflow** later. \n",
|
||||
"\n",
|
||||
"For Python3, replace **pip** with **pip3**.\n",
|
||||
"\n",
|
||||
"For OSX users we recommend, after having installed Xcode, to\n",
|
||||
"install **brew**. Brew allows for a seamless installation of additional\n",
|
||||
"software via for example \n",
|
||||
"\n",
|
||||
"1. brew install python3\n",
|
||||
"\n",
|
||||
"For Linux users, with its variety of distributions like for example the widely popular Ubuntu distribution,\n",
|
||||
"you can use **pip** as well and simply install Python as \n",
|
||||
"\n",
|
||||
"1. sudo apt-get install python3 (or python for Python2.7)\n",
|
||||
"\n",
|
||||
"If you don't want to perform these operations separately and venture\n",
|
||||
"into the hassle of exploring how to set up dependencies and paths, we\n",
|
||||
"recommend two widely used distrubutions which set up all relevant\n",
|
||||
"dependencies for Python, namely \n",
|
||||
"\n",
|
||||
"* [Anaconda](https://docs.anaconda.com/), \n",
|
||||
"\n",
|
||||
"which is an open source\n",
|
||||
"distribution of the Python and R programming languages for large-scale\n",
|
||||
"data processing, predictive analytics, and scientific computing, that\n",
|
||||
"aims to simplify package management and deployment. Package versions\n",
|
||||
"are managed by the package management system **conda**. \n",
|
||||
"\n",
|
||||
"* [Enthought canopy](https://www.enthought.com/product/canopy/) \n",
|
||||
"\n",
|
||||
"is a Python\n",
|
||||
"distribution for scientific and analytic computing distribution and\n",
|
||||
"analysis environment, available for free and under a commercial\n",
|
||||
"license.\n",
|
||||
"\n",
|
||||
"We recommend using **Anaconda** if you are not too familiar with setting paths in a terminal environment."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5b1c397f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 2: making your own data and exploring scikit-learn\n",
|
||||
"\n",
|
||||
"We will generate our own dataset for a function $y(x)$ where $x \\in [0,1]$ and defined by random numbers computed with the uniform distribution. The function $y$ is a quadratic polynomial in $x$ with added stochastic noise according to the normal distribution $\\cal {N}(0,1)$.\n",
|
||||
"The following simple Python instructions define our $x$ and $y$ values (with 100 data points)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "eb74f73c",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import numpy as np\n",
|
||||
"x = np.random.rand(100,1)\n",
|
||||
"y = 2.0+5*x*x+0.1*np.random.randn(100,1)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "49257b1a",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"1. Write your own code (following the examples under the [regression notes](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter1.html)) for computing the parametrization of the data set fitting a second-order polynomial. \n",
|
||||
"\n",
|
||||
"2. Use thereafter **scikit-learn** (see again the examples in the regression slides) and compare with your own code. \n",
|
||||
"\n",
|
||||
"3. Using scikit-learn, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3bfa5d7b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"MSE(\\boldsymbol{y},\\boldsymbol{\\tilde{y}}) = \\frac{1}{n}\n",
|
||||
"\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5df6889b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"and the $R^2$ score function.\n",
|
||||
"If $\\tilde{\\boldsymbol{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2f021f71",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"R^2(\\boldsymbol{y}, \\tilde{\\boldsymbol{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7b91a34b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"where we have defined the mean value of $\\boldsymbol{y}$ as"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "58da66a3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2d3dc22b",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions. \n",
|
||||
"Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4527a3e2",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"## Exercise 3: Split data in test and training data\n",
|
||||
"\n",
|
||||
"In this exercise we want you to to compute the MSE for the training\n",
|
||||
"data and the test data as function of the complexity of a polynomial,\n",
|
||||
"that is the degree of a given polynomial.\n",
|
||||
"\n",
|
||||
"The aim is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf).\n",
|
||||
"\n",
|
||||
"Our data is defined by $x\\in [-3,3]$ with a total of for example $n=100$ data points. You should try to vary the number of data points $n$ in your analysis."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "87ac310b",
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"np.random.seed()\n",
|
||||
"n = 100\n",
|
||||
"# Make data set.\n",
|
||||
"x = np.linspace(-3, 3, n).reshape(-1, 1)\n",
|
||||
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "2b923c6f",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"where $y$ is the function we want to fit with a given polynomial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "e62af227",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**a)**\n",
|
||||
"Write a first code which sets up a design matrix $X$ defined by a fifth-order polynomial and split your data set in training and test data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "8df2cfd3",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**b)**\n",
|
||||
"Write thereafter (using either **scikit-learn** or your matrix inversion code using for example **numpy**)\n",
|
||||
"and perform an ordinary least squares fitting and compute the mean squared error for the training data and the test data. These calculations should apply to a model given by a fifth-order polynomial."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "01a227d5",
|
||||
"metadata": {
|
||||
"editable": true
|
||||
},
|
||||
"source": [
|
||||
"**c)**\n",
|
||||
"Add now a model which allows you to make polynomials up to degree $15$. Perform a standard OLS fitting of the training data and compute the MSE for the training and test data and plot both test and training data MSE as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 18 KiB |
@@ -41,5 +41,6 @@ parts:
|
||||
- caption: Weekly material, notes and exercises
|
||||
numbered: false
|
||||
chapters:
|
||||
- file: exercisesweek34.ipynb
|
||||
- file: E1.ipynb
|
||||
- file: week34.ipynb
|
||||
- file: E2.ipynb
|
||||
|
||||
Reference in New Issue
Block a user