Morten Hjorth-Jensen
373 lines
11 KiB
Plaintext
373 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba6ac392",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html exercisesweek36.do.txt -->\n",
|
|
"<!-- dom:TITLE: Exercises week 36 -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b6e88f71",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Exercises week 36\n",
|
|
"**September 2-6, 2024**\n",
|
|
"\n",
|
|
"Date: **Deadline is Friday September 6 at midnight**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bdcd6bb3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Overarching aims of the exercises this week\n",
|
|
"\n",
|
|
"This set of exercises form an important part of the first project. The\n",
|
|
"analytical exercises deal with the material covered last week on the\n",
|
|
"mathematical interpretations of ordinary least squares and of Ridge\n",
|
|
"regression. The numerical exercises can be seen as a continuation of\n",
|
|
"exercise 3 from week 35, with the inclusion of Ridge regression. This\n",
|
|
"material enters also the discussions of the first project."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "06e77343",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Exercise 1: Analytical exercises\n",
|
|
"\n",
|
|
"The aim here is to derive the expression for the optimal parameters\n",
|
|
"using Ridge regression. Furthermore, using the singular value\n",
|
|
"decomposition, we will analyze the difference between the ordinary\n",
|
|
"least squares approach and Ridge regression.\n",
|
|
"\n",
|
|
"The expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, was given by the\n",
|
|
"optimization problem"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5025274e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "900798b7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which we can also write as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c3916302",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
|
|
"{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "52a3d66c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where we have used the definition of a norm-2 vector, that is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e537f20e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "71e5f3a3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"By minimizing the above equation with respect to the parameters\n",
|
|
"$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
|
|
"parameters $\\boldsymbol{\\beta}$.\n",
|
|
"\n",
|
|
"We can add a regularization parameter $\\lambda$ by\n",
|
|
"defining a new cost function to be optimized, that is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "26fe46d4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
|
|
"{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "13bacbe7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"which leads to the Ridge regression minimization problem. One can require as part of the optimization problem \n",
|
|
"that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
|
|
"a finite number larger than zero. We will not implement that here."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3ca7c852",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### a) Expression for Ridge regression\n",
|
|
"\n",
|
|
"Show that the optimal parameters"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c6c84292",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "15007b26",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix.\n",
|
|
"\n",
|
|
"The ordinary least squares result is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "58f20595",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "aab6662a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### b) The singular value decomposition\n",
|
|
"\n",
|
|
"Here we will use the singular value decomposition of an $n\\times p$ matrix $\\boldsymbol{X}$ (our design matrix)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "50358368",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\boldsymbol{X}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a87efdb8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"to study properties of Ridge regression and ordinary least squares regression.\n",
|
|
"Here $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are orthogonal matrices of dimensions\n",
|
|
"$n\\times n$ and $p\\times p$, respectively, and $\\boldsymbol{\\Sigma}$ is an\n",
|
|
"$n\\times p$ matrix which contains the singular values only. This material was discussed during the lectures of week 35.\n",
|
|
"\n",
|
|
"Show that you can write the \n",
|
|
"OLS solutions in terms of the eigenvectors (the columns) of the orthogonal matrix $\\boldsymbol{U}$ as"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "23d5e69c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} = \\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\boldsymbol{y}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f1086e51",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"For Ridge regression, show that the corresponding equation is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8cad7d30",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a7e167a0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$. \n",
|
|
"\n",
|
|
"Give an interpretation of the results. [Section 3.4 of Hastie et al's textbook gives a good discussion of the above results](https://link.springer.com/book/10.1007/978-0-387-84858-7)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a7482960",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Exercise 2: Adding Ridge Regression\n",
|
|
"\n",
|
|
"This exercise is a continuation of exercise 3 from week 35, see <https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek35.html>. We will use the same function to\n",
|
|
"generate our data set, still staying with a simple function $y(x)$\n",
|
|
"which we want to fit using linear regression, but now extending the\n",
|
|
"analysis to include the Ridge regression method.\n",
|
|
"\n",
|
|
"In this exercise you need to include the same elements from last week, that is\n",
|
|
"1. scale your data by subtracting the mean value from each column in the design matrix.\n",
|
|
"\n",
|
|
"2. perform a split of the data in a training set and a test set.\n",
|
|
"\n",
|
|
"The addition to the analysis this time is the introduction of the hyperparameter $\\lambda$ when introducing Ridge regression.\n",
|
|
"\n",
|
|
"Extend the code from exercise 3 from [week 35](https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/exercisesweek35.html) to include Ridge regression with the hyperparameter $\\lambda$. The optimal parameters $\\hat{\\beta}$ for Ridge regression can be obtained by matrix inversion in a similar way as done for ordinary least squares. You need to add to your code the following equations"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abcb7cf9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bb76fdeb",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The ordinary least squares result you encoded last week is given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "034da514",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8b5613a0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Use these results to compute the mean squared error for ordinary least\n",
|
|
"squares and Ridge regression first for a polynomial of degree five\n",
|
|
"with $n=100$ data points and five selected values of\n",
|
|
"$\\lambda=[0.0001,0.001, 0.01,0.1,1.0]$. Compute thereafter the mean\n",
|
|
"squared error for the same values of $\\lambda$ for polynomials of degree ten\n",
|
|
"and $15$. Discuss your results for the training MSE and test MSE with\n",
|
|
"Ridge regression and ordinary least squares."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|