diff --git a/doc/.DS_Store b/doc/.DS_Store index 83d1eeb6c..b09063e1b 100644 Binary files a/doc/.DS_Store and b/doc/.DS_Store differ diff --git a/doc/pub/week35/html/._week35-bs001.html b/doc/pub/week35/html/._week35-bs001.html index 0660798bd..0ccd71fcd 100644 --- a/doc/pub/week35/html/._week35-bs001.html +++ b/doc/pub/week35/html/._week35-bs001.html @@ -376,6 +376,7 @@ MathJax.Hub.Config({
  1. These lecture notes
  2. Video of lecture
    3. +
  3. Video for exercises week 35
  4. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)
  5. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.
  6. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week.
    7. diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 894906df6..c12fab77a 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -209,6 +209,7 @@ MathJax.Hub.Config({

    1. These lecture notes

    2. Video of lecture
      3. +

    3. Video for exercises week 35

    4. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)

    5. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.

    6. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week.
      7. diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index f7139726d..0133b9afc 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -325,6 +325,7 @@ MathJax.Hub.Config({
      1. These lecture notes
      2. Video of lecture
        3. +
      3. Video for exercises week 35
      4. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)
      5. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.
      6. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week.
        7. diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index f2ade0867..91d4d2d60 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -402,6 +402,7 @@ MathJax.Hub.Config({
        1. These lecture notes
        2. Video of lecture
          3. +
        3. Video for exercises week 35
        4. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)
        5. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.
        6. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week.
          7. diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 48782683d..ca74b965d 100644 Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 6c8f151eb..a2a42e29e 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "a4c8f3e7", - "metadata": {}, + "id": "4056272f", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "959e7c4f", - "metadata": {}, + "id": "bed7fa64", + "metadata": { + "editable": true + }, "source": [ "# Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo\n", @@ -23,8 +27,10 @@ }, { "cell_type": "markdown", - "id": "b6849cc6", - "metadata": {}, + "id": "f651b769", + "metadata": { + "editable": true + }, "source": [ "## Plans for week 35\n", "\n", @@ -43,8 +49,10 @@ }, { "cell_type": "markdown", - "id": "227526ee", - "metadata": {}, + "id": "3cc28c2f", + "metadata": { + "editable": true + }, "source": [ "### Reading recommendations:\n", "\n", @@ -52,17 +60,21 @@ "\n", "2. [Video of lecture](https://youtu.be/VKakN-e4aUA)\n", "\n", - "3. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)\n", + "3. [Video for exercises week 35](https://youtu.be/yiY0OltU1s8)\n", "\n", - "4. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.\n", + "4. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background)\n", "\n", - "5. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL\"https://mml-book.github.io/\" (section 5.5 on derivatives) is very useful for exercise 1 this coming week." + "5. Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4.\n", + "\n", + "6. For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL\"https://mml-book.github.io/\" (section 5.5 on derivatives) is very useful for exercise 1 this coming week." ] }, { "cell_type": "markdown", - "id": "7a23a33d", - "metadata": {}, + "id": "4ce22826", + "metadata": { + "editable": true + }, "source": [ "## For exercise sessions: Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week\n", "\n", @@ -93,8 +105,10 @@ }, { "cell_type": "markdown", - "id": "53a199d1", - "metadata": {}, + "id": "1d9df4fe", + "metadata": { + "editable": true + }, "source": [ "## The equations for ordinary least squares\n", "\n", @@ -107,8 +121,10 @@ }, { "cell_type": "markdown", - "id": "9b572594", - "metadata": {}, + "id": "7bc9d9b8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i=f(x_i)+\\epsilon_i,\n", @@ -117,16 +133,20 @@ }, { "cell_type": "markdown", - "id": "baddcb7b", - "metadata": {}, + "id": "16d390b6", + "metadata": { + "editable": true + }, "source": [ "or in general" ] }, { "cell_type": "markdown", - "id": "4831a2c7", - "metadata": {}, + "id": "4e4a28d3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}=f(\\boldsymbol{x})+\\boldsymbol{\\epsilon},\n", @@ -135,8 +155,10 @@ }, { "cell_type": "markdown", - "id": "7855acbd", - "metadata": {}, + "id": "eb6ac14f", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\epsilon}$ represents some noise which is normally assumed to\n", "be distributed via a normal probability distribution with zero mean\n", @@ -154,8 +176,10 @@ }, { "cell_type": "markdown", - "id": "e489c877", - "metadata": {}, + "id": "4da31a11", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -164,8 +188,10 @@ }, { "cell_type": "markdown", - "id": "8962de54", - "metadata": {}, + "id": "2231b9b8", + "metadata": { + "editable": true + }, "source": [ "and in order to find the optimal parameters $\\beta_i$ we defined a function which\n", "gives a measure of the spread between the values $y_i$ (which\n", @@ -175,8 +201,10 @@ }, { "cell_type": "markdown", - "id": "839560c0", - "metadata": {}, + "id": "dc2e84eb", + "metadata": { + "editable": true + }, "source": [ "## The cost/loss function\n", "\n", @@ -185,8 +213,10 @@ }, { "cell_type": "markdown", - "id": "19671b87", - "metadata": {}, + "id": "a2cd53fd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", @@ -195,16 +225,20 @@ }, { "cell_type": "markdown", - "id": "3b9e52d5", - "metadata": {}, + "id": "142e9e9d", + "metadata": { + "editable": true + }, "source": [ "or using the matrix $\\boldsymbol{X}$ and in a more compact matrix-vector notation as" ] }, { "cell_type": "markdown", - "id": "10c51b3d", - "metadata": {}, + "id": "a3d1bc30", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -213,8 +247,10 @@ }, { "cell_type": "markdown", - "id": "2beea594", - "metadata": {}, + "id": "4824b7b6", + "metadata": { + "editable": true + }, "source": [ "This function represents one of many possible ways to define the so-called cost function.\n", "\n", @@ -224,8 +260,10 @@ }, { "cell_type": "markdown", - "id": "45aa8758", - "metadata": {}, + "id": "aac7541e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n", @@ -234,16 +272,20 @@ }, { "cell_type": "markdown", - "id": "517f3b5e", - "metadata": {}, + "id": "68cb773e", + "metadata": { + "editable": true + }, "source": [ "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out." ] }, { "cell_type": "markdown", - "id": "1afd62d4", - "metadata": {}, + "id": "558ff41a", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "\n", @@ -252,8 +294,10 @@ }, { "cell_type": "markdown", - "id": "a4b4bb62", - "metadata": {}, + "id": "29815637", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n", @@ -262,8 +306,10 @@ }, { "cell_type": "markdown", - "id": "d8f50b66", - "metadata": {}, + "id": "48dac5ad", + "metadata": { + "editable": true + }, "source": [ "can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. \n", "When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value" @@ -271,8 +317,10 @@ }, { "cell_type": "markdown", - "id": "79350ccb", - "metadata": {}, + "id": "b06e9d96", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n", @@ -281,8 +329,10 @@ }, { "cell_type": "markdown", - "id": "c3d2eef3", - "metadata": {}, + "id": "edd5397a", + "metadata": { + "editable": true + }, "source": [ "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", "till now we have treated $y_i$ as the exact value. Normally, the\n", @@ -298,8 +348,10 @@ }, { "cell_type": "markdown", - "id": "80a776b1", - "metadata": {}, + "id": "5684ef05", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -309,16 +361,20 @@ }, { "cell_type": "markdown", - "id": "83e65671", - "metadata": {}, + "id": "3a97a3c6", + "metadata": { + "editable": true + }, "source": [ "In practical terms it means we will require" ] }, { "cell_type": "markdown", - "id": "dcd401a6", - "metadata": {}, + "id": "49ef3525", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n", @@ -327,16 +383,20 @@ }, { "cell_type": "markdown", - "id": "af49643b", - "metadata": {}, + "id": "9ecb7f5e", + "metadata": { + "editable": true + }, "source": [ "which results in" ] }, { "cell_type": "markdown", - "id": "f6cd4986", - "metadata": {}, + "id": "d817f6e0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n", @@ -345,16 +405,20 @@ }, { "cell_type": "markdown", - "id": "93a3774a", - "metadata": {}, + "id": "8bfba457", + "metadata": { + "editable": true + }, "source": [ "or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivation below)" ] }, { "cell_type": "markdown", - "id": "5cacc270", - "metadata": {}, + "id": "295374b3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", @@ -363,8 +427,10 @@ }, { "cell_type": "markdown", - "id": "affb01c0", - "metadata": {}, + "id": "a286d821", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "We can rewrite, see the derivations below," @@ -372,8 +438,10 @@ }, { "cell_type": "markdown", - "id": "35e50283", - "metadata": {}, + "id": "b82376cc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n", @@ -382,16 +450,20 @@ }, { "cell_type": "markdown", - "id": "7e137ec8", - "metadata": {}, + "id": "1bc91e3c", + "metadata": { + "editable": true + }, "source": [ "as" ] }, { "cell_type": "markdown", - "id": "9168a472", - "metadata": {}, + "id": "ef57a496", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -400,16 +472,20 @@ }, { "cell_type": "markdown", - "id": "42261983", - "metadata": {}, + "id": "ef6b79b3", + "metadata": { + "editable": true + }, "source": [ "and if the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ is invertible we have the solution" ] }, { "cell_type": "markdown", - "id": "ac934e5f", - "metadata": {}, + "id": "2bfe6ba6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -418,8 +494,10 @@ }, { "cell_type": "markdown", - "id": "a2018e5e", - "metadata": {}, + "id": "1ea9631f", + "metadata": { + "editable": true + }, "source": [ "We note also that since our design matrix is defined as $\\boldsymbol{X}\\in\n", "{\\mathbb{R}}^{n\\times p}$, the product $\\boldsymbol{X}^T\\boldsymbol{X} \\in\n", @@ -436,8 +514,10 @@ }, { "cell_type": "markdown", - "id": "f3d10b4e", - "metadata": {}, + "id": "40df8223", + "metadata": { + "editable": true + }, "source": [ "## Some useful matrix and vector expressions\n", "\n", @@ -461,8 +541,10 @@ }, { "cell_type": "markdown", - "id": "9ffdf547", - "metadata": {}, + "id": "e45dec6e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}=f(\\boldsymbol{x}).\n", @@ -471,8 +553,10 @@ }, { "cell_type": "markdown", - "id": "06fe59d3", - "metadata": {}, + "id": "044bc60d", + "metadata": { + "editable": true + }, "source": [ "## The Jacobian\n", "\n", @@ -481,8 +565,10 @@ }, { "cell_type": "markdown", - "id": "4fd53261", - "metadata": {}, + "id": "b54ca0eb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{J}=\\frac{\\partial \\boldsymbol{y}}{\\partial \\boldsymbol{x}}=\\begin{bmatrix} \\frac{\\partial y_0}{\\partial x_0} & \\frac{\\partial y_0}{\\partial x_1} & \\frac{\\partial y_0}{\\partial x_2} & \\dots & \\dots & \\frac{\\partial y_0}{\\partial x_{n-1}} \\\\ \\frac{\\partial y_1}{\\partial x_0} & \\frac{\\partial y_1}{\\partial x_1} & \\frac{\\partial y_1}{\\partial x_2} & \\dots & \\dots & \\frac{\\partial y_1}{\\partial x_{n-1}} \\\\\n", @@ -495,8 +581,10 @@ }, { "cell_type": "markdown", - "id": "9dac1f85", - "metadata": {}, + "id": "9bc22dd0", + "metadata": { + "editable": true + }, "source": [ "which is an $m\\times n$ matrix. If $\\boldsymbol{x}$ is a scalar, then the\n", "Jacobian is only a single-column vector, or an $m\\times 1$ matrix. If\n", @@ -511,8 +599,10 @@ }, { "cell_type": "markdown", - "id": "4ddbe0f3", - "metadata": {}, + "id": "9af4fcfe", + "metadata": { + "editable": true + }, "source": [ "## Derivatives, example 1\n", "\n", @@ -521,8 +611,10 @@ }, { "cell_type": "markdown", - "id": "b493f27d", - "metadata": {}, + "id": "84760094", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i = \\sum_{j=0}^{n-1}a_{ij}x_j,\n", @@ -531,8 +623,10 @@ }, { "cell_type": "markdown", - "id": "d8e52971", - "metadata": {}, + "id": "0d9a7ac6", + "metadata": { + "editable": true + }, "source": [ "with $\\forall i=0,1,2,\\dots,m-1$. The individual matrix elements of $\\boldsymbol{A}$ are given by the symbol $a_{ij}$.\n", "It follows that the partial derivatives of $y_i$ with respect to $x_k$" @@ -540,8 +634,10 @@ }, { "cell_type": "markdown", - "id": "2467254e", - "metadata": {}, + "id": "ccbe4c4e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial y_i }{\\partial x_k}= a_{ik} \\forall i=0,1,2,\\dots,m-1.\n", @@ -550,16 +646,20 @@ }, { "cell_type": "markdown", - "id": "ad1bb07b", - "metadata": {}, + "id": "1c8a9e7d", + "metadata": { + "editable": true + }, "source": [ "From this we have, using the definition of the Jacobian" ] }, { "cell_type": "markdown", - "id": "910a1e9c", - "metadata": {}, + "id": "ecb14896", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\boldsymbol{y} }{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n", @@ -568,8 +668,10 @@ }, { "cell_type": "markdown", - "id": "e21b9cd4", - "metadata": {}, + "id": "6578e470", + "metadata": { + "editable": true + }, "source": [ "## Example 2\n", "\n", @@ -580,8 +682,10 @@ }, { "cell_type": "markdown", - "id": "d7859263", - "metadata": {}, + "id": "227ea794", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{y}^T\\boldsymbol{A}\\boldsymbol{x},\n", @@ -590,8 +694,10 @@ }, { "cell_type": "markdown", - "id": "a73d4636", - "metadata": {}, + "id": "d88f1a80", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{y}$ a vector of length $m$, $\\boldsymbol{A}$ an $m\\times n$ matrix and $\\boldsymbol{x}$ a vector of length $n$. We assume also that $\\boldsymbol{A}$ does not depend on any of the two vectors.\n", "In order to find the derivative of $\\alpha$ with respect to the two vectors, we define an intermediate vector $\\boldsymbol{z}$. We define first\n", @@ -600,8 +706,10 @@ }, { "cell_type": "markdown", - "id": "ce30bf27", - "metadata": {}, + "id": "c6f9c72f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{z}^T\\boldsymbol{x},\n", @@ -610,16 +718,20 @@ }, { "cell_type": "markdown", - "id": "d38383b9", - "metadata": {}, + "id": "014a0ee3", + "metadata": { + "editable": true + }, "source": [ "which means that (using our previous example and keeping track of our definition of the derivative of a scalar) we have" ] }, { "cell_type": "markdown", - "id": "a5e969fe", - "metadata": {}, + "id": "bbd1cd50", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{x}} = \\frac{\\partial \\boldsymbol{z}^T\\boldsymbol{x}}{\\partial \\boldsymbol{x}}=\\boldsymbol{z}^T.\n", @@ -628,8 +740,10 @@ }, { "cell_type": "markdown", - "id": "05ad78b6", - "metadata": {}, + "id": "7129292e", + "metadata": { + "editable": true + }, "source": [ "Note that the resulting vector elements are the same for $\\boldsymbol{z}^T$ and $\\boldsymbol{z}$, the only difference is that one is just the transpose of the other. We have the transposed here since we have used that the inner product of two vectors is a scalar.\n", "\n", @@ -638,8 +752,10 @@ }, { "cell_type": "markdown", - "id": "c200bbd6", - "metadata": {}, + "id": "63264651", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{y}} = \\boldsymbol{z}^T=\\boldsymbol{x}^T\\boldsymbol{A}^T.\n", @@ -648,8 +764,10 @@ }, { "cell_type": "markdown", - "id": "0e76e759", - "metadata": {}, + "id": "d86227f5", + "metadata": { + "editable": true + }, "source": [ "## Example 3\n", "\n", @@ -660,8 +778,10 @@ }, { "cell_type": "markdown", - "id": "20f0ef9e", - "metadata": {}, + "id": "4b24b1a0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x},\n", @@ -670,8 +790,10 @@ }, { "cell_type": "markdown", - "id": "ccd6ef29", - "metadata": {}, + "id": "0329d0e5", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{x}$ a vector of length $n$.\n", "\n", @@ -680,8 +802,10 @@ }, { "cell_type": "markdown", - "id": "3d0021d4", - "metadata": {}, + "id": "60fdb497", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\sum_{i=0}^{n-1}\\sum_{j=0}^{n-1}x_i a_{ij}x_j,\n", @@ -690,16 +814,20 @@ }, { "cell_type": "markdown", - "id": "b93aec90", - "metadata": {}, + "id": "23ddc055", + "metadata": { + "editable": true + }, "source": [ "taking the derivative of $\\alpha$ with respect to a given component $x_k$ we get the two sums" ] }, { "cell_type": "markdown", - "id": "d7b53094", - "metadata": {}, + "id": "3cd9c608", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial x_k} = \\sum_{i=0}^{n-1}a_{ik}x_i+\\sum_{j=0}^{n-1}a_{kj}x_j,\n", @@ -708,16 +836,20 @@ }, { "cell_type": "markdown", - "id": "374bedac", - "metadata": {}, + "id": "798f4dbf", + "metadata": { + "editable": true + }, "source": [ "for $\\forall k =0,1,2,\\dots,n-1$. We identify these sums as" ] }, { "cell_type": "markdown", - "id": "578e857b", - "metadata": {}, + "id": "d4c7b4f8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{x}} = \\boldsymbol{x}^T\\left(\\boldsymbol{A}^T+\\boldsymbol{A}\\right).\n", @@ -726,16 +858,20 @@ }, { "cell_type": "markdown", - "id": "44b8e0d5", - "metadata": {}, + "id": "aaea62fe", + "metadata": { + "editable": true + }, "source": [ "If the matrix $\\boldsymbol{A}$ is symmetric, that is $\\boldsymbol{A}=\\boldsymbol{A}^T$, we have" ] }, { "cell_type": "markdown", - "id": "83cf4934", - "metadata": {}, + "id": "558bd5db", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{x}} = 2\\boldsymbol{x}^T\\boldsymbol{A}.\n", @@ -744,8 +880,10 @@ }, { "cell_type": "markdown", - "id": "54d87a04", - "metadata": {}, + "id": "7196cbcc", + "metadata": { + "editable": true + }, "source": [ "## Example 4\n", "\n", @@ -754,8 +892,10 @@ }, { "cell_type": "markdown", - "id": "75b5c120", - "metadata": {}, + "id": "b3aeae7c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{y}^T\\boldsymbol{x},\n", @@ -764,8 +904,10 @@ }, { "cell_type": "markdown", - "id": "349aed8a", - "metadata": {}, + "id": "b08d19a4", + "metadata": { + "editable": true + }, "source": [ "where both $\\boldsymbol{y}$ and $\\boldsymbol{x}$ have the same length $n$, or if we\n", "wish to think of them as column vectors, they have dimensions $n\\times\n", @@ -777,8 +919,10 @@ }, { "cell_type": "markdown", - "id": "d66cc9ad", - "metadata": {}, + "id": "04ec54c8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\sum_{i=0}^{n-1}y_ix_i,\n", @@ -787,16 +931,20 @@ }, { "cell_type": "markdown", - "id": "2e4f3835", - "metadata": {}, + "id": "15b70672", + "metadata": { + "editable": true + }, "source": [ "and the partial derivative" ] }, { "cell_type": "markdown", - "id": "79d6c4a1", - "metadata": {}, + "id": "baabd91a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial z_k} = \\sum_{i=0}^{n-1}\\left(x_i\\frac{\\partial y_i}{\\partial z_k}+y_i\\frac{\\partial x_i}{\\partial z_k}\\right),\n", @@ -805,16 +953,20 @@ }, { "cell_type": "markdown", - "id": "fa478123", - "metadata": {}, + "id": "c31bcade", + "metadata": { + "editable": true + }, "source": [ "for $\\forall k =0,1,2,\\dots,n-1$. We can rewrite the partial derivative in a more compact form as" ] }, { "cell_type": "markdown", - "id": "5baf3cf3", - "metadata": {}, + "id": "0c70fe69", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{z}} = \\boldsymbol{x}^T\\frac{\\partial \\boldsymbol{y}}{\\partial \\boldsymbol{z}}+\\boldsymbol{y}^T\\frac{\\partial \\boldsymbol{x}}{\\partial \\boldsymbol{z}},\n", @@ -823,16 +975,20 @@ }, { "cell_type": "markdown", - "id": "55bccb8f", - "metadata": {}, + "id": "8311f527", + "metadata": { + "editable": true + }, "source": [ "and if $\\boldsymbol{y}=\\boldsymbol{x}$ we have" ] }, { "cell_type": "markdown", - "id": "e5d72571", - "metadata": {}, + "id": "38a430f0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{z}} = 2\\boldsymbol{x}^T\\frac{\\partial \\boldsymbol{x}}{\\partial \\boldsymbol{z}}.\n", @@ -841,8 +997,10 @@ }, { "cell_type": "markdown", - "id": "c5d24b37", - "metadata": {}, + "id": "9cacb60b", + "metadata": { + "editable": true + }, "source": [ "## The mean squared error and its derivative\n", "\n", @@ -851,8 +1009,10 @@ }, { "cell_type": "markdown", - "id": "034723eb", - "metadata": {}, + "id": "48c8af1a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", @@ -861,16 +1021,20 @@ }, { "cell_type": "markdown", - "id": "99f84d2d", - "metadata": {}, + "id": "3d5b6042", + "metadata": { + "editable": true + }, "source": [ "or using the design/feature matrix $\\boldsymbol{X}$ we have the more compact matrix-vector" ] }, { "cell_type": "markdown", - "id": "84a5648d", - "metadata": {}, + "id": "1d50d4ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -879,8 +1043,10 @@ }, { "cell_type": "markdown", - "id": "71302309", - "metadata": {}, + "id": "f3ef4fda", + "metadata": { + "editable": true + }, "source": [ "We note that the design matrix $\\boldsymbol{X}$ does not depend on the unknown parameters defined by the vector $\\boldsymbol{\\beta}$.\n", "We are now interested in minimizing the cost function with respect to the unknown parameters $\\boldsymbol{\\beta}$.\n", @@ -890,8 +1056,10 @@ }, { "cell_type": "markdown", - "id": "d3c880b6", - "metadata": {}, + "id": "2540968d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{w}=\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -900,16 +1068,20 @@ }, { "cell_type": "markdown", - "id": "0ad3e27b", - "metadata": {}, + "id": "3bc106fa", + "metadata": { + "editable": true + }, "source": [ "which depends on $\\boldsymbol{\\beta}$. We rewrite the cost function as" ] }, { "cell_type": "markdown", - "id": "ddf58d6d", - "metadata": {}, + "id": "35c769d4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\boldsymbol{w}^T\\boldsymbol{w},\n", @@ -918,16 +1090,20 @@ }, { "cell_type": "markdown", - "id": "fce48c5c", - "metadata": {}, + "id": "f964b8d2", + "metadata": { + "editable": true + }, "source": [ "with partial derivative" ] }, { "cell_type": "markdown", - "id": "0bb38fd4", - "metadata": {}, + "id": "278df999", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=\\frac{2}{n}\\boldsymbol{w}^T\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\beta}},\n", @@ -936,16 +1112,20 @@ }, { "cell_type": "markdown", - "id": "8d03c742", - "metadata": {}, + "id": "ef003c39", + "metadata": { + "editable": true + }, "source": [ "and using that" ] }, { "cell_type": "markdown", - "id": "5d1bd0dd", - "metadata": {}, + "id": "f4bae399", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\beta}}=-\\boldsymbol{X},\n", @@ -954,16 +1134,20 @@ }, { "cell_type": "markdown", - "id": "5f118679", - "metadata": {}, + "id": "e4161bf9", + "metadata": { + "editable": true + }, "source": [ "where we used the result from example two above. Inserting the last expression we obtain" ] }, { "cell_type": "markdown", - "id": "e1c3e5ab", - "metadata": {}, + "id": "a5badddc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\boldsymbol{X},\n", @@ -972,16 +1156,20 @@ }, { "cell_type": "markdown", - "id": "51937c66", - "metadata": {}, + "id": "3a9bcee4", + "metadata": { + "editable": true + }, "source": [ "or as" ] }, { "cell_type": "markdown", - "id": "55f8871a", - "metadata": {}, + "id": "742dd6d4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T}=-\\frac{2}{n}\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", @@ -990,8 +1178,10 @@ }, { "cell_type": "markdown", - "id": "a7697cc9", - "metadata": {}, + "id": "07447ef0", + "metadata": { + "editable": true + }, "source": [ "## Other useful relations\n", "\n", @@ -1000,8 +1190,10 @@ }, { "cell_type": "markdown", - "id": "fc537a1b", - "metadata": {}, + "id": "42809d8f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial (\\boldsymbol{x}^T\\boldsymbol{a})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", @@ -1010,8 +1202,10 @@ }, { "cell_type": "markdown", - "id": "147601fc", - "metadata": {}, + "id": "3de9e409", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{x})}{\\partial \\boldsymbol{x}} = \\boldsymbol{a}^T,\n", @@ -1020,8 +1214,10 @@ }, { "cell_type": "markdown", - "id": "5eccde34", - "metadata": {}, + "id": "976c9ed9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial tr(\\boldsymbol{B}\\boldsymbol{A})}{\\partial \\boldsymbol{A}} = \\boldsymbol{B}^T,\n", @@ -1030,8 +1226,10 @@ }, { "cell_type": "markdown", - "id": "9e817edd", - "metadata": {}, + "id": "598bfc4d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\log{\\vert\\boldsymbol{A}\\vert}}{\\partial \\boldsymbol{A}} = (\\boldsymbol{A}^{-1})^T.\n", @@ -1040,8 +1238,10 @@ }, { "cell_type": "markdown", - "id": "e025864d", - "metadata": {}, + "id": "bdbc0fea", + "metadata": { + "editable": true + }, "source": [ "## Meet the Hessian Matrix\n", "\n", @@ -1054,8 +1254,10 @@ }, { "cell_type": "markdown", - "id": "f8c3180f", - "metadata": {}, + "id": "11a6a7b9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -1064,16 +1266,20 @@ }, { "cell_type": "markdown", - "id": "41e51518", - "metadata": {}, + "id": "4587c0c5", + "metadata": { + "editable": true + }, "source": [ "The Hessian matrix plays an important role and is defined here as" ] }, { "cell_type": "markdown", - "id": "2b8b6511", - "metadata": {}, + "id": "300ea026", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -1082,8 +1288,10 @@ }, { "cell_type": "markdown", - "id": "5ba4b948", - "metadata": {}, + "id": "18a45342", + "metadata": { + "editable": true + }, "source": [ "For ordinary least squares, it is inversely proportional (derivation\n", "next week) with the variance of the optimal parameters\n", @@ -1098,8 +1306,10 @@ }, { "cell_type": "markdown", - "id": "dacd56e9", - "metadata": {}, + "id": "b1538a53", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "\n", @@ -1108,8 +1318,10 @@ }, { "cell_type": "markdown", - "id": "ff3e1ed3", - "metadata": {}, + "id": "80afef85", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -1118,16 +1330,20 @@ }, { "cell_type": "markdown", - "id": "d41ee98a", - "metadata": {}, + "id": "f2175720", + "metadata": { + "editable": true + }, "source": [ "and with" ] }, { "cell_type": "markdown", - "id": "4b041a5e", - "metadata": {}, + "id": "f47c3870", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", @@ -1136,16 +1352,20 @@ }, { "cell_type": "markdown", - "id": "c4c19d87", - "metadata": {}, + "id": "997e4516", + "metadata": { + "editable": true + }, "source": [ "we have" ] }, { "cell_type": "markdown", - "id": "aea64900", - "metadata": {}, + "id": "1860f06d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", @@ -1154,16 +1374,20 @@ }, { "cell_type": "markdown", - "id": "e78e5c70", - "metadata": {}, + "id": "bed5b5bf", + "metadata": { + "editable": true + }, "source": [ "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals." ] }, { "cell_type": "markdown", - "id": "f83f72a6", - "metadata": {}, + "id": "9ddbc1b9", + "metadata": { + "editable": true + }, "source": [ "## Example relevant for the exercises\n", "\n", @@ -1174,8 +1398,10 @@ }, { "cell_type": "markdown", - "id": "274c7763", - "metadata": {}, + "id": "e3182bc2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_i = \\beta_0+\\beta_1x_i+\\beta_2x_i^2+\\beta_3x_i^3+\\beta_4x_i^4.\n", @@ -1184,8 +1410,10 @@ }, { "cell_type": "markdown", - "id": "f8b11c79", - "metadata": {}, + "id": "fc7ad1a4", + "metadata": { + "editable": true + }, "source": [ "we have five predictors/features. The first is the intercept $\\beta_0$. The other terms are $\\beta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an \n", "$n\\times p$ matrix $\\boldsymbol{X}$." @@ -1193,8 +1421,10 @@ }, { "cell_type": "markdown", - "id": "99f38f15", - "metadata": {}, + "id": "b2d8e764", + "metadata": { + "editable": true + }, "source": [ "## Own code for Ordinary Least Squares\n", "\n", @@ -1203,16 +1433,19 @@ }, { "cell_type": "code", - "execution_count": 36, - "id": "dace9ae7", - "metadata": {}, + "execution_count": 1, + "id": "b550fbbe", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# matrix inversion to find beta\n", "# First we set up the data\n", "import numpy as np\n", "x = np.random.rand(100)\n", - "y = 2.0+5*x*x+1.0*np.random.randn(100)\n", + "y = 2.0+5*x*x+0.1*np.random.randn(100)\n", "# and then the design matrix X including the intercept\n", "# The design matrix now as function of a fourth-order polynomial\n", "X = np.zeros((len(x),5))\n", @@ -1228,17 +1461,22 @@ }, { "cell_type": "markdown", - "id": "fdf2525a", - "metadata": {}, + "id": "b7390330", + "metadata": { + "editable": true + }, "source": [ "Alternatively, you can use the least squares functionality in **Numpy** as" ] }, { "cell_type": "code", - "execution_count": 37, - "id": "11868544", - "metadata": {}, + "execution_count": 2, + "id": "2b0ea997", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "fit = np.linalg.lstsq(X, y, rcond =None)[0]\n", @@ -1247,8 +1485,10 @@ }, { "cell_type": "markdown", - "id": "a6afa437", - "metadata": {}, + "id": "a58e344f", + "metadata": { + "editable": true + }, "source": [ "## Adding error analysis and training set up\n", "\n", @@ -1258,9 +1498,12 @@ }, { "cell_type": "code", - "execution_count": 38, - "id": "1a8e4684", - "metadata": {}, + "execution_count": 3, + "id": "ae3d0b17", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def R2(y_data, y_model):\n", @@ -1269,52 +1512,46 @@ }, { "cell_type": "markdown", - "id": "a7d9c395", - "metadata": {}, + "id": "39bebaf1", + "metadata": { + "editable": true + }, "source": [ "and we would be using it as" ] }, { "cell_type": "code", - "execution_count": 39, - "id": "a86e7913", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0.6729150449047916\n" - ] - } - ], + "execution_count": 4, + "id": "9b073a72", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "print(R2(y,ytilde))" ] }, { "cell_type": "markdown", - "id": "e219dc1e", - "metadata": {}, + "id": "b8af0dc1", + "metadata": { + "editable": true + }, "source": [ "We can easily add our **MSE** score as" ] }, { "cell_type": "code", - "execution_count": 40, - "id": "f75682d9", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0.949955776248515\n" - ] - } - ], + "execution_count": 5, + "id": "7ad18065", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "def MSE(y_data,y_model):\n", " n = np.size(y_model)\n", @@ -1325,8 +1562,10 @@ }, { "cell_type": "markdown", - "id": "f68ecfd9", - "metadata": {}, + "id": "5f6880a5", + "metadata": { + "editable": true + }, "source": [ "and finally the relative error as" ] @@ -1334,8 +1573,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "4a92d7ed", - "metadata": {}, + "id": "105de2f9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def RelativeError(y_data,y_model):\n", @@ -1345,8 +1587,10 @@ }, { "cell_type": "markdown", - "id": "c5465350", - "metadata": {}, + "id": "fb5d5828", + "metadata": { + "editable": true + }, "source": [ "## Splitting our Data in Training and Test data\n", "\n", @@ -1364,34 +1608,23 @@ }, { "cell_type": "markdown", - "id": "1cb9fb17", - "metadata": {}, + "id": "5d3571b0", + "metadata": { + "editable": true + }, "source": [ "## The complete code with a simple data set" ] }, { "cell_type": "code", - "execution_count": 41, - "id": "b0e0ec9d", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[ 1.9604947 0.26950852 3.61021966 2.57284577 -1.44467108]\n", - "Training R2\n", - "0.9964127933024364\n", - "Training MSE\n", - "0.008205373380288369\n", - "Test R2\n", - "0.9977591706039226\n", - "Test MSE\n", - "0.005966417834794856\n" - ] - } - ], + "execution_count": 7, + "id": "50c74297", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -1439,8 +1672,10 @@ }, { "cell_type": "markdown", - "id": "83618bc5", - "metadata": {}, + "id": "81d56642", + "metadata": { + "editable": true + }, "source": [ "## Making your own test-train splitting" ] @@ -1448,8 +1683,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "cc6480ea", - "metadata": {}, + "id": "2effb9ff", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# equivalently in numpy\n", @@ -1470,8 +1708,10 @@ }, { "cell_type": "markdown", - "id": "44f4ce6e", - "metadata": {}, + "id": "994b52a7", + "metadata": { + "editable": true + }, "source": [ "But since **scikit-learn** has its own function for doing this and since\n", "it interfaces easily with **tensorflow** and other libraries, we\n", @@ -1480,8 +1720,10 @@ }, { "cell_type": "markdown", - "id": "74508a97", - "metadata": {}, + "id": "99f996fd", + "metadata": { + "editable": true + }, "source": [ "## Reducing the number of degrees of freedom, overarching view\n", "\n", @@ -1507,8 +1749,10 @@ }, { "cell_type": "markdown", - "id": "b93e1823", - "metadata": {}, + "id": "32b2f8e0", + "metadata": { + "editable": true + }, "source": [ "## Preprocessing our data\n", "\n", @@ -1530,8 +1774,10 @@ }, { "cell_type": "markdown", - "id": "df747aed", - "metadata": {}, + "id": "d837dc89", + "metadata": { + "editable": true + }, "source": [ "## Functionality in Scikit-Learn\n", "\n", @@ -1548,8 +1794,10 @@ }, { "cell_type": "markdown", - "id": "6c576050", - "metadata": {}, + "id": "f3f3ec16", + "metadata": { + "editable": true + }, "source": [ "## More preprocessing\n", "\n", @@ -1573,8 +1821,10 @@ }, { "cell_type": "markdown", - "id": "4d579715", - "metadata": {}, + "id": "bcbbea15", + "metadata": { + "editable": true + }, "source": [ "## Frequently used scaling functions\n", "\n", @@ -1584,8 +1834,10 @@ }, { "cell_type": "markdown", - "id": "bdc67cc0", - "metadata": {}, + "id": "dce3c751", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_j^{(i)} \\rightarrow \\frac{x_j^{(i)} - \\overline{x}_j}{\\sigma(x_j)},\n", @@ -1594,8 +1846,10 @@ }, { "cell_type": "markdown", - "id": "d1a91553", - "metadata": {}, + "id": "e3a1dd2c", + "metadata": { + "editable": true + }, "source": [ "where $\\overline{x}_j$ and $\\sigma(x_j)$ are the mean and standard deviation, respectively, of the feature $x_j$.\n", "This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one." @@ -1603,8 +1857,10 @@ }, { "cell_type": "markdown", - "id": "0bfbf206", - "metadata": {}, + "id": "00ead977", + "metadata": { + "editable": true + }, "source": [ "## Example of own Standard scaling\n", "\n", @@ -1617,8 +1873,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "c5d921c9", - "metadata": {}, + "id": "f48cac43", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import sklearn.linear_model as skl\n", @@ -1648,16 +1907,20 @@ }, { "cell_type": "markdown", - "id": "d266b771", - "metadata": {}, + "id": "0eae8922", + "metadata": { + "editable": true + }, "source": [ "Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives." ] }, { "cell_type": "markdown", - "id": "08d80e06", - "metadata": {}, + "id": "709832eb", + "metadata": { + "editable": true + }, "source": [ "## Min-Max Scaling\n", "\n", @@ -1669,8 +1932,10 @@ }, { "cell_type": "markdown", - "id": "cb1b8d9c", - "metadata": {}, + "id": "5660e410", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_j^{(i)} \\rightarrow (b-a)\\frac{x_j^{(i)} - \\min(x_j)}{\\max(x_j) - \\min(x_j)} - a\n", @@ -1679,16 +1944,20 @@ }, { "cell_type": "markdown", - "id": "0d175f62", - "metadata": {}, + "id": "d44e5934", + "metadata": { + "editable": true + }, "source": [ "where $\\min(x_j)$ and $\\max(x_j)$ return the minimum and maximum value of $x_j$ over the data set, respectively." ] }, { "cell_type": "markdown", - "id": "d14473d2", - "metadata": {}, + "id": "98d863c6", + "metadata": { + "editable": true + }, "source": [ "## Testing the Means Squared Error as function of Complexity\n", "\n", @@ -1700,23 +1969,28 @@ }, { "cell_type": "code", - "execution_count": 44, - "id": "2f49f528", - "metadata": {}, + "execution_count": 10, + "id": "d11f262e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "np.random.seed()\n", - "n = 50\n", - "maxdegree = 5\n", + "n = 100\n", + "maxdegree = 14\n", "# Make data set.\n", "x = np.linspace(-3, 3, n).reshape(-1, 1)\n", - "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)#+ np.random.normal(0, 0.1, x.shape)" + "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)" ] }, { "cell_type": "markdown", - "id": "dfc96064", - "metadata": {}, + "id": "f69a30ce", + "metadata": { + "editable": true + }, "source": [ "where $y$ is the function we want to fit with a given polynomial.\n", "\n", @@ -1725,21 +1999,13 @@ }, { "cell_type": "code", - "execution_count": 49, - "id": "48b86b87", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi4AAAGsCAYAAAD62iyRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA5kklEQVR4nO3deXRU9f3/8ddNSCZhybCaRUJkRxZjDEgCQqFCWCoF9Ujab0VQ1EPVVpr6FePCIq3RVjQgqKWlptQ2RBsWrViM3wqBgv0JTZRvpRRsajAmXwqVTBKaDCT39wdkYMhCJgTm3szzcc49J/OZz9x53+s9zpvPapimaQoAAMAGgvwdAAAAQEuRuAAAANsgcQEAALZB4gIAAGyDxAUAANgGiQsAALANEhcAAGAbJC4AAMA2SFwAAIBtkLgAAADbaHeJS35+vmbMmKGYmBgZhqHNmzf79Pnq6mrNmzdPI0aMUIcOHTRr1qwGdTZu3KjJkyerV69eioiIUHJysrZt29Y2FwAAAJrU7hKXqqoqxcfHa/Xq1a36fG1trcLDw/X9739fkyZNarROfn6+Jk+erK1bt2rfvn2aOHGiZsyYoYKCgksJHQAAXITRnjdZNAxDmzZt8mo1cbvdevLJJ/Wb3/xGJ06c0PDhw/Xcc89pwoQJDT4/b948nThxokWtNsOGDVNqaqoWL17cdhcAAAC8dPB3AFfa3XffrX/+85/asGGDYmJitGnTJk2dOlX79+/XwIEDW3XOuro6VVRUqHv37m0cLQAAOF+76ypqzmeffabs7Gy9+eabGjdunPr3769HHnlEN910k1577bVWn3fFihWqqqrS7Nmz2zBaAABwoYBqcfnLX/4i0zQ1aNAgr/Kamhr16NGjVefMzs7W0qVLtWXLFl111VVtESYAAGhCQCUudXV1Cg4O1r59+xQcHOz1XufOnX0+X05OjubPn68333yzyYG8AACg7QRU4pKQkKDa2lodPXpU48aNu6RzZWdn65577lF2dra+8Y1vtFGEAACgOe0ucamsrNThw4c9r4uKilRYWKju3btr0KBB+s53vqO77rpLK1asUEJCgo4dO6Y//vGPGjFihKZPny5J+vTTT+V2u/Xvf/9bFRUVKiwslCRdf/31ks4kLXfddZdWrlyppKQklZWVSZLCw8PldDqv6PUCABBI2t106O3bt2vixIkNyufOnausrCydOnVKP/rRj7R+/XqVlJSoR48eSk5O1rJlyzRixAhJ0jXXXKPPP/+8wTnqb9WECRO0Y8eOJr8DAABcHu0ucQEAAO1XQE2HBgAA9kbiAgAAbKPdDM6tq6vTl19+qS5dusgwDH+HAwAAWsA0TVVUVCgmJkZBQRdvT2k3icuXX36p2NhYf4cBAABa4ciRI+rdu/dF67WbxKVLly6Szlx4RESEn6MBAAAt4XK5FBsb6/kdv5h2k7jUdw9FRESQuAAAYDMtHebB4FwAAGAbJC4AAMA2fE5c8vPzNWPGDMXExMgwDG3evLnZ+vPmzZNhGA2OYcOGeepkZWU1Wqe6utrnCwIAAO2Xz2NcqqqqFB8fr7vvvlu33377ReuvXLlSzz77rOf16dOnFR8frzvuuMOrXkREhA4ePOhVFhYW5mt4zaqtrdWpU6fa9JywhuDgYHXo0IGp8ADQzvmcuEybNk3Tpk1rcX2n0+m18eDmzZv11Vdf6e677/aqZxiGoqKifA2nxSorK/XFF1+IHQ7ar44dOyo6OlqhoaH+DgUAcJlc8VlF69at06RJkxQXF+dVXllZqbi4ONXW1ur666/X8uXLlZCQ0OR5ampqVFNT43ntcrmarFtbW6svvvhCHTt2VK9evfhXeTtjmqbcbrf+9a9/qaioSAMHDmzRIkYAAPu5oolLaWmp3n33Xf32t7/1Kh8yZIiysrI0YsQIuVwurVy5UmPHjtXHH3+sgQMHNnqujIwMLVu2rEXfe+rUKZmmqV69eik8PPySrwPWEx4erpCQEH3++edyu91t3s0IALCGK/rP0qysLHXt2lWzZs3yKk9KStKdd96p+Ph4jRs3Tm+88YYGDRqkl156qclzpaenq7y83HMcOXLkot9PS0v7RisLALR/V6zFxTRN/fKXv9ScOXMuOgYhKChIo0aN0qFDh5qs43A45HA42jpMAABgYVfsn6g7duzQ4cOHNX/+/IvWNU1ThYWFio6OvgKRAQAAu/C5xaWyslKHDx/2vC4qKlJhYaG6d++uPn36KD09XSUlJVq/fr3X59atW6fRo0dr+PDhDc65bNkyJSUlaeDAgXK5XFq1apUKCwu1Zs2aVlwSAABor3xucdm7d68SEhI8M37S0tKUkJCgxYsXSzozALe4uNjrM+Xl5crNzW2yteXEiRO6//77de211yolJUUlJSXKz8/XjTfe6Gt47UZjC/Kdf8ybN6/V577mmmuUmZnZonqNfff56/IAAHAl+dziMmHChGbXQsnKympQ5nQ6dfLkySY/8+KLL+rFF1/0NZR2rbS01PN3Tk6OFi9e7LVA35WaHfX000/rvvvu8ypragdP0zRVW1urDh28Hyu3292qtVVa+zkAQNvKfP/vqqw+rbuSr1GfHh39GktATsMwTVMn3af9crR0AbyoqCjP4XQ6PQv01R/5+flKTExUWFiY+vXrp2XLlun06dOezy9dulR9+vSRw+FQTEyMvv/970s6k3h+/vnn+sEPfuBpQWlOly5dvL43KipKnTp1kiRt375dhmFo27ZtGjlypBwOh3bu3KkJEybooYceUlpamnr27KnJkydLOjPO6cYbb5TD4VB0dLQee+wxr5ib+hwAwL9+t+8L/WJXkY5V1Vy88mV2xRegs4L/nKrV0MXb/PLdnz49RR1DL+22b9u2TXfeeadWrVqlcePG6bPPPtP9998vSVqyZIl+97vf6cUXX9SGDRs0bNgwlZWV6eOPP5Ykbdy4UfHx8br//vsbtKS01qOPPqrnn39e/fr1U9euXSVJv/rVr/Td735Xf/rTn2SapkpKSjR9+nTNmzdP69ev19/+9jfdd999CgsL09KlSz3nuvBzAAD/q6o584/Mzg7/pw3+jwA++/GPf6zHHntMc+fOlST169dPy5cv16OPPqolS5aouLhYUVFRmjRpkkJCQtSnTx/PeKHu3bsrODjY05JyMYsWLdKTTz7pVfb73/9eEyZM8Lx++umnG7SODBgwQD/5yU88r5944gnFxsZq9erVMgxDQ4YM0ZdffqlFixZp8eLFnjVYLvwcAMD/qmpqJZG4+E14SLA+fXqK3777Uu3bt08fffSRfvzjH3vKamtrVV1drZMnT+qOO+5QZmam+vXrp6lTp2r69OmaMWNGg7EnLfHf//3fDQYCX3311V6vR44c2eBzF5YdOHBAycnJXl1TY8eO9ewh1adPnybPBQDwH/fpOrlr6yRJnUhc/MMwjEvurvGnuro6LVu2TLfddluD98LCwhQbG6uDBw8qLy9P77//vh544AH99Kc/1Y4dOxQSEuLTd/Xs2VMDBgxotk79mJfmykzTbDCepr4r6Pzyxs4FAPCf+m4iSeoUeun/+L5U9v31DmA33HCDDh482GxCER4erm9+85v65je/qQcffFBDhgzR/v37dcMNNyg0NFS1tbVXMGJp6NChys3N9Upgdu/erS5dujRowQEAWEfl2cQlLCRIHYL9P6eHxMWGFi9erFtuuUWxsbG64447FBQUpE8++UT79+/Xj370I2VlZam2tlajR49Wx44d9etf/1rh4eGeHbmvueYa5efn61vf+pYcDod69uzZ5HdVVFSorKzMq6xjx46KiIjwKeYHHnhAmZmZ+t73vqeHHnpIBw8e1JIlS5SWlsYeQwBgYVVu6wzMlQJ0OrTdTZkyRb///e+Vl5enUaNGKSkpSS+88IInMenatat+/vOfa+zYsbruuuv0P//zP3r77bfVo0cPSWcG0/7zn/9U//791atXr2a/a/HixYqOjvY6Hn30UZ9jvvrqq7V161b9v//3/xQfH68FCxZo/vz5DQb+AgCspb6ryArjWyTJMNvJnFOXyyWn06ny8vIGrQHV1dUqKipS3759FRYW5qcIcbnx3xkA2t72g0c177WPNDQ6QlsfHtfm52/u97sxtLgAAIAmWWkqtETiAgAAmuFZfC6MxAUAAFhcpcXGuJC4AACAJp1b7t//a7hIJC4AAKAZlWenQ3eyyMKtJC4AAKBJVpsOTeICAACaxKwiAABgGxXVtLgAAACbONdVxOBc+MGECRO0cOFCf4cBALAJ9ipCixiG0ewxb968Vp1348aNWr58+SXFNm/evEZjmjp16iWdFwBgPZU11kpcrBEFGigtLfX8nZOTo8WLF+vgwYOesvDwcK/6p06dUkhIyEXP27179zaJb+rUqXrttde8yhwOR5P1G4uvpTG35FwAgMuDWUVWYJqSu8o/Rwv3tIyKivIcTqdThmF4XldXV6tr16564403NGHCBIWFhen111/X8ePH9e1vf1u9e/dWx44dNWLECGVnZ3ud98KuomuuuUbPPPOM7rnnHnXp0kV9+vTR2rVrLxqfw+HwijEqKkrdunXzvG8Yhl599VXNnDlTnTp10o9+9CMtXbpU119/vX75y1+qX79+cjgcMk1TxcXFmjlzpjp37qyIiAjNnj1b//d//+c5V1OfAwBcflabVWSNKK60UyelZ2L8892PfymFdmqTUy1atEgrVqzQa6+9JofDoerqaiUmJmrRokWKiIjQO++8ozlz5qhfv34aPXp0k+dZsWKFli9frscff1y/+93v9N3vflfjx4/XkCFDLim+JUuWKCMjQy+++KKCg4P12muv6fDhw3rjjTeUm5ur4OAzA71mzZqlTp06aceOHTp9+rQeeOABpaamavv27Z5zNfY5AMDlZZqmZ4yLVVpcrBEFWmXhwoW67bbbvMoeeeQRz9/f+9739Ic//EFvvvlms4nL9OnT9cADD0g6kwy9+OKL2r59e7OJy+9//3t17tzZq2zRokV66qmnPK//67/+S/fcc49XHbfbrV//+tfq1auXJCkvL0+ffPKJioqKFBsbK0n69a9/rWHDhumjjz7SqFGjGv0cAODyO+mu9XQU0OLiTyEdz7R8+Ou728jIkSO9XtfW1urZZ59VTk6OSkpKVFNTo5qaGnXq1HwLz3XXXef5u75L6ujRo81+ZuLEiXrllVe8yi4cP3NhfJIUFxfnlXwcOHBAsbGxnqRFkoYOHaquXbvqwIEDnsTlws8BAC6/+oG5QYYUFmKN0SWBmbgYRpt11/jThQnJihUr9OKLLyozM1MjRoxQp06dtHDhQrnd7mbPc+FAV8MwVFdXd9HvHjBggE/xNVZmmqYMw2hQ78LyiyVfAIC2d/7O0I39v9ofAjNxaad27typmTNn6s4775Qk1dXV6dChQ7r22mv9HFnThg4dquLiYh05csTT6vLpp5+qvLzc0nEDQCCosthUaInEpV0ZMGCAcnNztXv3bnXr1k0vvPCCysrKLksCUFNTo7KyMq+yDh06qGfPnj6dZ9KkSbruuuv0ne98R5mZmZ7BuV/72tca7WoCAFw5VlvDRQrU6dDt1FNPPaUbbrhBU6ZM0YQJExQVFaVZs2Zdlu/6wx/+oOjoaK/jpptu8vk8hmFo8+bN6tatm8aPH69JkyapX79+ysnJuQxRAwB8UT8V2ioziiTJMNvJghgul0tOp1Pl5eWKiIjweq+6ulpFRUXq27evwsLC/BQhLjf+OwNA29pcUKKFOYW6aUBPvX5v07NTL0Vzv9+NocUFAAA0qtJiGyxKJC4AAKAJVlvuXyJxAQAATbDirCISFwAA0KgKWlz8q52MQ0YT+O8LAG2LFhc/qd+U72IryMLeTp48KanhSsAAgNax2s7QUoAsQNehQwd17NhR//rXvxQSEqKgoIDI1wKGaZo6efKkjh49qq5du7J7NAC0kUoLdhVZJ5LLyDAMRUdHq6ioSJ9//rm/w8Fl0rVrV0VFRfk7DABoN851FVnnH4Q+Jy75+fn66U9/qn379qm0tFSbNm1qdnXW7du3a+LEiQ3KDxw4oCFDhnhe5+bm6qmnntJnn32m/v3768c//rFuvfVWX8NrUmhoqAYOHEh3UTsVEhJCSwsAtLF20eJSVVWl+Ph43X333br99ttb/LmDBw96rYjXq1cvz9979uxRamqqli9frltvvVWbNm3S7NmztWvXLo0e3XYr9QUFBbGiKgAALVTlbgeJy7Rp0zRt2jSfv+iqq65S165dG30vMzNTkydPVnp6uiQpPT1dO3bsUGZmprKzs33+LgAAcOmsODj3io1STUhIUHR0tG6++WZ98MEHXu/t2bNHKSkpXmVTpkzR7t27mzxfTU2NXC6X1wEAANpOZbX1Wlwue+ISHR2ttWvXKjc3Vxs3btTgwYN18803Kz8/31OnrKxMkZGRXp+LjIxUWVlZk+fNyMiQ0+n0HLGxsZftGgAACDTu03Vy19ZJkjqHWidxueyRDB48WIMHD/a8Tk5O1pEjR/T8889r/PjxnnLDMLw+Z5pmg7LzpaenKy0tzfPa5XKRvAAA0EbqZxRJbLKopKQkHTp0yPM6KiqqQevK0aNHG7TCnM/hcCgiIsLrAAAAbaN+RlFYSJA6BFtn/TO/RFJQUKDo6GjP6+TkZOXl5XnVee+99zRmzJgrHRoAANC5GUVWGpgrtaKrqLKyUocPH/a8LioqUmFhobp3764+ffooPT1dJSUlWr9+vaQzM4auueYaDRs2TG63W6+//rpyc3OVm5vrOcfDDz+s8ePH67nnntPMmTO1ZcsWvf/++9q1a1cbXCIAAPBVlQXXcJFakbjs3bvXa0G5+nEmc+fOVVZWlkpLS1VcXOx53+1265FHHlFJSYnCw8M1bNgwvfPOO5o+fbqnzpgxY7RhwwY9+eSTeuqpp9S/f3/l5OS06RouAACg5SrPToXuZKGBuZJkmO1kS12XyyWn06ny8nLGuwAAcIm27i/VA7/5i268prveWJB82b7H199v64y2AQAAlnFuDRfrzCiSSFwAAEAjrLhPkUTiAgAAGnFuZ2gSFwAAYHGVFp0OTeICAAAasOp0aBIXAADQgBV3hpZIXAAAQCMYnAsAAGzjXFcR06EBAIDFVTKrCAAA2AVdRQAAwDZYxwUAANgGs4oAAIAtmKapKjddRQAAwAZOumtlmmf+psUFAABYWv34liBDCguxVqpgrWgAAIDfnT+jyDAMP0fjjcQFAAB4seoaLhKJCwAAuIBV13CRSFwAAMAF6qdCk7gAAADLqx+c24XEBQAAWF2lRTdYlEhcAADABaoY4wIAAOzCqvsUSSQuAADgApUMzgUAAHZBiwsAALANz+DcUAbnAgAAi2MBOgAAYBuedVzCSFwAAIDF0eICAABso8pN4gIAAGyifq8iZhUBAADLo6sIAADYgvt0ndyn6yRJnUNJXAAAgIXVzyiS2GQRAABYXH03kaNDkDoEWy9NsF5EAADAb+pnFFlxDReJxAUAAJynysIDcyUSFwAAcB7PztAWHJgrtSJxyc/P14wZMxQTEyPDMLR58+Zm62/cuFGTJ09Wr169FBERoeTkZG3bts2rTlZWlgzDaHBUV1f7Gh4AALgEVt4ZWmpF4lJVVaX4+HitXr26RfXz8/M1efJkbd26Vfv27dPEiRM1Y8YMFRQUeNWLiIhQaWmp1xEWFuZreAAA4BKcW8PFejOKJMnndGratGmaNm1ai+tnZmZ6vX7mmWe0ZcsWvf3220pISPCUG4ahqKgoX8MBAABtiDEuF6irq1NFRYW6d+/uVV5ZWam4uDj17t1bt9xyS4MWmQvV1NTI5XJ5HQAA4NJUVrezrqJLtWLFClVVVWn27NmesiFDhigrK0tvvfWWsrOzFRYWprFjx+rQoUNNnicjI0NOp9NzxMbGXonwAQBo1yotvMGidIUTl+zsbC1dulQ5OTm66qqrPOVJSUm68847FR8fr3HjxumNN97QoEGD9NJLLzV5rvT0dJWXl3uOI0eOXIlLAACgXbN6V9EViyonJ0fz58/Xm2++qUmTJjVbNygoSKNGjWq2xcXhcMjhcLR1mAAABLT6naG7WDRxuSItLtnZ2Zo3b55++9vf6hvf+MZF65umqcLCQkVHR1+B6AAAQD0r7wwttaLFpbKyUocPH/a8LioqUmFhobp3764+ffooPT1dJSUlWr9+vaQzSctdd92llStXKikpSWVlZZKk8PBwOZ1OSdKyZcuUlJSkgQMHyuVyadWqVSosLNSaNWva4hoBAEALVVl8OrTPLS579+5VQkKCZypzWlqaEhIStHjxYklSaWmpiouLPfV/9rOf6fTp03rwwQcVHR3tOR5++GFPnRMnTuj+++/Xtddeq5SUFJWUlCg/P1833njjpV4fAADwgdUXoDNM0zT9HURbcLlccjqdKi8vV0REhL/DAQDAlm5esV2f/atKG+5PUlK/Hpf9+3z9/WavIgAA4FFp8RYXEhcAAOBRP6vIqoNzSVwAAICkM7N6q9ztbHAuAABon066a1U/8rWLI8S/wTSBxAUAAEg6N6MoyJDCQqyZIlgzKgAAcMWdv/icYRh+jqZxJC4AAEDSuYG5Vp1RJJG4AACAs6y+3L9E4gIAAM4icQEAALZxbrl/a06FlkhcAADAWZ4Wl1BaXAAAgMV5WlzCSFwAAIDFWX1naInEBQAAnFVp8X2KJBIXAABwFi0uAADANirrN1gMZVYRAACwuCrWcQEAAHZRWU1XEQAAsAlWzgUAALZR5WYdFwAAYBPsDg0AAGyDriIAAGALp2rr5D5dJ0nqzF5FAADAyuqnQktSJ3aHBgAAVlbfTeToEKQOwdZND6wbGQAAuGIqbbDcv0TiAgAAZI9VcyUSFwAAoHM7Q9PiAgAALM8OO0NLJC4AAEDnr+Fi3RlFEokLAAAQY1wAAICN0FUEAABso35wLi0uAADA8iprTkkicQEAADZwbmdoBucCAACLs8PO0BKJCwAAEINzAQCAjbTbxCU/P18zZsxQTEyMDMPQ5s2bL/qZHTt2KDExUWFhYerXr59effXVBnVyc3M1dOhQORwODR06VJs2bfI1NAAA0ErttquoqqpK8fHxWr16dYvqFxUVafr06Ro3bpwKCgr0+OOP6/vf/75yc3M9dfbs2aPU1FTNmTNHH3/8sebMmaPZs2frz3/+s6/hAQCAVqiyyV5FhmmaZqs/bBjatGmTZs2a1WSdRYsW6a233tKBAwc8ZQsWLNDHH3+sPXv2SJJSU1Plcrn07rvveupMnTpV3bp1U3Z2doticblccjqdKi8vV0REROsuCACAADViyTZV1JzWB49MUN+ena7Y9/r6+33Zx7js2bNHKSkpXmVTpkzR3r17derUqWbr7N69u8nz1tTUyOVyeR0AAMB3pmmq0s1eRZKksrIyRUZGepVFRkbq9OnTOnbsWLN1ysrKmjxvRkaGnE6n54iNjW374AEACAAn3bWq73+xelfRFZlVZBiG1+v63qnzyxurc2HZ+dLT01VeXu45jhw50oYRAwAQOOpnFAUZUniItVtcLntaFRUV1aDl5OjRo+rQoYN69OjRbJ0LW2HO53A45HA42j5gAAACzPkzipprNLCCy97ikpycrLy8PK+y9957TyNHjlRISEizdcaMGXO5wwMAIODZZUaR1IoWl8rKSh0+fNjzuqioSIWFherevbv69Omj9PR0lZSUaP369ZLOzCBavXq10tLSdN9992nPnj1at26d12yhhx9+WOPHj9dzzz2nmTNnasuWLXr//fe1a9euNrhEAADQHLus4SK1osVl7969SkhIUEJCgiQpLS1NCQkJWrx4sSSptLRUxcXFnvp9+/bV1q1btX37dl1//fVavny5Vq1apdtvv91TZ8yYMdqwYYNee+01XXfddcrKylJOTo5Gjx59qdcHAAAuospGicslreNiJazjAgBA62wpLNHDGwo1dkAP/ebepCv63ZZbxwUAAFibp6so1PotLiQuAAAEuMpqe2ywKJG4AAAQ8Ow0xoXEBQCAAFdZPx06jMQFAABYXH2LC11FAADA8jwbLIZae7l/icQFAICAxxgXAABgG3QVAQAA26gfnEuLCwAAsLzKmlOSSFwAAIAN2Gl3aBIXAAACXP2S/6zjAgAALO1UbZ3cp+skSZ3ZqwgAAFhZ/YwiSerkYB0XAABgYfXdRI4OQeoQbP20wPoRAgCAy8ZOA3MlEhcAAAJapY1WzZVIXAAACGgkLgAAwDbOLfdv/YG5EokLAAABrdJG+xRJJC4AAAQ0O+0MLZG4AAAQ0Oy0M7RE4gIAQECz087QEokLAAABja4iAABgG8wqAgAAtlFBiwsAALALBucCAADb8IxxCSVxAQAAFudZgC6MxAUAAFgcu0MDAADbYDo0AACwBdM0VeWuT1yYDg0AACzsP6dqVWee+ZuuIgAAYGmV1WdaW4IMKTyEFhcAAGBhledNhTYMw8/RtAyJCwAAAarKZhssSiQuAAAELLut4SKRuAAAELDsNhVaamXi8vLLL6tv374KCwtTYmKidu7c2WTdefPmyTCMBsewYcM8dbKyshqtU11d3ZrwAABAC9RPhbbLztBSKxKXnJwcLVy4UE888YQKCgo0btw4TZs2TcXFxY3WX7lypUpLSz3HkSNH1L17d91xxx1e9SIiIrzqlZaWKiwsrHVXBQAALqrSZvsUSa1IXF544QXNnz9f9957r6699lplZmYqNjZWr7zySqP1nU6noqKiPMfevXv11Vdf6e677/aqZxiGV72oqKjWXREAAGgRu+0MLfmYuLjdbu3bt08pKSle5SkpKdq9e3eLzrFu3TpNmjRJcXFxXuWVlZWKi4tT7969dcstt6igoKDZ89TU1MjlcnkdAACg5Srb+6yiY8eOqba2VpGRkV7lkZGRKisru+jnS0tL9e677+ree+/1Kh8yZIiysrL01ltvKTs7W2FhYRo7dqwOHTrU5LkyMjLkdDo9R2xsrC+XAgBAwKtfgK7dJi71LlykxjTNFi1ck5WVpa5du2rWrFle5UlJSbrzzjsVHx+vcePG6Y033tCgQYP00ksvNXmu9PR0lZeXe44jR4605lIAAAhY57qK7DM416cUq2fPngoODm7QunL06NEGrTAXMk1Tv/zlLzVnzhyFhoY2WzcoKEijRo1qtsXF4XDI4XC0PHgAAOCl0t3Ox7iEhoYqMTFReXl5XuV5eXkaM2ZMs5/dsWOHDh8+rPnz51/0e0zTVGFhoaKjo30JDwAA+MCO67j4HGlaWprmzJmjkSNHKjk5WWvXrlVxcbEWLFgg6UwXTklJidavX+/1uXXr1mn06NEaPnx4g3MuW7ZMSUlJGjhwoFwul1atWqXCwkKtWbOmlZcFAAAuxo6zinyONDU1VcePH9fTTz+t0tJSDR8+XFu3bvXMEiotLW2wpkt5eblyc3O1cuXKRs954sQJ3X///SorK5PT6VRCQoLy8/N14403tuKSAABAS9hxVpFhmqbp7yDagsvlktPpVHl5uSIiIvwdDgAAljf+Jx+o+N8nlfvdMUqM6+aXGHz9/WavIgAAApQdu4pIXAAACFAVnsG59pkOTeICAEAAOlVbJ/fpOkm0uAAAAIur7yaS7DU4l8QFAIAAVL8ztKNDkEKC7ZMO2CdSAADQZqrOToW2UzeRROICAEBAqrThqrkSiQsAAAHJjsv9SyQuAAAEJDvuDC2RuAAAEJAqaHEBAAB2QVcRAACwDU9XUSiJCwAAsLj6naE7h5G4AAAAi6OrCAAA2AazigAAgG2wAB0AALCNKnd9iwuJCwAAsLjK6rMtLswqAgAAVkdXEQAAsA12hwYAALbhmVXEOi4AAMDKTNP0DM7txHRoAABgZf85Vas688zfdBUBAABLqx+YG2RI4SG0uAAAAAurH5jbKbSDDMPwczS+IXEBACDA2HWfIonEBQCAgFNRbc+BuRKJCwAAAefcBou0uAAAAIvz7FNkszVcJBIXAAACjme5f5vtUySRuAAAEHDoKgIAALZRWT8dmsQFAABYHdOhAQCAbZzrKmI6NAAAsLgKWlwAAIBd0FUEAABsoz5x6ULiAgAArC7gZhW9/PLL6tu3r8LCwpSYmKidO3c2WXf79u0yDKPB8be//c2rXm5uroYOHSqHw6GhQ4dq06ZNrQkNAABcREB1FeXk5GjhwoV64oknVFBQoHHjxmnatGkqLi5u9nMHDx5UaWmp5xg4cKDnvT179ig1NVVz5szRxx9/rDlz5mj27Nn685//7PsVAQCAZtl5ATrDNE3Tlw+MHj1aN9xwg1555RVP2bXXXqtZs2YpIyOjQf3t27dr4sSJ+uqrr9S1a9dGz5mamiqXy6V3333XUzZ16lR169ZN2dnZLYrL5XLJ6XSqvLxcERERvlwSAAABZfCT76rmdJ12LZqo3t06+jUWX3+/fWpxcbvd2rdvn1JSUrzKU1JStHv37mY/m5CQoOjoaN1888364IMPvN7bs2dPg3NOmTKl2XPW1NTI5XJ5HQAAoHmnautUc7pOkj1bXHxKXI4dO6ba2lpFRkZ6lUdGRqqsrKzRz0RHR2vt2rXKzc3Vxo0bNXjwYN18883Kz8/31CkrK/PpnJKUkZEhp9PpOWJjY325FAAAAlJ9N5FkzzEurYrYMAyv16ZpNiirN3jwYA0ePNjzOjk5WUeOHNHzzz+v8ePHt+qckpSenq60tDTPa5fLRfICAMBF1O8MHdohSCHB9ptc7FPEPXv2VHBwcIOWkKNHjzZoMWlOUlKSDh065HkdFRXl8zkdDociIiK8DgAA0Lyqs1Oh7biGi+Rj4hIaGqrExETl5eV5lefl5WnMmDEtPk9BQYGio6M9r5OTkxuc87333vPpnAAA4OIqbTwVWmpFV1FaWprmzJmjkSNHKjk5WWvXrlVxcbEWLFgg6UwXTklJidavXy9JyszM1DXXXKNhw4bJ7Xbr9ddfV25urnJzcz3nfPjhhzV+/Hg999xzmjlzprZs2aL3339fu3btaqPLBAAAkr3XcJFakbikpqbq+PHjevrpp1VaWqrhw4dr69atiouLkySVlpZ6renidrv1yCOPqKSkROHh4Ro2bJjeeecdTZ8+3VNnzJgx2rBhg5588kk99dRT6t+/v3JycjR69Og2uEQAAFDPzjtDS61Yx8WqWMcFAICLe3PvEf337z7RhMG9lHX3jf4O5/Ku4wIAAOzN7l1FJC4AAASQ+sG5nUNJXAAAgMXZeWdoicQFAICAYvfBuSQuAAAEEE/iEkaLCwAAsDi7L0BH4gIAQACpctd3FZG4AAAAi/MMzmVWEQAAsDrWcQEAALZRWU1XEQAAsIlzLS5MhwYAABZmmiaDcwEAgD3851St6s5urcw6LgAAwNLq13AJMqTwELqKAACAhVWdNxXaMAw/R9M6JC4AAAQIu0+FlkhcAAAIGJU2n1EkkbgAABAwzu0MTYsLAACwOLtvsCiRuAAAEDBIXAAAgG3UdxV1IXEBAABW59kZmsQFAABYHdOhAQCAbZybVcR0aAAAYHEMzgUAALZBVxEAALCNShagAwAAdsGsIgAAYBss+Q8AAGyDxAUAANgGu0MDAABbOF1bp5rTdZJocQEAABZXdXZgrsTgXAAAYHGV7jPdRKEdghQSbN+ff/tGDgAAWqyy2v4DcyUSFwAAAkJ7GJgrkbgAABAQzk2FDvFzJJeGxAUAgADQHnaGllqZuLz88svq27evwsLClJiYqJ07dzZZd+PGjZo8ebJ69eqliIgIJScna9u2bV51srKyZBhGg6O6uro14QEAgAu0h52hpVYkLjk5OVq4cKGeeOIJFRQUaNy4cZo2bZqKi4sbrZ+fn6/Jkydr69at2rdvnyZOnKgZM2aooKDAq15ERIRKS0u9jrCwsNZdFQAA8NIedoaWJJ+jf+GFFzR//nzde++9kqTMzExt27ZNr7zyijIyMhrUz8zM9Hr9zDPPaMuWLXr77beVkJDgKTcMQ1FRUb6GAwAAWqDKfWYdl86h9k5cfGpxcbvd2rdvn1JSUrzKU1JStHv37hado66uThUVFerevbtXeWVlpeLi4tS7d2/dcsstDVpkLlRTUyOXy+V1AACAxgVkV9GxY8dUW1uryMhIr/LIyEiVlZW16BwrVqxQVVWVZs+e7SkbMmSIsrKy9NZbbyk7O1thYWEaO3asDh061OR5MjIy5HQ6PUdsbKwvlwIAQEA5t45LAA7ONQzD67Vpmg3KGpOdna2lS5cqJydHV111lac8KSlJd955p+Lj4zVu3Di98cYbGjRokF566aUmz5Wenq7y8nLPceTIkdZcCgAAASEgx7j07NlTwcHBDVpXjh492qAV5kI5OTmaP3++3nzzTU2aNKnZukFBQRo1alSzLS4Oh0MOh6PlwQMAEMACsqsoNDRUiYmJysvL8yrPy8vTmDFjmvxcdna25s2bp9/+9rf6xje+cdHvMU1ThYWFio6O9iU8AADQhKqzexV1CbN34uJz9GlpaZozZ45Gjhyp5ORkrV27VsXFxVqwYIGkM104JSUlWr9+vaQzSctdd92llStXKikpydNaEx4eLqfTKUlatmyZkpKSNHDgQLlcLq1atUqFhYVas2ZNW10nAAABrfLs7tCdbD6ryOfoU1NTdfz4cT399NMqLS3V8OHDtXXrVsXFxUmSSktLvdZ0+dnPfqbTp0/rwQcf1IMPPugpnzt3rrKysiRJJ06c0P3336+ysjI5nU4lJCQoPz9fN9544yVeHgAAkNrPGBfDNE3T30G0BZfLJafTqfLyckVERPg7HAAALCU5439UWl6ttx+6SSN6O/0djoevv9/sVQQAQABgd2gAAGALpmmet8mivbuKSFwAAGjn/nOqVnVnB4bYfYwLiQsAAO1cfTeRYUgdQ+kqAgAAFlZVc26DxZasdG9lJC4AALRz7WUqtETiAgBAu9deZhRJJC4AALR77WVGkUTiAgBAu9deNliUSFwAAGj3jrpqJJG4AAAAi6s+VavX/lQkSUrq18PP0Vw6EhcAANqx1z/8XF+WVyvaGabvjO7j73AuGYkLAADtVEX1Ka354LAkaeGkgQoLYVYRAACwqF/sLNJXJ0+pX69Ouv2G3v4Op02QuAAA0A4dr6zRL3b+Q5L0SMpgdQhuHz/57eMqAACAlzUffKYqd61GXO3UtOFR/g6nzZC4AADQznzx1Um9/uHnkqRHpw62/f5E5yNxAQCgnVn5/iG5a+uU3K+HbhrQ09/htCkSFwAA2pFD/1eh3L98Ian9tbZIJC4AALQrK977u+pMKWVopBL6dPN3OG2OxAUAgHai8MgJ/eGvZQoypEemDPZ3OJcFiQsAAO3ET7f9TZJ0a0JvDYrs4udoLg8SFwAA2oFdh47pT4ePKzQ4SAsnDfR3OJcNiQsAADZnmqZ+cra15TtJfRTbvaOfI7p8SFwAALC5P/xvmT75olwdQ4P14MQB/g7nsiJxAQDAxk7X1un59w5Kku4d1089Ozv8HNHlReICAICNbfxLiT77V5W6dQzRfeP6+jucy47EBQAAm6o+VavM9/8uSXpw4gB1CQvxc0SXH4kLAAA29fqHn+vL8mpFO8N0Z1Kcv8O5IkhcAACwoYrqU3p5+2eSpIWTBiosJNjPEV0ZJC4AANjQL3YW6d9VbvXr1Um339Db3+FcMSQuAADYzPHKGv1i5z8kSY+kDFaH4MD5OQ+cKwUAoJ1Y88FnqnLXasTVTk0bHuXvcK4oEhcAAGyk5MR/9PqHn0uSHp06WIZh+DmiK4vEBQAAG8nM+7vctXVK7tdDNw3o6e9wrjgSFwAAbOLw0Qrl/uULSYHZ2iKRuAAAYBvPb/u76kwpZWikEvp083c4fkHiAgCADXx85IT+8NcyBRnSI1MG+zscv2lV4vLyyy+rb9++CgsLU2Jionbu3Nls/R07digxMVFhYWHq16+fXn311QZ1cnNzNXToUDkcDg0dOlSbNm1qTWgAALQLdXWm/vGvSm0pLNHy33+q72UXSJJuTeitQZFd/Byd/3Tw9QM5OTlauHChXn75ZY0dO1Y/+9nPNG3aNH366afq06dPg/pFRUWaPn267rvvPr3++uv605/+pAceeEC9evXS7bffLknas2ePUlNTtXz5ct16663atGmTZs+erV27dmn06NGXfpUAAFiYaZoq/vdJffJFufaXlGv/F+X635JyVdSc9qoXEdZBCycN9FOU1mCYpmn68oHRo0frhhtu0CuvvOIpu/baazVr1ixlZGQ0qL9o0SK99dZbOnDggKdswYIF+vjjj7Vnzx5JUmpqqlwul959911PnalTp6pbt27Kzs5uUVwul0tOp1Pl5eWKiIjw5ZKaZprSqZNtcy4AAHQmSSk5Ua2/flmu/y1x6dMvy/W/X5bLVX26QV1HhyANiY7Q8OgIDbvaqTEDeiiyS5gfoj4rpKPUxgOCff399qnFxe12a9++fXrssce8ylNSUrR79+5GP7Nnzx6lpKR4lU2ZMkXr1q3TqVOnFBISoj179ugHP/hBgzqZmZlNxlJTU6OamhrPa5fL5cultMypk9IzMW1/XgBAwDIk9T57TDn/jabykX+dPT65vHG1yONfSqGd/BqCT2Ncjh07ptraWkVGRnqVR0ZGqqysrNHPlJWVNVr/9OnTOnbsWLN1mjqnJGVkZMjpdHqO2NhYXy4FAADYkM9jXCQ1mDdummazc8kbq39hua/nTE9PV1pamue1y+Vq++QlpKM2TP5QZeXVbXteAIAMGRcWNPfSq4vCkBRkGAoypKCgM+We14YhwzjzuxJknPkXev1rwzjzvcFB9fUNBQcZCgoyFHze+YIMQ0Fn6wTXv9aZvyWpzjTPHlJt/d91Z17X1Zkyz3vPNE3Vnn3PkNT/qs4aFNlZjg423M05pKO/I/AtcenZs6eCg4MbtIQcPXq0QYtJvaioqEbrd+jQQT169Gi2TlPnlCSHwyGHw+FL+L4zDH1r7LWX9zsAAECL+dRVFBoaqsTEROXl5XmV5+XlacyYMY1+Jjk5uUH99957TyNHjlRISEizdZo6JwAACEw+dxWlpaVpzpw5GjlypJKTk7V27VoVFxdrwYIFks504ZSUlGj9+vWSzswgWr16tdLS0nTfffdpz549WrdunddsoYcffljjx4/Xc889p5kzZ2rLli16//33tWvXrja6TAAA0B74nLikpqbq+PHjevrpp1VaWqrhw4dr69atiouLkySVlpaquLjYU79v377aunWrfvCDH2jNmjWKiYnRqlWrPGu4SNKYMWO0YcMGPfnkk3rqqafUv39/5eTksIYLAADw4vM6LlZ1WdZxAQAAl5Wvv9/sVQQAAGyDxAUAANgGiQsAALANEhcAAGAbJC4AAMA2SFwAAIBtkLgAAADbIHEBAAC2QeICAABsw+cl/62qfgFgl8vl50gAAEBL1f9ut3Qh/3aTuFRUVEiSYmNj/RwJAADwVUVFhZxO50XrtZu9iurq6vTll1+qS5cuMgyjzc7rcrkUGxurI0eOsAeSD7hvrcN98x33rHW4b63DfWud5u6baZqqqKhQTEyMgoIuPoKl3bS4BAUFqXfv3pft/BERETykrcB9ax3um++4Z63DfWsd7lvrNHXfWtLSUo/BuQAAwDZIXAAAgG2QuFyEw+HQkiVL5HA4/B2KrXDfWof75jvuWetw31qH+9Y6bXnf2s3gXAAA0P7R4gIAAGyDxAUAANgGiQsAALANEhcAAGAbJC4X8fLLL6tv374KCwtTYmKidu7c6e+QLG3p0qUyDMPriIqK8ndYlpKfn68ZM2YoJiZGhmFo8+bNXu+bpqmlS5cqJiZG4eHhmjBhgv7617/6J1gLudh9mzdvXoNnLykpyT/BWkRGRoZGjRqlLl266KqrrtKsWbN08OBBrzo8bw215L7xvDX0yiuv6LrrrvMsMpecnKx3333X835bPWskLs3IycnRwoUL9cQTT6igoEDjxo3TtGnTVFxc7O/QLG3YsGEqLS31HPv37/d3SJZSVVWl+Ph4rV69utH3f/KTn+iFF17Q6tWr9dFHHykqKkqTJ0/27McVqC523yRp6tSpXs/e1q1br2CE1rNjxw49+OCD+vDDD5WXl6fTp08rJSVFVVVVnjo8bw215L5JPG8X6t27t5599lnt3btXe/fu1de//nXNnDnTk5y02bNmokk33nijuWDBAq+yIUOGmI899pifIrK+JUuWmPHx8f4OwzYkmZs2bfK8rqurM6Oiosxnn33WU1ZdXW06nU7z1Vdf9UOE1nThfTNN05w7d645c+ZMv8RjF0ePHjUlmTt27DBNk+etpS68b6bJ89ZS3bp1M3/xi1+06bNGi0sT3G639u3bp5SUFK/ylJQU7d69209R2cOhQ4cUExOjvn376lvf+pb+8Y9/+Dsk2ygqKlJZWZnXc+dwOPS1r32N564Ftm/frquuukqDBg3Sfffdp6NHj/o7JEspLy+XJHXv3l0Sz1tLXXjf6vG8Na22tlYbNmxQVVWVkpOT2/RZI3FpwrFjx1RbW6vIyEiv8sjISJWVlfkpKusbPXq01q9fr23btunnP/+5ysrKNGbMGB0/ftzfodlC/bPFc+e7adOm6Te/+Y3++Mc/asWKFfroo4/09a9/XTU1Nf4OzRJM01RaWppuuukmDR8+XBLPW0s0dt8knrem7N+/X507d5bD4dCCBQu0adMmDR06tE2ftXazO/TlYhiG12vTNBuU4Zxp06Z5/h4xYoSSk5PVv39//epXv1JaWpofI7MXnjvfpaamev4ePny4Ro4cqbi4OL3zzju67bbb/BiZNTz00EP65JNPtGvXrgbv8bw1ran7xvPWuMGDB6uwsFAnTpxQbm6u5s6dqx07dnjeb4tnjRaXJvTs2VPBwcENMsGjR482yBjRtE6dOmnEiBE6dOiQv0OxhfoZWDx3ly46OlpxcXE8e5K+973v6a233tIHH3yg3r17e8p53prX1H1rDM/bGaGhoRowYIBGjhypjIwMxcfHa+XKlW36rJG4NCE0NFSJiYnKy8vzKs/Ly9OYMWP8FJX91NTU6MCBA4qOjvZ3KLbQt29fRUVFeT13brdbO3bs4Lnz0fHjx3XkyJGAfvZM09RDDz2kjRs36o9//KP69u3r9T7PW+Mudt8aw/PWONM0VVNT07bPWhsNHG6XNmzYYIaEhJjr1q0zP/30U3PhwoVmp06dzH/+85/+Ds2yfvjDH5rbt283//GPf5gffvihecstt5hdunThnp2noqLCLCgoMAsKCkxJ5gsvvGAWFBSYn3/+uWmapvnss8+aTqfT3Lhxo7l//37z29/+thkdHW26XC4/R+5fzd23iooK84c//KG5e/dus6ioyPzggw/M5ORk8+qrrw7o+/bd737XdDqd5vbt283S0lLPcfLkSU8dnreGLnbfeN4al56ebubn55tFRUXmJ598Yj7++ONmUFCQ+d5775mm2XbPGonLRaxZs8aMi4szQ0NDzRtuuMFrOhwaSk1NNaOjo82QkBAzJibGvO2228y//vWv/g7LUj744ANTUoNj7ty5pmmemaK6ZMkSMyoqynQ4HOb48ePN/fv3+zdoC2juvp08edJMSUkxe/XqZYaEhJh9+vQx586daxYXF/s7bL9q7H5JMl977TVPHZ63hi5233jeGnfPPfd4fi979epl3nzzzZ6kxTTb7lkzTNM0W9kCBAAAcEUxxgUAANgGiQsAALANEhcAAGAbJC4AAMA2SFwAAIBtkLgAAADbIHEBAAC2QeICAABsg8QFAADYBokLAACwDRIXAABgGyQuAADANv4/UjXIoov3qXkAAAAASUVORK5CYII=", - "text/plain": [ - "
          " - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 11, + "id": "b038687c", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", @@ -1751,7 +2017,7 @@ "\n", "np.random.seed(2018)\n", "n = 50\n", - "maxdegree = 30\n", + "maxdegree = 5\n", "# Make data set.\n", "x = np.linspace(-3, 3, n).reshape(-1, 1)\n", "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n", @@ -1781,8 +2047,10 @@ }, { "cell_type": "markdown", - "id": "1ed6b276", - "metadata": {}, + "id": "36f38098", + "metadata": { + "editable": true + }, "source": [ "## More preprocessing examples, two-dimensional example, the Franke function" ] @@ -1790,8 +2058,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "b6da6f80", - "metadata": {}, + "id": "46512f53", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -1890,8 +2161,10 @@ }, { "cell_type": "markdown", - "id": "a920b2aa", - "metadata": {}, + "id": "afb7e208", + "metadata": { + "editable": true + }, "source": [ "## To think about, first part\n", "\n", @@ -1916,8 +2189,10 @@ }, { "cell_type": "markdown", - "id": "5b2edf17", - "metadata": {}, + "id": "9aea8569", + "metadata": { + "editable": true + }, "source": [ "## More thinking\n", "\n", @@ -1949,8 +2224,10 @@ }, { "cell_type": "markdown", - "id": "9895401e", - "metadata": {}, + "id": "3c3ee7a6", + "metadata": { + "editable": true + }, "source": [ "## Still thinking\n", "\n", @@ -1962,8 +2239,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "5c9604e8", - "metadata": {}, + "id": "610ed7c9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "#Model training, we compute the mean value of y and X\n", @@ -1984,8 +2264,10 @@ }, { "cell_type": "markdown", - "id": "bb31dcc4", - "metadata": {}, + "id": "5aaa0c35", + "metadata": { + "editable": true + }, "source": [ "## What does centering (subtracting the mean values) mean mathematically?\n", "\n", @@ -1998,8 +2280,10 @@ }, { "cell_type": "markdown", - "id": "69732dbc", - "metadata": {}, + "id": "d446b89c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", @@ -2008,8 +2292,10 @@ }, { "cell_type": "markdown", - "id": "fe76e2ee", - "metadata": {}, + "id": "198b6351", + "metadata": { + "editable": true + }, "source": [ "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n", "\n", @@ -2021,8 +2307,10 @@ }, { "cell_type": "markdown", - "id": "7aeb050c", - "metadata": {}, + "id": "f87a12ef", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", @@ -2031,16 +2319,20 @@ }, { "cell_type": "markdown", - "id": "b9beeff0", - "metadata": {}, + "id": "9c9aeaff", + "metadata": { + "editable": true + }, "source": [ "for all $j$. For $\\beta_0$ we have" ] }, { "cell_type": "markdown", - "id": "dcd138b2", - "metadata": {}, + "id": "f4c3cf71", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", @@ -2049,16 +2341,20 @@ }, { "cell_type": "markdown", - "id": "ca257ffd", - "metadata": {}, + "id": "bfc99033", + "metadata": { + "editable": true + }, "source": [ "Multiplying away the constant $2/n$, we obtain" ] }, { "cell_type": "markdown", - "id": "e8fa46c2", - "metadata": {}, + "id": "7982def4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", @@ -2067,8 +2363,10 @@ }, { "cell_type": "markdown", - "id": "bd766850", - "metadata": {}, + "id": "3610d0b0", + "metadata": { + "editable": true + }, "source": [ "## Further Manipulations\n", "\n", @@ -2078,8 +2376,10 @@ }, { "cell_type": "markdown", - "id": "bf315cd2", - "metadata": {}, + "id": "8bdee448", + "metadata": { + "editable": true + }, "source": [ "$$\n", "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", @@ -2088,16 +2388,20 @@ }, { "cell_type": "markdown", - "id": "92da7a90", - "metadata": {}, + "id": "a6a3d795", + "metadata": { + "editable": true + }, "source": [ "We obtain then" ] }, { "cell_type": "markdown", - "id": "206731ce", - "metadata": {}, + "id": "ecf6d960", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", @@ -2106,16 +2410,20 @@ }, { "cell_type": "markdown", - "id": "4ceacbfb", - "metadata": {}, + "id": "3477f330", + "metadata": { + "editable": true + }, "source": [ "If we define" ] }, { "cell_type": "markdown", - "id": "9dd26f70", - "metadata": {}, + "id": "3faf2dd4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n", @@ -2124,16 +2432,20 @@ }, { "cell_type": "markdown", - "id": "2deddeaa", - "metadata": {}, + "id": "c0eb3a77", + "metadata": { + "editable": true + }, "source": [ "and if we define the mean value of the outputs as" ] }, { "cell_type": "markdown", - "id": "aa10bb43", - "metadata": {}, + "id": "f9884460", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n", @@ -2142,16 +2454,20 @@ }, { "cell_type": "markdown", - "id": "fce60eb7", - "metadata": {}, + "id": "94e51a90", + "metadata": { + "editable": true + }, "source": [ "we have" ] }, { "cell_type": "markdown", - "id": "f4ec9b91", - "metadata": {}, + "id": "a3c36461", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n", @@ -2160,16 +2476,20 @@ }, { "cell_type": "markdown", - "id": "77a212bc", - "metadata": {}, + "id": "571b9c21", + "metadata": { + "editable": true + }, "source": [ "In the general case, that is we have more parameters than $\\beta_0$ and $\\beta_1$, we have" ] }, { "cell_type": "markdown", - "id": "62a38cf1", - "metadata": {}, + "id": "a7497fca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", @@ -2178,16 +2498,20 @@ }, { "cell_type": "markdown", - "id": "d8bbd737", - "metadata": {}, + "id": "eceeabf7", + "metadata": { + "editable": true + }, "source": [ "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)" ] }, { "cell_type": "markdown", - "id": "9a0d298a", - "metadata": {}, + "id": "36d559dd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", @@ -2196,8 +2520,10 @@ }, { "cell_type": "markdown", - "id": "d111103f", - "metadata": {}, + "id": "51b9a971", + "metadata": { + "editable": true + }, "source": [ "## Wrapping it up\n", "\n", @@ -2206,8 +2532,10 @@ }, { "cell_type": "markdown", - "id": "92479700", - "metadata": {}, + "id": "1d5cc9a6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", @@ -2216,8 +2544,10 @@ }, { "cell_type": "markdown", - "id": "462e5487", - "metadata": {}, + "id": "53ea8a95", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", @@ -2227,8 +2557,10 @@ }, { "cell_type": "markdown", - "id": "17ac9c99", - "metadata": {}, + "id": "786d0fee", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", @@ -2237,16 +2569,20 @@ }, { "cell_type": "markdown", - "id": "73463eed", - "metadata": {}, + "id": "4192a1c6", + "metadata": { + "editable": true + }, "source": [ "What does this mean? And why do we insist on all this? Let us look at some examples." ] }, { "cell_type": "markdown", - "id": "79b970d8", - "metadata": {}, + "id": "da29b29a", + "metadata": { + "editable": true + }, "source": [ "## Linear Regression code, Intercept handling first\n", "\n", @@ -2257,8 +2593,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "bc93a61f", - "metadata": {}, + "id": "7e1a63ba", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -2351,8 +2690,10 @@ }, { "cell_type": "markdown", - "id": "67ed06a9", - "metadata": {}, + "id": "93c527ef", + "metadata": { + "editable": true + }, "source": [ "The intercept is the value of our output/target variable\n", "when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n", @@ -2370,8 +2711,10 @@ }, { "cell_type": "markdown", - "id": "c1e2182e", - "metadata": {}, + "id": "95f1207f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", @@ -2380,16 +2723,20 @@ }, { "cell_type": "markdown", - "id": "aca3fa40", - "metadata": {}, + "id": "28924a72", + "metadata": { + "editable": true + }, "source": [ "but when we take out the intercept, this equation becomes" ] }, { "cell_type": "markdown", - "id": "126ed4ec", - "metadata": {}, + "id": "2335128f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", @@ -2398,16 +2745,20 @@ }, { "cell_type": "markdown", - "id": "c2769dc4", - "metadata": {}, + "id": "04fcb5bb", + "metadata": { + "editable": true + }, "source": [ "For Lasso regression we have" ] }, { "cell_type": "markdown", - "id": "e8e8cb90", - "metadata": {}, + "id": "d5795062", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", @@ -2416,8 +2767,10 @@ }, { "cell_type": "markdown", - "id": "58a1cb4a", - "metadata": {}, + "id": "2dc0f045", + "metadata": { + "editable": true + }, "source": [ "It means that, when scaling the design matrix and the outputs/targets,\n", "by subtracting the mean values, we have an optimization problem which\n", @@ -2430,16 +2783,20 @@ }, { "cell_type": "markdown", - "id": "9e5a31cc", - "metadata": {}, + "id": "6cad8214", + "metadata": { + "editable": true + }, "source": [ "## Material for lecture Monday, August 26" ] }, { "cell_type": "markdown", - "id": "77639bfa", - "metadata": {}, + "id": "2bad21b9", + "metadata": { + "editable": true + }, "source": [ "## Mathematical Interpretation of Ordinary Least Squares\n", "\n", @@ -2450,8 +2807,10 @@ }, { "cell_type": "markdown", - "id": "9a83aba8", - "metadata": {}, + "id": "644c911c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -2460,8 +2819,10 @@ }, { "cell_type": "markdown", - "id": "209e5dfb", - "metadata": {}, + "id": "e8fd561c", + "metadata": { + "editable": true + }, "source": [ "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n", "\n", @@ -2470,8 +2831,10 @@ }, { "cell_type": "markdown", - "id": "a3c4e2aa", - "metadata": {}, + "id": "727edd59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -2480,16 +2843,20 @@ }, { "cell_type": "markdown", - "id": "1dfcaa45", - "metadata": {}, + "id": "8c0e5f99", + "metadata": { + "editable": true + }, "source": [ "We now define a matrix" ] }, { "cell_type": "markdown", - "id": "d652144f", - "metadata": {}, + "id": "2a29fb9d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T.\n", @@ -2498,16 +2865,20 @@ }, { "cell_type": "markdown", - "id": "f9f01db8", - "metadata": {}, + "id": "2a148675", + "metadata": { + "editable": true + }, "source": [ "We can rewrite" ] }, { "cell_type": "markdown", - "id": "846edb22", - "metadata": {}, + "id": "e35bcb63", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n", @@ -2516,8 +2887,10 @@ }, { "cell_type": "markdown", - "id": "6e7169a8", - "metadata": {}, + "id": "23e05e57", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n", "We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix." @@ -2525,8 +2898,10 @@ }, { "cell_type": "markdown", - "id": "76a4f3af", - "metadata": {}, + "id": "26d2aa42", + "metadata": { + "editable": true + }, "source": [ "## Residual Error\n", "\n", @@ -2535,8 +2910,10 @@ }, { "cell_type": "markdown", - "id": "174980e4", - "metadata": {}, + "id": "c0a9222f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=\\left[\\boldsymbol{I}-\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\right]\\boldsymbol{y}.\n", @@ -2545,16 +2922,20 @@ }, { "cell_type": "markdown", - "id": "5f09c446", - "metadata": {}, + "id": "8144224f", + "metadata": { + "editable": true + }, "source": [ "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal component of the space defined by the column vectors of $\\boldsymbol{X}$." ] }, { "cell_type": "markdown", - "id": "8c89d3c7", - "metadata": {}, + "id": "f3669729", + "metadata": { + "editable": true + }, "source": [ "## Simple case\n", "\n", @@ -2563,8 +2944,10 @@ }, { "cell_type": "markdown", - "id": "dffb0161", - "metadata": {}, + "id": "67dccbf0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{X}\\boldsymbol{X}^T = \\boldsymbol{I}.\n", @@ -2573,16 +2956,20 @@ }, { "cell_type": "markdown", - "id": "e413c54c", - "metadata": {}, + "id": "243c515d", + "metadata": { + "editable": true + }, "source": [ "In this case the matrix $\\boldsymbol{A}$ becomes" ] }, { "cell_type": "markdown", - "id": "ba8769cf", - "metadata": {}, + "id": "3025d27b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n", @@ -2591,16 +2978,20 @@ }, { "cell_type": "markdown", - "id": "94c722f1", - "metadata": {}, + "id": "8d03e62a", + "metadata": { + "editable": true + }, "source": [ "and we have the obvious case" ] }, { "cell_type": "markdown", - "id": "6dc80f7c", - "metadata": {}, + "id": "6a26e615", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=0.\n", @@ -2609,16 +3000,20 @@ }, { "cell_type": "markdown", - "id": "2d53c537", - "metadata": {}, + "id": "6b6112b3", + "metadata": { + "editable": true + }, "source": [ "This serves also as a useful test of our codes." ] }, { "cell_type": "markdown", - "id": "0894bac8", - "metadata": {}, + "id": "3d09419a", + "metadata": { + "editable": true + }, "source": [ "## The singular value decomposition\n", "\n", @@ -2655,8 +3050,10 @@ }, { "cell_type": "markdown", - "id": "0e569502", - "metadata": {}, + "id": "d0b08069", + "metadata": { + "editable": true + }, "source": [ "## Linear Regression Problems\n", "\n", @@ -2670,8 +3067,10 @@ }, { "cell_type": "markdown", - "id": "497ed48c", - "metadata": {}, + "id": "fb79434f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -2691,8 +3090,10 @@ }, { "cell_type": "markdown", - "id": "6ec2d053", - "metadata": {}, + "id": "7454590c", + "metadata": { + "editable": true + }, "source": [ "The columns of $\\boldsymbol{X}$ are linearly dependent. We see this easily since the \n", "the first column is the row-wise sum of the other two columns. The rank (more correct,\n", @@ -2706,8 +3107,10 @@ }, { "cell_type": "markdown", - "id": "f21de103", - "metadata": {}, + "id": "e5607549", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -2723,8 +3126,10 @@ }, { "cell_type": "markdown", - "id": "41a3e5f1", - "metadata": {}, + "id": "7cb60f09", + "metadata": { + "editable": true + }, "source": [ "We see easily that $\\mbox{det}(\\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \\times (-1) - 1 \\times (-1) = 0$. Hence, $\\mathbf{X}$ is singular and its inverse is undefined.\n", "This is equivalent to saying that the matrix $\\boldsymbol{X}$ has at least an eigenvalue which is zero." @@ -2732,8 +3137,10 @@ }, { "cell_type": "markdown", - "id": "562e348f", - "metadata": {}, + "id": "4ffab4c8", + "metadata": { + "editable": true + }, "source": [ "## Fixing the singularity\n", "\n", @@ -2742,8 +3149,10 @@ }, { "cell_type": "markdown", - "id": "d9925625", - "metadata": {}, + "id": "9398c2a3", + "metadata": { + "editable": true + }, "source": [ "\n", "
          \n", @@ -2758,8 +3167,10 @@ }, { "cell_type": "markdown", - "id": "ea7a41cc", - "metadata": {}, + "id": "d4eb85a1", + "metadata": { + "editable": true + }, "source": [ "has linearly dependent column vectors, we will not be able to compute the inverse\n", "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n", @@ -2772,8 +3183,10 @@ }, { "cell_type": "markdown", - "id": "c5bdb149", - "metadata": {}, + "id": "7826f21f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^{T} \\boldsymbol{X} \\rightarrow \\boldsymbol{X}^{T} \\boldsymbol{X}+\\lambda \\boldsymbol{I},\n", @@ -2782,16 +3195,20 @@ }, { "cell_type": "markdown", - "id": "fac1e4ef", - "metadata": {}, + "id": "f8e53e32", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{I}$ is the identity matrix. When we discuss **Ridge** regression this is actually what we end up evaluating. The parameter $\\lambda$ is called a hyperparameter. More about this later." ] }, { "cell_type": "markdown", - "id": "8b6cfbf1", - "metadata": {}, + "id": "9fdadcf2", + "metadata": { + "editable": true + }, "source": [ "## Basic math of the SVD\n", "\n", @@ -2803,8 +3220,10 @@ }, { "cell_type": "markdown", - "id": "6d652283", - "metadata": {}, + "id": "bb0f1a50", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\lambda_1,\\boldsymbol{u}_1),\\dots, (\\lambda_n,\\boldsymbol{u}_n),\n", @@ -2813,16 +3232,20 @@ }, { "cell_type": "markdown", - "id": "ff6cb23a", - "metadata": {}, + "id": "38de8410", + "metadata": { + "editable": true + }, "source": [ "and the eigenvalues are given by the diagonal matrix" ] }, { "cell_type": "markdown", - "id": "8166fb7b", - "metadata": {}, + "id": "c2e79f6c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\\mathrm{Diag}(\\lambda_1, \\dots,\\lambda_n).\n", @@ -2831,16 +3254,20 @@ }, { "cell_type": "markdown", - "id": "c274bcb2", - "metadata": {}, + "id": "5062f0d2", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ can be written in terms of an orthogonal/unitary transformation $\\boldsymbol{U}$" ] }, { "cell_type": "markdown", - "id": "2fc3a9b7", - "metadata": {}, + "id": "e82354ce", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -2849,8 +3276,10 @@ }, { "cell_type": "markdown", - "id": "3be571f3", - "metadata": {}, + "id": "8df6f261", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ or $\\boldsymbol{U}\\boldsymbol{U}^{\\dagger}=\\boldsymbol{I}$.\n", "\n", @@ -2859,8 +3288,10 @@ }, { "cell_type": "markdown", - "id": "c2f222d7", - "metadata": {}, + "id": "be644b5b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\begin{bmatrix} \n", @@ -2872,8 +3303,10 @@ }, { "cell_type": "markdown", - "id": "06c9a819", - "metadata": {}, + "id": "3a7b74dd", + "metadata": { + "editable": true + }, "source": [ "is not diagonalizable, it is a so-called [defective matrix](https://en.wikipedia.org/wiki/Defective_matrix). It is easy to see that the condition\n", "$\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ is not fulfilled." @@ -2881,8 +3314,10 @@ }, { "cell_type": "markdown", - "id": "889d8b92", - "metadata": {}, + "id": "c0f0b528", + "metadata": { + "editable": true + }, "source": [ "## The SVD, a Fantastic Algorithm\n", "\n", @@ -2899,8 +3334,10 @@ }, { "cell_type": "markdown", - "id": "6b7610f5", - "metadata": {}, + "id": "db75edea", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\n", @@ -2909,16 +3346,20 @@ }, { "cell_type": "markdown", - "id": "7a5006eb", - "metadata": {}, + "id": "7cc08b68", + "metadata": { + "editable": true + }, "source": [ "As an example, the above defective matrix can be decomposed as" ] }, { "cell_type": "markdown", - "id": "c26d0b45", - "metadata": {}, + "id": "eb30435e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& 1 \\\\ 1& -1\\\\ \\end{bmatrix} \\begin{bmatrix} 2& 0 \\\\ 0& 0\\\\ \\end{bmatrix} \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& -1 \\\\ 1& 1\\\\ \\end{bmatrix}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -2927,8 +3368,10 @@ }, { "cell_type": "markdown", - "id": "c63b7581", - "metadata": {}, + "id": "57f3ab78", + "metadata": { + "editable": true + }, "source": [ "with eigenvalues $\\sigma_1=2$ and $\\sigma_2=0$. \n", "The SVD exits always! \n", @@ -2954,8 +3397,10 @@ }, { "cell_type": "markdown", - "id": "7e8b925c", - "metadata": {}, + "id": "c66c4798", + "metadata": { + "editable": true + }, "source": [ "## Economy-size SVD\n", "\n", @@ -2979,8 +3424,10 @@ }, { "cell_type": "markdown", - "id": "5982c0d9", - "metadata": {}, + "id": "2db6a3d6", + "metadata": { + "editable": true + }, "source": [ "## Codes for the SVD" ] @@ -2988,8 +3435,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "522ca752", - "metadata": {}, + "id": "c7e2d452", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -3025,8 +3475,10 @@ }, { "cell_type": "markdown", - "id": "1084a7d6", - "metadata": {}, + "id": "4876d7fb", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ has columns that are linearly dependent. The first\n", "column is the row-wise sum of the other two columns. The rank of a\n", @@ -3040,8 +3492,10 @@ }, { "cell_type": "markdown", - "id": "798189f9", - "metadata": {}, + "id": "60a1b266", + "metadata": { + "editable": true + }, "source": [ "## Note about SVD Calculations\n", "\n", @@ -3061,8 +3515,10 @@ }, { "cell_type": "markdown", - "id": "f8220dcb", - "metadata": {}, + "id": "f19eab4b", + "metadata": { + "editable": true + }, "source": [ "## Mathematics of the SVD and implications\n", "\n", @@ -3073,8 +3529,10 @@ }, { "cell_type": "markdown", - "id": "c72578b5", - "metadata": {}, + "id": "d5e735b4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -3090,16 +3548,20 @@ }, { "cell_type": "markdown", - "id": "b6dd939c", - "metadata": {}, + "id": "2911819b", + "metadata": { + "editable": true + }, "source": [ "We can SVD decompose our matrix as" ] }, { "cell_type": "markdown", - "id": "f4368817", - "metadata": {}, + "id": "7e22b8f3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -3108,8 +3570,10 @@ }, { "cell_type": "markdown", - "id": "273a1ee3", - "metadata": {}, + "id": "aa21f027", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{U}$ is an orthogonal matrix of dimension $n\\times n$, meaning that $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{I}_n$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $n \\times n$.\n", "\n", @@ -3120,8 +3584,10 @@ }, { "cell_type": "markdown", - "id": "bd90c4cd", - "metadata": {}, + "id": "0ae64e85", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sigma_0 > \\sigma_1 > \\sigma_2 > \\dots > \\sigma_{p-1} > 0.\n", @@ -3130,16 +3596,20 @@ }, { "cell_type": "markdown", - "id": "7e755f46", - "metadata": {}, + "id": "25f04f52", + "metadata": { + "editable": true + }, "source": [ "All values beyond $p-1$ are all zero." ] }, { "cell_type": "markdown", - "id": "d79120e6", - "metadata": {}, + "id": "b9507486", + "metadata": { + "editable": true + }, "source": [ "## Example Matrix\n", "\n", @@ -3148,8 +3618,10 @@ }, { "cell_type": "markdown", - "id": "e653c8e7", - "metadata": {}, + "id": "a4f09fd9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\n", @@ -3163,16 +3635,20 @@ }, { "cell_type": "markdown", - "id": "30061962", - "metadata": {}, + "id": "89a4dd64", + "metadata": { + "editable": true + }, "source": [ "The singular values are $\\sigma_0=2$ and $\\sigma_1=1$. It is common to rewrite the matrix $\\boldsymbol{\\Sigma}$ as" ] }, { "cell_type": "markdown", - "id": "dd0fb7b8", - "metadata": {}, + "id": "e12c7f1f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\n", @@ -3185,16 +3661,20 @@ }, { "cell_type": "markdown", - "id": "1691c8c0", - "metadata": {}, + "id": "ce657a98", + "metadata": { + "editable": true + }, "source": [ "where" ] }, { "cell_type": "markdown", - "id": "e6ce2bff", - "metadata": {}, + "id": "bf65318f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{\\Sigma}}=\n", @@ -3207,16 +3687,20 @@ }, { "cell_type": "markdown", - "id": "9ec5da0d", - "metadata": {}, + "id": "ab7198f2", + "metadata": { + "editable": true + }, "source": [ "contains only the singular values. Note also (and we will use this below) that" ] }, { "cell_type": "markdown", - "id": "393a349c", - "metadata": {}, + "id": "58b82bec", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\n", @@ -3229,16 +3713,20 @@ }, { "cell_type": "markdown", - "id": "1f7f6262", - "metadata": {}, + "id": "f92bbe1a", + "metadata": { + "editable": true + }, "source": [ "which is a $2\\times 2 $ matrix while" ] }, { "cell_type": "markdown", - "id": "8d584adc", - "metadata": {}, + "id": "36e0b936", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T=\n", @@ -3252,8 +3740,10 @@ }, { "cell_type": "markdown", - "id": "25640918", - "metadata": {}, + "id": "183f7286", + "metadata": { + "editable": true + }, "source": [ "is a $3\\times 3 $ matrix. The last row and column of this last matrix\n", "contain only zeros. This will have important consequences for our SVD\n", @@ -3262,8 +3752,10 @@ }, { "cell_type": "markdown", - "id": "f4fe4493", - "metadata": {}, + "id": "df9df90f", + "metadata": { + "editable": true + }, "source": [ "## Setting up the Matrix to be inverted\n", "\n", @@ -3272,8 +3764,10 @@ }, { "cell_type": "markdown", - "id": "9f4fe3e3", - "metadata": {}, + "id": "c90899a8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -3282,16 +3776,20 @@ }, { "cell_type": "markdown", - "id": "11e6a3a0", - "metadata": {}, + "id": "983cdbd6", + "metadata": { + "editable": true + }, "source": [ "and using the orthogonality of the matrix $\\boldsymbol{U}$ we have" ] }, { "cell_type": "markdown", - "id": "407eaa57", - "metadata": {}, + "id": "0f436284", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -3300,8 +3798,10 @@ }, { "cell_type": "markdown", - "id": "bb2b3779", - "metadata": {}, + "id": "7269e493", + "metadata": { + "editable": true + }, "source": [ "We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\\tilde{\\boldsymbol{\\Sigma}}^2$ which is a diagonal matrix containing only the singular values squared. It has dimensionality $p \\times p$.\n", "\n", @@ -3310,8 +3810,10 @@ }, { "cell_type": "markdown", - "id": "918381fb", - "metadata": {}, + "id": "21eac5a7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -3320,16 +3822,20 @@ }, { "cell_type": "markdown", - "id": "5756c858", - "metadata": {}, + "id": "7c9459fd", + "metadata": { + "editable": true + }, "source": [ "and using our SVD decomposition of $\\boldsymbol{X}$ we have" ] }, { "cell_type": "markdown", - "id": "d93b3599", - "metadata": {}, + "id": "631bf89c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\left(\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^{2}(\\boldsymbol{V}^T\\right)^{-1}\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{y},\n", @@ -3338,16 +3844,20 @@ }, { "cell_type": "markdown", - "id": "dbfc1c41", - "metadata": {}, + "id": "f0e2dabd", + "metadata": { + "editable": true + }, "source": [ "which gives us, using the orthogonality of the matrix $\\boldsymbol{V}$," ] }, { "cell_type": "markdown", - "id": "b1087142", - "metadata": {}, + "id": "ede39b3f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}^T_i\\boldsymbol{y},\n", @@ -3356,8 +3866,10 @@ }, { "cell_type": "markdown", - "id": "e82fa7f7", - "metadata": {}, + "id": "7f555ff2", + "metadata": { + "editable": true + }, "source": [ "It means that the ordinary least square model (with the optimal\n", "parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal\n", @@ -3370,8 +3882,10 @@ }, { "cell_type": "markdown", - "id": "1642a934", - "metadata": {}, + "id": "4142188d", + "metadata": { + "editable": true + }, "source": [ "## Further properties (important for our analyses later)\n", "\n", @@ -3380,8 +3894,10 @@ }, { "cell_type": "markdown", - "id": "39c0e39f", - "metadata": {}, + "id": "097c9279", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -3390,16 +3906,20 @@ }, { "cell_type": "markdown", - "id": "b732098b", - "metadata": {}, + "id": "b1592a88", + "metadata": { + "editable": true + }, "source": [ "If we now multiply from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "id": "95c2fdfd", - "metadata": {}, + "id": "4f07ced0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}.\n", @@ -3408,8 +3928,10 @@ }, { "cell_type": "markdown", - "id": "dae64b7c", - "metadata": {}, + "id": "d669c781", + "metadata": { + "editable": true + }, "source": [ "This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n", "with eigenvalues given by the singular values squared, that is" @@ -3417,8 +3939,10 @@ }, { "cell_type": "markdown", - "id": "9bcbfe47", - "metadata": {}, + "id": "e63ecc04", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -3427,16 +3951,20 @@ }, { "cell_type": "markdown", - "id": "57d8f11b", - "metadata": {}, + "id": "a2ab776d", + "metadata": { + "editable": true + }, "source": [ "Similarly, if we use the SVD decomposition for the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$, we have" ] }, { "cell_type": "markdown", - "id": "23c75625", - "metadata": {}, + "id": "9fde1543", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T.\n", @@ -3445,16 +3973,20 @@ }, { "cell_type": "markdown", - "id": "2dadc3b4", - "metadata": {}, + "id": "253cd0ab", + "metadata": { + "editable": true + }, "source": [ "If we now multiply from the right with $\\boldsymbol{U}$ (using the orthogonality of $\\boldsymbol{U}$) we get" ] }, { "cell_type": "markdown", - "id": "c660e94c", - "metadata": {}, + "id": "935f5017", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T.\n", @@ -3463,8 +3995,10 @@ }, { "cell_type": "markdown", - "id": "650dbaa5", - "metadata": {}, + "id": "03ac6053", + "metadata": { + "editable": true + }, "source": [ "This means the vectors $\\boldsymbol{u}_i$ of the orthogonal matrix $\\boldsymbol{U}$ are the eigenvectors of the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "with eigenvalues given by the singular values squared, that is" @@ -3472,8 +4006,10 @@ }, { "cell_type": "markdown", - "id": "ba036597", - "metadata": {}, + "id": "82e8e71f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{u}_i=\\boldsymbol{u}_i\\sigma_i^2.\n", @@ -3482,8 +4018,10 @@ }, { "cell_type": "markdown", - "id": "15f22850", - "metadata": {}, + "id": "53f7343a", + "metadata": { + "editable": true + }, "source": [ "**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n", "$n\\times p$ matrix. In most supervised learning cases we have that $n\n", @@ -3498,8 +4036,10 @@ }, { "cell_type": "markdown", - "id": "90a608b8", - "metadata": {}, + "id": "8e6dd667", + "metadata": { + "editable": true + }, "source": [ "## Meet the Covariance Matrix\n", "\n", @@ -3512,8 +4052,10 @@ }, { "cell_type": "markdown", - "id": "0031b6a1", - "metadata": {}, + "id": "e57b1e19", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -3522,8 +4064,10 @@ }, { "cell_type": "markdown", - "id": "50f9a4be", - "metadata": {}, + "id": "431d052c", + "metadata": { + "editable": true + }, "source": [ "This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n", "\n", @@ -3532,8 +4076,10 @@ }, { "cell_type": "markdown", - "id": "dd8f36d1", - "metadata": {}, + "id": "bfb5acc9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -3542,8 +4088,10 @@ }, { "cell_type": "markdown", - "id": "17767821", - "metadata": {}, + "id": "52b1fcc3", + "metadata": { + "editable": true + }, "source": [ "The Hessian matrix for ordinary least squares is also proportional to\n", "the covariance matrix. This means also that we can use the SVD to find\n", @@ -3553,8 +4101,10 @@ }, { "cell_type": "markdown", - "id": "cd378ce2", - "metadata": {}, + "id": "c0b02536", + "metadata": { + "editable": true + }, "source": [ "## Introducing the Covariance and Correlation functions\n", "\n", @@ -3567,8 +4117,10 @@ }, { "cell_type": "markdown", - "id": "25f962ae", - "metadata": {}, + "id": "f3b67f67", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -3579,16 +4131,20 @@ }, { "cell_type": "markdown", - "id": "6e016c3e", - "metadata": {}, + "id": "c4130d0b", + "metadata": { + "editable": true + }, "source": [ "where for example" ] }, { "cell_type": "markdown", - "id": "3f0949a7", - "metadata": {}, + "id": "926b7b65", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n", @@ -3597,16 +4153,20 @@ }, { "cell_type": "markdown", - "id": "68ee6732", - "metadata": {}, + "id": "4d5eb320", + "metadata": { + "editable": true + }, "source": [ "With this definition and recalling that the variance is defined as" ] }, { "cell_type": "markdown", - "id": "7925746b", - "metadata": {}, + "id": "e0544424", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{var}[\\boldsymbol{x}]=\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})^2,\n", @@ -3615,16 +4175,20 @@ }, { "cell_type": "markdown", - "id": "93cccb87", - "metadata": {}, + "id": "422f75af", + "metadata": { + "editable": true + }, "source": [ "we can rewrite the covariance matrix as" ] }, { "cell_type": "markdown", - "id": "af5dc54e", - "metadata": {}, + "id": "9593498d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -3635,8 +4199,10 @@ }, { "cell_type": "markdown", - "id": "409097c5", - "metadata": {}, + "id": "7f13e100", + "metadata": { + "editable": true + }, "source": [ "**Note:** we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. \n", "What you will find in essentially all statistics texts are equations\n", @@ -3650,8 +4216,10 @@ }, { "cell_type": "markdown", - "id": "ce35f8cd", - "metadata": {}, + "id": "9b6613bd", + "metadata": { + "editable": true + }, "source": [ "## Covariance and Correlation Matrix\n", "\n", @@ -3664,8 +4232,10 @@ }, { "cell_type": "markdown", - "id": "14e59b18", - "metadata": {}, + "id": "1eeb6559", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]=\\frac{\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}]}{\\sqrt{\\mathrm{var}[\\boldsymbol{x}] \\mathrm{var}[\\boldsymbol{y}]}}.\n", @@ -3674,8 +4244,10 @@ }, { "cell_type": "markdown", - "id": "d695a41f", - "metadata": {}, + "id": "e484d492", + "metadata": { + "editable": true + }, "source": [ "The correlation function is then given by values $\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]\n", "\\in [-1,1]$. This avoids eventual problems with too large values. We\n", @@ -3685,8 +4257,10 @@ }, { "cell_type": "markdown", - "id": "98c0aa9e", - "metadata": {}, + "id": "f5e67ca7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{K}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} 1 & \\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -3697,16 +4271,20 @@ }, { "cell_type": "markdown", - "id": "d5ec88a2", - "metadata": {}, + "id": "3ea9bbff", + "metadata": { + "editable": true + }, "source": [ "In the above example this is the function we constructed using **pandas**." ] }, { "cell_type": "markdown", - "id": "b8e87fc7", - "metadata": {}, + "id": "cdf01f7e", + "metadata": { + "editable": true + }, "source": [ "## Correlation Function and Design/Feature Matrix\n", "\n", @@ -3716,8 +4294,10 @@ }, { "cell_type": "markdown", - "id": "73f8a37c", - "metadata": {}, + "id": "0ec98540", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -3733,8 +4313,10 @@ }, { "cell_type": "markdown", - "id": "b5685f73", - "metadata": {}, + "id": "cc502dc5", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors/features $p$ refering to the column numbers and the\n", "entries $n$ being the row elements.\n", @@ -3743,8 +4325,10 @@ }, { "cell_type": "markdown", - "id": "d33818a6", - "metadata": {}, + "id": "97c54b6b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} \\boldsymbol{x}_0 & \\boldsymbol{x}_1 & \\boldsymbol{x}_2 & \\dots & \\dots & \\boldsymbol{x}_{p-1}\\end{bmatrix},\n", @@ -3753,16 +4337,20 @@ }, { "cell_type": "markdown", - "id": "a8ac336a", - "metadata": {}, + "id": "af4b9fb5", + "metadata": { + "editable": true + }, "source": [ "with a given vector" ] }, { "cell_type": "markdown", - "id": "3b3df3bd", - "metadata": {}, + "id": "781e1060", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T = \\begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \\dots & \\dots x_{n-1,i}\\end{bmatrix}.\n", @@ -3771,8 +4359,10 @@ }, { "cell_type": "markdown", - "id": "3e6823a3", - "metadata": {}, + "id": "975af21b", + "metadata": { + "editable": true + }, "source": [ "With these definitions, we can now rewrite our $2\\times 2$\n", "correlation/covariance matrix in terms of a moe general design/feature\n", @@ -3782,8 +4372,10 @@ }, { "cell_type": "markdown", - "id": "140d91a6", - "metadata": {}, + "id": "e020b717", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix}\n", @@ -3799,16 +4391,20 @@ }, { "cell_type": "markdown", - "id": "777762cc", - "metadata": {}, + "id": "71796e08", + "metadata": { + "editable": true + }, "source": [ "and the correlation matrix" ] }, { "cell_type": "markdown", - "id": "491754e5", - "metadata": {}, + "id": "bcf53807", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{K}[\\boldsymbol{x}] = \\begin{bmatrix}\n", @@ -3824,8 +4420,10 @@ }, { "cell_type": "markdown", - "id": "28c5c1ef", - "metadata": {}, + "id": "0d110719", + "metadata": { + "editable": true + }, "source": [ "## Covariance Matrix Examples\n", "\n", @@ -3840,8 +4438,10 @@ }, { "cell_type": "markdown", - "id": "bb7447a4", - "metadata": {}, + "id": "7bc1b5db", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{W} = \\begin{bmatrix} x_0 & x_1 & x_2 & \\dots & x_{n-2} & x_{n-1} \\\\\n", @@ -3852,8 +4452,10 @@ }, { "cell_type": "markdown", - "id": "87eb961e", - "metadata": {}, + "id": "7fe635ef", + "metadata": { + "editable": true + }, "source": [ "which in turn is converted into into the $2\\times 2$ covariance matrix\n", "$\\boldsymbol{C}$ via the Numpy function **np.cov()**. We note that we can also calculate\n", @@ -3865,8 +4467,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "61ab5106", - "metadata": {}, + "id": "cde46031", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -3883,8 +4488,10 @@ }, { "cell_type": "markdown", - "id": "6bf630cb", - "metadata": {}, + "id": "13b2f4b2", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix\n", "\n", @@ -3898,8 +4505,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "52a37ecd", - "metadata": {}, + "id": "3b76f958", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -3927,8 +4537,10 @@ }, { "cell_type": "markdown", - "id": "dc629cda", - "metadata": {}, + "id": "41d7f745", + "metadata": { + "editable": true + }, "source": [ "We see that the matrix elements along the diagonal are one as they\n", "should be and that the matrix is symmetric. Furthermore, diagonalizing\n", @@ -3939,8 +4551,10 @@ }, { "cell_type": "markdown", - "id": "2d746b53", - "metadata": {}, + "id": "e984c719", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix with Pandas\n", "\n", @@ -3950,8 +4564,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "f9300182", - "metadata": {}, + "id": "acb65991", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -3972,16 +4589,20 @@ }, { "cell_type": "markdown", - "id": "baa289ef", - "metadata": {}, + "id": "6b800aaa", + "metadata": { + "editable": true + }, "source": [ "We expand this model to the Franke function discussed above." ] }, { "cell_type": "markdown", - "id": "5b453957", - "metadata": {}, + "id": "c6ff301f", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix with Pandas and the Franke function" ] @@ -3989,8 +4610,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "3e58c060", - "metadata": {}, + "id": "c24073b5", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -4040,8 +4664,10 @@ }, { "cell_type": "markdown", - "id": "d2ed77e1", - "metadata": {}, + "id": "7efb5b00", + "metadata": { + "editable": true + }, "source": [ "We note here that the covariance is zero for the first rows and\n", "columns since all matrix elements in the design matrix were set to one\n", @@ -4055,8 +4681,10 @@ }, { "cell_type": "markdown", - "id": "59fec8cf", - "metadata": {}, + "id": "bf2bb0dd", + "metadata": { + "editable": true + }, "source": [ "## Rewriting the Covariance and/or Correlation Matrix\n", "\n", @@ -4065,8 +4693,10 @@ }, { "cell_type": "markdown", - "id": "03ef8b54", - "metadata": {}, + "id": "c6200de6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}] = \\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X}= \\mathbb{E}[\\boldsymbol{X}^T\\boldsymbol{X}].\n", @@ -4075,16 +4705,20 @@ }, { "cell_type": "markdown", - "id": "42df6819", - "metadata": {}, + "id": "0b0725eb", + "metadata": { + "editable": true + }, "source": [ "To see this let us simply look at a design matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{2\\times 2}$" ] }, { "cell_type": "markdown", - "id": "93f18132", - "metadata": {}, + "id": "c84487a4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -4098,16 +4732,20 @@ }, { "cell_type": "markdown", - "id": "7ad53332", - "metadata": {}, + "id": "b33a3bb0", + "metadata": { + "editable": true + }, "source": [ "If we then compute the expectation value (note the $1/n$ factor instead of $1/(n-1)$)" ] }, { "cell_type": "markdown", - "id": "a71a4506", - "metadata": {}, + "id": "7864a2c5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E}[\\boldsymbol{X}^T\\boldsymbol{X}] = \\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X}=\\frac{1}{n}\\begin{bmatrix}\n", @@ -4119,16 +4757,20 @@ }, { "cell_type": "markdown", - "id": "ba124e79", - "metadata": {}, + "id": "45f8a421", + "metadata": { + "editable": true + }, "source": [ "which is just" ] }, { "cell_type": "markdown", - "id": "1a8dc304", - "metadata": {}, + "id": "bb1ed4b2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] = \\boldsymbol{C}[\\boldsymbol{x}]=\\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}_0] & \\mathrm{cov}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] \\\\\n", @@ -4139,8 +4781,10 @@ }, { "cell_type": "markdown", - "id": "e262eb52", - "metadata": {}, + "id": "d4953f79", + "metadata": { + "editable": true + }, "source": [ "where we wrote $$\\boldsymbol{C}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] = \\boldsymbol{C}[\\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors $\\boldsymbol{x}$ of the design/feature matrix $\\boldsymbol{X}$.\n", "\n", @@ -4149,8 +4793,10 @@ }, { "cell_type": "markdown", - "id": "d9bb7592", - "metadata": {}, + "id": "cf7fb557", + "metadata": { + "editable": true + }, "source": [ "## Linking with the SVD\n", "\n", @@ -4159,8 +4805,10 @@ }, { "cell_type": "markdown", - "id": "c104c5a2", - "metadata": {}, + "id": "97c2518c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -4169,16 +4817,20 @@ }, { "cell_type": "markdown", - "id": "f7b56297", - "metadata": {}, + "id": "2d654a0f", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix" ] }, { "cell_type": "markdown", - "id": "476df6a8", - "metadata": {}, + "id": "4d325cac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -4187,16 +4839,20 @@ }, { "cell_type": "markdown", - "id": "708c9dde", - "metadata": {}, + "id": "b8d4cc1c", + "metadata": { + "editable": true + }, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] }, { "cell_type": "markdown", - "id": "5aa7a295", - "metadata": {}, + "id": "d4ae3be8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -4210,16 +4866,20 @@ }, { "cell_type": "markdown", - "id": "c280709e", - "metadata": {}, + "id": "eecc1756", + "metadata": { + "editable": true + }, "source": [ "meaning we can write" ] }, { "cell_type": "markdown", - "id": "6fb708e6", - "metadata": {}, + "id": "eed91ae9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -4228,16 +4888,20 @@ }, { "cell_type": "markdown", - "id": "9bd89cad", - "metadata": {}, + "id": "add135b7", + "metadata": { + "editable": true + }, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "id": "66d52818", - "metadata": {}, + "id": "1b5cd68c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -4246,8 +4910,10 @@ }, { "cell_type": "markdown", - "id": "697358ef", - "metadata": {}, + "id": "37574dab", + "metadata": { + "editable": true + }, "source": [ "## What does it mean?\n", "\n", @@ -4258,8 +4924,10 @@ }, { "cell_type": "markdown", - "id": "c7835825", - "metadata": {}, + "id": "f93e4f56", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -4268,8 +4936,10 @@ }, { "cell_type": "markdown", - "id": "dc60e78e", - "metadata": {}, + "id": "d029f9dd", + "metadata": { + "editable": true + }, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -4288,8 +4958,10 @@ }, { "cell_type": "markdown", - "id": "b842c6f8", - "metadata": {}, + "id": "c0b9cede", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -4298,8 +4970,10 @@ }, { "cell_type": "markdown", - "id": "39430147", - "metadata": {}, + "id": "552c3234", + "metadata": { + "editable": true + }, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", @@ -4311,8 +4985,10 @@ }, { "cell_type": "markdown", - "id": "a1a9de0f", - "metadata": {}, + "id": "3fbd26d9", + "metadata": { + "editable": true + }, "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", @@ -4321,8 +4997,10 @@ }, { "cell_type": "markdown", - "id": "6ec35e26", - "metadata": {}, + "id": "bd716e28", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -4331,16 +5009,20 @@ }, { "cell_type": "markdown", - "id": "5c766892", - "metadata": {}, + "id": "7bbcf98d", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] }, { "cell_type": "markdown", - "id": "208c9a9d", - "metadata": {}, + "id": "fe830533", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -4349,16 +5031,20 @@ }, { "cell_type": "markdown", - "id": "bdb0d77b", - "metadata": {}, + "id": "8a6ba504", + "metadata": { + "editable": true + }, "source": [ "leading to" ] }, { "cell_type": "markdown", - "id": "0307040c", - "metadata": {}, + "id": "d21aa636", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -4367,16 +5053,20 @@ }, { "cell_type": "markdown", - "id": "470ec977", - "metadata": {}, + "id": "94080758", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] }, { "cell_type": "markdown", - "id": "c792ed2e", - "metadata": {}, + "id": "7393d0d6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -4385,8 +5075,10 @@ }, { "cell_type": "markdown", - "id": "f2afdfb3", - "metadata": {}, + "id": "5b57e226", + "metadata": { + "editable": true + }, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -4400,8 +5092,10 @@ }, { "cell_type": "markdown", - "id": "44e96378", - "metadata": {}, + "id": "908cb642", + "metadata": { + "editable": true + }, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -4411,8 +5105,10 @@ }, { "cell_type": "markdown", - "id": "41570d44", - "metadata": {}, + "id": "ea466969", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -4421,16 +5117,20 @@ }, { "cell_type": "markdown", - "id": "3800f6aa", - "metadata": {}, + "id": "84aceaf9", + "metadata": { + "editable": true + }, "source": [ "or we can state it as" ] }, { "cell_type": "markdown", - "id": "c730feea", - "metadata": {}, + "id": "e1f62ccc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -4440,16 +5140,20 @@ }, { "cell_type": "markdown", - "id": "0f42466b", - "metadata": {}, + "id": "864ec2d0", + "metadata": { + "editable": true + }, "source": [ "where we have used the definition of a norm-2 vector, that is" ] }, { "cell_type": "markdown", - "id": "6a59d510", - "metadata": {}, + "id": "9e98ecef", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -4458,8 +5162,10 @@ }, { "cell_type": "markdown", - "id": "a220eac6", - "metadata": {}, + "id": "fa772655", + "metadata": { + "editable": true + }, "source": [ "By minimizing the above equation with respect to the parameters\n", "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", @@ -4469,8 +5175,10 @@ }, { "cell_type": "markdown", - "id": "b1552898", - "metadata": {}, + "id": "4c175716", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -4480,8 +5188,10 @@ }, { "cell_type": "markdown", - "id": "c1fb4d79", - "metadata": {}, + "id": "0a717474", + "metadata": { + "editable": true + }, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -4490,8 +5200,10 @@ }, { "cell_type": "markdown", - "id": "f74a50f3", - "metadata": {}, + "id": "7a411718", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4500,16 +5212,20 @@ }, { "cell_type": "markdown", - "id": "d91b2af5", - "metadata": {}, + "id": "37e0a2d7", + "metadata": { + "editable": true + }, "source": [ "we have a new optimization equation" ] }, { "cell_type": "markdown", - "id": "3223ff5e", - "metadata": {}, + "id": "332f3492", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -4519,8 +5235,10 @@ }, { "cell_type": "markdown", - "id": "996d1643", - "metadata": {}, + "id": "0c9fccd1", + "metadata": { + "editable": true + }, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -4529,8 +5247,10 @@ }, { "cell_type": "markdown", - "id": "b95d5ee7", - "metadata": {}, + "id": "97f20b29", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -4539,8 +5259,10 @@ }, { "cell_type": "markdown", - "id": "50508cfb", - "metadata": {}, + "id": "aa95ae99", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -4549,8 +5271,10 @@ }, { "cell_type": "markdown", - "id": "5ba4add4", - "metadata": {}, + "id": "75d137ff", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -4559,8 +5283,10 @@ }, { "cell_type": "markdown", - "id": "54263127", - "metadata": {}, + "id": "c4d95105", + "metadata": { + "editable": true + }, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -4571,8 +5297,10 @@ }, { "cell_type": "markdown", - "id": "adef9008", - "metadata": {}, + "id": "dcdc73f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -4581,16 +5309,20 @@ }, { "cell_type": "markdown", - "id": "d7bd0ebe", - "metadata": {}, + "id": "72118165", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] }, { "cell_type": "markdown", - "id": "24e49828", - "metadata": {}, + "id": "2e5cdc63", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -4599,8 +5331,10 @@ }, { "cell_type": "markdown", - "id": "94eca210", - "metadata": {}, + "id": "f68206a9", + "metadata": { + "editable": true + }, "source": [ "with $t$ a finite positive number. \n", "\n", @@ -4609,8 +5343,10 @@ }, { "cell_type": "markdown", - "id": "c3d3f41e", - "metadata": {}, + "id": "5daca0e9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -4619,8 +5355,10 @@ }, { "cell_type": "markdown", - "id": "35d2e488", - "metadata": {}, + "id": "6a6c67f6", + "metadata": { + "editable": true + }, "source": [ "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", "\n", @@ -4629,8 +5367,10 @@ }, { "cell_type": "markdown", - "id": "f55212bc", - "metadata": {}, + "id": "8842c60f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -4639,8 +5379,10 @@ }, { "cell_type": "markdown", - "id": "11eb2ecd", - "metadata": {}, + "id": "d84a6f2b", + "metadata": { + "editable": true + }, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", @@ -4656,8 +5398,10 @@ }, { "cell_type": "markdown", - "id": "ac0b37e4", - "metadata": {}, + "id": "20c11ca3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -4666,16 +5410,20 @@ }, { "cell_type": "markdown", - "id": "12a2abf9", - "metadata": {}, + "id": "2e3e3cf9", + "metadata": { + "editable": true + }, "source": [ "For Ridge regression this becomes" ] }, { "cell_type": "markdown", - "id": "58bbe0b5", - "metadata": {}, + "id": "4a413908", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -4684,16 +5432,20 @@ }, { "cell_type": "markdown", - "id": "4a9a4758", - "metadata": {}, + "id": "69332e69", + "metadata": { + "editable": true + }, "source": [ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." ] }, { "cell_type": "markdown", - "id": "75f7eb2e", - "metadata": {}, + "id": "cda5f635", + "metadata": { + "editable": true + }, "source": [ "## Interpreting the Ridge results\n", "\n", @@ -4702,8 +5454,10 @@ }, { "cell_type": "markdown", - "id": "3cd08131", - "metadata": {}, + "id": "0cbb43fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -4712,8 +5466,10 @@ }, { "cell_type": "markdown", - "id": "94ea5e61", - "metadata": {}, + "id": "5f5f8d8d", + "metadata": { + "editable": true + }, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -4726,8 +5482,10 @@ }, { "cell_type": "markdown", - "id": "9d10638a", - "metadata": {}, + "id": "5a3bd8c2", + "metadata": { + "editable": true + }, "source": [ "## More interpretations\n", "\n", @@ -4736,8 +5494,10 @@ }, { "cell_type": "markdown", - "id": "6c69c6c3", - "metadata": {}, + "id": "fd0c9a5e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -4746,16 +5506,20 @@ }, { "cell_type": "markdown", - "id": "fcb5619d", - "metadata": {}, + "id": "1251007f", + "metadata": { + "editable": true + }, "source": [ "In this case the standard OLS results in" ] }, { "cell_type": "markdown", - "id": "26cfaed5", - "metadata": {}, + "id": "b6ed7a8f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", @@ -4764,16 +5528,20 @@ }, { "cell_type": "markdown", - "id": "94f13112", - "metadata": {}, + "id": "2ac0389a", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "293d4f99", - "metadata": {}, + "id": "bd13cdb1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -4782,8 +5550,10 @@ }, { "cell_type": "markdown", - "id": "9b67616d", - "metadata": {}, + "id": "e51668ec", + "metadata": { + "editable": true + }, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -4797,8 +5567,10 @@ }, { "cell_type": "markdown", - "id": "891450ee", - "metadata": {}, + "id": "b9747ef9", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Lasso Regression Equations\n", "\n", @@ -4807,8 +5579,10 @@ }, { "cell_type": "markdown", - "id": "e0238724", - "metadata": {}, + "id": "bdcce776", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4817,16 +5591,20 @@ }, { "cell_type": "markdown", - "id": "b922897d", - "metadata": {}, + "id": "49fc8f41", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] }, { "cell_type": "markdown", - "id": "6e697628", - "metadata": {}, + "id": "6499d8b0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\beta}=\\mathrm{sgn}(\\beta)=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -4835,16 +5613,20 @@ }, { "cell_type": "markdown", - "id": "185f8feb", - "metadata": {}, + "id": "3a5f4cf1", + "metadata": { + "editable": true + }, "source": [ "we have that the derivative of the cost function is" ] }, { "cell_type": "markdown", - "id": "16657f90", - "metadata": {}, + "id": "def20ba0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -4853,16 +5635,20 @@ }, { "cell_type": "markdown", - "id": "5a43b0bc", - "metadata": {}, + "id": "b10f7158", + "metadata": { + "editable": true + }, "source": [ "and reordering we have" ] }, { "cell_type": "markdown", - "id": "f17f7e9c", - "metadata": {}, + "id": "cb04be1f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\frac{n}{2}\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -4871,16 +5657,20 @@ }, { "cell_type": "markdown", - "id": "2c47798c", - "metadata": {}, + "id": "04c55218", + "metadata": { + "editable": true + }, "source": [ "We can redefine $\\lambda$ to absorb the constant $n/2$ and we rewrite the last equation as" ] }, { "cell_type": "markdown", - "id": "a1870073", - "metadata": {}, + "id": "5bf45a1e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -4889,32 +5679,16 @@ }, { "cell_type": "markdown", - "id": "887602a2", - "metadata": {}, + "id": "b6e3c60b", + "metadata": { + "editable": true + }, "source": [ "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.18" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index c94075211..d4c12e518 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -18,6 +18,7 @@ o Monday: Ridge and Lasso regression and Singular Value Decomposition o These lecture notes o "Video of lecture":"https://youtu.be/VKakN-e4aUA" + o "Video for exercises week 35":"https://youtu.be/yiY0OltU1s8" o Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics (background) o Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4. o For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week.