From 8ba5a0e89b1cf25a3d092cc4457523d7ec9d0a8f Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 6 Sep 2021 21:58:48 +0200 Subject: [PATCH] final update of week 35 --- doc/pub/week35/html/week35-bs.html | 13 +++- doc/pub/week35/html/week35-reveal.html | 47 +++++++++++- doc/pub/week35/html/week35-solarized.html | 43 ++++++++++- doc/pub/week35/html/week35.html | 43 ++++++++++- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 74 ++++++++++++++++++- doc/src/week35/week35.do.txt | 37 +++++++++- 7 files changed, 233 insertions(+), 24 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index b5ed0e960..6ab140ab4 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -248,6 +248,10 @@ Automatically generated HTML file from DocOnce source None, 'interpreting-the-ridge-results'), ('More interpretations', 2, None, 'more-interpretations'), + ('Deriving the Lasso Regression Equations', + 2, + None, + 'deriving-the-lasso-regression-equations'), ('Exercises for week 36, September 6-10', 2, None, @@ -364,9 +368,10 @@ MathJax.Hub.Config({
  • Deriving the Ridge Regression Equations
  • Interpreting the Ridge results
  • More interpretations
    • -
  • Exercises for week 36, September 6-10
    • -
  • Exercise 1: Adding Ridge and Lasso Regression
    • -
  •    Exercise: Linear Regression for a two-dimensional function
    • +
  • Deriving the Lasso Regression Equations
    • +
  • Exercises for week 36, September 6-10
    • +
  • Exercise 1: Adding Ridge and Lasso Regression
    • +
  •    Exercise: Linear Regression for a two-dimensional function
    • @@ -425,7 +430,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 69
    • +
  • 70
  • »
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index bff8750bf..5a67ece0e 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -2899,16 +2899,16 @@ $$

    Deriving the Ridge Regression Equations

    -Using the matrix-vector expression for Ridge regression, +Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

     
    $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$

     
    -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +and +taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters @@ -3039,6 +3039,45 @@ Similarly, Mehta et a +

    +

    Deriving the Lasso Regression Equations

    + +

    +Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function + +

     
    +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ +

     
    + +

    +Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +

     
    +$$ +\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. +$$ +

     
    + +we have that the derivative of the cost function is + +

     
    +$$ +\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, +$$ +

     
    + +and reordering we have +

     
    +$$ +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. +$$ +

     
    + +This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package     CVXOPT. We will discuss this later. +

    + +

    Exercises for week 36, September 6-10

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index f4ef35128..9bfbcb50b 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -268,6 +268,10 @@ div { text-align: justify; text-justify: inter-word; } None, 'interpreting-the-ridge-results'), ('More interpretations', 2, None, 'more-interpretations'), + ('Deriving the Lasso Regression Equations', + 2, + None, + 'deriving-the-lasso-regression-equations'), ('Exercises for week 36, September 6-10', 2, None, @@ -2863,14 +2867,14 @@ $$

    Deriving the Ridge Regression Equations

    -Using the matrix-vector expression for Ridge regression, +Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +and +taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters @@ -2984,6 +2988,37 @@ Similarly, Mehta et a











    +

    Deriving the Lasso Regression Equations

    + +

    +Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +

    +Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +$$ +\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. +$$ + +we have that the derivative of the cost function is + +$$ +\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, +$$ + +and reordering we have +$$ +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. +$$ + +This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. + +

    +









    +

    Exercises for week 36, September 6-10

    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index fc18d1a1d..816c6c1f6 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -273,6 +273,10 @@ div { text-align: justify; text-justify: inter-word; } None, 'interpreting-the-ridge-results'), ('More interpretations', 2, None, 'more-interpretations'), + ('Deriving the Lasso Regression Equations', + 2, + None, + 'deriving-the-lasso-regression-equations'), ('Exercises for week 36, September 6-10', 2, None, @@ -2868,14 +2872,14 @@ $$

    Deriving the Ridge Regression Equations

    -Using the matrix-vector expression for Ridge regression, +Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +and +taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters @@ -2989,6 +2993,37 @@ Similarly, Mehta et a











    +

    Deriving the Lasso Regression Equations

    + +

    +Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +

    +Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +$$ +\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. +$$ + +we have that the derivative of the cost function is + +$$ +\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, +$$ + +and reordering we have +$$ +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. +$$ + +This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. + +

    +









    +

    Exercises for week 36, September 6-10

    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 4e7f655066fbf837b31ea327f0e5b4a97ac74d55..8093a6f54c5dc4e1d4734a321bdac744b1a5ae22 100644 GIT binary patch delta 36 scmX@Wcz}^jzMF&LVbMgkK+b8GW&|pjJku7Nm=M8{FyCkq0|x^G0OK7CcK`qY delta 36 scmX@Wcz}^jzMF%=-+3ZiAm=fS<(dLgpF@QvCPZ)~Up%>kfrEho0Lgy~4gdfE diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index a1a54d36b..3695b0526 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -3790,7 +3790,7 @@ "source": [ "## Deriving the Ridge Regression Equations\n", "\n", - "Using the matrix-vector expression for Ridge regression," + "Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have" ] }, { @@ -3798,7 +3798,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", "$$" ] }, @@ -3806,7 +3806,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "by taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", + "and \n", + "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", "a slightly modified matrix inversion problem which for finite values\n", "of $\\lambda$ does not suffer from singularity problems. We obtain\n", "the optimal parameters" @@ -3991,8 +3992,73 @@ "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", "\n", + "## Deriving the Lasso Regression Equations\n", "\n", - "\n", + "Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following **cost** function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\ 0 & \\beta =0\\\\-1 & \\beta < 0, \\end{array}\\right.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we have that the derivative of the cost function is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and reordering we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T(\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n", "\n", "## Exercises for week 36, September 6-10\n", "\n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 8f168817a..9d06e92eb 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -2357,15 +2357,15 @@ Here we have defined the norm-1 as !split ===== Deriving the Ridge Regression Equations ===== -Using the matrix-vector expression for Ridge regression, +Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have !bt \[ -C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, \] !et - -by taking the derivatives with respect to $\bm{\beta}$ we obtain then +and +taking the derivatives with respect to $\bm{\beta}$ we obtain then a slightly modified matrix inversion problem which for finite values of $\lambda$ does not suffer from singularity problems. We obtain the optimal parameters @@ -2476,8 +2476,37 @@ We will come back to more interpreations after we have gone through some of the For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended. Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended. +!split +===== Deriving the Lasso Regression Equations ===== +Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following _cost_ function +!bt +\[ +C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1, +\] +!et + +Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +!bt +\[ +\frac{d \vert \beta\vert}{d \bm{\beta}}=\mathrm{sgn}(\bm{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. +\] +!et +we have that the derivative of the cost function is + +!bt +\[ +\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-2\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0, +\] +!et +and reordering we have +!bt +\[ +\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}. +\] +!et +This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later. !split ===== Exercises for week 36, September 6-10 =====