From 2bcc525906683faf12abddb2f2dfc72f5337c2fe Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Wed, 8 Sep 2021 06:08:36 +0200 Subject: [PATCH] update week 36 --- doc/pub/week36/html/week36-bs.html | 4 +- doc/pub/week36/html/week36-reveal.html | 29 ++++++++++-- doc/pub/week36/html/week36-solarized.html | 27 +++++++++-- doc/pub/week36/html/week36.html | 27 +++++++++-- doc/pub/week36/ipynb/ipynb-week36-src.tar.gz | Bin 191 -> 192 bytes doc/pub/week36/ipynb/week36.ipynb | 47 +++++++++++++++++-- doc/src/week36/week36.do.txt | 26 ++++++++-- 7 files changed, 135 insertions(+), 25 deletions(-) diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index 239aafe78..3c356d49c 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -83,7 +83,7 @@ Automatically generated HTML file from DocOnce source None, 'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'), ('Ridge Regression', 2, None, 'ridge-regression'), - ('Lasso Rgeression', 2, None, 'lasso-rgeression'), + ('Lasso Regression', 2, None, 'lasso-regression'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -238,7 +238,7 @@ MathJax.Hub.Config({
  • Deriving the Lasso Regression Equations
  • Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression
  • Ridge Regression
    • -
  • Lasso Rgeression
    • +
  • Lasso Regression
  • Linking the regression analysis with a statistical interpretation
  • Assumptions made
  • Expectation value and variance
    • diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index 5c79e75b5..2451a7d21 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -608,7 +608,7 @@ $$ For Ridge regression our cost function is

     
    $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$

     
    @@ -622,17 +622,38 @@ $$

    -

    Lasso Rgeression

    +

    Lasso Regression

    -For Ridge regression our cost function is +For Lasso regression our cost function is

     
    $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$

     
    and minimizing we have that +

     
    +$$ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +$$ +

     
    + +which leads to +

     
    +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +$$ +

     
    + +

    +Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). + +

    +We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).

    diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html index 6deff0044..1e86b8369 100644 --- a/doc/pub/week36/html/week36-solarized.html +++ b/doc/pub/week36/html/week36-solarized.html @@ -103,7 +103,7 @@ div { text-align: justify; text-justify: inter-word; } None, 'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'), ('Ridge Regression', 2, None, 'ridge-regression'), - ('Lasso Rgeression', 2, None, 'lasso-rgeression'), + ('Lasso Regression', 2, None, 'lasso-regression'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -632,7 +632,7 @@ $$

    For Ridge regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ and minimizing we have that @@ -643,15 +643,32 @@ $$











    -

    Lasso Rgeression

    +

    Lasso Regression

    -For Ridge regression our cost function is +For Lasso regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ and minimizing we have that +$$ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +$$ + +which leads to +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +$$ + +

    +Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). + +

    +We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).

    diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index dde24d44d..5d9cdcfc9 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -108,7 +108,7 @@ div { text-align: justify; text-justify: inter-word; } None, 'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'), ('Ridge Regression', 2, None, 'ridge-regression'), - ('Lasso Rgeression', 2, None, 'lasso-rgeression'), + ('Lasso Regression', 2, None, 'lasso-regression'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -637,7 +637,7 @@ $$

    For Ridge regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ and minimizing we have that @@ -648,15 +648,32 @@ $$











    -

    Lasso Rgeression

    +

    Lasso Regression

    -For Ridge regression our cost function is +For Lasso regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ and minimizing we have that +$$ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +$$ + +which leads to +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +$$ + +

    +Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). + +

    +We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).

    diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 1a865aee4a9ffda4fd2e9b9800a1c48a8ef3ce32..1413466271d342cb2ecbad013ab60078aafae093 100644 GIT binary patch literal 192 zcmV;x06+g9iwFQVH#lJc1MSbv3c@f92k@Qu6nTQt+O0ba?%+WX@dY}TxjNU*wnO*! z?gR9sco`z}cli?%LUPE~n_U*Uy9*XW2;p%IW2QWrlC0)>LTLgtrj!%LV$Ok3maNhM z$Z{vWbk+$ooYGWhR2J2{xnZm1q(qL5B%B`#rW^vQ_EM&YlY@jTD-zV-kWL9-bE2mk=54OxBw literal 191 zcmV;w06_mAiwFS5GdN)Y1MSbv3c@f92k@Qu6nTP??xQ;kdhj5K_yQfvT%ESscIe*T zeSn@6FGGa>E`LHoNOr5maB>xAh~IoC&0mejkseyplK?3rGHXa0$OEiJ6KovXD1r5zPoH*h0r5+2D*rwS-F t*`Wo7uB|i#LUka@QaY)XIE5FZ4@R7A6#n`d&+|O*YY(7l8IAx5004JiR<8g6 diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 96657f4bc..f2bf4eaaa 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -694,7 +694,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n", + "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", "$$" ] }, @@ -718,9 +718,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Lasso Rgeression\n", + "## Lasso Regression\n", "\n", - "For Ridge regression our cost function is" + "For Lasso regression our cost function is" ] }, { @@ -728,7 +728,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n", + "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", "$$" ] }, @@ -736,7 +736,44 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "and minimizing we have that\n", + "and minimizing we have that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which leads to" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", + " y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n", + "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\verty_i\\vert\\le \\frac{\\lambda}{2}\\\\\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the hand the values of $\\beta_i$ as function of $\\lambda$.\n", + "\n", + "We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.\n", + "We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n", "\n", "\n", "\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index fd9b5cbd3..4299321aa 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -385,7 +385,7 @@ and minimizing we have that For Ridge regression our cost function is !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, \] !et and minimizing we have that @@ -397,15 +397,33 @@ and minimizing we have that !split -===== Lasso Rgeression ===== +===== Lasso Regression ===== -For Ridge regression our cost function is +For Lasso regression our cost function is !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, \] !et and minimizing we have that +!bt +\[ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +\] +!et +which leads to +!bt +\[ +\hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +\] +!et + +Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the hand the values of $\beta_i$ as function of $\lambda$. + +We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$. !split