$$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
$$
-For Ridge regression our cost function is
+For Lasso regression our cost function is
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
+
+
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
+We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
@@ -622,17 +622,38 @@ $$
Lasso Rgeression
+Lasso Regression
$$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
$$
and minimizing we have that
+
+$$
+-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
+$$
+
+
+which leads to
+
+$$
+\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
+ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
+ 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\
+$$
+
+
+
For Ridge regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ and minimizing we have that @@ -643,15 +643,32 @@ $$
-
-For Ridge regression our cost function is +For Lasso regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ and minimizing we have that +$$ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +$$ + +which leads to +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +$$ + +
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). + +
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index dde24d44d..5d9cdcfc9 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -108,7 +108,7 @@ div { text-align: justify; text-justify: inter-word; } None, 'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'), ('Ridge Regression', 2, None, 'ridge-regression'), - ('Lasso Rgeression', 2, None, 'lasso-rgeression'), + ('Lasso Regression', 2, None, 'lasso-regression'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -637,7 +637,7 @@ $$
For Ridge regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ and minimizing we have that @@ -648,15 +648,32 @@ $$
-
-For Ridge regression our cost function is +For Lasso regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ and minimizing we have that +$$ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +$$ + +which leads to +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +$$ + +
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). + +
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index 1a865aee4..141346627 100644 Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 96657f4bc..f2bf4eaaa 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -694,7 +694,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n", + "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", "$$" ] }, @@ -718,9 +718,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Lasso Rgeression\n", + "## Lasso Regression\n", "\n", - "For Ridge regression our cost function is" + "For Lasso regression our cost function is" ] }, { @@ -728,7 +728,7 @@ "metadata": {}, "source": [ "$$\n", - "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n", + "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", "$$" ] }, @@ -736,7 +736,44 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "and minimizing we have that\n", + "and minimizing we have that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which leads to" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", + " y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n", + "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\verty_i\\vert\\le \\frac{\\lambda}{2}\\\\\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the hand the values of $\\beta_i$ as function of $\\lambda$.\n", + "\n", + "We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.\n", + "We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n", "\n", "\n", "\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index fd9b5cbd3..4299321aa 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -385,7 +385,7 @@ and minimizing we have that For Ridge regression our cost function is !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, \] !et and minimizing we have that @@ -397,15 +397,33 @@ and minimizing we have that !split -===== Lasso Rgeression ===== +===== Lasso Regression ===== -For Ridge regression our cost function is +For Lasso regression our cost function is !bt \[ -C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,, +C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, \] !et and minimizing we have that +!bt +\[ +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +\] +!et +which leads to +!bt +\[ +\hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\ +\] +!et + +Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the hand the values of $\beta_i$ as function of $\lambda$. + +We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. +We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$. !split