update week 36

doc/pub/week36/html/week36-bs.html

@@ -83,7 +83,7 @@ Automatically generated HTML file from DocOnce source
                None,
                'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
               ('Ridge Regression', 2, None, 'ridge-regression'),
-              ('Lasso Rgeression', 2, None, 'lasso-rgeression'),
+              ('Lasso Regression', 2, None, 'lasso-regression'),
               ('Linking the regression analysis with a statistical '
                'interpretation',
                2,
@@ -238,7 +238,7 @@ MathJax.Hub.Config({
      <!-- navigation toc: --> <li><a href="._week36-bs012.html#deriving-the-lasso-regression-equations" style="font-size: 80%;">Deriving the  Lasso Regression Equations</a></li>
      <!-- navigation toc: --> <li><a href="._week36-bs013.html#simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression" style="font-size: 80%;">Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression</a></li>
      <!-- navigation toc: --> <li><a href="._week36-bs014.html#ridge-regression" style="font-size: 80%;">Ridge Regression</a></li>
-     <!-- navigation toc: --> <li><a href="._week36-bs015.html#lasso-rgeression" style="font-size: 80%;">Lasso Rgeression</a></li>
+     <!-- navigation toc: --> <li><a href="._week36-bs015.html#lasso-regression" style="font-size: 80%;">Lasso Regression</a></li>
      <!-- navigation toc: --> <li><a href="._week36-bs016.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
      <!-- navigation toc: --> <li><a href="._week36-bs017.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
      <!-- navigation toc: --> <li><a href="._week36-bs018.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>

doc/pub/week36/html/week36-reveal.html

@@ -608,7 +608,7 @@ $$
 For Ridge regression our cost function is
 <p>&nbsp;<br>
 $$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
 $$
 <p>&nbsp;<br>
 
@@ -622,17 +622,38 @@ $$
 
 
 <section>
-<h2 id="lasso-rgeression">Lasso Rgeression </h2>
+<h2 id="lasso-regression">Lasso Regression </h2>
 
 <p>
-For Ridge regression our cost function is
+For Lasso regression our cost function is
 <p>&nbsp;<br>
 $$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
 $$
 <p>&nbsp;<br>
 
 and minimizing we have that
+<p>&nbsp;<br>
+$$
+-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
+$$
+<p>&nbsp;<br>
+
+which leads to 
+<p>&nbsp;<br>
+$$
+\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
+                                                          y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
+							  0 &\mathrm{if} & \verty_i\vert\le  \frac{\lambda}{2}\\
+$$
+<p>&nbsp;<br>
+
+<p>
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
+
+<p>
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
+We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
 </section>
 
 

doc/pub/week36/html/week36-solarized.html

@@ -103,7 +103,7 @@ div { text-align: justify; text-justify: inter-word; }
                None,
                'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
               ('Ridge Regression', 2, None, 'ridge-regression'),
-              ('Lasso Rgeression', 2, None, 'lasso-rgeression'),
+              ('Lasso Regression', 2, None, 'lasso-regression'),
               ('Linking the regression analysis with a statistical '
                'interpretation',
                2,
@@ -632,7 +632,7 @@ $$
 <p>
 For Ridge regression our cost function is
 $$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
 $$
 
 and minimizing we have that
@@ -643,15 +643,32 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
 
-<h2 id="lasso-rgeression">Lasso Rgeression </h2>
+<h2 id="lasso-regression">Lasso Regression </h2>
 
 <p>
-For Ridge regression our cost function is
+For Lasso regression our cost function is
 $$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
 $$
 
 and minimizing we have that
+$$
+-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
+$$
+
+which leads to 
+$$
+\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
+                                                          y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
+							  0 &\mathrm{if} & \verty_i\vert\le  \frac{\lambda}{2}\\
+$$
+
+<p>
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
+
+<p>
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
+We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
 
 <p>
 <!-- !split  -->

doc/pub/week36/html/week36.html

@@ -108,7 +108,7 @@ div { text-align: justify; text-justify: inter-word; }
                None,
                'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
               ('Ridge Regression', 2, None, 'ridge-regression'),
-              ('Lasso Rgeression', 2, None, 'lasso-rgeression'),
+              ('Lasso Regression', 2, None, 'lasso-regression'),
               ('Linking the regression analysis with a statistical '
                'interpretation',
                2,
@@ -637,7 +637,7 @@ $$
 <p>
 For Ridge regression our cost function is
 $$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
 $$
 
 and minimizing we have that
@@ -648,15 +648,32 @@ $$
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
 
-<h2 id="lasso-rgeression">Lasso Rgeression </h2>
+<h2 id="lasso-regression">Lasso Regression </h2>
 
 <p>
-For Ridge regression our cost function is
+For Lasso regression our cost function is
 $$
-C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
 $$
 
 and minimizing we have that
+$$
+-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
+$$
+
+which leads to 
+$$
+\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
+                                                          y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
+							  0 &\mathrm{if} & \verty_i\vert\le  \frac{\lambda}{2}\\
+$$
+
+<p>
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
+
+<p>
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
+We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \).
 
 <p>
 <!-- !split  -->

doc/pub/week36/ipynb/week36.ipynb

@@ -694,7 +694,7 @@
    "metadata": {},
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n",
+    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n",
     "$$"
    ]
   },
@@ -718,9 +718,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Lasso Rgeression\n",
+    "## Lasso Regression\n",
     "\n",
-    "For Ridge regression our cost function is"
+    "For Lasso regression our cost function is"
    ]
   },
   {
@@ -728,7 +728,7 @@
    "metadata": {},
    "source": [
     "$$\n",
-    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n",
+    "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n",
     "$$"
    ]
   },
@@ -736,7 +736,44 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "and minimizing we have that\n",
+    "and minimizing we have that"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "which leads to"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "$$\n",
+    "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
+    "                                                          y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n",
+    "\t\t\t\t\t\t\t  0 &\\mathrm{if} & \\verty_i\\vert\\le  \\frac{\\lambda}{2}\\\\\n",
+    "$$"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the hand the values of $\\beta_i$ as function of $\\lambda$.\n",
+    "\n",
+    "We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.\n",
+    "We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n",
     "\n",
     "\n",
     "<!-- !split  -->\n",

doc/src/week36/week36.do.txt

@@ -385,7 +385,7 @@ and minimizing we have that
 For Ridge regression our cost function is
 !bt
 \[
-C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
 \]
 !et
 and minimizing we have that
@@ -397,15 +397,33 @@ and minimizing we have that
 
 
 !split
-===== Lasso Rgeression =====
+===== Lasso Regression =====
 
-For Ridge regression our cost function is
+For Lasso regression our cost function is
 !bt
 \[
-C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
 \]
 !et
 and minimizing we have that
+!bt
+\[
+-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
+\]
+!et
+which leads to 
+!bt
+\[
+\hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
+                                                          y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\
+							  0 &\mathrm{if} & \verty_i\vert\le  \frac{\lambda}{2}\\
+\]
+!et
+
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the hand the values of $\beta_i$ as function of $\lambda$.
+
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
+We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$.
 
 
 !split