update week 36

doc/src/week36/week36.do.txt

@@ -385,7 +385,7 @@ and minimizing we have that
 For Ridge regression our cost function is
 !bt
 \[
-C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
 \]
 !et
 and minimizing we have that
@@ -397,15 +397,33 @@ and minimizing we have that
 
 
 !split
-===== Lasso Rgeression =====
+===== Lasso Regression =====
 
-For Ridge regression our cost function is
+For Lasso regression our cost function is
 !bt
 \[
-C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
 \]
 !et
 and minimizing we have that
+!bt
+\[
+-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
+\]
+!et
+which leads to 
+!bt
+\[
+\hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
+                                                          y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\
+							  0 &\mathrm{if} & \verty_i\vert\le  \frac{\lambda}{2}\\
+\]
+!et
+
+Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the hand the values of $\beta_i$ as function of $\lambda$.
+
+We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
+We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$.
 
 
 !split