final update of week 35

doc/src/week35/week35.do.txt

@@ -2357,15 +2357,15 @@ Here we have defined the norm-1 as
 !split
 ===== Deriving the  Ridge Regression Equations =====
 
-Using the matrix-vector expression for Ridge regression,
+Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have
 
 !bt
 \[
-C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
+C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
 \]
 !et
-
-by taking the derivatives with respect to $\bm{\beta}$ we obtain then
+and 
+taking the derivatives with respect to $\bm{\beta}$ we obtain then
 a slightly modified matrix inversion problem which for finite values
 of $\lambda$ does not suffer from singularity problems. We obtain
 the optimal parameters
@@ -2476,8 +2476,37 @@ We will come back to more interpreations after we have gone through some of the
 For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
 Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended.
 
+!split
+===== Deriving the  Lasso Regression Equations =====
 
+Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following _cost_ function
 
+!bt
+\[
+C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
+\]
+!et
+
+Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+!bt
+\[
+\frac{d \vert \beta\vert}{d \bm{\beta}}=\mathrm{sgn}(\bm{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right.
+\]
+!et
+we have that the derivative of the cost function is
+
+!bt
+\[
+\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-2\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
+\]
+!et
+and reordering we have
+!bt
+\[
+\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}.
+\]
+!et
+This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later. 
 
 !split
 ===== Exercises for week 36, September 6-10 =====