updates

2023-09-03 13:57:59 +02:00
parent 22c8178dfb
commit 22739ad2c9
3 changed files with 630 additions and 824 deletions
@@ -2,7 +2,7 @@
 "cells": [
  {
   "cell_type": "markdown",
-   "id": "6bb78ffe",
+   "id": "50b2fbbe",
   "metadata": {
    "editable": true
   },
@@ -14,7 +14,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "a756b88a",
+   "id": "da488c0c",
   "metadata": {
    "editable": true
   },
@@ -27,7 +27,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "4c64515c",
+   "id": "b84d6b13",
   "metadata": {
    "editable": true
   },
@@ -44,7 +44,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "6d83ecbd",
+   "id": "96c9c28e",
   "metadata": {
    "editable": true
   },
@@ -62,7 +62,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "96444623",
+   "id": "439f1456",
   "metadata": {
    "editable": true
   },
@@ -74,7 +74,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "66ea8e0c",
+   "id": "b51e09f7",
   "metadata": {
    "editable": true
   },
@@ -84,7 +84,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "90823663",
+   "id": "02c45981",
   "metadata": {
    "editable": true
   },
@@ -97,7 +97,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "812b9485",
+   "id": "cc0e91ea",
   "metadata": {
    "editable": true
   },
@@ -107,7 +107,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "b0bf6d67",
+   "id": "b5805f35",
   "metadata": {
    "editable": true
   },
@@ -119,7 +119,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "737ebbd9",
+   "id": "6e3095bf",
   "metadata": {
    "editable": true
   },
@@ -134,7 +134,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "d0089b18",
+   "id": "da90fe04",
   "metadata": {
    "editable": true
   },
@@ -147,7 +147,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "14388b10",
+   "id": "1a106e07",
   "metadata": {
    "editable": true
   },
@@ -159,7 +159,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "a8bc91a0",
+   "id": "3917877b",
   "metadata": {
    "editable": true
   },
@@ -171,7 +171,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "9652f49f",
+   "id": "78226f28",
   "metadata": {
    "editable": true
   },
@@ -183,7 +183,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "80519c5d",
+   "id": "951dfffa",
   "metadata": {
    "editable": true
   },
@@ -193,7 +193,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "43f58011",
+   "id": "21d2770e",
   "metadata": {
    "editable": true
   },
@@ -205,7 +205,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "4f336993",
+   "id": "ec212498",
   "metadata": {
    "editable": true
   },
@@ -217,7 +217,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "b9dc87be",
+   "id": "4ffabf6c",
   "metadata": {
    "editable": true
   },
@@ -229,7 +229,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "7dc4483d",
+   "id": "f97a6f45",
   "metadata": {
    "editable": true
   },
@@ -241,7 +241,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "e45e9fa3",
+   "id": "8761ed23",
   "metadata": {
    "editable": true
   },
@@ -253,7 +253,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "acb74b76",
+   "id": "92f8479e",
   "metadata": {
    "editable": true
   },
@@ -268,7 +268,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "67537508",
+   "id": "9df91bda",
   "metadata": {
    "editable": true
   },
@@ -280,7 +280,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "c2de46e7",
+   "id": "e6de0312",
   "metadata": {
    "editable": true
   },
@@ -290,7 +290,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "0ead662e",
+   "id": "8e09d132",
   "metadata": {
    "editable": true
   },
@@ -302,7 +302,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "d8b43f14",
+   "id": "a9c924ab",
   "metadata": {
    "editable": true
   },
@@ -314,7 +314,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "d90dde5d",
+   "id": "3b9328a1",
   "metadata": {
    "editable": true
   },
@@ -338,7 +338,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "e303dad2",
+   "id": "5b54b7b4",
   "metadata": {
    "editable": true
   },
@@ -350,7 +350,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "4fa7dde4",
+   "id": "a7ebe31b",
   "metadata": {
    "editable": true
   },
@@ -360,7 +360,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "1e1b12fc",
+   "id": "874e0dd3",
   "metadata": {
    "editable": true
   },
@@ -372,7 +372,7 @@
  },
  {
   "cell_type": "markdown",
-   "id": "a6ea7503",
+   "id": "9adcc39f",
   "metadata": {
    "editable": true
   },
@@ -6,19 +6,18 @@ DATE: September 4-8, 2023
 !split
 ===== Plans for week 36 =====

-o Material for the active learning sessions on Tuesday and Wednesday
-  o Summary from last week on discussion of SVD, Ridge and Lasso linear regression.
-  o Recommended Reading: Hastie et al chapter 3, see URL:"https://link.springer.com/book/10.1007/978-0-387-84858-7"
-  o Presentation and discussion of first project
-o Material for the lecture on Thursday September 7
-  o Linear Regression and links with Statistics, Resampling methods
-  o Recommended Reading: Hastie et al chapter 3, see URL:"https://link.springer.com/book/10.1007/978-0-387-84858-7"
+* Material for the active learning sessions on Tuesday and Wednesday
+  * Summary from last week on discussion of SVD, Ridge and Lasso linear regression.
+  * Recommended Reading: Hastie et al chapter 3, see URL:"https://link.springer.com/book/10.1007/978-0-387-84858-7"
+  * Presentation and discussion of first project
+* Material for the lecture on Thursday September 7
+  * Linear Regression and links with Statistics, Resampling methods
+  * Recommended Reading: Hastie et al chapter 3, see URL:"https://link.springer.com/book/10.1007/978-0-387-84858-7"

 !split
 ===== Material for the active learning sessions Tuesday and Wednesday =====

-!split
-===== Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples =====
+The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples 

 !split
 ===== Linear Regression and  the SVD =====
@@ -233,7 +232,7 @@ C = SVDinv(A)
 print(np.abs(C-B))

 !ec
-As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by _Numpy_.
+As you can see from this example, our own decomposition based on the SVD agrees with  the pseudoinverse algorithm provided by _Numpy_.



@@ -278,7 +277,9 @@ defining a new cost function to be optimized, that is

 which leads to the Ridge regression minimization problem where we
 require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is
-a finite number larger than zero. By defining
+a finite number larger than zero. We do not include such a constraints in the discussions here.
+
+By defining

 !bt
 \[
@@ -448,34 +449,34 @@ Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also re
 !split
 ===== Deriving the  Lasso Regression Equations =====

-Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard mean squared error equation, we have the following _cost_ function
+Using the matrix-vector expression for Lasso regression, we have the following _cost_ function

 !bt
 \[
-C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
+C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
 \]
 !et

-Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicity)
 !bt
 \[
-\frac{d \vert \beta\vert}{d \bm{\beta}}=\mathrm{sgn}(\bm{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
 \]
 !et
 we have that the derivative of the cost function is

 !bt
 \[
-\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-2\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
+\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
 \]
 !et
 and reordering we have
 !bt
 \[
-\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
+\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=\bm{X}^T\bm{y}.
 \]
 !et
-This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later. 
+This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. We have absorbed the factor $2/n$ in a redefinition of the parameter $\lambda$. We will solve this type of problems using libraries like _scikit-learn_.



@@ -812,7 +813,7 @@ for i in range(nlambdas):
    # and then make the prediction
    ypredictRidge = X @ Ridgebeta
    MSERidgePredict[i] = MSE(y,ypredictRidge)
-    RegLasso = linear_model.Lasso(lmb)
+    RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
    RegLasso.fit(X,y)
    ypredictLasso = RegLasso.predict(X)
    print(RegLasso.coef_)