From 79156f215035924488ae256520d7f5463d6b6119 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 9 Sep 2021 06:19:09 +0200 Subject: [PATCH] update --- doc/pub/week36/html/week36-bs.html | 79 ++++++----- doc/pub/week36/html/week36-reveal.html | 92 ++++++++++-- doc/pub/week36/html/week36-solarized.html | 82 +++++++++-- doc/pub/week36/html/week36.html | 82 +++++++++-- doc/pub/week36/ipynb/ipynb-week36-src.tar.gz | Bin 192 -> 191 bytes doc/pub/week36/ipynb/week36.ipynb | 142 ++++++++++++++++++- doc/src/week36/week36.do.txt | 77 +++++++++- 7 files changed, 481 insertions(+), 73 deletions(-) diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index 11aacc49a..63cff1914 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -88,6 +88,11 @@ Automatically generated HTML file from DocOnce source ('Yet another Example', 2, None, 'yet-another-example'), ('The OLS case', 2, None, 'the-ols-case'), ('The Ridge case', 2, None, 'the-ridge-case'), + ('Writing the Cost Function', + 2, + None, + 'writing-the-cost-function'), + ('Lasso case', 2, None, 'lasso-case'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -255,40 +260,42 @@ MathJax.Hub.Config({
  • Yet another Example
  • The OLS case
  • The Ridge case
    • -
  • Linking the regression analysis with a statistical interpretation
    • -
  • Assumptions made
    • -
  • Expectation value and variance
    • -
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • -
  • Deriving OLS from a probability distribution
    • -
  • Independent and Identically Distrubuted (iid)
    • -
  • Friday September 10
    • -
  • Why resampling methods
    • -
  • Resampling methods
    • -
  • Resampling approaches can be computationally expensive
    • -
  • Why resampling methods ?
    • -
  • Statistical analysis
    • -
  • Resampling methods
    • -
  • Resampling methods: Jackknife and Bootstrap
    • -
  • Resampling methods: Jackknife
    • -
  • Jackknife code example
    • -
  • Resampling methods: Bootstrap
    • -
  • Resampling methods: Bootstrap background
    • -
  • Resampling methods: More Bootstrap background
    • -
  • Resampling methods: Bootstrap approach
    • -
  • Resampling methods: Bootstrap steps
    • -
  • Code example for the Bootstrap method
    • -
  • Various steps in cross-validation
    • -
  • How to set up the cross-validation for Ridge and/or Lasso
    • -
  • Cross-validation in brief
    • -
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • -
  • The bias-variance tradeoff
    • -
  • Example code for Bias-Variance tradeoff
    • -
  • Understanding what happens
    • -
  • Summing up
    • -
  • Another Example from Scikit-Learn's Repository
    • -
  • More examples on bootstrap and cross-validation and errors
    • -
  • The same example but now with cross-validation
    • -
  • Cross-validation with Ridge
    • +
  • Writing the Cost Function
    • +
  • Lasso case
    • +
  • Linking the regression analysis with a statistical interpretation
    • +
  • Assumptions made
    • +
  • Expectation value and variance
    • +
  • Expectation value and variance for \( \boldsymbol{\beta} \)
    • +
  • Deriving OLS from a probability distribution
    • +
  • Independent and Identically Distrubuted (iid)
    • +
  • Friday September 10
    • +
  • Why resampling methods
    • +
  • Resampling methods
    • +
  • Resampling approaches can be computationally expensive
    • +
  • Why resampling methods ?
    • +
  • Statistical analysis
    • +
  • Resampling methods
    • +
  • Resampling methods: Jackknife and Bootstrap
    • +
  • Resampling methods: Jackknife
    • +
  • Jackknife code example
    • +
  • Resampling methods: Bootstrap
    • +
  • Resampling methods: Bootstrap background
    • +
  • Resampling methods: More Bootstrap background
    • +
  • Resampling methods: Bootstrap approach
    • +
  • Resampling methods: Bootstrap steps
    • +
  • Code example for the Bootstrap method
    • +
  • Various steps in cross-validation
    • +
  • How to set up the cross-validation for Ridge and/or Lasso
    • +
  • Cross-validation in brief
    • +
  • Code Example for Cross-validation and \( k \)-fold Cross-validation
    • +
  • The bias-variance tradeoff
    • +
  • Example code for Bias-Variance tradeoff
    • +
  • Understanding what happens
    • +
  • Summing up
    • +
  • Another Example from Scikit-Learn's Repository
    • +
  • More examples on bootstrap and cross-validation and errors
    • +
  • The same example but now with cross-validation
    • +
  • Cross-validation with Ridge
    • @@ -323,7 +330,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 8, 2021

    +

    Sep 9, 2021


    @@ -347,7 +354,7 @@ MathJax.Hub.Config({

  • 9
  • 10
  • ...
    • -
  • 54
    • +
  • 56
  • »
    • diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index d5e64189d..a62d4be6a 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

     
    -

    Sep 8, 2021

    +

    Sep 9, 2021


    @@ -654,7 +654,7 @@ $$

     

    -Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). +Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

    We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. @@ -677,7 +677,7 @@ $$ and our inputs as a \( 3\times 2 \) design matrix

     
    $$ -\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$

     
    @@ -701,13 +701,10 @@ Inserting the above values we obtain that

     
    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$

     
    -

    -Computing the mean squared error we obtian a value of \( 0.27 \). -

    The code which implements this simpler case is presented after the discussion of Ridge and Lasso. @@ -729,7 +726,7 @@ Inserting the above values we obtain that

     
    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$

     
    @@ -742,6 +739,85 @@ To see this, let us write the cost function for Ridge regression. +

    +

    Writing the Cost Function

    + +

    +We define the MSE without the \( 1/n \) factor and have then, using that +

     
    +$$ +\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +$$ +

     
    + +

     
    +$$ +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +$$ +

     
    + +and taking the derivative with respect to \( \beta_0 \) we get +

     
    +$$ +\beta_0=\frac{8}{4+\lambda}, +$$ +

     
    + +and for \( \beta_1 \) we have +

     
    +$$ +\beta_1=\frac{2}{1+\lambda}, +$$ +

     
    + +

    +Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving +

     
    +$$ +\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, +$$ +

     
    + +which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \). +

    + + +
    +

    Lasso case

    + +

    +For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \) +and \( beta_1 \). This gives us the following derivatives of the cost function +

     
    +$$ +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), +$$ +

     
    + +

     
    +$$ +\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0, +$$ +

     
    + +and +

     
    +$$ +\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0. +$$ +

     
    + +We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely + +

      +

    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
      2. +

    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
      3. +

    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
      4. +

    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \),
      5. +
    +
    + +

    Linking the regression analysis with a statistical interpretation

    diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html index 1bccf3b26..8fdeadf2e 100644 --- a/doc/pub/week36/html/week36-solarized.html +++ b/doc/pub/week36/html/week36-solarized.html @@ -108,6 +108,11 @@ div { text-align: justify; text-justify: inter-word; } ('Yet another Example', 2, None, 'yet-another-example'), ('The OLS case', 2, None, 'the-ols-case'), ('The Ridge case', 2, None, 'the-ridge-case'), + ('Writing the Cost Function', + 2, + None, + 'writing-the-cost-function'), + ('Lasso case', 2, None, 'lasso-case'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -260,7 +265,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 8, 2021

    +

    Sep 9, 2021












    @@ -681,7 +686,7 @@ $$ $$

    -Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). +Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

    We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. @@ -701,7 +706,7 @@ $$ and our inputs as a \( 3\times 2 \) design matrix $$ -\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$ meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. @@ -721,12 +726,9 @@ $$ Inserting the above values we obtain that $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$ -

    -Computing the mean squared error we obtian a value of \( 0.27 \). -

    The code which implements this simpler case is presented after the discussion of Ridge and Lasso. @@ -745,7 +747,7 @@ $$ Inserting the above values we obtain that $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$

    @@ -756,6 +758,70 @@ Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. To see this, let us write the cost function for Ridge regression.

    +









    + +

    Writing the Cost Function

    + +

    +We define the MSE without the \( 1/n \) factor and have then, using that +$$ +\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +$$ + + +$$ +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +$$ + +and taking the derivative with respect to \( \beta_0 \) we get +$$ +\beta_0=\frac{8}{4+\lambda}, +$$ + +and for \( \beta_1 \) we have +$$ +\beta_1=\frac{2}{1+\lambda}, +$$ + +

    +Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving +$$ +\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, +$$ + +which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \). + +

    +









    + +

    Lasso case

    + +

    +For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \) +and \( beta_1 \). This gives us the following derivatives of the cost function +$$ +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), +$$ + + +$$ +\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0, +$$ + +and +$$ +\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0. +$$ + +We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely + +

      +
    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
      2. +
    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
      3. +
    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
      4. +
    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \),
      5. +
    +

    Linking the regression analysis with a statistical interpretation

    diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index fabda1bf3..293abf5ef 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -113,6 +113,11 @@ div { text-align: justify; text-justify: inter-word; } ('Yet another Example', 2, None, 'yet-another-example'), ('The OLS case', 2, None, 'the-ols-case'), ('The Ridge case', 2, None, 'the-ridge-case'), + ('Writing the Cost Function', + 2, + None, + 'writing-the-cost-function'), + ('Lasso case', 2, None, 'lasso-case'), ('Linking the regression analysis with a statistical ' 'interpretation', 2, @@ -265,7 +270,7 @@ MathJax.Hub.Config({
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 8, 2021

    +

    Sep 9, 2021












    @@ -686,7 +691,7 @@ $$ $$

    -Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \). +Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

    We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. @@ -706,7 +711,7 @@ $$ and our inputs as a \( 3\times 2 \) design matrix $$ -\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$ meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. @@ -726,12 +731,9 @@ $$ Inserting the above values we obtain that $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$ -

    -Computing the mean squared error we obtian a value of \( 0.27 \). -

    The code which implements this simpler case is presented after the discussion of Ridge and Lasso. @@ -750,7 +752,7 @@ $$ Inserting the above values we obtain that $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$

    @@ -761,6 +763,70 @@ Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint. To see this, let us write the cost function for Ridge regression.

    +









    + +

    Writing the Cost Function

    + +

    +We define the MSE without the \( 1/n \) factor and have then, using that +$$ +\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +$$ + + +$$ +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +$$ + +and taking the derivative with respect to \( \beta_0 \) we get +$$ +\beta_0=\frac{8}{4+\lambda}, +$$ + +and for \( \beta_1 \) we have +$$ +\beta_1=\frac{2}{1+\lambda}, +$$ + +

    +Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving +$$ +\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, +$$ + +which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \). + +

    +









    + +

    Lasso case

    + +

    +For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \) +and \( beta_1 \). This gives us the following derivatives of the cost function +$$ +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), +$$ + + +$$ +\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0, +$$ + +and +$$ +\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0. +$$ + +We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely + +

      +
    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
      2. +
    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
      3. +
    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
      4. +
    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \),
      5. +
    +

    Linking the regression analysis with a statistical interpretation

    diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index fae1ac5a5906c936172561f6e9d272586008fb43..c989468ec9b6d5462edd55e28003c790acc13b5f 100644 GIT binary patch literal 191 zcmV;w06_mAiwFQSi#cHc1MSaC3c@fD2H>uHia9|^nulw_E?fvAULd8ZjkQTlQna_X z573q3rihSl^E1pa%p9`yW|t-Y?xV#ZggB)z=8Cf^Q99QXj2U2xF(-&oP5@!TQXTwd?K@bGN*B%Y2vhx54005?RSYH4D literal 192 zcmV;x06+g9iwFQb#yDXB1MSbv3c@f92k@Qu6nTQtx~@A5?%+WX@dY}TxjNU*wnO*! z?gR9sco`z}cli?%LUPE~n_U*Uy9*XW2;q#um?=-DB&)feP?`W`lyb^qk}@DPPF5@c zvfN28opr(tr!>_Wl|}V#ZWt@e4}0cU;F*8oSV;@peeWu*Kxv1$)-~J^>zGHf?Ntti uMmx5^;I)$mL8u-?QAj7X5|^+w`ea07qwv?yc%J8ZUwZ(}n(to#2mk<-X<3*6 diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 2c0731daa..920d58234 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 8, 2021**\n", + "Date: **Sep 9, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -772,7 +772,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the hand the values of $\\beta_i$ as function of $\\lambda$.\n", + "Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n", "\n", "We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.\n", "We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n", @@ -804,7 +804,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 1 & 0\\end{bmatrix},\n", + "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n", "$$" ] }, @@ -840,7 +840,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}\\frac{11}{5} \\\\ 2\\end{bmatrix},\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", "$$" ] }, @@ -848,8 +848,6 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Computing the mean squared error we obtian a value of $0.27$. \n", - "\n", "The code which implements this simpler case is presented after the discussion of Ridge and Lasso.\n", "\n", "## The Ridge case\n", @@ -878,7 +876,7 @@ "metadata": {}, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{11}{5+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", + "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", "$$" ] }, @@ -892,6 +890,136 @@ "To see this, let us write the cost function for Ridge regression. \n", "\n", "\n", + "## Writing the Cost Function\n", + "\n", + "We define the MSE without the $1/n$ factor and have then, using that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and taking the derivative with respect to $\\beta_0$ we get" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\beta_0=\\frac{8}{4+\\lambda},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and for $\\beta_1$ we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\beta_1=\\frac{2}{1+\\lambda},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$.\n", + "\n", + "## Lasso case\n", + "\n", + "For Lasso we need now, keeping the same constraint on $\\beta_0^2+\\beta_1^2=1$, to take the derivative of the absolute values of $\\beta_0$\n", + "and $beta_1$. This gives us the following derivatives of the cost function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-2(4-2\\beta_0)+\\lambda\\mathrm{sgn}\\beta_0=0,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}\\beta_1=0.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", + "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", + "\n", + "2. $\\beta_0 > 0$ and $\\beta_1 < 0$,\n", + "\n", + "3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n", + "\n", + "4. $\\beta_0 < 0$ and $\\beta_1 < 0$,\n", + "\n", "\n", "## Linking the regression analysis with a statistical interpretation\n", "\n", diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index aef89b5ad..35f0d253d 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -423,7 +423,7 @@ which leads to \] !et -Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the hand the values of $\beta_i$ as function of $\lambda$. +Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the other hand the values of $\beta_i$ as function of $\lambda$. We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$. @@ -442,7 +442,7 @@ Let us assume we have a data set with outputs/targets given by the vector and our inputs as a $3\times 2$ design matrix !bt \[ -\bm{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix}, +\bm{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, \] !et meaning that we have two features and two unknown parameters $\beta_0$ and $\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression. @@ -461,12 +461,10 @@ Inserting the above values we obtain that !bt \[ -\hat{\bm{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix}, +\hat{\bm{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, \] !et -Computing the mean squared error we obtian a value of $0.27$. - The code which implements this simpler case is presented after the discussion of Ridge and Lasso. !split @@ -483,7 +481,7 @@ Inserting the above values we obtain that !bt \[ -\hat{\bm{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\hat{\bm{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, \] !et @@ -493,6 +491,73 @@ Let us for simplicity assume now that $\beta_0^2+\beta_1^2=1$ as constraint. Thi To see this, let us write the cost function for Ridge regression. +!split +===== Writing the Cost Function ===== + +We define the MSE without the $1/n$ factor and have then, using that +!bt +\[ +\bm{X}\bm{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, +\] +!et + +!bt +\[ +C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +\] +!et +and taking the derivative with respect to $\beta_0$ we get +!bt +\[ +\beta_0=\frac{8}{4+\lambda}, +\] +!et +and for $\beta_1$ we have +!bt +\[ +\beta_1=\frac{2}{1+\lambda}, +\] +!et + +Using the constraint for $\beta_0^2+\beta_1^2=1$ we can constrain $\lambda$ by solving +!bt +\[ +\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, +\] +!et +which gives $\lambda=4.571$ and $\beta_0=0.933$ and $\beta_1=0.359$. + +!split +===== Lasso case ===== + +For Lasso we need now, keeping the same constraint on $\beta_0^2+\beta_1^2=1$, to take the derivative of the absolute values of $\beta_0$ +and $beta_1$. This gives us the following derivatives of the cost function +!bt +\[ +C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), +\] +!et + +!bt +\[ +\frac{\partialC(\bm{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0, +\] +!et +and +!bt +\[ +\frac{\partialC(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0. +\] +!et +We have now four cases to solve besides the trivial cases $\beta_0$ and/or $\beta_1$ are zero, namely +o $\beta_0 > 0$ and $\beta_1 > 0$, +o $\beta_0 > 0$ and $\beta_1 < 0$, +o $\beta_0 < 0$ and $\beta_1 > 0$, +o $\beta_0 < 0$ and $\beta_1 < 0$, + + + + !split ===== Linking the regression analysis with a statistical interpretation =====