From 2944749151e7a2a4ac62a2c2e33c138759b20ab3 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 6 Sep 2021 09:54:06 +0200 Subject: [PATCH] rewriting slides from week 35 --- doc/pub/week35/html/week35-bs.html | 62 +- doc/pub/week35/html/week35-reveal.html | 508 ++++++------- doc/pub/week35/html/week35-solarized.html | 472 ++++++------ doc/pub/week35/html/week35.html | 472 ++++++------ doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 738 +++++++++---------- doc/src/week35/week35.do.txt | 442 +++++------ 7 files changed, 1347 insertions(+), 1347 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 4f0ce4205..2d850f8d1 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -200,24 +200,6 @@ Automatically generated HTML file from DocOnce source 2, None, 'meet-the-covariance-matrix'), - ('Ridge and LASSO Regression', - 2, - None, - 'ridge-and-lasso-regression'), - ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), - ('Interpreting the Ridge results', - 2, - None, - 'interpreting-the-ridge-results'), - ('More interpretations', 2, None, 'more-interpretations'), - ('A better understanding of regularization', - 2, - None, - 'a-better-understanding-of-regularization'), - ('Decomposing the OLS and Ridge expressions', - 2, - None, - 'decomposing-the-ols-and-ridge-expressions'), ('Introducing the Covariance and Correlation functions', 2, None, @@ -243,6 +225,24 @@ Automatically generated HTML file from DocOnce source 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), + ('Interpreting the Ridge results', + 2, + None, + 'interpreting-the-ridge-results'), + ('More interpretations', 2, None, 'more-interpretations'), + ('A better understanding of regularization', + 2, + None, + 'a-better-understanding-of-regularization'), + ('Decomposing the OLS and Ridge expressions', + 2, + None, + 'decomposing-the-ols-and-ridge-expressions'), ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Exercises for week 36, September 6-10', 2, @@ -345,19 +345,19 @@ MathJax.Hub.Config({
  • Setting up the Matrix to be inverted
  • Further properties (important for our analyses later)
  • Meet the Covariance Matrix
    • -
  • Ridge and LASSO Regression
    • -
  • More on Ridge Regression
    • -
  • Interpreting the Ridge results
    • -
  • More interpretations
    • -
  • A better understanding of regularization
    • -
  • Decomposing the OLS and Ridge expressions
    • -
  • Introducing the Covariance and Correlation functions
    • -
  • Correlation Function and Design/Feature Matrix
    • -
  • Covariance Matrix Examples
    • -
  • Correlation Matrix
    • -
  • Correlation Matrix with Pandas
    • -
  • Correlation Matrix with Pandas and the Franke function
    • -
  • Rewriting the Covariance and/or Correlation Matrix
    • +
  • Introducing the Covariance and Correlation functions
    • +
  • Correlation Function and Design/Feature Matrix
    • +
  • Covariance Matrix Examples
    • +
  • Correlation Matrix
    • +
  • Correlation Matrix with Pandas
    • +
  • Correlation Matrix with Pandas and the Franke function
    • +
  • Rewriting the Covariance and/or Correlation Matrix
    • +
  • Ridge and LASSO Regression
    • +
  • More on Ridge Regression
    • +
  • Interpreting the Ridge results
    • +
  • More interpretations
    • +
  • A better understanding of regularization
    • +
  • Decomposing the OLS and Ridge expressions
  • Mathematical Properties
  • Exercises for week 36, September 6-10
  • Exercise 1: Adding Ridge and Lasso Regression
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 9a530cc19..5e14eaf2e 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -2283,265 +2283,12 @@ terms of the singular values. -
    -

    Ridge and LASSO Regression

    - -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is -

     
    -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. -$$ -

     
    - -or we can state it as -

     
    -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, -$$ -

     
    - -where we have used the definition of a norm-2 vector, that is -

     
    -$$ -\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. -$$ -

     
    - -

    -By minimizing the above equation with respect to the parameters -\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the -parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by -defining a new cost function to be optimized, that is - -

     
    -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 -$$ -

     
    - -

    -which leads to the Ridge regression minimization problem where we -require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is -a finite number larger than zero. By defining - -

     
    -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, -$$ -

     
    - -

    -we have a new optimization equation -

     
    -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 -$$ -

     
    - -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. - -

    -Here we have defined the norm-1 as -

     
    -$$ -\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. -$$ -

     
    -

    - - -
    -

    More on Ridge Regression

    - -

    -Using the matrix-vector expression for Ridge regression, - -

     
    -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, -$$ -

     
    - -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain - -

     
    -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ -

     
    - -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that - -

     
    -$$ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, -$$ -

     
    - -

    -with \( t \) a finite positive number. - -

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had -

     
    -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. -$$ -

     
    - -

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as -

     
    -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} -$$ -

     
    - -

    -For Ridge regression this becomes - -

     
    -$$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, -$$ -

     
    - -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). -

    - - -
    -

    Interpreting the Ridge results

    - -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have - -

     
    -$$ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. -$$ -

     
    - -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the -orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by -\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has -eigenvalues ordered in a descending way, that is \( \sigma_i \geq -\sigma_{i+1} \). - -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. -Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). -With a parameter \( \lambda \) we can thus shrink the role of specific parameters. -

    - - -
    -

    More interpretations

    - -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is - -

     
    -$$ -\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. -$$ -

     
    - -

    -In this case the standard OLS results in -

     
    -$$ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, -$$ -

     
    - -

    -and - -

     
    -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, -$$ -

     
    - -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. - -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. - -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. -Similarly, Mehta et al's article is also recommended. -

    - - -
    -

    A better understanding of regularization

    - -

    -The parameter \( \lambda \) that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -

    -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters \( \boldsymbol{\beta} \) is -affected by changing the parameter \( \lambda \). -

    - - -
    -

    Decomposing the OLS and Ridge expressions

    - -

    -We have our design matrix - \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as - -

     
    -$$ -\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, -$$ -

     
    - -

    -with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) -and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). - -

    -The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). -

    - -

    Introducing the Covariance and Correlation functions

    Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about -the definition of the covariance and the correlation function. These are quantities +the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.

    Suppose we have defined two vectors @@ -2914,6 +2661,259 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t

    +
    +

    Ridge and LASSO Regression

    + +

    +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +

     
    +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ +

     
    + +or we can state it as +

     
    +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ +

     
    + +where we have used the definition of a norm-2 vector, that is +

     
    +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ +

     
    + +

    +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +

     
    +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ +

     
    + +

    +which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +

     
    +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ +

     
    + +

    +we have a new optimization equation +

     
    +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ +

     
    + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

    +Here we have defined the norm-1 as +

     
    +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ +

     
    +

    + + +
    +

    More on Ridge Regression

    + +

    +Using the matrix-vector expression for Ridge regression, + +

     
    +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ +

     
    + +

    +by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +

     
    +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ +

     
    + +

    +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +

     
    +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ +

     
    + +

    +with \( t \) a finite positive number. + +

    +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance tradeoff +are rather interesting. + +

    +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +

     
    +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ +

     
    + +

    +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

     
    +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ +

     
    + +

    +For Ridge regression this becomes + +

     
    +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ +

     
    + +

    +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). +

    + + +
    +

    Interpreting the Ridge results

    + +

    +Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +

     
    +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ +

     
    + +

    +Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

    +For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. +

    + + +
    +

    More interpretations

    + +

    +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +

     
    +$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ +

     
    + +

    +In this case the standard OLS results in +

     
    +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ +

     
    + +

    +and + +

     
    +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ +

     
    + +

    +that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

    +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

    +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. +

    + + +
    +

    A better understanding of regularization

    + +

    +The parameter \( \lambda \) that we have introduced in the Ridge (and +Lasso as well) regression is often called a regularization parameter +or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? + +

    +Here we will first look at how to analyze the difference between the +standard OLS equations and the Ridge expressions in terms of a linear +algebra analysis using the SVD algorithm. Thereafter, we will link +(see the material on the bias-variance tradeoff below) these +observation to the statisical analysis of the results. In particular +we consider how the variance of the parameters \( \boldsymbol{\beta} \) is +affected by changing the parameter \( \lambda \). +

    + + +
    +

    Decomposing the OLS and Ridge expressions

    + +

    +We have our design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as + +

     
    +$$ +\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, +$$ +

     
    + +

    +with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) +and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). + +

    +The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). +

    + +

    Mathematical Properties

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index d3945fa43..ce90fe276 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -220,24 +220,6 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'meet-the-covariance-matrix'), - ('Ridge and LASSO Regression', - 2, - None, - 'ridge-and-lasso-regression'), - ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), - ('Interpreting the Ridge results', - 2, - None, - 'interpreting-the-ridge-results'), - ('More interpretations', 2, None, 'more-interpretations'), - ('A better understanding of regularization', - 2, - None, - 'a-better-understanding-of-regularization'), - ('Decomposing the OLS and Ridge expressions', - 2, - None, - 'decomposing-the-ols-and-ridge-expressions'), ('Introducing the Covariance and Correlation functions', 2, None, @@ -263,6 +245,24 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), + ('Interpreting the Ridge results', + 2, + None, + 'interpreting-the-ridge-results'), + ('More interpretations', 2, None, 'more-interpretations'), + ('A better understanding of regularization', + 2, + None, + 'a-better-understanding-of-regularization'), + ('Decomposing the OLS and Ridge expressions', + 2, + None, + 'decomposing-the-ols-and-ridge-expressions'), ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Exercises for week 36, September 6-10', 2, @@ -2315,228 +2315,11 @@ terms of the singular values.











    -

    Ridge and LASSO Regression

    - -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. -$$ - -or we can state it as -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, -$$ - -where we have used the definition of a norm-2 vector, that is -$$ -\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. -$$ - -

    -By minimizing the above equation with respect to the parameters -\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the -parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by -defining a new cost function to be optimized, that is - -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 -$$ - -

    -which leads to the Ridge regression minimization problem where we -require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is -a finite number larger than zero. By defining - -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, -$$ - -

    -we have a new optimization equation -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 -$$ - -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. - -

    -Here we have defined the norm-1 as -$$ -\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. -$$ - -

    -









    - -

    More on Ridge Regression

    - -

    -Using the matrix-vector expression for Ridge regression, - -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, -$$ - -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain - -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that - -$$ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, -$$ - -

    -with \( t \) a finite positive number. - -

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. -$$ - -

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} -$$ - -

    -For Ridge regression this becomes - -$$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, -$$ - -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). - -

    -









    - -

    Interpreting the Ridge results

    - -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have - -$$ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. -$$ - -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the -orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by -\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has -eigenvalues ordered in a descending way, that is \( \sigma_i \geq -\sigma_{i+1} \). - -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. -Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). -With a parameter \( \lambda \) we can thus shrink the role of specific parameters. - -

    -









    - -

    More interpretations

    - -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is - -$$ -\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. -$$ - -

    -In this case the standard OLS results in -$$ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, -$$ - -

    -and - -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, -$$ - -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. - -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. - -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. -Similarly, Mehta et al's article is also recommended. - -

    - - -

    A better understanding of regularization

    - -

    -The parameter \( \lambda \) that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -

    -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters \( \boldsymbol{\beta} \) is -affected by changing the parameter \( \lambda \). - -

    -









    - -

    Decomposing the OLS and Ridge expressions

    - -

    -We have our design matrix - \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as - -$$ -\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, -$$ - -

    -with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) -and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). - -

    -The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). - -

    -









    -

    Introducing the Covariance and Correlation functions

    Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about -the definition of the covariance and the correlation function. These are quantities +the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.

    Suppose we have defined two vectors @@ -2875,6 +2658,223 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t











    +

    Ridge and LASSO Regression

    + +

    +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +or we can state it as +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +

    +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ + +

    +which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

    +we have a new optimization equation +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

    +Here we have defined the norm-1 as +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +

    +









    + +

    More on Ridge Regression

    + +

    +Using the matrix-vector expression for Ridge regression, + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ + +

    +by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +

    +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ + +

    +with \( t \) a finite positive number. + +

    +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance tradeoff +are rather interesting. + +

    +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ + +

    +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ + +

    +For Ridge regression this becomes + +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

    +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). + +

    +









    + +

    Interpreting the Ridge results

    + +

    +Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ + +

    +Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

    +For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. + +

    +









    + +

    More interpretations

    + +

    +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ + +

    +In this case the standard OLS results in +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ + +

    +and + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

    +that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

    +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

    +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

    + + +

    A better understanding of regularization

    + +

    +The parameter \( \lambda \) that we have introduced in the Ridge (and +Lasso as well) regression is often called a regularization parameter +or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? + +

    +Here we will first look at how to analyze the difference between the +standard OLS equations and the Ridge expressions in terms of a linear +algebra analysis using the SVD algorithm. Thereafter, we will link +(see the material on the bias-variance tradeoff below) these +observation to the statisical analysis of the results. In particular +we consider how the variance of the parameters \( \boldsymbol{\beta} \) is +affected by changing the parameter \( \lambda \). + +

    +









    + +

    Decomposing the OLS and Ridge expressions

    + +

    +We have our design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as + +$$ +\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, +$$ + +

    +with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) +and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). + +

    +The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). + +

    +









    +

    Mathematical Properties

    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 25d8e5fa9..6e4ea5930 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -225,24 +225,6 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'meet-the-covariance-matrix'), - ('Ridge and LASSO Regression', - 2, - None, - 'ridge-and-lasso-regression'), - ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), - ('Interpreting the Ridge results', - 2, - None, - 'interpreting-the-ridge-results'), - ('More interpretations', 2, None, 'more-interpretations'), - ('A better understanding of regularization', - 2, - None, - 'a-better-understanding-of-regularization'), - ('Decomposing the OLS and Ridge expressions', - 2, - None, - 'decomposing-the-ols-and-ridge-expressions'), ('Introducing the Covariance and Correlation functions', 2, None, @@ -268,6 +250,24 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'rewriting-the-covariance-and-or-correlation-matrix'), + ('Ridge and LASSO Regression', + 2, + None, + 'ridge-and-lasso-regression'), + ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), + ('Interpreting the Ridge results', + 2, + None, + 'interpreting-the-ridge-results'), + ('More interpretations', 2, None, 'more-interpretations'), + ('A better understanding of regularization', + 2, + None, + 'a-better-understanding-of-regularization'), + ('Decomposing the OLS and Ridge expressions', + 2, + None, + 'decomposing-the-ols-and-ridge-expressions'), ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Exercises for week 36, September 6-10', 2, @@ -2320,228 +2320,11 @@ terms of the singular values.











    -

    Ridge and LASSO Regression

    - -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. -$$ - -or we can state it as -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, -$$ - -where we have used the definition of a norm-2 vector, that is -$$ -\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. -$$ - -

    -By minimizing the above equation with respect to the parameters -\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the -parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by -defining a new cost function to be optimized, that is - -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 -$$ - -

    -which leads to the Ridge regression minimization problem where we -require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is -a finite number larger than zero. By defining - -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, -$$ - -

    -we have a new optimization equation -$$ -{\displaystyle \min_{\boldsymbol{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 -$$ - -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. - -

    -Here we have defined the norm-1 as -$$ -\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. -$$ - -

    -









    - -

    More on Ridge Regression

    - -

    -Using the matrix-vector expression for Ridge regression, - -$$ -C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, -$$ - -

    -by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then -a slightly modified matrix inversion problem which for finite values -of \( \lambda \) does not suffer from singularity problems. We obtain - -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that - -$$ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, -$$ - -

    -with \( t \) a finite positive number. - -

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. -$$ - -

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} -$$ - -

    -For Ridge regression this becomes - -$$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, -$$ - -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). - -

    -









    - -

    Interpreting the Ridge results

    - -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have - -$$ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. -$$ - -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the -orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by -\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has -eigenvalues ordered in a descending way, that is \( \sigma_i \geq -\sigma_{i+1} \). - -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. -Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). -With a parameter \( \lambda \) we can thus shrink the role of specific parameters. - -

    -









    - -

    More interpretations

    - -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is - -$$ -\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. -$$ - -

    -In this case the standard OLS results in -$$ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, -$$ - -

    -and - -$$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, -$$ - -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. - -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. - -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. -Similarly, Mehta et al's article is also recommended. - -

    - - -

    A better understanding of regularization

    - -

    -The parameter \( \lambda \) that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -

    -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters \( \boldsymbol{\beta} \) is -affected by changing the parameter \( \lambda \). - -

    -









    - -

    Decomposing the OLS and Ridge expressions

    - -

    -We have our design matrix - \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as - -$$ -\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, -$$ - -

    -with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) -and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). - -

    -The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). - -

    -









    -

    Introducing the Covariance and Correlation functions

    Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about -the definition of the covariance and the correlation function. These are quantities +the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.

    Suppose we have defined two vectors @@ -2880,6 +2663,223 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t











    +

    Ridge and LASSO Regression

    + +

    +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. +$$ + +or we can state it as +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, +$$ + +where we have used the definition of a norm-2 vector, that is +$$ +\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +$$ + +

    +By minimizing the above equation with respect to the parameters +\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the +parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by +defining a new cost function to be optimized, that is + +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 +$$ + +

    +which leads to the Ridge regression minimization problem where we +require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is +a finite number larger than zero. By defining + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, +$$ + +

    +we have a new optimization equation +$$ +{\displaystyle \min_{\boldsymbol{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 +$$ + +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +

    +Here we have defined the norm-1 as +$$ +\vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. +$$ + +

    +









    + +

    More on Ridge Regression

    + +

    +Using the matrix-vector expression for Ridge regression, + +$$ +C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, +$$ + +

    +by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then +a slightly modified matrix inversion problem which for finite values +of \( \lambda \) does not suffer from singularity problems. We obtain + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +

    +with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that + +$$ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +$$ + +

    +with \( t \) a finite positive number. + +

    +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The +consequences, in particular for our discussion of the bias-variance tradeoff +are rather interesting. + +

    +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +$$ +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +$$ + +

    +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +$$ +\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +$$ + +

    +For Ridge regression this becomes + +$$ +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +$$ + +

    +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). + +

    +









    + +

    Interpreting the Ridge results

    + +

    +Since \( \lambda \geq 0 \), it means that compared to OLS, we have + +$$ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +$$ + +

    +Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). + +

    +For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of \( \boldsymbol{X}\boldsymbol{v}_j \) shows that this quantity is equal to \( \sigma_j^2/n \). +With a parameter \( \lambda \) we can thus shrink the role of specific parameters. + +

    +









    + +

    More interpretations

    + +

    +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +$$ +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +$$ + +

    +In this case the standard OLS results in +$$ +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, +$$ + +

    +and + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

    +that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +

    +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +

    +For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. + +

    + + +

    A better understanding of regularization

    + +

    +The parameter \( \lambda \) that we have introduced in the Ridge (and +Lasso as well) regression is often called a regularization parameter +or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? + +

    +Here we will first look at how to analyze the difference between the +standard OLS equations and the Ridge expressions in terms of a linear +algebra analysis using the SVD algorithm. Thereafter, we will link +(see the material on the bias-variance tradeoff below) these +observation to the statisical analysis of the results. In particular +we consider how the variance of the parameters \( \boldsymbol{\beta} \) is +affected by changing the parameter \( \lambda \). + +

    +









    + +

    Decomposing the OLS and Ridge expressions

    + +

    +We have our design matrix + \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as + +$$ +\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, +$$ + +

    +with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) +and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). + +

    +The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). + +

    +









    +

    Mathematical Properties

    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 9e24a81847fa4f8169227a25c4566c6c42f789cf..37755550c2bf27e2c4618c230ca2227810c70489 100644 GIT binary patch delta 173 zcmV;e08;l$u|b;2Xr_9}-$qaDA%;I)$mL8u-?QAj7X5|^+w b`ea07qwv?qc%J8ZUwZ)fYX;)f00;m8k+4x@ delta 173 zcmV;e08;J2q7uwFlLq}QxZ16CzK{Y`A}m{7|VdrIAJURvfN59opr(tt2EUal|}WgZ)hva z4|C>I;F*8oSV;@peeWu*Kxv1$)-~J^>o|#I+p8Q3jduJ3gV#\n", - "## A better understanding of regularization\n", - "\n", - "The parameter $\\lambda$ that we have introduced in the Ridge (and\n", - "Lasso as well) regression is often called a regularization parameter\n", - "or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?\n", - "\n", - "Here we will first look at how to analyze the difference between the\n", - "standard OLS equations and the Ridge expressions in terms of a linear\n", - "algebra analysis using the SVD algorithm. Thereafter, we will link\n", - "(see the material on the bias-variance tradeoff below) these\n", - "observation to the statisical analysis of the results. In particular\n", - "we consider how the variance of the parameters $\\boldsymbol{\\beta}$ is\n", - "affected by changing the parameter $\\lambda$.\n", - "\n", - "## Decomposing the OLS and Ridge expressions\n", - "\n", - "We have our design matrix\n", - " $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. With the SVD we decompose it as" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "\\boldsymbol{X} = \\boldsymbol{U\\Sigma V^T},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "with $\\boldsymbol{U}\\in {\\mathbb{R}}^{n\\times n}$, $\\boldsymbol{\\Sigma}\\in {\\mathbb{R}}^{n\\times p}$\n", - "and $\\boldsymbol{V}\\in {\\mathbb{R}}^{p\\times p}$.\n", - "\n", - "The matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ and $\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{I}$.\n", - "\n", - "\n", - "\n", "## Introducing the Covariance and Correlation functions\n", "\n", "Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about\n", - "the definition of the covariance and the correlation function. These are quantities \n", + "the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.\n", "\n", "Suppose we have defined two vectors\n", "$\\hat{x}$ and $\\hat{y}$ with $n$ elements each. The covariance matrix $\\boldsymbol{C}$ is defined as" @@ -3796,6 +3428,374 @@ "It is easy to generalize this to a matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$.\n", "\n", "\n", + "\n", + "\n", + "\n", + "## Ridge and LASSO Regression\n", + "\n", + "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", + "our optimization problem is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "or we can state it as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "where we have used the definition of a norm-2 vector, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "By minimizing the above equation with respect to the parameters\n", + "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", + "parameters $\\boldsymbol{\\beta}$. We can add a regularization parameter $\\lambda$ by\n", + "defining a new cost function to be optimized, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which leads to the Ridge regression minimization problem where we\n", + "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", + "a finite number larger than zero. By defining" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "we have a new optimization equation" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", + "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", + "\n", + "Here we have defined the norm-1 as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## More on Ridge Regression\n", + "\n", + "Using the matrix-vector expression for Ridge regression," + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "by taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", + "a slightly modified matrix inversion problem which for finite values\n", + "of $\\lambda$ does not suffer from singularity problems. We obtain" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $t$ a finite positive number. \n", + "\n", + "We see that Ridge regression is nothing but the standard\n", + "OLS with a modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The\n", + "consequences, in particular for our discussion of the bias-variance tradeoff \n", + "are rather interesting.\n", + "\n", + "Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For Ridge regression this becomes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$. \n", + "\n", + "## Interpreting the Ridge results\n", + "\n", + "Since $\\lambda \\geq 0$, it means that compared to OLS, we have" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", + "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", + "$\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}$. Recall that the SVD has\n", + "eigenvalues ordered in a descending way, that is $\\sigma_i \\geq\n", + "\\sigma_{i+1}$.\n", + "\n", + "For small eigenvalues $\\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom.\n", + "Actually, calculating the variance of $\\boldsymbol{X}\\boldsymbol{v}_j$ shows that this quantity is equal to $\\sigma_j^2/n$.\n", + "With a parameter $\\lambda$ we can thus shrink the role of specific parameters. \n", + "\n", + "\n", + "## More interpretations\n", + "\n", + "For the sake of simplicity, let us assume that the design matrix is orthonormal, that is" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this case the standard OLS results in" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\boldsymbol{y},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "and" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", + "the Ridge estimator converges to zero when the hyperparameter goes to\n", + "infinity.\n", + "\n", + "We will come back to more interpreations after we have gone through some of the statistical analysis part. \n", + "\n", + "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", + "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", + "\n", + "\n", + "\n", + "## A better understanding of regularization\n", + "\n", + "The parameter $\\lambda$ that we have introduced in the Ridge (and\n", + "Lasso as well) regression is often called a regularization parameter\n", + "or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?\n", + "\n", + "Here we will first look at how to analyze the difference between the\n", + "standard OLS equations and the Ridge expressions in terms of a linear\n", + "algebra analysis using the SVD algorithm. Thereafter, we will link\n", + "(see the material on the bias-variance tradeoff below) these\n", + "observation to the statisical analysis of the results. In particular\n", + "we consider how the variance of the parameters $\\boldsymbol{\\beta}$ is\n", + "affected by changing the parameter $\\lambda$.\n", + "\n", + "## Decomposing the OLS and Ridge expressions\n", + "\n", + "We have our design matrix\n", + " $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. With the SVD we decompose it as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X} = \\boldsymbol{U\\Sigma V^T},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "with $\\boldsymbol{U}\\in {\\mathbb{R}}^{n\\times n}$, $\\boldsymbol{\\Sigma}\\in {\\mathbb{R}}^{n\\times p}$\n", + "and $\\boldsymbol{V}\\in {\\mathbb{R}}^{p\\times p}$.\n", + "\n", + "The matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ and $\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{I}$.\n", + "\n", + "\n", + "\n", + "\n", "## Mathematical Properties\n", "\n", "There are several interesting mathematical properties which will be\n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 8882cf666..a7de5d036 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -1821,231 +1821,11 @@ the eigenvalues of the covariance matrix and the Hessian matrix in terms of the singular values. - - - - -!split -===== Ridge and LASSO Regression ===== - -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is -our optimization problem is -!bt -\[ -{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. -\] -!et -or we can state it as -!bt -\[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, -\] -!et -where we have used the definition of a norm-2 vector, that is -!bt -\[ -\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. -\] -!et - -By minimizing the above equation with respect to the parameters -$\bm{\beta}$ we could then obtain an analytical expression for the -parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by -defining a new cost function to be optimized, that is - -!bt -\[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 -\] -!et - -which leads to the Ridge regression minimization problem where we -require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is -a finite number larger than zero. By defining - -!bt -\[ -C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, -\] -!et - -we have a new optimization equation -!bt -\[ -{\displaystyle \min_{\bm{\beta}\in -{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 -\] -!et -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. - -Here we have defined the norm-1 as -!bt -\[ -\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert. -\] -!et - - -!split -===== More on Ridge Regression ===== - -Using the matrix-vector expression for Ridge regression, - -!bt -\[ -C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, -\] -!et - -by taking the derivatives with respect to $\bm{\beta}$ we obtain then -a slightly modified matrix inversion problem which for finite values -of $\lambda$ does not suffer from singularity problems. We obtain - -!bt -\[ -\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, -\] -!et - -with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that - -!bt -\[ -\sum_{i=0}^{p-1} \beta_i^2 \leq t, -\] -!et - -with $t$ a finite positive number. - -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had -!bt -\[ -(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. -\] -!et - -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as -!bt -\[ -\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} -\] -!et - - -For Ridge regression this becomes - -!bt -\[ -\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, -\] -!et - -with the vectors $\bm{u}_j$ being the columns of $\bm{U}$. - -!split -===== Interpreting the Ridge results ===== - -Since $\lambda \geq 0$, it means that compared to OLS, we have - -!bt -\[ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. -\] -!et - -Ridge regression finds the coordinates of $\bm{y}$ with respect to the -orthonormal basis $\bm{U}$, it then shrinks the coordinates by -$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has -eigenvalues ordered in a descending way, that is $\sigma_i \geq -\sigma_{i+1}$. - -For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. -Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$. -With a parameter $\lambda$ we can thus shrink the role of specific parameters. - - -!split -===== More interpretations ===== - -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is - -!bt -\[ -\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}. -\] -!et - -In this case the standard OLS results in -!bt -\[ -\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}, -\] -!et - -and - -!bt -\[ -\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, -\] -!et - -that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. - -We will come back to more interpreations after we have gone through some of the statistical analysis part. - -For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended. -Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended. - - -!split -===== A better understanding of regularization ===== - -The parameter $\lambda$ that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters $\bm{\beta}$ is -affected by changing the parameter $\lambda$. - -!split -===== Decomposing the OLS and Ridge expressions ===== - -We have our design matrix - $\bm{X}\in {\mathbb{R}}^{n\times p}$. With the SVD we decompose it as - -!bt -\[ -\bm{X} = \bm{U\Sigma V^T}, -\] -!et - -with $\bm{U}\in {\mathbb{R}}^{n\times n}$, $\bm{\Sigma}\in {\mathbb{R}}^{n\times p}$ -and $\bm{V}\in {\mathbb{R}}^{p\times p}$. - -The matrices $\bm{U}$ and $\bm{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\bm{U}^T\bm{U}=\bm{U}\bm{U}^T=\bm{I}$ and $\bm{V}^T\bm{V}=\bm{V}\bm{V}^T=\bm{I}$. - - - !split ===== Introducing the Covariance and Correlation functions ===== Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about -the definition of the covariance and the correlation function. These are quantities +the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods. Suppose we have defined two vectors $\hat{x}$ and $\hat{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as @@ -2376,6 +2156,226 @@ where we wrote $$\bm{C}[\bm{x}_0,\bm{x}_1] = \bm{C}[\bm{x}]$$ to indicate that t It is easy to generalize this to a matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. + + + +!split +===== Ridge and LASSO Regression ===== + +Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +our optimization problem is +!bt +\[ +{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}. +\] +!et +or we can state it as +!bt +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2, +\] +!et +where we have used the definition of a norm-2 vector, that is +!bt +\[ +\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. +\] +!et + +By minimizing the above equation with respect to the parameters +$\bm{\beta}$ we could then obtain an analytical expression for the +parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by +defining a new cost function to be optimized, that is + +!bt +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2 +\] +!et + +which leads to the Ridge regression minimization problem where we +require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is +a finite number larger than zero. By defining + +!bt +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1, +\] +!et + +we have a new optimization equation +!bt +\[ +{\displaystyle \min_{\bm{\beta}\in +{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1 +\] +!et +which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. + +Here we have defined the norm-1 as +!bt +\[ +\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert. +\] +!et + + +!split +===== More on Ridge Regression ===== + +Using the matrix-vector expression for Ridge regression, + +!bt +\[ +C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta}, +\] +!et + +by taking the derivatives with respect to $\bm{\beta}$ we obtain then +a slightly modified matrix inversion problem which for finite values +of $\lambda$ does not suffer from singularity problems. We obtain + +!bt +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\] +!et + +with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that + +!bt +\[ +\sum_{i=0}^{p-1} \beta_i^2 \leq t, +\] +!et + +with $t$ a finite positive number. + +We see that Ridge regression is nothing but the standard +OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The +consequences, in particular for our discussion of the bias-variance tradeoff +are rather interesting. + +Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +!bt +\[ +(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. +\] +!et + +We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as +!bt +\[ +\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} +\] +!et + + +For Ridge regression this becomes + +!bt +\[ +\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\] +!et + +with the vectors $\bm{u}_j$ being the columns of $\bm{U}$. + +!split +===== Interpreting the Ridge results ===== + +Since $\lambda \geq 0$, it means that compared to OLS, we have + +!bt +\[ +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\] +!et + +Ridge regression finds the coordinates of $\bm{y}$ with respect to the +orthonormal basis $\bm{U}$, it then shrinks the coordinates by +$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has +eigenvalues ordered in a descending way, that is $\sigma_i \geq +\sigma_{i+1}$. + +For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. +Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$. +With a parameter $\lambda$ we can thus shrink the role of specific parameters. + + +!split +===== More interpretations ===== + +For the sake of simplicity, let us assume that the design matrix is orthonormal, that is + +!bt +\[ +\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}. +\] +!et + +In this case the standard OLS results in +!bt +\[ +\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y}, +\] +!et + +and + +!bt +\[ +\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}}, +\] +!et + +that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. + +We will come back to more interpreations after we have gone through some of the statistical analysis part. + +For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended. +Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended. + + +!split +===== A better understanding of regularization ===== + +The parameter $\lambda$ that we have introduced in the Ridge (and +Lasso as well) regression is often called a regularization parameter +or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? + +Here we will first look at how to analyze the difference between the +standard OLS equations and the Ridge expressions in terms of a linear +algebra analysis using the SVD algorithm. Thereafter, we will link +(see the material on the bias-variance tradeoff below) these +observation to the statisical analysis of the results. In particular +we consider how the variance of the parameters $\bm{\beta}$ is +affected by changing the parameter $\lambda$. + +!split +===== Decomposing the OLS and Ridge expressions ===== + +We have our design matrix + $\bm{X}\in {\mathbb{R}}^{n\times p}$. With the SVD we decompose it as + +!bt +\[ +\bm{X} = \bm{U\Sigma V^T}, +\] +!et + +with $\bm{U}\in {\mathbb{R}}^{n\times n}$, $\bm{\Sigma}\in {\mathbb{R}}^{n\times p}$ +and $\bm{V}\in {\mathbb{R}}^{p\times p}$. + +The matrices $\bm{U}$ and $\bm{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\bm{U}^T\bm{U}=\bm{U}\bm{U}^T=\bm{I}$ and $\bm{V}^T\bm{V}=\bm{V}\bm{V}^T=\bm{I}$. + + + + !split ===== Mathematical Properties =====