From f9e7f36c5ea8a2f3edb9a3086f43760c31554df5 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 6 Sep 2021 18:13:44 +0200 Subject: [PATCH] more updates --- doc/pub/week35/html/week35-bs.html | 27 +-- doc/pub/week35/html/week35-reveal.html | 176 +++------------- doc/pub/week35/html/week35-solarized.html | 156 +++----------- doc/pub/week35/html/week35.html | 156 +++----------- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 201 +++---------------- doc/src/week35/week35.do.txt | 136 +++---------- 7 files changed, 142 insertions(+), 710 deletions(-) diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index b5c92388f..b5ed0e960 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -239,21 +239,15 @@ Automatically generated HTML file from DocOnce source 2, None, 'ridge-and-lasso-regression'), - ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Interpreting the Ridge results', 2, None, 'interpreting-the-ridge-results'), ('More interpretations', 2, None, 'more-interpretations'), - ('A better understanding of regularization', - 2, - None, - 'a-better-understanding-of-regularization'), - ('Decomposing the OLS and Ridge expressions', - 2, - None, - 'decomposing-the-ols-and-ridge-expressions'), - ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Exercises for week 36, September 6-10', 2, None, @@ -367,15 +361,12 @@ MathJax.Hub.Config({
  • What does it mean?
  • And finally \( \boldsymbol{X}\boldsymbol{X}^T \)
  • Ridge and LASSO Regression
    • -
  • More on Ridge Regression
    • +
  • Deriving the Ridge Regression Equations
  • Interpreting the Ridge results
  • More interpretations
    • -
  • A better understanding of regularization
    • -
  • Decomposing the OLS and Ridge expressions
    • -
  • Mathematical Properties
    • -
  • Exercises for week 36, September 6-10
    • -
  • Exercise 1: Adding Ridge and Lasso Regression
    • -
  •    Exercise: Linear Regression for a two-dimensional function
    • +
  • Exercises for week 36, September 6-10
    • +
  • Exercise 1: Adding Ridge and Lasso Regression
    • +
  •    Exercise: Linear Regression for a two-dimensional function
    • @@ -434,7 +425,7 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 72
    • +
  • 69
  • »
    • diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 43a84655c..20d027e5d 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -2809,10 +2809,15 @@ $\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \ti $$

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \). +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +the non-zero singular values plus now a series of zeros. The column +vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and +measure how much correlations are contained in the rows of \( \boldsymbol{X} \).

    -Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +Since we will mainly be interested in the correlations among the features +of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular +values and the column vectors of \( \boldsymbol{V} \). @@ -2889,7 +2894,7 @@ $$

    -

    More on Ridge Regression

    +

    Deriving the Ridge Regression Equations

    Using the matrix-vector expression for Ridge regression, @@ -2904,10 +2909,10 @@ $$ by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain - +the optimal parameters

     
    $$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$

     
    @@ -2924,24 +2929,28 @@ $$ with \( t \) a finite positive number.

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +When we compare this with the ordinary least squares result we have

     
    $$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$

     
    +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. + +

    +Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as

     
    $$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$

     
    @@ -2950,7 +2959,7 @@ For Ridge regression this becomes

     
    $$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$

     
    @@ -3028,141 +3037,6 @@ Similarly, Mehta et a

    -
    -

    A better understanding of regularization

    - -

    -The parameter \( \lambda \) that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -

    -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters \( \boldsymbol{\beta} \) is -affected by changing the parameter \( \lambda \). -

    - - -
    -

    Decomposing the OLS and Ridge expressions

    - -

    -We have our design matrix - \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as - -

     
    -$$ -\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, -$$ -

     
    - -

    -with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) -and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). - -

    -The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). -

    - - -
    -

    Mathematical Properties

    - -

    -There are several interesting mathematical properties which will be -relevant when we are going to discuss the differences between say -ordinary least squares (OLS) and Ridge regression. - -

    -We have from OLS that the parameters of the linear approximation are given by -

     
    -$$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. -$$ -

     
    - -

    -The matrix to invert can be rewritten in terms of our SVD decomposition as - -

     
    -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. -$$ -

     
    - -Using the orthogonality properties of \( \boldsymbol{U} \) we have - -

     
    -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, -$$ -

     
    - -with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. - -

    -This means that -

     
    -$$ -(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, -$$ -

     
    - -that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that -

     
    -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, -$$ -

     
    - -that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. - -

    -Going back to our OLS equation we have -

     
    -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. -$$ -

     
    - -We will come back to this expression when we discuss Ridge regression. - -

     
    -$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ -

     
    and for Ridge we have  - -

     
    -$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ -

     
    .  - -

    -It is indeed the economy-sized SVD, note the summation runs up tp

     
    -$$p$$ -

     
    only and not

     
    -$$n$$ -

     
    .  - -

    -Here we have that

     
    -$${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$ -

     
    , with

     
    -$$\Sigma$$ -

     
    being an

     
    -$$ n\times p$$ -

     
    matrix and

     
    -$${\bf V}$$ -

     
    being a

     
    -$$ p\times p$$ -

     
    matrix. We also have assumed here that

     
    -$$ n > p$$ -

     
    .  -

    - -

    Exercises for week 36, September 6-10

    diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index bd108cb8e..9e956bfbe 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -259,21 +259,15 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'ridge-and-lasso-regression'), - ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Interpreting the Ridge results', 2, None, 'interpreting-the-ridge-results'), ('More interpretations', 2, None, 'more-interpretations'), - ('A better understanding of regularization', - 2, - None, - 'a-better-understanding-of-regularization'), - ('Decomposing the OLS and Ridge expressions', - 2, - None, - 'decomposing-the-ols-and-ridge-expressions'), - ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Exercises for week 36, September 6-10', 2, None, @@ -2795,10 +2789,15 @@ $\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \ti $$

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \). +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +the non-zero singular values plus now a series of zeros. The column +vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and +measure how much correlations are contained in the rows of \( \boldsymbol{X} \).

    -Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +Since we will mainly be interested in the correlations among the features +of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular +values and the column vectors of \( \boldsymbol{V} \).











    @@ -2861,7 +2860,7 @@ $$











    -

    More on Ridge Regression

    +

    Deriving the Ridge Regression Equations

    Using the matrix-vector expression for Ridge regression, @@ -2874,9 +2873,9 @@ $$ by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain - +the optimal parameters $$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$

    @@ -2890,28 +2889,32 @@ $$ with \( t \) a finite positive number.

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +When we compare this with the ordinary least squares result we have $$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. + +

    +Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as $$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$

    For Ridge regression this becomes $$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$

    @@ -2978,107 +2981,6 @@ We will come back to more interpreations after we have gone through some of the For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. -

    - - -

    A better understanding of regularization

    - -

    -The parameter \( \lambda \) that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -

    -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters \( \boldsymbol{\beta} \) is -affected by changing the parameter \( \lambda \). - -

    -









    - -

    Decomposing the OLS and Ridge expressions

    - -

    -We have our design matrix - \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as - -$$ -\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, -$$ - -

    -with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) -and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). - -

    -The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). - -

    -









    - -

    Mathematical Properties

    - -

    -There are several interesting mathematical properties which will be -relevant when we are going to discuss the differences between say -ordinary least squares (OLS) and Ridge regression. - -

    -We have from OLS that the parameters of the linear approximation are given by -$$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. -$$ - -

    -The matrix to invert can be rewritten in terms of our SVD decomposition as - -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. -$$ - -Using the orthogonality properties of \( \boldsymbol{U} \) we have - -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, -$$ - -with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. - -

    -This means that -$$ -(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, -$$ - -that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, -$$ - -that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. - -

    -Going back to our OLS equation we have -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. -$$ - -We will come back to this expression when we discuss Ridge regression. - -$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ and for Ridge we have  - -$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ .  - -

    -It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$.  - -

    -Here we have that $${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$, with $$\Sigma$$ being an $$ n\times p$$ matrix and $${\bf V}$$ being a $$ p\times p$$ matrix. We also have assumed here that $$ n > p$$.  -











    diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index ba815752f..6644f495b 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -264,21 +264,15 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'ridge-and-lasso-regression'), - ('More on Ridge Regression', 2, None, 'more-on-ridge-regression'), + ('Deriving the Ridge Regression Equations', + 2, + None, + 'deriving-the-ridge-regression-equations'), ('Interpreting the Ridge results', 2, None, 'interpreting-the-ridge-results'), ('More interpretations', 2, None, 'more-interpretations'), - ('A better understanding of regularization', - 2, - None, - 'a-better-understanding-of-regularization'), - ('Decomposing the OLS and Ridge expressions', - 2, - None, - 'decomposing-the-ols-and-ridge-expressions'), - ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Exercises for week 36, September 6-10', 2, None, @@ -2800,10 +2794,15 @@ $\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \ti $$

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \). +It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +the non-zero singular values plus now a series of zeros. The column +vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and +measure how much correlations are contained in the rows of \( \boldsymbol{X} \).

    -Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +Since we will mainly be interested in the correlations among the features +of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular +values and the column vectors of \( \boldsymbol{V} \).











    @@ -2866,7 +2865,7 @@ $$











    -

    More on Ridge Regression

    +

    Deriving the Ridge Regression Equations

    Using the matrix-vector expression for Ridge regression, @@ -2879,9 +2878,9 @@ $$ by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain - +the optimal parameters $$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$

    @@ -2895,28 +2894,32 @@ $$ with \( t \) a finite positive number.

    -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -

    -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +When we compare this with the ordinary least squares result we have $$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}. +\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. + +

    +Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as $$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y} +\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$

    For Ridge regression this becomes $$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$

    @@ -2983,107 +2986,6 @@ We will come back to more interpreations after we have gone through some of the For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. -

    - - -

    A better understanding of regularization

    - -

    -The parameter \( \lambda \) that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -

    -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters \( \boldsymbol{\beta} \) is -affected by changing the parameter \( \lambda \). - -

    -









    - -

    Decomposing the OLS and Ridge expressions

    - -

    -We have our design matrix - \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as - -$$ -\boldsymbol{X} = \boldsymbol{U\Sigma V^T}, -$$ - -

    -with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \) -and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \). - -

    -The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \). - -

    -









    - -

    Mathematical Properties

    - -

    -There are several interesting mathematical properties which will be -relevant when we are going to discuss the differences between say -ordinary least squares (OLS) and Ridge regression. - -

    -We have from OLS that the parameters of the linear approximation are given by -$$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. -$$ - -

    -The matrix to invert can be rewritten in terms of our SVD decomposition as - -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T. -$$ - -Using the orthogonality properties of \( \boldsymbol{U} \) we have - -$$ -\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T, -$$ - -with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared. - -

    -This means that -$$ -(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D}, -$$ - -that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that -$$ -(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}, -$$ - -that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same. - -

    -Going back to our OLS equation we have -$$ -\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. -$$ - -We will come back to this expression when we discuss Ridge regression. - -$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ and for Ridge we have  - -$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ .  - -

    -It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$.  - -

    -Here we have that $${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$, with $$\Sigma$$ being an $$ n\times p$$ matrix and $${\bf V}$$ being a $$ p\times p$$ matrix. We also have assumed here that $$ n > p$$.  -











    diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 70692391492588e3664c6c314bbc2e3b459c04c2..e4a7ec63de36e1f1f6ce9aad2c5daad1ebd74b3f 100644 GIT binary patch delta 169 zcmV;a09OCN0l)zzABzY8D?T=100ZsMOA5j;5C-6`dx|+hNlntO1-oz|ht=Fd`hyK-;+4z5VN5&ni5Wd#FVd+0LXGHy>!+IGpy27XH*u|yS|~VEI-Vd zPl0FtiDM-#Z1=sZv;w6a=33WqL#!!}WZSD83XOLB0)y918U&$w5Je%K)Jj~!*65QF Xjg7)zALDtR=Y8z~3Xuz)00;m8tL;h1 delta 169 zcmV;a09OCN0l)zzABzY8Fg7+}00ZsMOA5j;5C-6`dx|+hnKnsX3wGf`5b**jO_3!a zdqhkrg)vttn-Y`xJ&9umF&!!;1XBcz2}>{lTyCY8&N^;}RhsH7loje--_TZ;ALh)b zz%&2Ev62?H``%SrfznRoTGwzBq9!boZLe}Dw6NnB7;){S5x`XsyeOoTTJcNR8htXn Xu~GQzV?59Eystd~Xh=CZ00;m8rU*u1 diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 7db8aacb8..f80ae50ae 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -3653,9 +3653,14 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\\boldsymbol{U}$ are the eigenvectors of $\\boldsymbol{X}\\boldsymbol{X}^T$ and measure how much correlations are contained in the rows of $\\m{X}$.\n", + "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", + "the non-zero singular values plus now a series of zeros. The column\n", + "vectors of $\\boldsymbol{U}$ are the eigenvectors of $\\boldsymbol{X}\\boldsymbol{X}^T$ and\n", + "measure how much correlations are contained in the rows of $\\boldsymbol{X}$.\n", "\n", - "Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\\boldsymbol{V}$.\n", + "Since we will mainly be interested in the correlations among the features\n", + "of our data (the columns of $\\boldsymbol{X}$, the quantity of interest for us are the non-zero singular\n", + "values and the column vectors of $\\boldsymbol{V}$.\n", "\n", "\n", "## Ridge and LASSO Regression\n", @@ -3783,7 +3788,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## More on Ridge Regression\n", + "## Deriving the Ridge Regression Equations\n", "\n", "Using the matrix-vector expression for Ridge regression," ] @@ -3803,7 +3808,8 @@ "source": [ "by taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", "a slightly modified matrix inversion problem which for finite values\n", - "of $\\lambda$ does not suffer from singularity problems. We obtain" + "of $\\lambda$ does not suffer from singularity problems. We obtain\n", + "the optimal parameters" ] }, { @@ -3811,7 +3817,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", + "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", "$$" ] }, @@ -3837,12 +3843,7 @@ "source": [ "with $t$ a finite positive number. \n", "\n", - "We see that Ridge regression is nothing but the standard\n", - "OLS with a modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The\n", - "consequences, in particular for our discussion of the bias-variance tradeoff \n", - "are rather interesting.\n", - "\n", - "Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had" + "When we compare this with the ordinary least squares result we have" ] }, { @@ -3850,7 +3851,7 @@ "metadata": {}, "source": [ "$$\n", - "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D}.\n", + "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", "$$" ] }, @@ -3858,7 +3859,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as" + "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", + "\n", + "\n", + "We see that Ridge regression is nothing but the standard OLS with a\n", + "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", + "particular for our discussion of the bias-variance tradeoff are rather\n", + "interesting. We will see that for specific values of $\\lambda$, we may\n", + "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", + "\n", + "Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n", + "We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as" ] }, { @@ -3866,7 +3877,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}\n", + "\\boldsymbol{X}\\boldsymbol{\\beta} = =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", "$$" ] }, @@ -3882,7 +3893,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", + "\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", "$$" ] }, @@ -3981,166 +3992,6 @@ "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", "\n", "\n", - "\n", - "## A better understanding of regularization\n", - "\n", - "The parameter $\\lambda$ that we have introduced in the Ridge (and\n", - "Lasso as well) regression is often called a regularization parameter\n", - "or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?\n", - "\n", - "Here we will first look at how to analyze the difference between the\n", - "standard OLS equations and the Ridge expressions in terms of a linear\n", - "algebra analysis using the SVD algorithm. Thereafter, we will link\n", - "(see the material on the bias-variance tradeoff below) these\n", - "observation to the statisical analysis of the results. In particular\n", - "we consider how the variance of the parameters $\\boldsymbol{\\beta}$ is\n", - "affected by changing the parameter $\\lambda$.\n", - "\n", - "## Decomposing the OLS and Ridge expressions\n", - "\n", - "We have our design matrix\n", - " $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. With the SVD we decompose it as" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "\\boldsymbol{X} = \\boldsymbol{U\\Sigma V^T},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "with $\\boldsymbol{U}\\in {\\mathbb{R}}^{n\\times n}$, $\\boldsymbol{\\Sigma}\\in {\\mathbb{R}}^{n\\times p}$\n", - "and $\\boldsymbol{V}\\in {\\mathbb{R}}^{p\\times p}$.\n", - "\n", - "The matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ and $\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{I}$.\n", - "\n", - "\n", - "\n", - "\n", - "## Mathematical Properties\n", - "\n", - "There are several interesting mathematical properties which will be\n", - "relevant when we are going to discuss the differences between say\n", - "ordinary least squares (OLS) and **Ridge** regression.\n", - "\n", - "We have from OLS that the parameters of the linear approximation are given by" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The matrix to invert can be rewritten in terms of our SVD decomposition as" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "\\boldsymbol{X}^T\\boldsymbol{X} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Using the orthogonality properties of $\\boldsymbol{U}$ we have" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "\\boldsymbol{X}^T\\boldsymbol{X} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T = \\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T,\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "with $\\boldsymbol{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. \n", - "\n", - "This means that" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "(\\boldsymbol{X}^T\\boldsymbol{X})\\boldsymbol{V} = \\boldsymbol{V}\\boldsymbol{D},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "that is the eigenvectors of $(\\boldsymbol{X}^T\\boldsymbol{X})$ are given by the columns of the right singular matrix of $\\boldsymbol{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D},\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "that is, the eigenvectors of $(\\boldsymbol{X}\\boldsymbol{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. \n", - "\n", - "Going back to our OLS equation we have" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "$$\n", - "\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", - "$$" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We will come back to this expression when we discuss Ridge regression. \n", - "\n", - "\n", - "$$ \\tilde{y}^{OLS}={\\bf X}\\hat{\\beta}^{OLS}=\\sum_{j=1}^p {\\bf u}_j{\\bf u}_j^T{\\bf y}$$ and for Ridge we have \n", - "\n", - "$$ \\tilde{y}^{Ridge}={\\bf X}\\hat{\\beta}^{Ridge}=\\sum_{j=1}^p {\\bf u}_j\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}{\\bf u}_j^T{\\bf y}$$ . \n", - "\n", - "It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$. \n", - "\n", - "Here we have that $${\\bf X} = {\\bf U}{\\bf \\Sigma}{\\bf V}^T$$, with $$\\Sigma$$ being an $$ n\\times p$$ matrix and $${\\bf V}$$ being a $$ p\\times p$$ matrix. We also have assumed here that $$ n > p$$. \n", - "\n", - "\n", "\n", "\n", "## Exercises for week 36, September 6-10\n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index afd08ed6f..b41d1e28e 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -2281,9 +2281,14 @@ $\bm{X}\bm{X}^T$\bm{U}=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm \] !et -It means that the eigenvalues of $\bm{X}\bm{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\bm{U}$ are the eigenvectors of $\bm{X}\bm{X}^T$ and measure how much correlations are contained in the rows of $\m{X}$. +It means that the eigenvalues of $\bm{X}\bm{X}^T$ are again given by +the non-zero singular values plus now a series of zeros. The column +vectors of $\bm{U}$ are the eigenvectors of $\bm{X}\bm{X}^T$ and +measure how much correlations are contained in the rows of $\bm{X}$. -Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\bm{V}$. +Since we will mainly be interested in the correlations among the features +of our data (the columns of $\bm{X}$, the quantity of interest for us are the non-zero singular +values and the column vectors of $\bm{V}$. !split @@ -2350,7 +2355,7 @@ Here we have defined the norm-1 as !split -===== More on Ridge Regression ===== +===== Deriving the Ridge Regression Equations ===== Using the matrix-vector expression for Ridge regression, @@ -2363,10 +2368,10 @@ C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X} by taking the derivatives with respect to $\bm{\beta}$ we obtain then a slightly modified matrix inversion problem which for finite values of $\lambda$ does not suffer from singularity problems. We obtain - +the optimal parameters !bt \[ -\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, +\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}, \] !et @@ -2380,22 +2385,26 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that with $t$ a finite positive number. -We see that Ridge regression is nothing but the standard -OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The -consequences, in particular for our discussion of the bias-variance tradeoff -are rather interesting. - -Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had +When we compare this with the ordinary least squares result we have !bt \[ -(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}. +\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}, \] !et +which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\bm{X}^T\bm{X}$. -We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as + +We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to $\bm{X}^T\bm{X}$. The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of $\lambda$, we may +even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. + +Using our insights about the SVD of the design matrix $\bm{X}$ +We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as !bt \[ -\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y} +\bm{X}\bm{\beta} = =\bm{U}\bm{U}^T\bm{y}. \] !et @@ -2404,7 +2413,7 @@ For Ridge regression this becomes !bt \[ -\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, +\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y}, \] !et @@ -2468,103 +2477,6 @@ For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"ht Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended. -!split -===== A better understanding of regularization ===== - -The parameter $\lambda$ that we have introduced in the Ridge (and -Lasso as well) regression is often called a regularization parameter -or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically? - -Here we will first look at how to analyze the difference between the -standard OLS equations and the Ridge expressions in terms of a linear -algebra analysis using the SVD algorithm. Thereafter, we will link -(see the material on the bias-variance tradeoff below) these -observation to the statisical analysis of the results. In particular -we consider how the variance of the parameters $\bm{\beta}$ is -affected by changing the parameter $\lambda$. - -!split -===== Decomposing the OLS and Ridge expressions ===== - -We have our design matrix - $\bm{X}\in {\mathbb{R}}^{n\times p}$. With the SVD we decompose it as - -!bt -\[ -\bm{X} = \bm{U\Sigma V^T}, -\] -!et - -with $\bm{U}\in {\mathbb{R}}^{n\times n}$, $\bm{\Sigma}\in {\mathbb{R}}^{n\times p}$ -and $\bm{V}\in {\mathbb{R}}^{p\times p}$. - -The matrices $\bm{U}$ and $\bm{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\bm{U}^T\bm{U}=\bm{U}\bm{U}^T=\bm{I}$ and $\bm{V}^T\bm{V}=\bm{V}\bm{V}^T=\bm{I}$. - - - - -!split -===== Mathematical Properties ===== - -There are several interesting mathematical properties which will be -relevant when we are going to discuss the differences between say -ordinary least squares (OLS) and _Ridge_ regression. - -We have from OLS that the parameters of the linear approximation are given by -!bt -\[ -\bm{\tilde{y}} = \bm{X}\bm{\beta} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}. -\] -!et - -The matrix to invert can be rewritten in terms of our SVD decomposition as - -!bt -\[ -\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T. -\] -!et -Using the orthogonality properties of $\bm{U}$ we have - -!bt -\[ -\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T = \bm{V}\bm{D}\bm{V}^T, -\] -!et -with $\bm{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. - -This means that -!bt -\[ -(\bm{X}^T\bm{X})\bm{V} = \bm{V}\bm{D}, -\] -!et -that is the eigenvectors of $(\bm{X}^T\bm{X})$ are given by the columns of the right singular matrix of $\bm{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that -!bt -\[ -(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}, -\] -!et -that is, the eigenvectors of $(\bm{X}\bm{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. - -Going back to our OLS equation we have -!bt -\[ -\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}. -\] -!et -We will come back to this expression when we discuss Ridge regression. - - -$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ and for Ridge we have  - -$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ .  - -It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$.  - -Here we have that $${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$, with $$\Sigma$$ being an $$ n\times p$$ matrix and $${\bf V}$$ being a $$ p\times p$$ matrix. We also have assumed here that $$ n > p$$.  - - !split