rewriting slides from week 35
This commit is contained in:
Morten Hjorth-Jensen
+221
-221
@@ -1821,231 +1821,11 @@ the eigenvalues of the covariance matrix and the Hessian matrix in
|
||||
terms of the singular values.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Ridge and LASSO Regression =====
|
||||
|
||||
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
|
||||
our optimization problem is
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
|
||||
\]
|
||||
!et
|
||||
or we can state it as
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2,
|
||||
\]
|
||||
!et
|
||||
where we have used the definition of a norm-2 vector, that is
|
||||
!bt
|
||||
\[
|
||||
\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
|
||||
\]
|
||||
!et
|
||||
|
||||
By minimizing the above equation with respect to the parameters
|
||||
$\bm{\beta}$ we could then obtain an analytical expression for the
|
||||
parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
|
||||
defining a new cost function to be optimized, that is
|
||||
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2
|
||||
\]
|
||||
!et
|
||||
|
||||
which leads to the Ridge regression minimization problem where we
|
||||
require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is
|
||||
a finite number larger than zero. By defining
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1,
|
||||
\]
|
||||
!et
|
||||
|
||||
we have a new optimization equation
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1
|
||||
\]
|
||||
!et
|
||||
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
|
||||
|
||||
Here we have defined the norm-1 as
|
||||
!bt
|
||||
\[
|
||||
\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert.
|
||||
\]
|
||||
!et
|
||||
|
||||
|
||||
!split
|
||||
===== More on Ridge Regression =====
|
||||
|
||||
Using the matrix-vector expression for Ridge regression,
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
|
||||
\]
|
||||
!et
|
||||
|
||||
by taking the derivatives with respect to $\bm{\beta}$ we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of $\lambda$ does not suffer from singularity problems. We obtain
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that
|
||||
|
||||
!bt
|
||||
\[
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\]
|
||||
!et
|
||||
|
||||
with $t$ a finite positive number.
|
||||
|
||||
We see that Ridge regression is nothing but the standard
|
||||
OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The
|
||||
consequences, in particular for our discussion of the bias-variance tradeoff
|
||||
are rather interesting.
|
||||
|
||||
Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had
|
||||
!bt
|
||||
\[
|
||||
(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}.
|
||||
\]
|
||||
!et
|
||||
|
||||
We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
|
||||
!bt
|
||||
\[
|
||||
\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}
|
||||
\]
|
||||
!et
|
||||
|
||||
|
||||
For Ridge regression this becomes
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
with the vectors $\bm{u}_j$ being the columns of $\bm{U}$.
|
||||
|
||||
!split
|
||||
===== Interpreting the Ridge results =====
|
||||
|
||||
Since $\lambda \geq 0$, it means that compared to OLS, we have
|
||||
|
||||
!bt
|
||||
\[
|
||||
\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1.
|
||||
\]
|
||||
!et
|
||||
|
||||
Ridge regression finds the coordinates of $\bm{y}$ with respect to the
|
||||
orthonormal basis $\bm{U}$, it then shrinks the coordinates by
|
||||
$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has
|
||||
eigenvalues ordered in a descending way, that is $\sigma_i \geq
|
||||
\sigma_{i+1}$.
|
||||
|
||||
For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom.
|
||||
Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$.
|
||||
With a parameter $\lambda$ we can thus shrink the role of specific parameters.
|
||||
|
||||
|
||||
!split
|
||||
===== More interpretations =====
|
||||
|
||||
For the sake of simplicity, let us assume that the design matrix is orthonormal, that is
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}.
|
||||
\]
|
||||
!et
|
||||
|
||||
In this case the standard OLS results in
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
and
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}},
|
||||
\]
|
||||
!et
|
||||
|
||||
that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and
|
||||
the Ridge estimator converges to zero when the hyperparameter goes to
|
||||
infinity.
|
||||
|
||||
We will come back to more interpreations after we have gone through some of the statistical analysis part.
|
||||
|
||||
For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
|
||||
Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended.
|
||||
|
||||
|
||||
!split
|
||||
===== A better understanding of regularization =====
|
||||
|
||||
The parameter $\lambda$ that we have introduced in the Ridge (and
|
||||
Lasso as well) regression is often called a regularization parameter
|
||||
or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?
|
||||
|
||||
Here we will first look at how to analyze the difference between the
|
||||
standard OLS equations and the Ridge expressions in terms of a linear
|
||||
algebra analysis using the SVD algorithm. Thereafter, we will link
|
||||
(see the material on the bias-variance tradeoff below) these
|
||||
observation to the statisical analysis of the results. In particular
|
||||
we consider how the variance of the parameters $\bm{\beta}$ is
|
||||
affected by changing the parameter $\lambda$.
|
||||
|
||||
!split
|
||||
===== Decomposing the OLS and Ridge expressions =====
|
||||
|
||||
We have our design matrix
|
||||
$\bm{X}\in {\mathbb{R}}^{n\times p}$. With the SVD we decompose it as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X} = \bm{U\Sigma V^T},
|
||||
\]
|
||||
!et
|
||||
|
||||
with $\bm{U}\in {\mathbb{R}}^{n\times n}$, $\bm{\Sigma}\in {\mathbb{R}}^{n\times p}$
|
||||
and $\bm{V}\in {\mathbb{R}}^{p\times p}$.
|
||||
|
||||
The matrices $\bm{U}$ and $\bm{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\bm{U}^T\bm{U}=\bm{U}\bm{U}^T=\bm{I}$ and $\bm{V}^T\bm{V}=\bm{V}\bm{V}^T=\bm{I}$.
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Introducing the Covariance and Correlation functions =====
|
||||
|
||||
Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about
|
||||
the definition of the covariance and the correlation function. These are quantities
|
||||
the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods.
|
||||
|
||||
Suppose we have defined two vectors
|
||||
$\hat{x}$ and $\hat{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as
|
||||
@@ -2376,6 +2156,226 @@ where we wrote $$\bm{C}[\bm{x}_0,\bm{x}_1] = \bm{C}[\bm{x}]$$ to indicate that t
|
||||
It is easy to generalize this to a matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Ridge and LASSO Regression =====
|
||||
|
||||
Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is
|
||||
our optimization problem is
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
|
||||
\]
|
||||
!et
|
||||
or we can state it as
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2,
|
||||
\]
|
||||
!et
|
||||
where we have used the definition of a norm-2 vector, that is
|
||||
!bt
|
||||
\[
|
||||
\vert\vert \bm{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
|
||||
\]
|
||||
!et
|
||||
|
||||
By minimizing the above equation with respect to the parameters
|
||||
$\bm{\beta}$ we could then obtain an analytical expression for the
|
||||
parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
|
||||
defining a new cost function to be optimized, that is
|
||||
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2
|
||||
\]
|
||||
!et
|
||||
|
||||
which leads to the Ridge regression minimization problem where we
|
||||
require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is
|
||||
a finite number larger than zero. By defining
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1,
|
||||
\]
|
||||
!et
|
||||
|
||||
we have a new optimization equation
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1
|
||||
\]
|
||||
!et
|
||||
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
|
||||
|
||||
Here we have defined the norm-1 as
|
||||
!bt
|
||||
\[
|
||||
\vert\vert \bm{x}\vert\vert_1 = \sum_i \vert x_i\vert.
|
||||
\]
|
||||
!et
|
||||
|
||||
|
||||
!split
|
||||
===== More on Ridge Regression =====
|
||||
|
||||
Using the matrix-vector expression for Ridge regression,
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
|
||||
\]
|
||||
!et
|
||||
|
||||
by taking the derivatives with respect to $\bm{\beta}$ we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of $\lambda$ does not suffer from singularity problems. We obtain
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that
|
||||
|
||||
!bt
|
||||
\[
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\]
|
||||
!et
|
||||
|
||||
with $t$ a finite positive number.
|
||||
|
||||
We see that Ridge regression is nothing but the standard
|
||||
OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The
|
||||
consequences, in particular for our discussion of the bias-variance tradeoff
|
||||
are rather interesting.
|
||||
|
||||
Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had
|
||||
!bt
|
||||
\[
|
||||
(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}.
|
||||
\]
|
||||
!et
|
||||
|
||||
We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
|
||||
!bt
|
||||
\[
|
||||
\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}
|
||||
\]
|
||||
!et
|
||||
|
||||
|
||||
For Ridge regression this becomes
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
with the vectors $\bm{u}_j$ being the columns of $\bm{U}$.
|
||||
|
||||
!split
|
||||
===== Interpreting the Ridge results =====
|
||||
|
||||
Since $\lambda \geq 0$, it means that compared to OLS, we have
|
||||
|
||||
!bt
|
||||
\[
|
||||
\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1.
|
||||
\]
|
||||
!et
|
||||
|
||||
Ridge regression finds the coordinates of $\bm{y}$ with respect to the
|
||||
orthonormal basis $\bm{U}$, it then shrinks the coordinates by
|
||||
$\frac{\sigma_j^2}{\sigma_j^2+\lambda}$. Recall that the SVD has
|
||||
eigenvalues ordered in a descending way, that is $\sigma_i \geq
|
||||
\sigma_{i+1}$.
|
||||
|
||||
For small eigenvalues $\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom.
|
||||
Actually, calculating the variance of $\bm{X}\bm{v}_j$ shows that this quantity is equal to $\sigma_j^2/n$.
|
||||
With a parameter $\lambda$ we can thus shrink the role of specific parameters.
|
||||
|
||||
|
||||
!split
|
||||
===== More interpretations =====
|
||||
|
||||
For the sake of simplicity, let us assume that the design matrix is orthonormal, that is
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\bm{X}=(\bm{X}^T\bm{X})^{-1} =\bm{I}.
|
||||
\]
|
||||
!et
|
||||
|
||||
In this case the standard OLS results in
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_j\bm{u}_j^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
and
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}},
|
||||
\]
|
||||
!et
|
||||
|
||||
that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\lambda$, and
|
||||
the Ridge estimator converges to zero when the hyperparameter goes to
|
||||
infinity.
|
||||
|
||||
We will come back to more interpreations after we have gone through some of the statistical analysis part.
|
||||
|
||||
For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
|
||||
Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended.
|
||||
|
||||
|
||||
!split
|
||||
===== A better understanding of regularization =====
|
||||
|
||||
The parameter $\lambda$ that we have introduced in the Ridge (and
|
||||
Lasso as well) regression is often called a regularization parameter
|
||||
or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?
|
||||
|
||||
Here we will first look at how to analyze the difference between the
|
||||
standard OLS equations and the Ridge expressions in terms of a linear
|
||||
algebra analysis using the SVD algorithm. Thereafter, we will link
|
||||
(see the material on the bias-variance tradeoff below) these
|
||||
observation to the statisical analysis of the results. In particular
|
||||
we consider how the variance of the parameters $\bm{\beta}$ is
|
||||
affected by changing the parameter $\lambda$.
|
||||
|
||||
!split
|
||||
===== Decomposing the OLS and Ridge expressions =====
|
||||
|
||||
We have our design matrix
|
||||
$\bm{X}\in {\mathbb{R}}^{n\times p}$. With the SVD we decompose it as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X} = \bm{U\Sigma V^T},
|
||||
\]
|
||||
!et
|
||||
|
||||
with $\bm{U}\in {\mathbb{R}}^{n\times n}$, $\bm{\Sigma}\in {\mathbb{R}}^{n\times p}$
|
||||
and $\bm{V}\in {\mathbb{R}}^{p\times p}$.
|
||||
|
||||
The matrices $\bm{U}$ and $\bm{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\bm{U}^T\bm{U}=\bm{U}\bm{U}^T=\bm{I}$ and $\bm{V}^T\bm{V}=\bm{V}\bm{V}^T=\bm{I}$.
|
||||
|
||||
|
||||
|
||||
|
||||
!split
|
||||
===== Mathematical Properties =====
|
||||
|
||||
|
||||
Reference in New Issue
Block a user