updates
This commit is contained in:
Morten Hjorth-Jensen
@@ -405,16 +405,16 @@ doconce format html chapter2.do.txt --><section class="tex2jax_ignore mathjax_i
|
||||
<section id="mathematical-interpretation-of-ordinary-least-squares">
|
||||
<h2><span class="section-number">4.1. </span>Mathematical Interpretation of Ordinary Least Squares<a class="headerlink" href="#mathematical-interpretation-of-ordinary-least-squares" title="Link to this heading">#</a></h2>
|
||||
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).</p>
|
||||
<p>We have shown that in ordinary least squares (OLS) the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> are given by</p>
|
||||
<p>We have shown that in ordinary least squares (OLS) the optimal parameters <span class="math notranslate nohighlight">\(\theta\)</span> are given by</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>The <strong>hat</strong> over <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> means we have the optimal parameters after minimization of the cost function.</p>
|
||||
<p>The <strong>hat</strong> over <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> means we have the optimal parameters after minimization of the cost function.</p>
|
||||
<p>This means that our best model is defined as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>We now define a matrix</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -424,7 +424,7 @@ doconce format html chapter2.do.txt --><section class="tex2jax_ignore mathjax_i
|
||||
<p>We can rewrite</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>The matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> has the important property that <span class="math notranslate nohighlight">\(\boldsymbol{A}^2=\boldsymbol{A}\)</span>. This is the definition of a <a class="reference external" href="https://en.wikipedia.org/wiki/Projection_matrix">projection matrix</a>.
|
||||
We can then interpret our optimal model <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> as being represented by an orthogonal projection of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> onto a space defined by the column vectors of <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. In our case here the matrix <span class="math notranslate nohighlight">\(\boldsymbol{A}\)</span> is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.</p>
|
||||
@@ -525,15 +525,15 @@ This is equivalent to saying that the matrix <span class="math notranslate nohig
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{equation}
|
||||
\boldsymbol{\beta} = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\boldsymbol{\theta} = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\label{_auto1} \tag{1}
|
||||
\end{equation}
|
||||
\]</div>
|
||||
<p>has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\beta_i\)</span>.
|
||||
of <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span> and we cannot find the parameters (estimators) <span class="math notranslate nohighlight">\(\theta_i\)</span>.
|
||||
The estimators are only well-defined if <span class="math notranslate nohighlight">\((\boldsymbol{X}^{T}\boldsymbol{X})\)</span> can be inverted.
|
||||
This is more likely to happen when the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters <span class="math notranslate nohighlight">\(\beta_i\)</span> cannot be estimated.</p>
|
||||
the regression parameters <span class="math notranslate nohighlight">\(\theta_i\)</span> cannot be estimated.</p>
|
||||
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
@@ -923,7 +923,7 @@ least squares is proportional to the second derivative of the cost
|
||||
function, that is we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T\partial \boldsymbol{\theta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\]</div>
|
||||
<p>This quantity defines what is called the Hessian matrix (the second derivative of the cost function we want to optimize).</p>
|
||||
<p>The Hessian matrix plays an important role and is defined in this course as</p>
|
||||
@@ -1154,7 +1154,7 @@ def create_X(x, y, n ):
|
||||
y = np.ravel(y)
|
||||
|
||||
N = len(x)
|
||||
l = int((n+1)*(n+2)/2) # Number of elements in beta
|
||||
l = int((n+1)*(n+2)/2) # Number of elements in theta
|
||||
X = np.ones((N,l))
|
||||
|
||||
for i in range(1,n+1):
|
||||
@@ -1318,13 +1318,13 @@ values and the column vectors of <span class="math notranslate nohighlight">\(\b
|
||||
our optimization problem is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
\]</div>
|
||||
<p>or we can state it as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
|
||||
\]</div>
|
||||
<p>where we have used the definition of a norm-2 vector, that is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -1332,26 +1332,26 @@ our optimization problem is</p>
|
||||
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
|
||||
\]</div>
|
||||
<p>By minimizing the above equation with respect to the parameters
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we could then obtain an analytical expression for the
|
||||
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. We can add a regularization parameter <span class="math notranslate nohighlight">\(\lambda\)</span> by
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we could then obtain an analytical expression for the
|
||||
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. We can add a regularization parameter <span class="math notranslate nohighlight">\(\lambda\)</span> by
|
||||
defining a new cost function to be optimized, that is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
|
||||
\]</div>
|
||||
<p>which leads to the Ridge regression minimization problem where we
|
||||
require that <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t\)</span>, where <span class="math notranslate nohighlight">\(t\)</span> is
|
||||
require that <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t\)</span>, where <span class="math notranslate nohighlight">\(t\)</span> is
|
||||
a finite number larger than zero. By defining</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
|
||||
\]</div>
|
||||
<p>we have a new optimization equation</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
|
||||
\]</div>
|
||||
<p>which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.</p>
|
||||
<p>Here we have defined the norm-1 as</p>
|
||||
@@ -1362,44 +1362,44 @@ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\bolds
|
||||
<p>Using the matrix-vector expression for Ridge regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard means squared error equation, we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
|
||||
\]</div>
|
||||
<p>and
|
||||
taking the derivatives with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we obtain then
|
||||
taking the derivatives with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of <span class="math notranslate nohighlight">\(\lambda\)</span> does not suffer from singularity problems. We obtain
|
||||
the optimal parameters</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\]</div>
|
||||
<p>with <span class="math notranslate nohighlight">\(\boldsymbol{I}\)</span> being a <span class="math notranslate nohighlight">\(p\times p\)</span> identity matrix with the constraint that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
|
||||
\]</div>
|
||||
<p>with <span class="math notranslate nohighlight">\(t\)</span> a finite positive number.</p>
|
||||
<p>When we compare this with the ordinary least squares result we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\]</div>
|
||||
<p>which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>.</p>
|
||||
<p>We see that Ridge regression is nothing but the standard OLS with a
|
||||
modified diagonal term added to <span class="math notranslate nohighlight">\(\boldsymbol{X}^T\boldsymbol{X}\)</span>. The consequences, in
|
||||
particular for our discussion of the bias-variance tradeoff are rather
|
||||
interesting. We will see that for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>, we may
|
||||
even reduce the variance of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.</p>
|
||||
even reduce the variance of the optimal parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.</p>
|
||||
<p>Using our insights about the SVD of the design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>
|
||||
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>For Ridge regression this becomes</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
\]</div>
|
||||
<p>with the vectors <span class="math notranslate nohighlight">\(\boldsymbol{u}_j\)</span> being the columns of <span class="math notranslate nohighlight">\(\boldsymbol{U}\)</span> from the SVD of the matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. Note that the sums goes to <span class="math notranslate nohighlight">\(p-1\)</span> since.</p>
|
||||
<p>Since <span class="math notranslate nohighlight">\(\lambda \geq 0\)</span>, it means that compared to OLS, we have</p>
|
||||
@@ -1421,12 +1421,12 @@ eigenvalues ordered in a descending way, that is <span class="math notranslate n
|
||||
<p>In this case the standard OLS results in</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
\]</div>
|
||||
<p>and</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
|
||||
\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
|
||||
\]</div>
|
||||
<p>that is the Ridge estimator scales the OLS estimator by the inverse of a factor <span class="math notranslate nohighlight">\(1+\lambda\)</span>, and
|
||||
the Ridge estimator converges to zero when the hyperparameter goes to
|
||||
@@ -1435,64 +1435,64 @@ infinity.</p>
|
||||
<p>Using the matrix-vector expression for Lasso regression and dropping the parameter <span class="math notranslate nohighlight">\(1/n\)</span> in front of the standard mean squared error equation, we have the following <strong>cost</strong> function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
\]</div>
|
||||
<p>Taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<p>Taking the derivative with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
|
||||
\frac{d \vert \theta\vert}{d \boldsymbol{\theta}}=\mathrm{sgn}(\boldsymbol{\theta})=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
|
||||
\end{split}\]</div>
|
||||
<p>we have that the derivative of the cost function is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
|
||||
\]</div>
|
||||
<p>and reordering we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a class="reference external" href="https://cvxopt.org/">CVXOPT</a>. We will discuss this later.</p>
|
||||
<p>Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
|
||||
diagonal. In this case we have an equal number of rows and columns <span class="math notranslate nohighlight">\(n=p\)</span>.</p>
|
||||
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\beta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
|
||||
<p>Our model approximation is just <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}=\boldsymbol{\theta}\)</span> and the mean squared error and thereby the cost function for ordinary least squares (OLS) is then (we drop the term <span class="math notranslate nohighlight">\(1/n\)</span>)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
|
||||
C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2,
|
||||
\]</div>
|
||||
<p>and minimizing we have that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\beta}_i^{\mathrm{OLS}} = y_i.
|
||||
\hat{\theta}_i^{\mathrm{OLS}} = y_i.
|
||||
\]</div>
|
||||
<p>For Ridge regression our cost function is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,
|
||||
C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\theta_i^2,
|
||||
\]</div>
|
||||
<p>and minimizing we have that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}.
|
||||
\hat{\theta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}.
|
||||
\]</div>
|
||||
<p>For Lasso regression our cost function is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2},
|
||||
C(\boldsymbol{\theta})=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\theta_i\vert=\sum_{i=0}^{p-1}(y_i-\theta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\theta_i^2},
|
||||
\]</div>
|
||||
<p>and minimizing we have that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0,
|
||||
-2\sum_{i=0}^{p-1}(y_i-\theta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\theta_i)}{\vert\theta_i\vert}=0,
|
||||
\]</div>
|
||||
<p>which leads to</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
|
||||
\hat{\boldsymbol{\theta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
|
||||
y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\
|
||||
0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
|
||||
\end{split}\]</div>
|
||||
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\beta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\beta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>Plotting these results (<a class="reference external" href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf">figure in handwritten notes for week 36</a>) shows clearly that Lasso regression suppresses (sets to zero) values of <span class="math notranslate nohighlight">\(\theta_i\)</span> for specific values of <span class="math notranslate nohighlight">\(\lambda\)</span>. Ridge regression reduces on the other hand the values of <span class="math notranslate nohighlight">\(\theta_i\)</span> as function of <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>As another example,
|
||||
let us assume we have a data set with outputs/targets given by the vector</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -1504,99 +1504,99 @@ let us assume we have a data set with outputs/targets given by the vector</p>
|
||||
\[\begin{split}
|
||||
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix},
|
||||
\end{split}\]</div>
|
||||
<p>meaning that we have two features and two unknown parameters <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> to be determined either by ordinary least squares, Ridge or Lasso regression.</p>
|
||||
<p>meaning that we have two features and two unknown parameters <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> to be determined either by ordinary least squares, Ridge or Lasso regression.</p>
|
||||
<p>For ordinary least squares (OLS) we know that the optimal solution is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>Inserting the above values we obtain that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
|
||||
\end{split}\]</div>
|
||||
<p>The code which implements this simpler case is presented after the discussion of Ridge and Lasso.</p>
|
||||
<p>For Ridge regression we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\]</div>
|
||||
<p>Inserting the above values we obtain that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
|
||||
\end{split}\]</div>
|
||||
<p>There is normally a constraint on the value of <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\beta}\vert\vert_2\)</span> via the parameter <span class="math notranslate nohighlight">\(\lambda\)</span>.
|
||||
Let us for simplicity assume that <span class="math notranslate nohighlight">\(\beta_0^2+\beta_1^2=1\)</span> as constraint. This will allow us to find an expression for the optimal values of <span class="math notranslate nohighlight">\(\beta\)</span> and <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>There is normally a constraint on the value of <span class="math notranslate nohighlight">\(\vert\vert \boldsymbol{\theta}\vert\vert_2\)</span> via the parameter <span class="math notranslate nohighlight">\(\lambda\)</span>.
|
||||
Let us for simplicity assume that <span class="math notranslate nohighlight">\(\theta_0^2+\theta_1^2=1\)</span> as constraint. This will allow us to find an expression for the optimal values of <span class="math notranslate nohighlight">\(\theta\)</span> and <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>To see this, let us write the cost function for Ridge regression.</p>
|
||||
<p>We define the MSE without the <span class="math notranslate nohighlight">\(1/n\)</span> factor and have then, using that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
|
||||
\boldsymbol{X}\boldsymbol{\theta}=\begin{bmatrix} 2\theta_0 \\ \theta_1 \\0 \end{bmatrix},
|
||||
\end{split}\]</div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
|
||||
C(\boldsymbol{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\theta_0^2+\theta_1^2),
|
||||
\]</div>
|
||||
<p>and taking the derivative with respect to <span class="math notranslate nohighlight">\(\beta_0\)</span> we get</p>
|
||||
<p>and taking the derivative with respect to <span class="math notranslate nohighlight">\(\theta_0\)</span> we get</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\beta_0=\frac{8}{4+\lambda},
|
||||
\theta_0=\frac{8}{4+\lambda},
|
||||
\]</div>
|
||||
<p>and for <span class="math notranslate nohighlight">\(\beta_1\)</span> we obtain</p>
|
||||
<p>and for <span class="math notranslate nohighlight">\(\theta_1\)</span> we obtain</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\beta_1=\frac{2}{1+\lambda},
|
||||
\theta_1=\frac{2}{1+\lambda},
|
||||
\]</div>
|
||||
<p>Using the constraint for <span class="math notranslate nohighlight">\(\beta_0^2+\beta_1^2=1\)</span> we can constrain <span class="math notranslate nohighlight">\(\lambda\)</span> by solving</p>
|
||||
<p>Using the constraint for <span class="math notranslate nohighlight">\(\theta_0^2+\theta_1^2=1\)</span> we can constrain <span class="math notranslate nohighlight">\(\lambda\)</span> by solving</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
|
||||
\]</div>
|
||||
<p>which gives <span class="math notranslate nohighlight">\(\lambda=4.571\)</span> and <span class="math notranslate nohighlight">\(\beta_0=0.933\)</span> and <span class="math notranslate nohighlight">\(\beta_1=0.359\)</span>.</p>
|
||||
<p>For Lasso we need now, keeping a constraint on <span class="math notranslate nohighlight">\(\vert\beta_0\vert+\vert\beta_1\vert=1\)</span>, to take the derivative of the absolute values of <span class="math notranslate nohighlight">\(\beta_0\)</span>
|
||||
and <span class="math notranslate nohighlight">\(\beta_1\)</span>. This gives us the following derivatives of the cost function</p>
|
||||
<p>which gives <span class="math notranslate nohighlight">\(\lambda=4.571\)</span> and <span class="math notranslate nohighlight">\(\theta_0=0.933\)</span> and <span class="math notranslate nohighlight">\(\theta_1=0.359\)</span>.</p>
|
||||
<p>For Lasso we need now, keeping a constraint on <span class="math notranslate nohighlight">\(\vert\theta_0\vert+\vert\theta_1\vert=1\)</span>, to take the derivative of the absolute values of <span class="math notranslate nohighlight">\(\theta_0\)</span>
|
||||
and <span class="math notranslate nohighlight">\(\theta_1\)</span>. This gives us the following derivatives of the cost function</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
|
||||
C(\boldsymbol{\theta})=(4-2\theta_0)^2+(2-\theta_1)^2+\lambda(\vert\theta_0\vert+\vert\theta_1\vert),
|
||||
\]</div>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_0}=-4(4-2\theta_0)+\lambda\mathrm{sgn}(\theta_0)=0,
|
||||
\]</div>
|
||||
<p>and</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_1}=-2(2-\theta_1)+\lambda\mathrm{sgn}(\theta_1)=0.
|
||||
\]</div>
|
||||
<p>We have now four cases to solve besides the trivial cases <span class="math notranslate nohighlight">\(\beta_0\)</span> and/or <span class="math notranslate nohighlight">\(\beta_1\)</span> are zero, namely</p>
|
||||
<p>We have now four cases to solve besides the trivial cases <span class="math notranslate nohighlight">\(\theta_0\)</span> and/or <span class="math notranslate nohighlight">\(\theta_1\)</span> are zero, namely</p>
|
||||
<ol class="arabic simple">
|
||||
<li><p><span class="math notranslate nohighlight">\(\beta_0 > 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 > 0\)</span>,</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\beta_0 > 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 < 0\)</span>,</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\beta_0 < 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 > 0\)</span>,</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\beta_0 < 0\)</span> and <span class="math notranslate nohighlight">\(\beta_1 < 0\)</span>.</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\theta_0 > 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 > 0\)</span>,</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\theta_0 > 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 < 0\)</span>,</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\theta_0 < 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 > 0\)</span>,</p></li>
|
||||
<li><p><span class="math notranslate nohighlight">\(\theta_0 < 0\)</span> and <span class="math notranslate nohighlight">\(\theta_1 < 0\)</span>.</p></li>
|
||||
</ol>
|
||||
<p>If we consider the first case, we have then</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
-4(4-2\beta_0)+\lambda=0,
|
||||
-4(4-2\theta_0)+\lambda=0,
|
||||
\]</div>
|
||||
<p>and</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
-2(2-\beta_1)+\lambda=0.
|
||||
-2(2-\theta_1)+\lambda=0.
|
||||
\]</div>
|
||||
<p>which yields</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\beta_0=\frac{16+\lambda}{8},
|
||||
\theta_0=\frac{16+\lambda}{8},
|
||||
\]</div>
|
||||
<p>and</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\beta_1=\frac{4+\lambda}{2}.
|
||||
\theta_1=\frac{4+\lambda}{2}.
|
||||
\]</div>
|
||||
<p>Using the constraint on <span class="math notranslate nohighlight">\(\beta_0\)</span> and <span class="math notranslate nohighlight">\(\beta_1\)</span> we can then find the optimal value of <span class="math notranslate nohighlight">\(\lambda\)</span> for the different cases. We leave this as an exercise to you.</p>
|
||||
<p>Using the constraint on <span class="math notranslate nohighlight">\(\theta_0\)</span> and <span class="math notranslate nohighlight">\(\theta_1\)</span> we can then find the optimal value of <span class="math notranslate nohighlight">\(\lambda\)</span> for the different cases. We leave this as an exercise to you.</p>
|
||||
<p>Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of <span class="math notranslate nohighlight">\(\lambda\)</span>, meaning that we need to perform a search in order to find the optimal values.</p>
|
||||
<p>First we study and compare the OLS and Ridge results. The next code compares all three methods.
|
||||
We select values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\in [10^{-4},10^4]\)</span> and compute the predicted values for ordinary least squares and Ridge regression.</p>
|
||||
@@ -1623,14 +1623,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]])
|
||||
y = np.array( [4, 2, 3])
|
||||
|
||||
|
||||
# matrix inversion to find beta
|
||||
OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
|
||||
print(OLSbeta)
|
||||
# matrix inversion to find theta
|
||||
OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y
|
||||
print(OLStheta)
|
||||
# and then make the prediction
|
||||
ytildeOLS = X @ OLSbeta
|
||||
ytildeOLS = X @ OLStheta
|
||||
print("Training MSE for OLS")
|
||||
print(MSE(y,ytildeOLS))
|
||||
ypredictOLS = X @ OLSbeta
|
||||
ypredictOLS = X @ OLStheta
|
||||
|
||||
# Repeat now for Ridge regression and various values of the regularization parameter
|
||||
I = np.eye(2,2)
|
||||
@@ -1640,10 +1640,10 @@ MSEPredict = np.zeros(nlambdas)
|
||||
lambdas = np.logspace(-4, 4, nlambdas)
|
||||
for i in range(nlambdas):
|
||||
lmb = lambdas[i]
|
||||
Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
|
||||
# print(Ridgebeta)
|
||||
Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
|
||||
# print(Ridgetheta)
|
||||
# and then make the prediction
|
||||
ypredictRidge = X @ Ridgebeta
|
||||
ypredictRidge = X @ Ridgetheta
|
||||
MSEPredict[i] = MSE(y,ypredictRidge)
|
||||
# print(MSEPredict[i])
|
||||
# Now plot the results
|
||||
@@ -1657,9 +1657,9 @@ plt.show()
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
|
||||
<p>We see here that we reach a plateau for the Ridge results. Writing out the coefficients <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>, we observe that they are getting smaller and smaller and our error stabilizes since the predicted values of <span class="math notranslate nohighlight">\(\tilde{\boldsymbol{y}}\)</span> approach zero.</p>
|
||||
<p>This happens also for Lasso regression, as seen from the next code
|
||||
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\beta\)</span> to
|
||||
output. The difference is that Lasso shrinks the values of <span class="math notranslate nohighlight">\(\theta\)</span> to
|
||||
zero at a much earlier stage and the results flatten out. We see that
|
||||
Lasso gives also an excellent fit for small values of <span class="math notranslate nohighlight">\(\lambda\)</span> and
|
||||
shows the best performance of the three regression methods.</p>
|
||||
@@ -1685,14 +1685,14 @@ X = np.array( [ [ 2, 0], [0, 1], [0,0]])
|
||||
y = np.array( [4, 2, 3])
|
||||
|
||||
|
||||
# matrix inversion to find beta
|
||||
OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
|
||||
print(OLSbeta)
|
||||
# matrix inversion to find theta
|
||||
OLStheta = np.linalg.inv(X.T @ X) @ X.T @ y
|
||||
print(OLStheta)
|
||||
# and then make the prediction
|
||||
ytildeOLS = X @ OLSbeta
|
||||
ytildeOLS = X @ OLStheta
|
||||
print("Training MSE for OLS")
|
||||
print(MSE(y,ytildeOLS))
|
||||
ypredictOLS = X @ OLSbeta
|
||||
ypredictOLS = X @ OLStheta
|
||||
|
||||
# Repeat now for Ridge regression and various values of the regularization parameter
|
||||
I = np.eye(2,2)
|
||||
@@ -1703,10 +1703,10 @@ MSELassoPredict = np.zeros(nlambdas)
|
||||
lambdas = np.logspace(-4, 4, nlambdas)
|
||||
for i in range(nlambdas):
|
||||
lmb = lambdas[i]
|
||||
Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
|
||||
print(Ridgebeta)
|
||||
Ridgetheta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
|
||||
print(Ridgetheta)
|
||||
# and then make the prediction
|
||||
ypredictRidge = X @ Ridgebeta
|
||||
ypredictRidge = X @ Ridgetheta
|
||||
MSERidgePredict[i] = MSE(y,ypredictRidge)
|
||||
RegLasso = linear_model.Lasso(lmb)
|
||||
RegLasso.fit(X,y)
|
||||
@@ -1730,7 +1730,7 @@ three regression methods. Depending on the level of noise, we note
|
||||
that for small values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span> all three
|
||||
methods produce the same mean squared error. Again, Lasso shrinks the
|
||||
parameter values to zero much earlier than Ridge regression and the
|
||||
Lasso results flatten out much earlier since all <span class="math notranslate nohighlight">\(\beta_j=0\)</span> (check
|
||||
Lasso results flatten out much earlier since all <span class="math notranslate nohighlight">\(\theta_j=0\)</span> (check
|
||||
this by printing the values). This case is an example of where OLS
|
||||
performs best. Lasso and Ridge reproduce the OLS results for a limited
|
||||
set of <span class="math notranslate nohighlight">\(\lambda\)</span> values.</p>
|
||||
@@ -1767,14 +1767,14 @@ X[:,2] = x*x
|
||||
# We split the data in test and training data
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# matrix inversion to find beta
|
||||
OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
|
||||
print(OLSbeta)
|
||||
# matrix inversion to find theta
|
||||
OLStheta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
|
||||
print(OLStheta)
|
||||
# and then make the prediction
|
||||
ytildeOLS = X_train @ OLSbeta
|
||||
ytildeOLS = X_train @ OLStheta
|
||||
print("Training MSE for OLS")
|
||||
print(MSE(y_train,ytildeOLS))
|
||||
ypredictOLS = X_test @ OLSbeta
|
||||
ypredictOLS = X_test @ OLStheta
|
||||
print("Test MSE OLS")
|
||||
print(MSE(y_test,ypredictOLS))
|
||||
|
||||
@@ -1789,13 +1789,13 @@ MSELassoTrain = np.zeros(nlambdas)
|
||||
lambdas = np.logspace(-4, 4, nlambdas)
|
||||
for i in range(nlambdas):
|
||||
lmb = lambdas[i]
|
||||
Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
|
||||
Ridgetheta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
|
||||
# include lasso using Scikit-Learn
|
||||
RegLasso = linear_model.Lasso(lmb)
|
||||
RegLasso.fit(X_train,y_train)
|
||||
# and then make the prediction
|
||||
ytildeRidge = X_train @ Ridgebeta
|
||||
ypredictRidge = X_test @ Ridgebeta
|
||||
ytildeRidge = X_train @ Ridgetheta
|
||||
ypredictRidge = X_test @ Ridgetheta
|
||||
ytildeLasso = RegLasso.predict(X_train)
|
||||
ypredictLasso = RegLasso.predict(X_test)
|
||||
MSEPredict[i] = MSE(y_test,ypredictRidge)
|
||||
@@ -1832,7 +1832,7 @@ move from a linear algebra analysis to a statistical analysis. In
|
||||
particular, we will focus on what the regularization terms can result
|
||||
in. We will amongst other things show that the regularization
|
||||
parameter can reduce considerably the variance of the parameters
|
||||
<span class="math notranslate nohighlight">\(\beta\)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\theta\)</span>.</p>
|
||||
<p>The
|
||||
advantage of doing linear regression is that we actually end up with
|
||||
analytical expressions for several statistical quantities.<br />
|
||||
@@ -1853,7 +1853,7 @@ independent, i.e.:</p>
|
||||
<p>The randomness of <span class="math notranslate nohighlight">\(\varepsilon_i\)</span> implies that
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is also a random variable. In particular,
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> is normally distributed, because <span class="math notranslate nohighlight">\(\varepsilon_i \sim
|
||||
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\beta}\)</span> is a
|
||||
\mathcal{N}(0, \sigma^2)\)</span> and <span class="math notranslate nohighlight">\(\mathbf{X}_{i,\ast} \, \boldsymbol{\theta}\)</span> is a
|
||||
non-random scalar. To specify the parameters of the distribution of
|
||||
<span class="math notranslate nohighlight">\(\mathbf{y}_i\)</span> we need to calculate its first two moments.</p>
|
||||
<p>Recall that <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> is a matrix of dimensionality <span class="math notranslate nohighlight">\(n\times p\)</span>. The
|
||||
@@ -1870,15 +1870,15 @@ which describe our data</p>
|
||||
function <span class="math notranslate nohighlight">\(f\)</span> is approximated by <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> where we want to minimize <span class="math notranslate nohighlight">\((\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\)</span>, our MSE, with</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}.
|
||||
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\theta}.
|
||||
\]</div>
|
||||
<p>We can calculate the expectation value of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> for a given element <span class="math notranslate nohighlight">\(i\)</span></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\begin{align*}
|
||||
\mathbb{E}(y_i) & =
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta,
|
||||
\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\theta}) + \mathbb{E}(\varepsilon_i)
|
||||
\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \theta,
|
||||
\end{align*}
|
||||
\]</div>
|
||||
<p>while
|
||||
@@ -1888,80 +1888,80 @@ its variance is</p>
|
||||
\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i
|
||||
- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) -
|
||||
[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \,
|
||||
\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2
|
||||
\theta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 \\ &
|
||||
= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2 \varepsilon_i
|
||||
\mathbf{X}_{i, \ast} \, \boldsymbol{\theta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i,
|
||||
\ast} \, \theta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2 + 2
|
||||
\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\theta} +
|
||||
\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta})^2
|
||||
\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \,
|
||||
\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2.
|
||||
\end{align*}
|
||||
\end{split}\]</div>
|
||||
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
|
||||
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\beta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
|
||||
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> we can evaluate the expectation value</p>
|
||||
<p>Hence, <span class="math notranslate nohighlight">\(y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\theta}, \sigma^2)\)</span>, that is <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> follows a normal distribution with
|
||||
mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbol{\theta}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span> (not be confused with the singular values of the SVD).</p>
|
||||
<p>With the OLS expressions for the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> we can evaluate the expectation value</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
|
||||
\mathbb{E}(\boldsymbol{\theta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\theta}=\boldsymbol{\theta}.
|
||||
\]</div>
|
||||
<p>This means that the estimator of the regression parameters is unbiased.</p>
|
||||
<p>We can also calculate the variance</p>
|
||||
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is</p>
|
||||
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[\begin{split}
|
||||
\begin{eqnarray*}
|
||||
\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \}
|
||||
\mbox{Var}(\boldsymbol{\theta}) & = & \mathbb{E} \{ [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})] [\boldsymbol{\theta} - \mathbb{E}(\boldsymbol{\theta})]^{T} \}
|
||||
\\
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \}
|
||||
& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\theta}]^{T} \}
|
||||
\\
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\\
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
% \\
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1}
|
||||
% \\
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T
|
||||
% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\theta} \boldsymbol{\theta}^T
|
||||
\\
|
||||
& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T}
|
||||
& = & \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\theta} \, \boldsymbol{\theta}^{T}
|
||||
\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1},
|
||||
\end{eqnarray*}
|
||||
\end{split}\]</div>
|
||||
<p>where we have used that <span class="math notranslate nohighlight">\(\mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) =
|
||||
\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\beta}) = \sigma^2
|
||||
\mathbf{X} \, \boldsymbol{\theta} \, \boldsymbol{\theta}^{T} \, \mathbf{X}^{T} +
|
||||
\sigma^2 \, \mathbf{I}_{nn}\)</span>. From <span class="math notranslate nohighlight">\(\mbox{Var}(\boldsymbol{\theta}) = \sigma^2
|
||||
\, (\mathbf{X}^{T} \mathbf{X})^{-1}\)</span>, one obtains an estimate of the
|
||||
variance of the estimate of the <span class="math notranslate nohighlight">\(j\)</span>-th regression coefficient:
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\sigma}^2 (\boldsymbol{\theta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \)</span>. This may be used to
|
||||
construct a confidence interval for the estimates.</p>
|
||||
<p>In a similar way, we can obtain analytical expressions for say the
|
||||
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and their variance
|
||||
expectation values of the parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and their variance
|
||||
when we employ Ridge regression, allowing us again to define a confidence interval.</p>
|
||||
<p>It is rather straightforward to show that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}.
|
||||
\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\theta}^{\mathrm{OLS}}.
|
||||
\]</div>
|
||||
<p>We see clearly that
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda > 0\)</span>. We say then that the ridge estimator is biased.</p>
|
||||
<span class="math notranslate nohighlight">\(\mathbb{E} \big[ \boldsymbol{\theta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\theta}^{\mathrm{OLS}}\)</span> for any <span class="math notranslate nohighlight">\(\lambda > 0\)</span>. We say then that the ridge estimator is biased.</p>
|
||||
<p>We can also compute the variance as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T},
|
||||
\]</div>
|
||||
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> goes to zero.</p>
|
||||
<p>and it is easy to see that if the parameter <span class="math notranslate nohighlight">\(\lambda\)</span> goes to infinity then the variance of Ridge parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> goes to zero.</p>
|
||||
<p>With this, we can compute the difference</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\mbox{Var}[\boldsymbol{\theta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\theta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}.
|
||||
\]</div>
|
||||
<p>The difference is non-negative definite since each component of the
|
||||
matrix product is non-negative definite.
|
||||
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda > 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
|
||||
This means the variance we obtain with the standard OLS will always for <span class="math notranslate nohighlight">\(\lambda > 0\)</span> be larger than the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.</p>
|
||||
</section>
|
||||
<section id="deriving-ols-from-a-probability-distribution">
|
||||
<h2><span class="section-number">4.12. </span>Deriving OLS from a probability distribution<a class="headerlink" href="#deriving-ols-from-a-probability-distribution" title="Link to this heading">#</a></h2>
|
||||
@@ -1971,26 +1971,26 @@ that our output is determined by a given continuous function
|
||||
distribution with zero mean value and an undetermined variance
|
||||
<span class="math notranslate nohighlight">\(\sigma^2\)</span>.</p>
|
||||
<p>We found above that the outputs <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> have a mean value given by
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span> and variance <span class="math notranslate nohighlight">\(\sigma^2\)</span>. Since the entries to
|
||||
the design matrix are not stochastic variables, we can assume that the
|
||||
probability distribution of our targets is also a normal distribution
|
||||
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\beta}}\)</span>. This means that a
|
||||
but now with mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\hat{\boldsymbol{\theta}}\)</span>. This means that a
|
||||
single output <span class="math notranslate nohighlight">\(y_i\)</span> is given by the Gaussian distribution</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\theta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
<p>We assume now that the various <span class="math notranslate nohighlight">\(y_i\)</span> values are stochastically distributed according to the above Gaussian distribution.
|
||||
We define this distribution as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]},
|
||||
p(y_i, \boldsymbol{X}\vert\boldsymbol{\theta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]},
|
||||
\]</div>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<p>which reads as finding the likelihood of an event <span class="math notranslate nohighlight">\(y_i\)</span> with the input variables <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> given the parameters (to be determined) <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
|
||||
<p>Since these events are assumed to be independent and identically distributed we can build the probability distribution function (PDF) for all possible event <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as the product of the single events, that is we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}).
|
||||
p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta}).
|
||||
\]</div>
|
||||
<p>We will write this in a more compact form reserving <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> for the domain of events, including the ouputs (targets) and the inputs. That is
|
||||
in case we have a simple one-dimensional input and output case</p>
|
||||
@@ -2002,18 +2002,18 @@ in case we have a simple one-dimensional input and output case</p>
|
||||
We can now rewrite the above probability as</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
<p>It is a conditional probability (see below) and reads as the
|
||||
likelihood of a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> given a set of parameters
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>.</p>
|
||||
<p>In statistics, maximum likelihood estimation (MLE) is a method of
|
||||
estimating the parameters of an assumed probability distribution,
|
||||
given some observed data. This is achieved by maximizing a likelihood
|
||||
function so that, under the assumed statistical model, the observed
|
||||
data is the most probable.</p>
|
||||
<p>We will assume here that our events are given by the above Gaussian
|
||||
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\beta\)</span> by
|
||||
distribution and we will determine the optimal parameters <span class="math notranslate nohighlight">\(\theta\)</span> by
|
||||
maximizing the above PDF. However, computing the derivatives of a
|
||||
product function is cumbersome and can easily lead to overflow and/or
|
||||
underflowproblems, with potentials for loss of numerical precision.</p>
|
||||
@@ -2027,22 +2027,22 @@ is equivalent to the maximization/minimization of the function itself.</p>
|
||||
<p>We could now define a new cost function to minimize, namely the negative logarithm of the above PDF</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})},
|
||||
C(\boldsymbol{\theta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\theta})},
|
||||
\]</div>
|
||||
<p>which becomes</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}.
|
||||
C(\boldsymbol{\theta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}.
|
||||
\]</div>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\beta\)</span> we recognize our familiar OLS equation, namely</p>
|
||||
<p>Taking the derivative of the <em>new</em> cost function with respect to the parameters <span class="math notranslate nohighlight">\(\theta\)</span> we recognize our familiar OLS equation, namely</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0,
|
||||
\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right) =0,
|
||||
\]</div>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\beta\)</span></p>
|
||||
<p>which leads to the well-known OLS equation for the optimal paramters <span class="math notranslate nohighlight">\(\theta\)</span></p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\hat{\boldsymbol{\theta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}!
|
||||
\]</div>
|
||||
<p>Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.</p>
|
||||
<p>A central theorem in statistics is Bayes’ theorem. This theorem plays a similar role as the good old Pythagoras’ theorem in geometry.
|
||||
@@ -2160,9 +2160,9 @@ perhaps not that intuitive. The question is whether we can develop a
|
||||
more intuitive way of understanding what Ridge and Lasso express.</p>
|
||||
<p>Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.</p>
|
||||
<p>We will play around with a study of the values for the optimal
|
||||
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> using OLS, Ridge and Lasso regression. For
|
||||
parameters <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> using OLS, Ridge and Lasso regression. For
|
||||
OLS, you will notice as function of the noise and polynomial degree,
|
||||
that the parameters <span class="math notranslate nohighlight">\(\beta\)</span> will fluctuate from order to order in the
|
||||
that the parameters <span class="math notranslate nohighlight">\(\theta\)</span> will fluctuate from order to order in the
|
||||
polynomial fit and that for larger and larger polynomial degrees of
|
||||
freedom, the parameters will tend to increase in value for OLS.</p>
|
||||
<p>For Ridge and Lasso regression, the higher order parameters will
|
||||
@@ -2198,10 +2198,10 @@ for polydegree in range(1, Maxpolydegree):
|
||||
# We split the data in test and training data
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# matrix inversion to find beta
|
||||
OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
|
||||
print(OLSbeta)
|
||||
ypredictOLS = X_test @ OLSbeta
|
||||
# matrix inversion to find theta
|
||||
OLStheta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
|
||||
print(OLStheta)
|
||||
ypredictOLS = X_test @ OLStheta
|
||||
print("Test MSE OLS")
|
||||
print(MSE(y_test,ypredictOLS))
|
||||
# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
|
||||
@@ -2238,21 +2238,21 @@ plt.show()
|
||||
</div>
|
||||
</div>
|
||||
<p>How can we understand this?</p>
|
||||
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\beta_i\)</span> as functions
|
||||
<p>Let us write out the values of the coefficients <span class="math notranslate nohighlight">\(\theta_i\)</span> as functions
|
||||
of the polynomial degree and noise. We will focus only on the Ridge
|
||||
results and some few selected values of the hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>.</p>
|
||||
<p>If we don’t include any noise and run this code for different values
|
||||
of the polynomial degree, we notice that the results for <span class="math notranslate nohighlight">\(\beta_i\)</span> do
|
||||
of the polynomial degree, we notice that the results for <span class="math notranslate nohighlight">\(\theta_i\)</span> do
|
||||
not show great changes from one order to the next. This is an
|
||||
indication that for higher polynomial orders, our parameters become
|
||||
less important.</p>
|
||||
<p>If we however add noise, what happens is that the polynomial fit is
|
||||
trying to adjust the fit to traverse in the best possible way all data
|
||||
points. This can lead to large fluctuations in the parameters
|
||||
<span class="math notranslate nohighlight">\(\beta_i\)</span> as functions of polynomial order. It will also be reflected
|
||||
in a larger value of the variance of each parameter <span class="math notranslate nohighlight">\(\beta_i\)</span>. What
|
||||
<span class="math notranslate nohighlight">\(\theta_i\)</span> as functions of polynomial order. It will also be reflected
|
||||
in a larger value of the variance of each parameter <span class="math notranslate nohighlight">\(\theta_i\)</span>. What
|
||||
Ridge regression (and Lasso as well) are doing then is to try to
|
||||
quench the fluctuations in the parameters of <span class="math notranslate nohighlight">\(\beta_i\)</span> which have a
|
||||
quench the fluctuations in the parameters of <span class="math notranslate nohighlight">\(\theta_i\)</span> which have a
|
||||
large variance (normally for higher orders in the polynomial).</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -2291,26 +2291,26 @@ for i in range(nlambdas):
|
||||
# and then make the prediction
|
||||
ypredictRidge = RegRidge.predict(X_test)
|
||||
Coeffs = np.array(RegRidge.coef_)
|
||||
BetaValues = pd.DataFrame(Coeffs)
|
||||
BetaValues.columns = ['beta']
|
||||
display(BetaValues)
|
||||
ThetaValues = pd.DataFrame(Coeffs)
|
||||
ThetaValues.columns = ['theta']
|
||||
display(ThetaValues)
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p>As an exercise, repeat these calculations with ordinary least squares
|
||||
only with and without noise. Calculate thereafter the variance of the
|
||||
parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> as function of polynomial order and of the added
|
||||
parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> as function of polynomial order and of the added
|
||||
noise. Here we recommend to use <span class="math notranslate nohighlight">\(\sigma^2=1\)</span> as variance for the
|
||||
added noise (which follows a normal distribution with mean value zero).
|
||||
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span> vary more as function
|
||||
Comment your results. If you have a large noise term, do the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span> vary more as function
|
||||
of model complexity? And what about their variance?</p>
|
||||
</section>
|
||||
<section id="linking-bayes-theorem-with-ridge-and-lasso-regression">
|
||||
<h2><span class="section-number">4.14. </span>Linking Bayes’ Theorem with Ridge and Lasso Regression<a class="headerlink" href="#linking-bayes-theorem-with-ridge-and-lasso-regression" title="Link to this heading">#</a></h2>
|
||||
<p>We have seen that Ridge regression suppresses those features which
|
||||
have a small singular value. This corresponds to a feature which exhibits
|
||||
a large variance in the parameters <span class="math notranslate nohighlight">\(\beta_j\)</span>.
|
||||
a large variance in the parameters <span class="math notranslate nohighlight">\(\theta_j\)</span>.
|
||||
Our analysis hitherto has been based on linear algebra. To add to our intuition, we will use
|
||||
Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regression.</p>
|
||||
<p>For ordinary least squares we postulated that the maximum likelihood for the domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span> (one-dimensional case)</p>
|
||||
@@ -2321,74 +2321,74 @@ Bayes’ theorem in order to deepen our understanding of Ridge and Lasso regres
|
||||
<p>is given by</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}.
|
||||
p(\boldsymbol{D}\vert\boldsymbol{\theta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}.
|
||||
\]</div>
|
||||
<p>In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> given a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span>? That is, how can we define the posterior probability</p>
|
||||
<p>In Bayes’ theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> given a domain of events <span class="math notranslate nohighlight">\(\boldsymbol{D}\)</span>? That is, how can we define the posterior probability</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta}\vert\boldsymbol{D}).
|
||||
p(\boldsymbol{\theta}\vert\boldsymbol{D}).
|
||||
\]</div>
|
||||
<p>Bayes’ theorem comes to our rescue here since (omitting the normalization constant)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}).
|
||||
p(\boldsymbol{\theta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\theta})p(\boldsymbol{\theta}).
|
||||
\]</div>
|
||||
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\beta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\beta})\)</span>!</p>
|
||||
<p>We have a model for <span class="math notranslate nohighlight">\(p(\boldsymbol{D}\vert\boldsymbol{\theta})\)</span> but need one for the <strong>prior</strong> <span class="math notranslate nohighlight">\(p(\boldsymbol{\theta})\)</span>!</p>
|
||||
<p>With the posterior probability defined by a likelihood which we have
|
||||
already modeled and an unknown prior, we are now ready to make
|
||||
additional models for the prior.</p>
|
||||
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> and
|
||||
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is
|
||||
<p>We can, based on our discussions of the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> and
|
||||
the mean value, assume that the prior for the values <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span> is
|
||||
given by a Gaussian with mean value zero and variance <span class="math notranslate nohighlight">\(\tau^2\)</span>, that</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
|
||||
p(\boldsymbol{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}.
|
||||
\]</div>
|
||||
<p>Our posterior probability becomes then (omitting the normalization factor which is just a constant)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
|
||||
p(\boldsymbol{\theta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\theta_j^2}{2\tau^2}\right)}.
|
||||
\]</div>
|
||||
<p>We can now optimize this quantity with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. As we
|
||||
<p>We can now optimize this quantity with respect to <span class="math notranslate nohighlight">\(\boldsymbol{\theta}\)</span>. As we
|
||||
did for OLS, this is most conveniently done by taking the negative
|
||||
logarithm of the posterior probability. Doing so and leaving out the
|
||||
terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
|
||||
terms that do not depend on <span class="math notranslate nohighlight">\(\theta\)</span>, we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
|
||||
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\theta}\vert\vert_2^2,
|
||||
\]</div>
|
||||
<p>and replacing <span class="math notranslate nohighlight">\(1/2\tau^2\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
|
||||
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_2^2,
|
||||
\]</div>
|
||||
<p>which is our Ridge cost function! Nice, isn’t it?</p>
|
||||
<p>To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (<a class="reference external" href="https://en.wikipedia.org/wiki/Laplace_distribution">Laplace in this case</a>) with zero mean value, that is</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}.
|
||||
p(\boldsymbol{\theta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}.
|
||||
\]</div>
|
||||
<p>Our posterior probability becomes then (omitting the normalization factor which is just a constant)</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}.
|
||||
p(\boldsymbol{\theta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\theta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\theta_j\vert}{\tau}\right)}.
|
||||
\]</div>
|
||||
<p>Taking the negative
|
||||
logarithm of the posterior probability and leaving out the
|
||||
constants terms that do not depend on <span class="math notranslate nohighlight">\(\beta\)</span>, we have</p>
|
||||
constants terms that do not depend on <span class="math notranslate nohighlight">\(\theta\)</span>, we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
\]</div>
|
||||
<p>and replacing <span class="math notranslate nohighlight">\(1/\tau\)</span> with <span class="math notranslate nohighlight">\(\lambda\)</span> we have</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
\[
|
||||
C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{\theta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
\]</div>
|
||||
<p>which is our Lasso cost function!</p>
|
||||
<p>Plotting these prior functions shows us that we can use the parameter
|
||||
<span class="math notranslate nohighlight">\(\lambda\)</span> to shrink or increase the role of a given parameter
|
||||
<span class="math notranslate nohighlight">\(\beta_j\)</span>. The variance for the Laplace distribution is
|
||||
<span class="math notranslate nohighlight">\(\theta_j\)</span>. The variance for the Laplace distribution is
|
||||
<span class="math notranslate nohighlight">\(2\tau^2=1/\lambda\)</span> while for the Gaussian distribution it is
|
||||
<span class="math notranslate nohighlight">\(\sigma^2=1/(2\lambda)\)</span>. Thus, increasing the variance means
|
||||
decreasing <span class="math notranslate nohighlight">\(\lambda\)</span> and shrinking the variance means increasing
|
||||
|
||||
Reference in New Issue
Block a user