lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2025-08-24 22:31:49 +02:00
parent b68f2a90e3
commit d65046c133
23 changed files with 850 additions and 856 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/week35/week35.do.txt
+81 -82
View File
@@ -30,12 +30,11 @@ o For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Mar
!split
===== Reminder from last week =====
We need first a reminder from last week about linear regression.
We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters $\bm{\theta}$ and our first encounter is ordinary least squares.
Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$.
* Method of choice for fitting a continuous function!
* It is the method of choice for fitting a continuous function
* Gives an excellent introduction to central Machine Learning features with _understandable pedagogical_ links to other methods like _Neural Networks_, _Support Vector Machines_ etc
* Analytical expression for the fitting parameters $\bm{\beta}$
* Analytical expression for the fitting parameters $\bm{\theta}$
* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
* Analytical relation with probabilistic interpretations
* Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
@@ -76,20 +75,20 @@ value and a variance $\sigma^2$.
In linear regression we approximate the unknown function with another
continuous function $\tilde{\bm{y}}(\bm{x})$ which depends linearly on
some unknown parameters
$\bm{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}]$.
$\bm{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}]$.
Last week we introduced the so-called design matrix in order to define
the approximation $\bm{\tilde{y}}$ via the unknown quantity
$\bm{\beta}$ as
$\bm{\theta}$ as
!bt
\[
\bm{\tilde{y}}= \bm{X}\bm{\beta},
\bm{\tilde{y}}= \bm{X}\bm{\theta},
\]
!et
and in order to find the optimal parameters $\beta_i$ we defined a function which
and in order to find the optimal parameters $\theta_i$ we defined a function which
gives a measure of the spread between the values $y_i$ (which
represent the output values we want to reproduce) and the parametrized
values $\tilde{y}_i$, namely the so-called cost/loss function.
@@ -101,13 +100,13 @@ values $\tilde{y}_i$, namely the so-called cost/loss function.
We used the mean squared error to define the way we measure the quality of our model
!bt
\[
C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
\]
!et
or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as
!bt
\[
C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
This function represents one of many possible ways to define the so-called cost function.
@@ -119,10 +118,10 @@ the function $C$ as
!bt
\[
C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
C(\bm{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
\]
!et
since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out.
since when taking the first derivative with respect to the unknown parameters $\theta$, the factor of $2$ cancels out.
@@ -133,14 +132,14 @@ since when taking the first derivative with respect to the unknown parameters $\
The function
!bt
\[
C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\},
C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\},
\]
!et
can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value.
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value
!bt
\[
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
\]
!et
@@ -153,29 +152,29 @@ error estimate, often limited to a statistical error estimate given by
the standard deviation discussed earlier. In the discussion here we
will treat $y_i$ as our exact value for the response variable.
In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem
In order to find the parameters $\theta_i$ we will then minimize the spread of $C(\bm{\theta})$, that is we are going to solve the problem
!bt
\[
{\displaystyle \min_{\bm{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
{\displaystyle \min_{\bm{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
In practical terms it means we will require
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
\frac{\partial C(\bm{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
\]
!et
which results in
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
\frac{\partial C(\bm{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
\]
!et
or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivation below)
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right).
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right).
\]
!et
@@ -186,19 +185,19 @@ or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivatio
We can rewrite, see the derivations below,
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right),
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right),
\]
!et
as
!bt
\[
\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta},
\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\theta},
\]
!et
and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution
!bt
\[
\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\bm{\theta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
@@ -420,53 +419,53 @@ and if $\bm{y}=\bm{x}$ we have
We defined earlier a possible cost function using the mean squared error
!bt
\[
C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
\]
!et
or using the design/feature matrix $\bm{X}$ we have the more compact matrix-vector
!bt
\[
C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\beta}$.
We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\beta}$.
We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\theta}$.
We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\theta}$.
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
!bt
\[
\bm{w}=\bm{y}-\bm{X}\bm{\beta},
\bm{w}=\bm{y}-\bm{X}\bm{\theta},
\]
!et
which depends on $\bm{\beta}$. We rewrite the cost function as
which depends on $\bm{\theta}$. We rewrite the cost function as
!bt
\[
C(\bm{\beta})=\frac{1}{n}\bm{w}^T\bm{w},
C(\bm{\theta})=\frac{1}{n}\bm{w}^T\bm{w},
\]
!et
with partial derivative
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}}=\frac{2}{n}\bm{w}^T\frac{\partial \bm{w}}{\partial \bm{\beta}},
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}}=\frac{2}{n}\bm{w}^T\frac{\partial \bm{w}}{\partial \bm{\theta}},
\]
!et
and using that
!bt
\[
\frac{\partial \bm{w}}{\partial \bm{\beta}}=-\bm{X},
\frac{\partial \bm{w}}{\partial \bm{\theta}}=-\bm{X},
\]
!et
where we used the result from example two above. Inserting the last expression we obtain
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\bm{X},
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}}=-\frac{2}{n}\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\bm{X},
\]
!et
or as
!bt
\[
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T}=-\frac{2}{n}\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right).
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T}=-\frac{2}{n}\bm{X}^T\left(\bm{y}-\bm{X}\bm{\theta}\right).
\]
!et
@@ -505,13 +504,13 @@ We list here some other useful relations we may encounter (recall that vectors a
A very important matrix we will meet again and again in machine
learning is the Hessian. It is given by the second derivative of the
cost function with respect to the parameters $\bm{\beta}$. Using the above
cost function with respect to the parameters $\bm{\theta}$. Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the mean squared error as cost function is,
!bt
\[
\frac{\partial}{\partial \bm{\beta}}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} =\frac{\partial}{\partial \bm{\beta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
\frac{\partial}{\partial \bm{\theta}}\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} =\frac{\partial}{\partial \bm{\theta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
@@ -527,7 +526,7 @@ The Hessian matrix plays an important role and is defined here as
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
$\hat{\bm{\beta}}$. Furthermore, we will see later this week that it is
$\hat{\bm{\theta}}$. Furthermore, we will see later this week that it is
(aside the factor $1/n$) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
@@ -542,22 +541,22 @@ _Linear algebra question:_ Can we use the Hessian matrix to say something about
The residuals $\bm{\epsilon}$ are in turn given by
!bt
\[
\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta},
\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\theta},
\]
!et
and with
!bt
\[
\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0,
\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0,
\]
!et
we have
!bt
\[
\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0,
\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0,
\]
!et
meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals.
meaning that the solution for $\bm{\theta}$ is the one which minimizes the residuals.
!eblock
@@ -571,17 +570,17 @@ we condiser a simple polynomial fit.
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
!bt
\[
\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
\]
!et
we have five predictors/features. The first is the intercept $\beta_0$. The other terms are $\beta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an
we have five predictors/features. The first is the intercept $\theta_0$. The other terms are $\theta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an
$n\times p$ matrix $\bm{X}$.
!split
===== Own code for Ordinary Least Squares =====
It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\theta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
!bc pycod
# matrix inversion to find beta
# First we set up the data
@@ -952,21 +951,21 @@ plt.show()
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
We have shown that in ordinary least squares the optimal parameters $\beta$ are given by
We have shown that in ordinary least squares the optimal parameters $\theta$ are given by
!bt
\[
\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\hat{\bm{\theta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
The _hat_ over $\bm{\beta}$ means we have the optimal parameters after minimization of the cost function.
The _hat_ over $\bm{\theta}$ means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
!bt
\[
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
@@ -980,7 +979,7 @@ We now define a matrix
We can rewrite
!bt
\[
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}.
\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{A}\bm{y}.
\]
!et
@@ -1125,14 +1124,14 @@ This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue
If our design matrix $\bm{X}$ which enters the linear regression problem
!bt
\begin{align}
\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y},
\bm{\theta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y},
\end{align}
!et
has linearly dependent column vectors, we will not be able to compute the inverse
of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$.
of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\theta_i$.
The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits.
This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where
the regression parameters $\beta_i$ cannot be estimated.
the regression parameters $\theta_i$ cannot be estimated.
A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change
!bt
@@ -1541,7 +1540,7 @@ function, that is we have
!bt
\[
\frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} =\frac{2}{n}\bm{X}^T\bm{X}.
\frac{\partial^2 C(\bm{\theta})}{\partial \bm{\theta}\partial \bm{\theta}^T} =\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
@@ -2034,14 +2033,14 @@ Let us remind ourselves about the expression for the standard Mean Squared Error
our optimization problem is
!bt
\[
{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
{\displaystyle \min_{\bm{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
or we can state it as
!bt
\[
{\displaystyle \min_{\bm{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2,
{\displaystyle \min_{\bm{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2,
\]
!et
where we have used the definition of a norm-2 vector, that is
@@ -2052,32 +2051,32 @@ where we have used the definition of a norm-2 vector, that is
!et
By minimizing the above equation with respect to the parameters
$\bm{\beta}$ we could then obtain an analytical expression for the
parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
$\bm{\theta}$ we could then obtain an analytical expression for the
parameters $\bm{\theta}$. We can add a regularization parameter $\lambda$ by
defining a new cost function to be optimized, that is
!bt
\[
{\displaystyle \min_{\bm{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2
{\displaystyle \min_{\bm{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_2^2
\]
!et
which leads to the Ridge regression minimization problem where we
require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is
require that $\vert\vert \bm{\theta}\vert\vert_2^2\le t$, where $t$ is
a finite number larger than zero. By defining
!bt
\[
C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1,
C(\bm{X},\bm{\theta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1,
\]
!et
we have a new optimization equation
!bt
\[
{\displaystyle \min_{\bm{\beta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1
{\displaystyle \min_{\bm{\theta}\in
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1
\]
!et
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
@@ -2097,17 +2096,17 @@ Using the matrix-vector expression for Ridge regression and dropping the paramet
!bt
\[
C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
C(\bm{X},\bm{\theta})=\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\bm{\theta}^T\bm{\theta},
\]
!et
and
taking the derivatives with respect to $\bm{\beta}$ we obtain then
taking the derivatives with respect to $\bm{\theta}$ we obtain then
a slightly modified matrix inversion problem which for finite values
of $\lambda$ does not suffer from singularity problems. We obtain
the optimal parameters
!bt
\[
\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
@@ -2115,16 +2114,16 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that
!bt
\[
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
\]
!et
with $t$ a finite positive number.
If we keep the $1/n$ factor, the equation for the optimal $\beta$ changes to
If we keep the $1/n$ factor, the equation for the optimal $\theta$ changes to
!bt
\[
\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
@@ -2133,7 +2132,7 @@ In many textbooks the $1/n$ term is often omitted. Note that a library like _Sci
When we compare this with the ordinary least squares result we have
!bt
\[
\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
\hat{\bm{\theta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\bm{X}^T\bm{X}$.
@@ -2143,13 +2142,13 @@ We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to $\bm{X}^T\bm{X}$. The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of $\lambda$, we may
even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
even reduce the variance of the optimal parameters $\bm{\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix $\bm{X}$
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
!bt
\[
\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} =\bm{U}\bm{U}^T\bm{y}.
\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\theta} =\bm{U}\bm{U}^T\bm{y}.
\]
!et
@@ -2158,7 +2157,7 @@ For Ridge regression this becomes
!bt
\[
\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\beta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\theta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
\]
!et
@@ -2198,7 +2197,7 @@ For the sake of simplicity, let us assume that the design matrix is orthonormal,
In this case the standard OLS results in
!bt
\[
\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}_i^T\bm{y},
\bm{\theta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}_i^T\bm{y},
\]
!et
@@ -2206,7 +2205,7 @@ and
!bt
\[
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}},
\bm{\theta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\theta}^{\mathrm{OLS}},
\]
!et
@@ -2226,33 +2225,33 @@ Using the matrix-vector expression for Lasso regression, we have the following _
!bt
\[
C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
C(\bm{X},\bm{\theta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\vert\vert\bm{\theta}\vert\vert_1,
\]
!et
Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
Taking the derivative with respect to $\bm{\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
!bt
\[
\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
\]
!et
we have that the derivative of the cost function is
!bt
\[
\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
\frac{\partial C(\bm{X},\bm{\theta})}{\partial \bm{\theta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\theta})+\lambda sgn(\bm{\theta})=0,
\]
!et
and reordering we have
!bt
\[
\bm{X}^T\bm{X}\bm{\beta}+\frac{n}{2}\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
\bm{X}^T\bm{X}\bm{\theta}+\frac{n}{2}\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}.
\]
!et
We can redefine $\lambda$ to absorb the constant $n/2$ and we rewrite the last equation as
!bt
\[
\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
\bm{X}^T\bm{X}\bm{\theta}+\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}.
\]
!et