update
This commit is contained in:
Morten Hjorth-Jensen
@@ -323,13 +323,12 @@ MathJax.Hub.Config({
|
||||
<!-- !split -->
|
||||
<h2 id="reminder-from-last-week" class="anchor">Reminder from last week </h2>
|
||||
|
||||
<p>We need first a reminder from last week about linear regression. </p>
|
||||
<p>We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.</p>
|
||||
|
||||
<p>Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).</p>
|
||||
<ul>
|
||||
<li> Method of choice for fitting a continuous function!</li>
|
||||
<li> It is the method of choice for fitting a continuous function</li>
|
||||
<li> Gives an excellent introduction to central Machine Learning features with <b>understandable pedagogical</b> links to other methods like <b>Neural Networks</b>, <b>Support Vector Machines</b> etc</li>
|
||||
<li> Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)</li>
|
||||
<li> Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)</li>
|
||||
<li> Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</li>
|
||||
<li> Analytical relation with probabilistic interpretations</li>
|
||||
<li> Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</li>
|
||||
|
||||
@@ -346,19 +346,19 @@ value and a variance \( \sigma^2 \).
|
||||
<p>In linear regression we approximate the unknown function with another
|
||||
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
|
||||
some unknown parameters
|
||||
\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
|
||||
\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
|
||||
</p>
|
||||
|
||||
<p>Last week we introduced the so-called design matrix in order to define
|
||||
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
|
||||
\( \boldsymbol{\beta} \) as
|
||||
\( \boldsymbol{\theta} \) as
|
||||
</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and in order to find the optimal parameters \( \beta_i \) we defined a function which
|
||||
<p>and in order to find the optimal parameters \( \theta_i \) we defined a function which
|
||||
gives a measure of the spread between the values \( y_i \) (which
|
||||
represent the output values we want to reproduce) and the parametrized
|
||||
values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
|
||||
@@ -325,12 +325,12 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>We used the mean squared error to define the way we measure the quality of our model</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>This function represents one of many possible ways to define the so-called cost function.</p>
|
||||
@@ -340,10 +340,10 @@ the function \( C \) as
|
||||
</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
$$
|
||||
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. </p>
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out. </p>
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
|
||||
@@ -325,14 +325,14 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>The function </p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value.
|
||||
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
|
||||
</p>
|
||||
$$
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
$$
|
||||
|
||||
<p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
|
||||
@@ -345,25 +345,25 @@ the standard deviation discussed earlier. In the discussion here we
|
||||
will treat \( y_i \) as our exact value for the response variable.
|
||||
</p>
|
||||
|
||||
<p>In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem</p>
|
||||
<p>In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>In practical terms it means we will require</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
$$
|
||||
|
||||
<p>which results in</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
$$
|
||||
|
||||
<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
|
||||
|
||||
|
||||
@@ -327,17 +327,17 @@ MathJax.Hub.Config({
|
||||
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>We can rewrite, see the derivations below, </p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
|
||||
$$
|
||||
|
||||
<p>as</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution</p>
|
||||
$$
|
||||
\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
|
||||
|
||||
@@ -325,46 +325,46 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>We defined earlier a possible cost function using the mean squared error</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
|
||||
$$
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as</p>
|
||||
<p>which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
$$
|
||||
|
||||
<p>with partial derivative</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
|
||||
$$
|
||||
|
||||
<p>and using that</p>
|
||||
$$
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
|
||||
$$
|
||||
|
||||
<p>where we used the result from example two above. Inserting the last expression we obtain</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
|
||||
$$
|
||||
|
||||
<p>or as</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
|
||||
|
||||
|
||||
@@ -325,13 +325,13 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>A very important matrix we will meet again and again in machine
|
||||
learning is the Hessian. It is given by the second derivative of the
|
||||
cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
|
||||
cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
|
||||
expression for derivatives of vectors and matrices, we find that the
|
||||
second derivative of the mean squared error as cost function is,
|
||||
</p>
|
||||
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
<p>The Hessian matrix plays an important role and is defined here as</p>
|
||||
@@ -342,7 +342,7 @@ $$
|
||||
|
||||
<p>For ordinary least squares, it is inversely proportional (derivation
|
||||
next week) with the variance of the optimal parameters
|
||||
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
|
||||
\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
|
||||
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
|
||||
important role in optmization algorithms and Principal Component
|
||||
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
|
||||
|
||||
@@ -328,20 +328,20 @@ MathJax.Hub.Config({
|
||||
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<p>The residuals \( \boldsymbol{\epsilon} \) are in turn given by</p>
|
||||
$$
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and with </p>
|
||||
$$
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
|
||||
<p>we have</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
|
||||
<p>meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. </p>
|
||||
<p>meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals. </p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -328,10 +328,10 @@ we condiser a simple polynomial fit.
|
||||
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
|
||||
</p>
|
||||
$$
|
||||
\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
|
||||
\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
|
||||
$$
|
||||
|
||||
<p>we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
<p>we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
\( n\times p \) matrix \( \boldsymbol{X} \).
|
||||
</p>
|
||||
|
||||
|
||||
@@ -323,7 +323,7 @@ MathJax.Hub.Config({
|
||||
<!-- !split -->
|
||||
<h2 id="own-code-for-ordinary-least-squares" class="anchor">Own code for Ordinary Least Squares </h2>
|
||||
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="cell border-box-sizing code_cell rendered">
|
||||
|
||||
@@ -325,18 +325,18 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). </p>
|
||||
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by</p>
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by</p>
|
||||
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
|
||||
<p>This means that our best model is defined as</p>
|
||||
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We now define a matrix</p>
|
||||
@@ -346,7 +346,7 @@ $$
|
||||
|
||||
<p>We can rewrite</p>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
|
||||
|
||||
@@ -326,16 +326,16 @@ MathJax.Hub.Config({
|
||||
<p>If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem</p>
|
||||
$$
|
||||
\begin{align}
|
||||
\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\tag{1}
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
<p>has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
|
||||
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
|
||||
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters \( \beta_i \) cannot be estimated.
|
||||
the regression parameters \( \theta_i \) cannot be estimated.
|
||||
</p>
|
||||
|
||||
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
|
||||
|
||||
@@ -331,7 +331,7 @@ function, that is we have
|
||||
</p>
|
||||
|
||||
$$
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
<p>This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).</p>
|
||||
|
||||
@@ -327,13 +327,13 @@ MathJax.Hub.Config({
|
||||
our optimization problem is
|
||||
</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>or we can state it as</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
|
||||
$$
|
||||
|
||||
<p>where we have used the definition of a norm-2 vector, that is</p>
|
||||
@@ -342,29 +342,29 @@ $$
|
||||
$$
|
||||
|
||||
<p>By minimizing the above equation with respect to the parameters
|
||||
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
|
||||
\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
|
||||
defining a new cost function to be optimized, that is
|
||||
</p>
|
||||
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
|
||||
$$
|
||||
|
||||
<p>which leads to the Ridge regression minimization problem where we
|
||||
require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
a finite number larger than zero. By defining
|
||||
</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
|
||||
<p>we have a new optimization equation</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
|
||||
$$
|
||||
|
||||
<p>which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. </p>
|
||||
|
||||
@@ -326,37 +326,37 @@ MathJax.Hub.Config({
|
||||
<p>Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and
|
||||
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
|
||||
taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of \( \lambda \) does not suffer from singularity problems. We obtain
|
||||
the optimal parameters
|
||||
</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that</p>
|
||||
|
||||
$$
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
|
||||
$$
|
||||
|
||||
<p>with \( t \) a finite positive number. </p>
|
||||
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to</p>
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>In many textbooks the \( 1/n \) term is often omitted. Note that a library like <b>Scikit-Learn</b> does not include the \( 1/n \) factor in the setup of the cost function.</p>
|
||||
|
||||
<p>When we compare this with the ordinary least squares result we have</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).</p>
|
||||
@@ -365,20 +365,20 @@ $$
|
||||
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
|
||||
particular for our discussion of the bias-variance tradeoff are rather
|
||||
interesting. We will see that for specific values of \( \lambda \), we may
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
</p>
|
||||
|
||||
<p>Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
|
||||
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
|
||||
</p>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>For Ridge regression this becomes</p>
|
||||
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). </p>
|
||||
|
||||
@@ -331,13 +331,13 @@ $$
|
||||
|
||||
<p>In this case the standard OLS results in </p>
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>and</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
|
||||
\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
|
||||
$$
|
||||
|
||||
<p>that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
|
||||
|
||||
@@ -326,28 +326,28 @@ MathJax.Hub.Config({
|
||||
<p>Using the matrix-vector expression for Lasso regression, we have the following <b>cost</b> function</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
$$
|
||||
\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
|
||||
\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
|
||||
$$
|
||||
|
||||
<p>we have that the derivative of the cost function is</p>
|
||||
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
|
||||
$$
|
||||
|
||||
<p>and reordering we have</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_self">CVXOPT</a>. We will discuss this later. </p>
|
||||
|
||||
@@ -219,13 +219,12 @@ MathJax.Hub.Config({
|
||||
<section>
|
||||
<h2 id="reminder-from-last-week">Reminder from last week </h2>
|
||||
|
||||
<p>We need first a reminder from last week about linear regression. </p>
|
||||
<p>We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.</p>
|
||||
|
||||
<p>Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).</p>
|
||||
<ul>
|
||||
<p><li> Method of choice for fitting a continuous function!</li>
|
||||
<p><li> It is the method of choice for fitting a continuous function</li>
|
||||
<p><li> Gives an excellent introduction to central Machine Learning features with <b>understandable pedagogical</b> links to other methods like <b>Neural Networks</b>, <b>Support Vector Machines</b> etc</li>
|
||||
<p><li> Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)</li>
|
||||
<p><li> Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)</li>
|
||||
<p><li> Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</li>
|
||||
<p><li> Analytical relation with probabilistic interpretations</li>
|
||||
<p><li> Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</li>
|
||||
@@ -269,21 +268,21 @@ value and a variance \( \sigma^2 \).
|
||||
<p>In linear regression we approximate the unknown function with another
|
||||
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
|
||||
some unknown parameters
|
||||
\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
|
||||
\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
|
||||
</p>
|
||||
|
||||
<p>Last week we introduced the so-called design matrix in order to define
|
||||
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
|
||||
\( \boldsymbol{\beta} \) as
|
||||
\( \boldsymbol{\theta} \) as
|
||||
</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and in order to find the optimal parameters \( \beta_i \) we defined a function which
|
||||
<p>and in order to find the optimal parameters \( \theta_i \) we defined a function which
|
||||
gives a measure of the spread between the values \( y_i \) (which
|
||||
represent the output values we want to reproduce) and the parametrized
|
||||
values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
@@ -296,14 +295,14 @@ values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
<p>We used the mean squared error to define the way we measure the quality of our model</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -315,11 +314,11 @@ the function \( C \) as
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. </p>
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out. </p>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
@@ -328,7 +327,7 @@ $$
|
||||
<p>The function </p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -337,7 +336,7 @@ When linking (see the discussions next week) with the maximum likelihood approac
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -351,32 +350,32 @@ the standard deviation discussed earlier. In the discussion here we
|
||||
will treat \( y_i \) as our exact value for the response variable.
|
||||
</p>
|
||||
|
||||
<p>In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem</p>
|
||||
<p>In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem</p>
|
||||
<p> <br>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>In practical terms it means we will require</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>which results in</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
<p> <br>
|
||||
</section>
|
||||
@@ -389,21 +388,21 @@ $$
|
||||
<p>We can rewrite, see the derivations below, </p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>as</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -652,60 +651,60 @@ $$
|
||||
<p>We defined earlier a possible cost function using the mean squared error</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as</p>
|
||||
<p>which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as</p>
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>with partial derivative</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and using that</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>where we used the result from example two above. Inserting the last expression we obtain</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>or as</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
<p> <br>
|
||||
</section>
|
||||
@@ -744,14 +743,14 @@ $$
|
||||
|
||||
<p>A very important matrix we will meet again and again in machine
|
||||
learning is the Hessian. It is given by the second derivative of the
|
||||
cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
|
||||
cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
|
||||
expression for derivatives of vectors and matrices, we find that the
|
||||
second derivative of the mean squared error as cost function is,
|
||||
</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -765,7 +764,7 @@ $$
|
||||
|
||||
<p>For ordinary least squares, it is inversely proportional (derivation
|
||||
next week) with the variance of the optimal parameters
|
||||
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
|
||||
\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
|
||||
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
|
||||
important role in optmization algorithms and Principal Component
|
||||
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
|
||||
@@ -784,25 +783,25 @@ v
|
||||
<p>The residuals \( \boldsymbol{\epsilon} \) are in turn given by</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and with </p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>we have</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. </p>
|
||||
<p>meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals. </p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
@@ -815,11 +814,11 @@ We assume our data can represented by a fourth-order polynomial. For the $i$th c
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
|
||||
\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
<p>we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
\( n\times p \) matrix \( \boldsymbol{X} \).
|
||||
</p>
|
||||
</section>
|
||||
@@ -827,7 +826,7 @@ $$
|
||||
<section>
|
||||
<h2 id="own-code-for-ordinary-least-squares">Own code for Ordinary Least Squares </h2>
|
||||
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="cell border-box-sizing code_cell rendered">
|
||||
@@ -1426,21 +1425,21 @@ plt.show()
|
||||
|
||||
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). </p>
|
||||
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by</p>
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
|
||||
<p>This means that our best model is defined as</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -1454,7 +1453,7 @@ $$
|
||||
<p>We can rewrite</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -1610,17 +1609,17 @@ This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least a
|
||||
<p> <br>
|
||||
$$
|
||||
\begin{align}
|
||||
\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\tag{1}
|
||||
\end{align}
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
|
||||
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
|
||||
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters \( \beta_i \) cannot be estimated.
|
||||
the regression parameters \( \theta_i \) cannot be estimated.
|
||||
</p>
|
||||
|
||||
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
|
||||
@@ -2080,7 +2079,7 @@ function, that is we have
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2691,15 +2690,15 @@ our optimization problem is
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>or we can state it as</p>
|
||||
<p> <br>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2711,34 +2710,34 @@ $$
|
||||
<p> <br>
|
||||
|
||||
<p>By minimizing the above equation with respect to the parameters
|
||||
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
|
||||
\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
|
||||
defining a new cost function to be optimized, that is
|
||||
</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>which leads to the Ridge regression minimization problem where we
|
||||
require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
a finite number larger than zero. By defining
|
||||
</p>
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>we have a new optimization equation</p>
|
||||
<p> <br>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2759,19 +2758,19 @@ $$
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and
|
||||
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
|
||||
taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of \( \lambda \) does not suffer from singularity problems. We obtain
|
||||
the optimal parameters
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2779,16 +2778,16 @@ $$
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>with \( t \) a finite positive number. </p>
|
||||
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to</p>
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2797,7 +2796,7 @@ $$
|
||||
<p>When we compare this with the ordinary least squares result we have</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2807,7 +2806,7 @@ $$
|
||||
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
|
||||
particular for our discussion of the bias-variance tradeoff are rather
|
||||
interesting. We will see that for specific values of \( \lambda \), we may
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
</p>
|
||||
|
||||
<p>Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
|
||||
@@ -2815,7 +2814,7 @@ We have already analyzed the OLS solutions in terms of the eigenvectors (the col
|
||||
</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2823,7 +2822,7 @@ $$
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2865,7 +2864,7 @@ $$
|
||||
<p>In this case the standard OLS results in </p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2873,7 +2872,7 @@ $$
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
|
||||
\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2896,14 +2895,14 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
|
||||
\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2911,21 +2910,21 @@ $$
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>and reordering we have</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as</p>
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
|
||||
@@ -303,13 +303,12 @@ MathJax.Hub.Config({
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="reminder-from-last-week">Reminder from last week </h2>
|
||||
|
||||
<p>We need first a reminder from last week about linear regression. </p>
|
||||
<p>We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.</p>
|
||||
|
||||
<p>Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).</p>
|
||||
<ul>
|
||||
<li> Method of choice for fitting a continuous function!</li>
|
||||
<li> It is the method of choice for fitting a continuous function</li>
|
||||
<li> Gives an excellent introduction to central Machine Learning features with <b>understandable pedagogical</b> links to other methods like <b>Neural Networks</b>, <b>Support Vector Machines</b> etc</li>
|
||||
<li> Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)</li>
|
||||
<li> Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)</li>
|
||||
<li> Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</li>
|
||||
<li> Analytical relation with probabilistic interpretations</li>
|
||||
<li> Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</li>
|
||||
@@ -347,19 +346,19 @@ value and a variance \( \sigma^2 \).
|
||||
<p>In linear regression we approximate the unknown function with another
|
||||
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
|
||||
some unknown parameters
|
||||
\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
|
||||
\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
|
||||
</p>
|
||||
|
||||
<p>Last week we introduced the so-called design matrix in order to define
|
||||
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
|
||||
\( \boldsymbol{\beta} \) as
|
||||
\( \boldsymbol{\theta} \) as
|
||||
</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and in order to find the optimal parameters \( \beta_i \) we defined a function which
|
||||
<p>and in order to find the optimal parameters \( \theta_i \) we defined a function which
|
||||
gives a measure of the spread between the values \( y_i \) (which
|
||||
represent the output values we want to reproduce) and the parametrized
|
||||
values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
@@ -370,12 +369,12 @@ values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
|
||||
<p>We used the mean squared error to define the way we measure the quality of our model</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>This function represents one of many possible ways to define the so-called cost function.</p>
|
||||
@@ -385,24 +384,24 @@ the function \( C \) as
|
||||
</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
$$
|
||||
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. </p>
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out. </p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="interpretations-and-optimizing-our-parameters">Interpretations and optimizing our parameters </h2>
|
||||
|
||||
<p>The function </p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value.
|
||||
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
|
||||
</p>
|
||||
$$
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
$$
|
||||
|
||||
<p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
|
||||
@@ -415,25 +414,25 @@ the standard deviation discussed earlier. In the discussion here we
|
||||
will treat \( y_i \) as our exact value for the response variable.
|
||||
</p>
|
||||
|
||||
<p>In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem</p>
|
||||
<p>In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>In practical terms it means we will require</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
$$
|
||||
|
||||
<p>which results in</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
$$
|
||||
|
||||
<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
|
||||
|
||||
@@ -444,17 +443,17 @@ $$
|
||||
<p>
|
||||
<p>We can rewrite, see the derivations below, </p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
|
||||
$$
|
||||
|
||||
<p>as</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution</p>
|
||||
$$
|
||||
\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
|
||||
@@ -662,46 +661,46 @@ $$
|
||||
|
||||
<p>We defined earlier a possible cost function using the mean squared error</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
|
||||
$$
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as</p>
|
||||
<p>which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
$$
|
||||
|
||||
<p>with partial derivative</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
|
||||
$$
|
||||
|
||||
<p>and using that</p>
|
||||
$$
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
|
||||
$$
|
||||
|
||||
<p>where we used the result from example two above. Inserting the last expression we obtain</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
|
||||
$$
|
||||
|
||||
<p>or as</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
|
||||
|
||||
@@ -732,13 +731,13 @@ $$
|
||||
|
||||
<p>A very important matrix we will meet again and again in machine
|
||||
learning is the Hessian. It is given by the second derivative of the
|
||||
cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
|
||||
cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
|
||||
expression for derivatives of vectors and matrices, we find that the
|
||||
second derivative of the mean squared error as cost function is,
|
||||
</p>
|
||||
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
<p>The Hessian matrix plays an important role and is defined here as</p>
|
||||
@@ -749,7 +748,7 @@ $$
|
||||
|
||||
<p>For ordinary least squares, it is inversely proportional (derivation
|
||||
next week) with the variance of the optimal parameters
|
||||
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
|
||||
\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
|
||||
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
|
||||
important role in optmization algorithms and Principal Component
|
||||
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
|
||||
@@ -766,20 +765,20 @@ v
|
||||
<p>
|
||||
<p>The residuals \( \boldsymbol{\epsilon} \) are in turn given by</p>
|
||||
$$
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and with </p>
|
||||
$$
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
|
||||
<p>we have</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
|
||||
<p>meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. </p>
|
||||
<p>meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals. </p>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -791,17 +790,17 @@ we condiser a simple polynomial fit.
|
||||
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
|
||||
</p>
|
||||
$$
|
||||
\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
|
||||
\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
|
||||
$$
|
||||
|
||||
<p>we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
<p>we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
\( n\times p \) matrix \( \boldsymbol{X} \).
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="own-code-for-ordinary-least-squares">Own code for Ordinary Least Squares </h2>
|
||||
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
|
||||
<div class="cell border-box-sizing code_cell rendered">
|
||||
@@ -1391,18 +1390,18 @@ plt.show()
|
||||
|
||||
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). </p>
|
||||
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by</p>
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by</p>
|
||||
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
|
||||
<p>This means that our best model is defined as</p>
|
||||
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We now define a matrix</p>
|
||||
@@ -1412,7 +1411,7 @@ $$
|
||||
|
||||
<p>We can rewrite</p>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
|
||||
@@ -1550,16 +1549,16 @@ This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least a
|
||||
<p>If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem</p>
|
||||
$$
|
||||
\begin{align}
|
||||
\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\label{_auto1}
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
<p>has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
|
||||
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
|
||||
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters \( \beta_i \) cannot be estimated.
|
||||
the regression parameters \( \theta_i \) cannot be estimated.
|
||||
</p>
|
||||
|
||||
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
|
||||
@@ -1956,7 +1955,7 @@ function, that is we have
|
||||
</p>
|
||||
|
||||
$$
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
<p>This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).</p>
|
||||
@@ -2498,13 +2497,13 @@ values and the column vectors of \( \boldsymbol{V} \).
|
||||
our optimization problem is
|
||||
</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>or we can state it as</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
|
||||
$$
|
||||
|
||||
<p>where we have used the definition of a norm-2 vector, that is</p>
|
||||
@@ -2513,29 +2512,29 @@ $$
|
||||
$$
|
||||
|
||||
<p>By minimizing the above equation with respect to the parameters
|
||||
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
|
||||
\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
|
||||
defining a new cost function to be optimized, that is
|
||||
</p>
|
||||
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
|
||||
$$
|
||||
|
||||
<p>which leads to the Ridge regression minimization problem where we
|
||||
require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
a finite number larger than zero. By defining
|
||||
</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
|
||||
<p>we have a new optimization equation</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
|
||||
$$
|
||||
|
||||
<p>which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. </p>
|
||||
@@ -2552,37 +2551,37 @@ $$
|
||||
<p>Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and
|
||||
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
|
||||
taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of \( \lambda \) does not suffer from singularity problems. We obtain
|
||||
the optimal parameters
|
||||
</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that</p>
|
||||
|
||||
$$
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
|
||||
$$
|
||||
|
||||
<p>with \( t \) a finite positive number. </p>
|
||||
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to</p>
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>In many textbooks the \( 1/n \) term is often omitted. Note that a library like <b>Scikit-Learn</b> does not include the \( 1/n \) factor in the setup of the cost function.</p>
|
||||
|
||||
<p>When we compare this with the ordinary least squares result we have</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).</p>
|
||||
@@ -2591,20 +2590,20 @@ $$
|
||||
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
|
||||
particular for our discussion of the bias-variance tradeoff are rather
|
||||
interesting. We will see that for specific values of \( \lambda \), we may
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
</p>
|
||||
|
||||
<p>Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
|
||||
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
|
||||
</p>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>For Ridge regression this becomes</p>
|
||||
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). </p>
|
||||
@@ -2638,13 +2637,13 @@ $$
|
||||
|
||||
<p>In this case the standard OLS results in </p>
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>and</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
|
||||
\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
|
||||
$$
|
||||
|
||||
<p>that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
|
||||
@@ -2664,28 +2663,28 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
|
||||
<p>Using the matrix-vector expression for Lasso regression, we have the following <b>cost</b> function</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
$$
|
||||
\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
|
||||
\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
|
||||
$$
|
||||
|
||||
<p>we have that the derivative of the cost function is</p>
|
||||
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
|
||||
$$
|
||||
|
||||
<p>and reordering we have</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later. </p>
|
||||
|
||||
@@ -380,13 +380,12 @@ MathJax.Hub.Config({
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="reminder-from-last-week">Reminder from last week </h2>
|
||||
|
||||
<p>We need first a reminder from last week about linear regression. </p>
|
||||
<p>We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.</p>
|
||||
|
||||
<p>Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).</p>
|
||||
<ul>
|
||||
<li> Method of choice for fitting a continuous function!</li>
|
||||
<li> It is the method of choice for fitting a continuous function</li>
|
||||
<li> Gives an excellent introduction to central Machine Learning features with <b>understandable pedagogical</b> links to other methods like <b>Neural Networks</b>, <b>Support Vector Machines</b> etc</li>
|
||||
<li> Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)</li>
|
||||
<li> Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)</li>
|
||||
<li> Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more</li>
|
||||
<li> Analytical relation with probabilistic interpretations</li>
|
||||
<li> Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics</li>
|
||||
@@ -424,19 +423,19 @@ value and a variance \( \sigma^2 \).
|
||||
<p>In linear regression we approximate the unknown function with another
|
||||
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
|
||||
some unknown parameters
|
||||
\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
|
||||
\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
|
||||
</p>
|
||||
|
||||
<p>Last week we introduced the so-called design matrix in order to define
|
||||
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
|
||||
\( \boldsymbol{\beta} \) as
|
||||
\( \boldsymbol{\theta} \) as
|
||||
</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and in order to find the optimal parameters \( \beta_i \) we defined a function which
|
||||
<p>and in order to find the optimal parameters \( \theta_i \) we defined a function which
|
||||
gives a measure of the spread between the values \( y_i \) (which
|
||||
represent the output values we want to reproduce) and the parametrized
|
||||
values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
@@ -447,12 +446,12 @@ values \( \tilde{y}_i \), namely the so-called cost/loss function.
|
||||
|
||||
<p>We used the mean squared error to define the way we measure the quality of our model</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>This function represents one of many possible ways to define the so-called cost function.</p>
|
||||
@@ -462,24 +461,24 @@ the function \( C \) as
|
||||
</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
$$
|
||||
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. </p>
|
||||
<p>since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out. </p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="interpretations-and-optimizing-our-parameters">Interpretations and optimizing our parameters </h2>
|
||||
|
||||
<p>The function </p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value.
|
||||
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
|
||||
</p>
|
||||
$$
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
$$
|
||||
|
||||
<p>where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
|
||||
@@ -492,25 +491,25 @@ the standard deviation discussed earlier. In the discussion here we
|
||||
will treat \( y_i \) as our exact value for the response variable.
|
||||
</p>
|
||||
|
||||
<p>In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem</p>
|
||||
<p>In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>In practical terms it means we will require</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
$$
|
||||
|
||||
<p>which results in</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
$$
|
||||
|
||||
<p>or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
|
||||
|
||||
@@ -521,17 +520,17 @@ $$
|
||||
<p>
|
||||
<p>We can rewrite, see the derivations below, </p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
|
||||
$$
|
||||
|
||||
<p>as</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution</p>
|
||||
$$
|
||||
\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We note also that since our design matrix is defined as \( \boldsymbol{X}\in
|
||||
@@ -739,46 +738,46 @@ $$
|
||||
|
||||
<p>We defined earlier a possible cost function using the mean squared error</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
|
||||
$$
|
||||
|
||||
<p>or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
|
||||
<p>We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
|
||||
</p>
|
||||
|
||||
<p>The mean squared error is a scalar and if we use the results from example three above, we can define a new vector</p>
|
||||
$$
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as</p>
|
||||
<p>which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as</p>
|
||||
$$
|
||||
C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
|
||||
$$
|
||||
|
||||
<p>with partial derivative</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
|
||||
$$
|
||||
|
||||
<p>and using that</p>
|
||||
$$
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
|
||||
\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
|
||||
$$
|
||||
|
||||
<p>where we used the result from example two above. Inserting the last expression we obtain</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
|
||||
$$
|
||||
|
||||
<p>or as</p>
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
|
||||
\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
|
||||
$$
|
||||
|
||||
|
||||
@@ -809,13 +808,13 @@ $$
|
||||
|
||||
<p>A very important matrix we will meet again and again in machine
|
||||
learning is the Hessian. It is given by the second derivative of the
|
||||
cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
|
||||
cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
|
||||
expression for derivatives of vectors and matrices, we find that the
|
||||
second derivative of the mean squared error as cost function is,
|
||||
</p>
|
||||
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
<p>The Hessian matrix plays an important role and is defined here as</p>
|
||||
@@ -826,7 +825,7 @@ $$
|
||||
|
||||
<p>For ordinary least squares, it is inversely proportional (derivation
|
||||
next week) with the variance of the optimal parameters
|
||||
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
|
||||
\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
|
||||
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
|
||||
important role in optmization algorithms and Principal Component
|
||||
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
|
||||
@@ -843,20 +842,20 @@ v
|
||||
<p>
|
||||
<p>The residuals \( \boldsymbol{\epsilon} \) are in turn given by</p>
|
||||
$$
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
|
||||
\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and with </p>
|
||||
$$
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
|
||||
<p>we have</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
|
||||
\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
|
||||
$$
|
||||
|
||||
<p>meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. </p>
|
||||
<p>meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals. </p>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -868,17 +867,17 @@ we condiser a simple polynomial fit.
|
||||
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
|
||||
</p>
|
||||
$$
|
||||
\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
|
||||
\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
|
||||
$$
|
||||
|
||||
<p>we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
<p>we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
|
||||
\( n\times p \) matrix \( \boldsymbol{X} \).
|
||||
</p>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
<h2 id="own-code-for-ordinary-least-squares">Own code for Ordinary Least Squares </h2>
|
||||
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
<p>It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have </p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="cell border-box-sizing code_cell rendered">
|
||||
@@ -1468,18 +1467,18 @@ plt<span style="color: #666666">.</span>show()
|
||||
|
||||
<p>What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). </p>
|
||||
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by</p>
|
||||
<p>We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by</p>
|
||||
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
<p>The <b>hat</b> over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.</p>
|
||||
|
||||
<p>This means that our best model is defined as</p>
|
||||
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We now define a matrix</p>
|
||||
@@ -1489,7 +1488,7 @@ $$
|
||||
|
||||
<p>We can rewrite</p>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
|
||||
@@ -1627,16 +1626,16 @@ This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least a
|
||||
<p>If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem</p>
|
||||
$$
|
||||
\begin{align}
|
||||
\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
|
||||
\label{_auto1}
|
||||
\end{align}
|
||||
$$
|
||||
|
||||
<p>has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
|
||||
of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
|
||||
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
|
||||
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters \( \beta_i \) cannot be estimated.
|
||||
the regression parameters \( \theta_i \) cannot be estimated.
|
||||
</p>
|
||||
|
||||
<p>A cheap <em>ad hoc</em> approach is simply to add a small diagonal component to the matrix to invert, that is we change</p>
|
||||
@@ -2033,7 +2032,7 @@ function, that is we have
|
||||
</p>
|
||||
|
||||
$$
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
<p>This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).</p>
|
||||
@@ -2575,13 +2574,13 @@ values and the column vectors of \( \boldsymbol{V} \).
|
||||
our optimization problem is
|
||||
</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
|
||||
$$
|
||||
|
||||
<p>or we can state it as</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
|
||||
$$
|
||||
|
||||
<p>where we have used the definition of a norm-2 vector, that is</p>
|
||||
@@ -2590,29 +2589,29 @@ $$
|
||||
$$
|
||||
|
||||
<p>By minimizing the above equation with respect to the parameters
|
||||
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
|
||||
\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
|
||||
parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
|
||||
defining a new cost function to be optimized, that is
|
||||
</p>
|
||||
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
|
||||
$$
|
||||
|
||||
<p>which leads to the Ridge regression minimization problem where we
|
||||
require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
|
||||
a finite number larger than zero. By defining
|
||||
</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
|
||||
<p>we have a new optimization equation</p>
|
||||
$$
|
||||
{\displaystyle \min_{\boldsymbol{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
|
||||
{\displaystyle \min_{\boldsymbol{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
|
||||
$$
|
||||
|
||||
<p>which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. </p>
|
||||
@@ -2629,37 +2628,37 @@ $$
|
||||
<p>Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
|
||||
$$
|
||||
|
||||
<p>and
|
||||
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
|
||||
taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of \( \lambda \) does not suffer from singularity problems. We obtain
|
||||
the optimal parameters
|
||||
</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that</p>
|
||||
|
||||
$$
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
|
||||
$$
|
||||
|
||||
<p>with \( t \) a finite positive number. </p>
|
||||
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to</p>
|
||||
<p>If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>In many textbooks the \( 1/n \) term is often omitted. Note that a library like <b>Scikit-Learn</b> does not include the \( 1/n \) factor in the setup of the cost function.</p>
|
||||
|
||||
<p>When we compare this with the ordinary least squares result we have</p>
|
||||
$$
|
||||
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).</p>
|
||||
@@ -2668,20 +2667,20 @@ $$
|
||||
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
|
||||
particular for our discussion of the bias-variance tradeoff are rather
|
||||
interesting. We will see that for specific values of \( \lambda \), we may
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
</p>
|
||||
|
||||
<p>Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
|
||||
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
|
||||
</p>
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>For Ridge regression this becomes</p>
|
||||
|
||||
$$
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). </p>
|
||||
@@ -2715,13 +2714,13 @@ $$
|
||||
|
||||
<p>In this case the standard OLS results in </p>
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
|
||||
$$
|
||||
|
||||
<p>and</p>
|
||||
|
||||
$$
|
||||
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
|
||||
\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
|
||||
$$
|
||||
|
||||
<p>that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
|
||||
@@ -2741,28 +2740,28 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
|
||||
<p>Using the matrix-vector expression for Lasso regression, we have the following <b>cost</b> function</p>
|
||||
|
||||
$$
|
||||
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
|
||||
C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
|
||||
$$
|
||||
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
<p>Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)</p>
|
||||
$$
|
||||
\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
|
||||
\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
|
||||
$$
|
||||
|
||||
<p>we have that the derivative of the cost function is</p>
|
||||
|
||||
$$
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
|
||||
\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
|
||||
$$
|
||||
|
||||
<p>and reordering we have</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as</p>
|
||||
$$
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later. </p>
|
||||
|
||||
Binary file not shown.
+445
-446
File diff suppressed because it is too large
Load Diff
@@ -30,12 +30,11 @@ o For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Mar
|
||||
!split
|
||||
===== Reminder from last week =====
|
||||
|
||||
We need first a reminder from last week about linear regression.
|
||||
We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters $\bm{\theta}$ and our first encounter is ordinary least squares.
|
||||
|
||||
Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$.
|
||||
* Method of choice for fitting a continuous function!
|
||||
* It is the method of choice for fitting a continuous function
|
||||
* Gives an excellent introduction to central Machine Learning features with _understandable pedagogical_ links to other methods like _Neural Networks_, _Support Vector Machines_ etc
|
||||
* Analytical expression for the fitting parameters $\bm{\beta}$
|
||||
* Analytical expression for the fitting parameters $\bm{\theta}$
|
||||
* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
|
||||
* Analytical relation with probabilistic interpretations
|
||||
* Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
|
||||
@@ -76,20 +75,20 @@ value and a variance $\sigma^2$.
|
||||
In linear regression we approximate the unknown function with another
|
||||
continuous function $\tilde{\bm{y}}(\bm{x})$ which depends linearly on
|
||||
some unknown parameters
|
||||
$\bm{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}]$.
|
||||
$\bm{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}]$.
|
||||
|
||||
Last week we introduced the so-called design matrix in order to define
|
||||
the approximation $\bm{\tilde{y}}$ via the unknown quantity
|
||||
$\bm{\beta}$ as
|
||||
$\bm{\theta}$ as
|
||||
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{\tilde{y}}= \bm{X}\bm{\beta},
|
||||
\bm{\tilde{y}}= \bm{X}\bm{\theta},
|
||||
\]
|
||||
!et
|
||||
|
||||
and in order to find the optimal parameters $\beta_i$ we defined a function which
|
||||
and in order to find the optimal parameters $\theta_i$ we defined a function which
|
||||
gives a measure of the spread between the values $y_i$ (which
|
||||
represent the output values we want to reproduce) and the parametrized
|
||||
values $\tilde{y}_i$, namely the so-called cost/loss function.
|
||||
@@ -101,13 +100,13 @@ values $\tilde{y}_i$, namely the so-called cost/loss function.
|
||||
We used the mean squared error to define the way we measure the quality of our model
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
|
||||
C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
|
||||
\]
|
||||
!et
|
||||
or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
|
||||
C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
|
||||
\]
|
||||
!et
|
||||
This function represents one of many possible ways to define the so-called cost function.
|
||||
@@ -119,10 +118,10 @@ the function $C$ as
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
C(\bm{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
|
||||
\]
|
||||
!et
|
||||
since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out.
|
||||
since when taking the first derivative with respect to the unknown parameters $\theta$, the factor of $2$ cancels out.
|
||||
|
||||
|
||||
|
||||
@@ -133,14 +132,14 @@ since when taking the first derivative with respect to the unknown parameters $\
|
||||
The function
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\},
|
||||
C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\},
|
||||
\]
|
||||
!et
|
||||
can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value.
|
||||
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value
|
||||
!bt
|
||||
\[
|
||||
y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -153,29 +152,29 @@ error estimate, often limited to a statistical error estimate given by
|
||||
the standard deviation discussed earlier. In the discussion here we
|
||||
will treat $y_i$ as our exact value for the response variable.
|
||||
|
||||
In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem
|
||||
In order to find the parameters $\theta_i$ we will then minimize the spread of $C(\bm{\theta})$, that is we are going to solve the problem
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\bm{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
|
||||
\]
|
||||
!et
|
||||
In practical terms it means we will require
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
\frac{\partial C(\bm{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
|
||||
\]
|
||||
!et
|
||||
which results in
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
\frac{\partial C(\bm{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
|
||||
\]
|
||||
!et
|
||||
or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivation below)
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right).
|
||||
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right).
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -186,19 +185,19 @@ or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivatio
|
||||
We can rewrite, see the derivations below,
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right),
|
||||
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right),
|
||||
\]
|
||||
!et
|
||||
as
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta},
|
||||
\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\theta},
|
||||
\]
|
||||
!et
|
||||
and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\bm{\theta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -420,53 +419,53 @@ and if $\bm{y}=\bm{x}$ we have
|
||||
We defined earlier a possible cost function using the mean squared error
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
|
||||
C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
|
||||
\]
|
||||
!et
|
||||
or using the design/feature matrix $\bm{X}$ we have the more compact matrix-vector
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
|
||||
C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
|
||||
\]
|
||||
!et
|
||||
|
||||
We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\beta}$.
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\beta}$.
|
||||
We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\theta}$.
|
||||
We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\theta}$.
|
||||
|
||||
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
|
||||
!bt
|
||||
\[
|
||||
\bm{w}=\bm{y}-\bm{X}\bm{\beta},
|
||||
\bm{w}=\bm{y}-\bm{X}\bm{\theta},
|
||||
\]
|
||||
!et
|
||||
which depends on $\bm{\beta}$. We rewrite the cost function as
|
||||
which depends on $\bm{\theta}$. We rewrite the cost function as
|
||||
!bt
|
||||
\[
|
||||
C(\bm{\beta})=\frac{1}{n}\bm{w}^T\bm{w},
|
||||
C(\bm{\theta})=\frac{1}{n}\bm{w}^T\bm{w},
|
||||
\]
|
||||
!et
|
||||
with partial derivative
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}}=\frac{2}{n}\bm{w}^T\frac{\partial \bm{w}}{\partial \bm{\beta}},
|
||||
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}}=\frac{2}{n}\bm{w}^T\frac{\partial \bm{w}}{\partial \bm{\theta}},
|
||||
\]
|
||||
!et
|
||||
and using that
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial \bm{w}}{\partial \bm{\beta}}=-\bm{X},
|
||||
\frac{\partial \bm{w}}{\partial \bm{\theta}}=-\bm{X},
|
||||
\]
|
||||
!et
|
||||
where we used the result from example two above. Inserting the last expression we obtain
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\bm{X},
|
||||
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}}=-\frac{2}{n}\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\bm{X},
|
||||
\]
|
||||
!et
|
||||
or as
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T}=-\frac{2}{n}\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right).
|
||||
\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T}=-\frac{2}{n}\bm{X}^T\left(\bm{y}-\bm{X}\bm{\theta}\right).
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -505,13 +504,13 @@ We list here some other useful relations we may encounter (recall that vectors a
|
||||
|
||||
A very important matrix we will meet again and again in machine
|
||||
learning is the Hessian. It is given by the second derivative of the
|
||||
cost function with respect to the parameters $\bm{\beta}$. Using the above
|
||||
cost function with respect to the parameters $\bm{\theta}$. Using the above
|
||||
expression for derivatives of vectors and matrices, we find that the
|
||||
second derivative of the mean squared error as cost function is,
|
||||
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial}{\partial \bm{\beta}}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} =\frac{\partial}{\partial \bm{\beta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
|
||||
\frac{\partial}{\partial \bm{\theta}}\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} =\frac{\partial}{\partial \bm{\theta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -527,7 +526,7 @@ The Hessian matrix plays an important role and is defined here as
|
||||
|
||||
For ordinary least squares, it is inversely proportional (derivation
|
||||
next week) with the variance of the optimal parameters
|
||||
$\hat{\bm{\beta}}$. Furthermore, we will see later this week that it is
|
||||
$\hat{\bm{\theta}}$. Furthermore, we will see later this week that it is
|
||||
(aside the factor $1/n$) equal to the covariance matrix. It plays also a very
|
||||
important role in optmization algorithms and Principal Component
|
||||
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
|
||||
@@ -542,22 +541,22 @@ _Linear algebra question:_ Can we use the Hessian matrix to say something about
|
||||
The residuals $\bm{\epsilon}$ are in turn given by
|
||||
!bt
|
||||
\[
|
||||
\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta},
|
||||
\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\theta},
|
||||
\]
|
||||
!et
|
||||
and with
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0,
|
||||
\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0,
|
||||
\]
|
||||
!et
|
||||
we have
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0,
|
||||
\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0,
|
||||
\]
|
||||
!et
|
||||
meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals.
|
||||
meaning that the solution for $\bm{\theta}$ is the one which minimizes the residuals.
|
||||
|
||||
!eblock
|
||||
|
||||
@@ -571,17 +570,17 @@ we condiser a simple polynomial fit.
|
||||
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
|
||||
!bt
|
||||
\[
|
||||
\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
|
||||
\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
|
||||
\]
|
||||
!et
|
||||
we have five predictors/features. The first is the intercept $\beta_0$. The other terms are $\beta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an
|
||||
we have five predictors/features. The first is the intercept $\theta_0$. The other terms are $\theta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an
|
||||
$n\times p$ matrix $\bm{X}$.
|
||||
|
||||
|
||||
!split
|
||||
===== Own code for Ordinary Least Squares =====
|
||||
|
||||
It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
|
||||
It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\theta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
|
||||
!bc pycod
|
||||
# matrix inversion to find beta
|
||||
# First we set up the data
|
||||
@@ -952,21 +951,21 @@ plt.show()
|
||||
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
|
||||
|
||||
|
||||
We have shown that in ordinary least squares the optimal parameters $\beta$ are given by
|
||||
We have shown that in ordinary least squares the optimal parameters $\theta$ are given by
|
||||
|
||||
!bt
|
||||
\[
|
||||
\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\hat{\bm{\theta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
The _hat_ over $\bm{\beta}$ means we have the optimal parameters after minimization of the cost function.
|
||||
The _hat_ over $\bm{\theta}$ means we have the optimal parameters after minimization of the cost function.
|
||||
|
||||
This means that our best model is defined as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -980,7 +979,7 @@ We now define a matrix
|
||||
We can rewrite
|
||||
!bt
|
||||
\[
|
||||
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}.
|
||||
\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{A}\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -1125,14 +1124,14 @@ This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue
|
||||
If our design matrix $\bm{X}$ which enters the linear regression problem
|
||||
!bt
|
||||
\begin{align}
|
||||
\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y},
|
||||
\bm{\theta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y},
|
||||
\end{align}
|
||||
!et
|
||||
has linearly dependent column vectors, we will not be able to compute the inverse
|
||||
of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$.
|
||||
of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\theta_i$.
|
||||
The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits.
|
||||
This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where
|
||||
the regression parameters $\beta_i$ cannot be estimated.
|
||||
the regression parameters $\theta_i$ cannot be estimated.
|
||||
|
||||
A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change
|
||||
!bt
|
||||
@@ -1541,7 +1540,7 @@ function, that is we have
|
||||
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} =\frac{2}{n}\bm{X}^T\bm{X}.
|
||||
\frac{\partial^2 C(\bm{\theta})}{\partial \bm{\theta}\partial \bm{\theta}^T} =\frac{2}{n}\bm{X}^T\bm{X}.
|
||||
\]
|
||||
!et
|
||||
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
|
||||
@@ -2034,14 +2033,14 @@ Let us remind ourselves about the expression for the standard Mean Squared Error
|
||||
our optimization problem is
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
|
||||
{\displaystyle \min_{\bm{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
|
||||
\]
|
||||
!et
|
||||
or we can state it as
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2,
|
||||
{\displaystyle \min_{\bm{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2,
|
||||
\]
|
||||
!et
|
||||
where we have used the definition of a norm-2 vector, that is
|
||||
@@ -2052,32 +2051,32 @@ where we have used the definition of a norm-2 vector, that is
|
||||
!et
|
||||
|
||||
By minimizing the above equation with respect to the parameters
|
||||
$\bm{\beta}$ we could then obtain an analytical expression for the
|
||||
parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
|
||||
$\bm{\theta}$ we could then obtain an analytical expression for the
|
||||
parameters $\bm{\theta}$. We can add a regularization parameter $\lambda$ by
|
||||
defining a new cost function to be optimized, that is
|
||||
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2
|
||||
{\displaystyle \min_{\bm{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_2^2
|
||||
\]
|
||||
!et
|
||||
|
||||
which leads to the Ridge regression minimization problem where we
|
||||
require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is
|
||||
require that $\vert\vert \bm{\theta}\vert\vert_2^2\le t$, where $t$ is
|
||||
a finite number larger than zero. By defining
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1,
|
||||
C(\bm{X},\bm{\theta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1,
|
||||
\]
|
||||
!et
|
||||
|
||||
we have a new optimization equation
|
||||
!bt
|
||||
\[
|
||||
{\displaystyle \min_{\bm{\beta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1
|
||||
{\displaystyle \min_{\bm{\theta}\in
|
||||
{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1
|
||||
\]
|
||||
!et
|
||||
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
|
||||
@@ -2097,17 +2096,17 @@ Using the matrix-vector expression for Ridge regression and dropping the paramet
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
|
||||
C(\bm{X},\bm{\theta})=\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\bm{\theta}^T\bm{\theta},
|
||||
\]
|
||||
!et
|
||||
and
|
||||
taking the derivatives with respect to $\bm{\beta}$ we obtain then
|
||||
taking the derivatives with respect to $\bm{\theta}$ we obtain then
|
||||
a slightly modified matrix inversion problem which for finite values
|
||||
of $\lambda$ does not suffer from singularity problems. We obtain
|
||||
the optimal parameters
|
||||
!bt
|
||||
\[
|
||||
\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
|
||||
\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -2115,16 +2114,16 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that
|
||||
|
||||
!bt
|
||||
\[
|
||||
\sum_{i=0}^{p-1} \beta_i^2 \leq t,
|
||||
\sum_{i=0}^{p-1} \theta_i^2 \leq t,
|
||||
\]
|
||||
!et
|
||||
|
||||
with $t$ a finite positive number.
|
||||
|
||||
If we keep the $1/n$ factor, the equation for the optimal $\beta$ changes to
|
||||
If we keep the $1/n$ factor, the equation for the optimal $\theta$ changes to
|
||||
!bt
|
||||
\[
|
||||
\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -2133,7 +2132,7 @@ In many textbooks the $1/n$ term is often omitted. Note that a library like _Sci
|
||||
When we compare this with the ordinary least squares result we have
|
||||
!bt
|
||||
\[
|
||||
\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
|
||||
\hat{\bm{\theta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\bm{X}^T\bm{X}$.
|
||||
@@ -2143,13 +2142,13 @@ We see that Ridge regression is nothing but the standard OLS with a
|
||||
modified diagonal term added to $\bm{X}^T\bm{X}$. The consequences, in
|
||||
particular for our discussion of the bias-variance tradeoff are rather
|
||||
interesting. We will see that for specific values of $\lambda$, we may
|
||||
even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
even reduce the variance of the optimal parameters $\bm{\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
|
||||
|
||||
Using our insights about the SVD of the design matrix $\bm{X}$
|
||||
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
|
||||
!bt
|
||||
\[
|
||||
\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} =\bm{U}\bm{U}^T\bm{y}.
|
||||
\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\theta} =\bm{U}\bm{U}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -2158,7 +2157,7 @@ For Ridge regression this becomes
|
||||
|
||||
!bt
|
||||
\[
|
||||
\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\beta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
|
||||
\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\theta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -2198,7 +2197,7 @@ For the sake of simplicity, let us assume that the design matrix is orthonormal,
|
||||
In this case the standard OLS results in
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}_i^T\bm{y},
|
||||
\bm{\theta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}_i^T\bm{y},
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -2206,7 +2205,7 @@ and
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}},
|
||||
\bm{\theta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\theta}^{\mathrm{OLS}},
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -2226,33 +2225,33 @@ Using the matrix-vector expression for Lasso regression, we have the following _
|
||||
|
||||
!bt
|
||||
\[
|
||||
C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
|
||||
C(\bm{X},\bm{\theta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\vert\vert\bm{\theta}\vert\vert_1,
|
||||
\]
|
||||
!et
|
||||
|
||||
Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
|
||||
Taking the derivative with respect to $\bm{\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
|
||||
!bt
|
||||
\[
|
||||
\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
|
||||
\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
|
||||
\]
|
||||
!et
|
||||
we have that the derivative of the cost function is
|
||||
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
|
||||
\frac{\partial C(\bm{X},\bm{\theta})}{\partial \bm{\theta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\theta})+\lambda sgn(\bm{\theta})=0,
|
||||
\]
|
||||
!et
|
||||
and reordering we have
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\bm{X}\bm{\beta}+\frac{n}{2}\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
|
||||
\bm{X}^T\bm{X}\bm{\theta}+\frac{n}{2}\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
We can redefine $\lambda$ to absorb the constant $n/2$ and we rewrite the last equation as
|
||||
!bt
|
||||
\[
|
||||
\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
|
||||
\bm{X}^T\bm{X}\bm{\theta}+\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
|
||||
Reference in New Issue
Block a user