In linear regression we approximate the unknown function with another
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
+\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
Last week we introduced the so-called design matrix in order to define
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
-\( \boldsymbol{\beta} \) as
+\( \boldsymbol{\theta} \) as
$$
-\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
$$
-and in order to find the optimal parameters \( \beta_i \) we defined a function which
+
and in order to find the optimal parameters \( \theta_i \) we defined a function which
gives a measure of the spread between the values \( y_i \) (which
represent the output values we want to reproduce) and the parametrized
values \( \tilde{y}_i \), namely the so-called cost/loss function.
diff --git a/doc/pub/week35/html/._week35-bs004.html b/doc/pub/week35/html/._week35-bs004.html
index 4bb20012a..fa87622ea 100644
--- a/doc/pub/week35/html/._week35-bs004.html
+++ b/doc/pub/week35/html/._week35-bs004.html
@@ -325,12 +325,12 @@ MathJax.Hub.Config({
We used the mean squared error to define the way we measure the quality of our model
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
This function represents one of many possible ways to define the so-called cost function.
@@ -340,10 +340,10 @@ the function \( C \) as
$$
-C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
+C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
$$
-since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.
+since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out.
diff --git a/doc/pub/week35/html/._week35-bs005.html b/doc/pub/week35/html/._week35-bs005.html
index 07ab45faf..a3517a3a3 100644
--- a/doc/pub/week35/html/._week35-bs005.html
+++ b/doc/pub/week35/html/._week35-bs005.html
@@ -325,14 +325,14 @@ MathJax.Hub.Config({
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value.
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
$$
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
@@ -345,25 +345,25 @@ the standard deviation discussed earlier. In the discussion here we
will treat \( y_i \) as our exact value for the response variable.
-In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem
+In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
diff --git a/doc/pub/week35/html/._week35-bs006.html b/doc/pub/week35/html/._week35-bs006.html
index b7801c390..b02176fd6 100644
--- a/doc/pub/week35/html/._week35-bs006.html
+++ b/doc/pub/week35/html/._week35-bs006.html
@@ -327,17 +327,17 @@ MathJax.Hub.Config({
and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution
$$
-\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
We note also that since our design matrix is defined as \( \boldsymbol{X}\in
diff --git a/doc/pub/week35/html/._week35-bs013.html b/doc/pub/week35/html/._week35-bs013.html
index 7023a8f1b..cbc52604b 100644
--- a/doc/pub/week35/html/._week35-bs013.html
+++ b/doc/pub/week35/html/._week35-bs013.html
@@ -325,46 +325,46 @@ MathJax.Hub.Config({
or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
-We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
-We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
+
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
+We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
$$
-\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
-which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as
+which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
+C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
$$
where we used the result from example two above. Inserting the last expression we obtain
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
$$
A very important matrix we will meet again and again in machine
learning is the Hessian. It is given by the second derivative of the
-cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
+cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the mean squared error as cost function is,
$$
-\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
-\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
+\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
diff --git a/doc/pub/week35/html/._week35-bs016.html b/doc/pub/week35/html/._week35-bs016.html
index 8404e4964..e7a6b33d0 100644
--- a/doc/pub/week35/html/._week35-bs016.html
+++ b/doc/pub/week35/html/._week35-bs016.html
@@ -328,20 +328,20 @@ MathJax.Hub.Config({
meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals.
+meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals.
diff --git a/doc/pub/week35/html/._week35-bs017.html b/doc/pub/week35/html/._week35-bs017.html
index dc7390997..3e8bbca7f 100644
--- a/doc/pub/week35/html/._week35-bs017.html
+++ b/doc/pub/week35/html/._week35-bs017.html
@@ -328,10 +328,10 @@ we condiser a simple polynomial fit.
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
$$
-\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
+\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
$$
-we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
+
we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
\( n\times p \) matrix \( \boldsymbol{X} \).
diff --git a/doc/pub/week35/html/._week35-bs018.html b/doc/pub/week35/html/._week35-bs018.html
index 628fe5f2a..2a1386fe4 100644
--- a/doc/pub/week35/html/._week35-bs018.html
+++ b/doc/pub/week35/html/._week35-bs018.html
@@ -323,7 +323,7 @@ MathJax.Hub.Config({
It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
diff --git a/doc/pub/week35/html/._week35-bs031.html b/doc/pub/week35/html/._week35-bs031.html
index e5aadeb88..faa72a631 100644
--- a/doc/pub/week35/html/._week35-bs031.html
+++ b/doc/pub/week35/html/._week35-bs031.html
@@ -325,18 +325,18 @@ MathJax.Hub.Config({
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
-
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
+
We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by
$$
-\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
-
The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
+
The hat over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
We now define a matrix
@@ -346,7 +346,7 @@ $$
We can rewrite
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
$$
The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
diff --git a/doc/pub/week35/html/._week35-bs036.html b/doc/pub/week35/html/._week35-bs036.html
index 5f7291dff..f25cc3ac2 100644
--- a/doc/pub/week35/html/._week35-bs036.html
+++ b/doc/pub/week35/html/._week35-bs036.html
@@ -326,16 +326,16 @@ MathJax.Hub.Config({
If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem
$$
\begin{align}
-\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
+\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\tag{1}
\end{align}
$$
has linearly dependent column vectors, we will not be able to compute the inverse
-of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
-the regression parameters \( \beta_i \) cannot be estimated.
+the regression parameters \( \theta_i \) cannot be estimated.
A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change
diff --git a/doc/pub/week35/html/._week35-bs046.html b/doc/pub/week35/html/._week35-bs046.html
index eb4ff0c8d..342cb0303 100644
--- a/doc/pub/week35/html/._week35-bs046.html
+++ b/doc/pub/week35/html/._week35-bs046.html
@@ -331,7 +331,7 @@ function, that is we have
$$
-\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
diff --git a/doc/pub/week35/html/._week35-bs058.html b/doc/pub/week35/html/._week35-bs058.html
index 1db5e819c..a7342d68f 100644
--- a/doc/pub/week35/html/._week35-bs058.html
+++ b/doc/pub/week35/html/._week35-bs058.html
@@ -327,13 +327,13 @@ MathJax.Hub.Config({
our optimization problem is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
or we can state it as
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
$$
where we have used the definition of a norm-2 vector, that is
@@ -342,29 +342,29 @@ $$
$$
By minimizing the above equation with respect to the parameters
-\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
-parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
+\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
+parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
defining a new cost function to be optimized, that is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
$$
which leads to the Ridge regression minimization problem where we
-require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
+require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
a finite number larger than zero. By defining
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
$$
we have a new optimization equation
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
$$
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
diff --git a/doc/pub/week35/html/._week35-bs059.html b/doc/pub/week35/html/._week35-bs059.html
index 02cdd228c..3b2f2f61f 100644
--- a/doc/pub/week35/html/._week35-bs059.html
+++ b/doc/pub/week35/html/._week35-bs059.html
@@ -326,37 +326,37 @@ MathJax.Hub.Config({
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
+C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
$$
and
-taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
+taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
$$
-\sum_{i=0}^{p-1} \beta_i^2 \leq t,
+\sum_{i=0}^{p-1} \theta_i^2 \leq t,
$$
with \( t \) a finite positive number.
-
If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to
+
If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
When we compare this with the ordinary least squares result we have
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
@@ -365,20 +365,20 @@ $$
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
-even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
$$
-\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
For Ridge regression this becomes
$$
-\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
+\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).
diff --git a/doc/pub/week35/html/._week35-bs061.html b/doc/pub/week35/html/._week35-bs061.html
index d1fb26f51..f0e6527b8 100644
--- a/doc/pub/week35/html/._week35-bs061.html
+++ b/doc/pub/week35/html/._week35-bs061.html
@@ -331,13 +331,13 @@ $$
In this case the standard OLS results in
$$
-\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
+\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
$$
and
$$
-\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
+\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
$$
that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
diff --git a/doc/pub/week35/html/._week35-bs062.html b/doc/pub/week35/html/._week35-bs062.html
index 318c1563e..f1b1e7f4b 100644
--- a/doc/pub/week35/html/._week35-bs062.html
+++ b/doc/pub/week35/html/._week35-bs062.html
@@ -326,28 +326,28 @@ MathJax.Hub.Config({
Using the matrix-vector expression for Lasso regression, we have the following cost function
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
$$
-
Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+
Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
$$
-\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
$$
we have that the derivative of the cost function is
$$
-\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
+\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
$$
and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html
index 0d940c371..cca79a06f 100644
--- a/doc/pub/week35/html/week35-reveal.html
+++ b/doc/pub/week35/html/week35-reveal.html
@@ -219,13 +219,12 @@ MathJax.Hub.Config({
Reminder from last week
-We need first a reminder from last week about linear regression.
+We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.
-Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).
-- Method of choice for fitting a continuous function!
+- It is the method of choice for fitting a continuous function
- Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
-- Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
+- Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)
- Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
- Analytical relation with probabilistic interpretations
- Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
@@ -269,21 +268,21 @@ value and a variance \( \sigma^2 \).
In linear regression we approximate the unknown function with another
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
+\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
Last week we introduced the so-called design matrix in order to define
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
-\( \boldsymbol{\beta} \) as
+\( \boldsymbol{\theta} \) as
$$
-\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
$$
-
and in order to find the optimal parameters \( \beta_i \) we defined a function which
+
and in order to find the optimal parameters \( \theta_i \) we defined a function which
gives a measure of the spread between the values \( y_i \) (which
represent the output values we want to reproduce) and the parametrized
values \( \tilde{y}_i \), namely the so-called cost/loss function.
@@ -296,14 +295,14 @@ values \( \tilde{y}_i \), namely the so-called cost/loss function.
We used the mean squared error to define the way we measure the quality of our model
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
@@ -315,11 +314,11 @@ the function \( C \) as
$$
-C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
+C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
$$
-
since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.
+since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out.
@@ -328,7 +327,7 @@ $$
The function
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
$$
@@ -337,7 +336,7 @@ When linking (see the discussions next week) with the maximum likelihood approac
$$
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
$$
@@ -351,32 +350,32 @@ the standard deviation discussed earlier. In the discussion here we
will treat \( y_i \) as our exact value for the response variable.
-In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem
+In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
In practical terms it means we will require
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
@@ -389,21 +388,21 @@ $$
We can rewrite, see the derivations below,
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
$$
as
$$
-\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
$$
and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution
$$
-\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
@@ -652,60 +651,60 @@ $$
We defined earlier a possible cost function using the mean squared error
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
-
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
-We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
+
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
+We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
$$
-\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
-
which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as
+
which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
+C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
$$
with partial derivative
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
$$
and using that
$$
-\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
+\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
$$
where we used the result from example two above. Inserting the last expression we obtain
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
$$
or as
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
@@ -744,14 +743,14 @@ $$
A very important matrix we will meet again and again in machine
learning is the Hessian. It is given by the second derivative of the
-cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
+cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the mean squared error as cost function is,
$$
-\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
@@ -765,7 +764,7 @@ $$
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
-\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
+\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
@@ -784,25 +783,25 @@ v
The residuals \( \boldsymbol{\epsilon} \) are in turn given by
$$
-\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
and with
$$
-\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
+\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
$$
we have
$$
-\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
+\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
$$
-
meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals.
+
meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals.
we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
+
we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
\( n\times p \) matrix \( \boldsymbol{X} \).
@@ -827,7 +826,7 @@ $$
Own code for Ordinary Least Squares
-It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
@@ -1426,21 +1425,21 @@ plt.show()
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
-
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
+
We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by
$$
-\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
-
The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
+
The hat over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
@@ -1454,7 +1453,7 @@ $$
We can rewrite
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
$$
@@ -1610,17 +1609,17 @@ This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least a
$$
\begin{align}
-\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
+\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\tag{1}
\end{align}
$$
has linearly dependent column vectors, we will not be able to compute the inverse
-of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
-the regression parameters \( \beta_i \) cannot be estimated.
+the regression parameters \( \theta_i \) cannot be estimated.
A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change
@@ -2080,7 +2079,7 @@ function, that is we have
$$
-\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
@@ -2691,15 +2690,15 @@ our optimization problem is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
or we can state it as
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
$$
@@ -2711,34 +2710,34 @@ $$
By minimizing the above equation with respect to the parameters
-\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
-parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
+\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
+parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
defining a new cost function to be optimized, that is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
$$
which leads to the Ridge regression minimization problem where we
-require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
+require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
a finite number larger than zero. By defining
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
$$
we have a new optimization equation
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
$$
@@ -2759,19 +2758,19 @@ $$
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
+C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
$$
and
-taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
+taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
@@ -2779,16 +2778,16 @@ $$
$$
-\sum_{i=0}^{p-1} \beta_i^2 \leq t,
+\sum_{i=0}^{p-1} \theta_i^2 \leq t,
$$
with \( t \) a finite positive number.
-
If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to
+
If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
@@ -2797,7 +2796,7 @@ $$
When we compare this with the ordinary least squares result we have
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
@@ -2807,7 +2806,7 @@ $$
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
-even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
@@ -2815,7 +2814,7 @@ We have already analyzed the OLS solutions in terms of the eigenvectors (the col
$$
-\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
@@ -2823,7 +2822,7 @@ $$
$$
-\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
+\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
@@ -2865,7 +2864,7 @@ $$
In this case the standard OLS results in
$$
-\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
+\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
$$
@@ -2873,7 +2872,7 @@ $$
$$
-\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
+\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
$$
@@ -2896,14 +2895,14 @@ Similarly, Mehta et a
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
$$
-
Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
$$
-\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
$$
@@ -2911,21 +2910,21 @@ $$
$$
-\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
+\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
$$
and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html
index b458df946..c0b90c6f4 100644
--- a/doc/pub/week35/html/week35-solarized.html
+++ b/doc/pub/week35/html/week35-solarized.html
@@ -303,13 +303,12 @@ MathJax.Hub.Config({
Reminder from last week
-We need first a reminder from last week about linear regression.
+We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.
-Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).
-- Method of choice for fitting a continuous function!
+- It is the method of choice for fitting a continuous function
- Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
-- Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
+- Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)
- Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
- Analytical relation with probabilistic interpretations
- Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
@@ -347,19 +346,19 @@ value and a variance \( \sigma^2 \).
In linear regression we approximate the unknown function with another
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
+\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
Last week we introduced the so-called design matrix in order to define
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
-\( \boldsymbol{\beta} \) as
+\( \boldsymbol{\theta} \) as
$$
-\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
$$
-and in order to find the optimal parameters \( \beta_i \) we defined a function which
+
and in order to find the optimal parameters \( \theta_i \) we defined a function which
gives a measure of the spread between the values \( y_i \) (which
represent the output values we want to reproduce) and the parametrized
values \( \tilde{y}_i \), namely the so-called cost/loss function.
@@ -370,12 +369,12 @@ values \( \tilde{y}_i \), namely the so-called cost/loss function.
We used the mean squared error to define the way we measure the quality of our model
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
This function represents one of many possible ways to define the so-called cost function.
@@ -385,24 +384,24 @@ the function \( C \) as
$$
-C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
+C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
$$
-
since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.
+
since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out.
Interpretations and optimizing our parameters
The function
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
$$
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value.
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
$$
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
@@ -415,25 +414,25 @@ the standard deviation discussed earlier. In the discussion here we
will treat \( y_i \) as our exact value for the response variable.
-
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem
+
In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
In practical terms it means we will require
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
@@ -444,17 +443,17 @@ $$
We can rewrite, see the derivations below,
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
$$
as
$$
-\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
$$
and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution
$$
-\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
We note also that since our design matrix is defined as \( \boldsymbol{X}\in
@@ -662,46 +661,46 @@ $$
We defined earlier a possible cost function using the mean squared error
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
-
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
-We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
+
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
+We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
$$
-\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
-
which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as
+
which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
+C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
$$
with partial derivative
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
$$
and using that
$$
-\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
+\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
$$
where we used the result from example two above. Inserting the last expression we obtain
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
$$
or as
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
@@ -732,13 +731,13 @@ $$
A very important matrix we will meet again and again in machine
learning is the Hessian. It is given by the second derivative of the
-cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
+cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the mean squared error as cost function is,
$$
-\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
The Hessian matrix plays an important role and is defined here as
@@ -749,7 +748,7 @@ $$
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
-\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
+\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
@@ -766,20 +765,20 @@ v
The residuals \( \boldsymbol{\epsilon} \) are in turn given by
$$
-\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
and with
$$
-\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
+\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
$$
we have
$$
-\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
+\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
$$
-
meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals.
+
meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals.
we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
+
we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
\( n\times p \) matrix \( \boldsymbol{X} \).
Own code for Ordinary Least Squares
-It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
@@ -1391,18 +1390,18 @@ plt.show()
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
-
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
+
We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by
$$
-\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
-
The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
+
The hat over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
We now define a matrix
@@ -1412,7 +1411,7 @@ $$
We can rewrite
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
$$
The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
@@ -1550,16 +1549,16 @@ This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least a
If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem
$$
\begin{align}
-\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
+\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\label{_auto1}
\end{align}
$$
has linearly dependent column vectors, we will not be able to compute the inverse
-of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
-the regression parameters \( \beta_i \) cannot be estimated.
+the regression parameters \( \theta_i \) cannot be estimated.
A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change
@@ -1956,7 +1955,7 @@ function, that is we have
$$
-\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
@@ -2498,13 +2497,13 @@ values and the column vectors of \( \boldsymbol{V} \).
our optimization problem is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
or we can state it as
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
$$
where we have used the definition of a norm-2 vector, that is
@@ -2513,29 +2512,29 @@ $$
$$
By minimizing the above equation with respect to the parameters
-\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
-parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
+\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
+parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
defining a new cost function to be optimized, that is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
$$
which leads to the Ridge regression minimization problem where we
-require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
+require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
a finite number larger than zero. By defining
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
$$
we have a new optimization equation
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
$$
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
@@ -2552,37 +2551,37 @@ $$
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
+C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
$$
and
-taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
+taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
$$
-\sum_{i=0}^{p-1} \beta_i^2 \leq t,
+\sum_{i=0}^{p-1} \theta_i^2 \leq t,
$$
with \( t \) a finite positive number.
-
If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to
+
If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
When we compare this with the ordinary least squares result we have
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
@@ -2591,20 +2590,20 @@ $$
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
-even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
$$
-\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
For Ridge regression this becomes
$$
-\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
+\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).
@@ -2638,13 +2637,13 @@ $$
In this case the standard OLS results in
$$
-\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
+\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
$$
and
$$
-\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
+\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
$$
that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
@@ -2664,28 +2663,28 @@ Similarly, Mehta et a
Using the matrix-vector expression for Lasso regression, we have the following cost function
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
$$
-Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
$$
-\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
$$
we have that the derivative of the cost function is
$$
-\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
+\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
$$
and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html
index 4cefeb9d7..8391c4fdc 100644
--- a/doc/pub/week35/html/week35.html
+++ b/doc/pub/week35/html/week35.html
@@ -380,13 +380,12 @@ MathJax.Hub.Config({
Reminder from last week
-We need first a reminder from last week about linear regression.
+We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\theta} \) and our first encounter is ordinary least squares.
-Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).
-- Method of choice for fitting a continuous function!
+- It is the method of choice for fitting a continuous function
- Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
-- Analytical expression for the fitting parameters \( \boldsymbol{\beta} \)
+- Analytical expression for the fitting parameters \( \boldsymbol{\theta} \)
- Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
- Analytical relation with probabilistic interpretations
- Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
@@ -424,19 +423,19 @@ value and a variance \( \sigma^2 \).
In linear regression we approximate the unknown function with another
continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on
some unknown parameters
-\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).
+\( \boldsymbol{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}] \).
Last week we introduced the so-called design matrix in order to define
the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity
-\( \boldsymbol{\beta} \) as
+\( \boldsymbol{\theta} \) as
$$
-\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\theta},
$$
-and in order to find the optimal parameters \( \beta_i \) we defined a function which
+
and in order to find the optimal parameters \( \theta_i \) we defined a function which
gives a measure of the spread between the values \( y_i \) (which
represent the output values we want to reproduce) and the parametrized
values \( \tilde{y}_i \), namely the so-called cost/loss function.
@@ -447,12 +446,12 @@ values \( \tilde{y}_i \), namely the so-called cost/loss function.
We used the mean squared error to define the way we measure the quality of our model
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
This function represents one of many possible ways to define the so-called cost function.
@@ -462,24 +461,24 @@ the function \( C \) as
$$
-C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
+C(\boldsymbol{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
$$
-
since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.
+
since when taking the first derivative with respect to the unknown parameters \( \theta \), the factor of \( 2 \) cancels out.
Interpretations and optimizing our parameters
The function
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\},
$$
can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value.
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value
$$
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
$$
where \( \langle y_i \rangle \) is the mean value. Keep in mind also that
@@ -492,25 +491,25 @@ the standard deviation discussed earlier. In the discussion here we
will treat \( y_i \) as our exact value for the response variable.
-
In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem
+
In order to find the parameters \( \theta_i \) we will then minimize the spread of \( C(\boldsymbol{\theta}) \), that is we are going to solve the problem
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
In practical terms it means we will require
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
$$
which results in
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial C(\boldsymbol{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
$$
or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
@@ -521,17 +520,17 @@ $$
We can rewrite, see the derivations below,
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right),
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right),
$$
as
$$
-\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta},
$$
and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution
$$
-\boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{\theta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
We note also that since our design matrix is defined as \( \boldsymbol{X}\in
@@ -739,46 +738,46 @@ $$
We defined earlier a possible cost function using the mean squared error
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
+C(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\},
$$
or using the design/feature matrix \( \boldsymbol{X} \) we have the more compact matrix-vector
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+C(\boldsymbol{\theta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
-
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\beta} \).
-We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).
+
We note that the design matrix \( \boldsymbol{X} \) does not depend on the unknown parameters defined by the vector \( \boldsymbol{\theta} \).
+We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\theta} \).
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
$$
-\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
-
which depends on \( \boldsymbol{\beta} \). We rewrite the cost function as
+
which depends on \( \boldsymbol{\theta} \). We rewrite the cost function as
$$
-C(\boldsymbol{\beta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
+C(\boldsymbol{\theta})=\frac{1}{n}\boldsymbol{w}^T\boldsymbol{w},
$$
with partial derivative
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=\frac{2}{n}\boldsymbol{w}^T\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}},
$$
and using that
$$
-\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X},
+\frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\theta}}=-\boldsymbol{X},
$$
where we used the result from example two above. Inserting the last expression we obtain
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X},
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\boldsymbol{X},
$$
or as
$$
-\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right).
+\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T}=-\frac{2}{n}\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right).
$$
@@ -809,13 +808,13 @@ $$
A very important matrix we will meet again and again in machine
learning is the Hessian. It is given by the second derivative of the
-cost function with respect to the parameters \( \boldsymbol{\beta} \). Using the above
+cost function with respect to the parameters \( \boldsymbol{\theta} \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the mean squared error as cost function is,
$$
-\frac{\partial}{\partial \boldsymbol{\beta}}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial}{\partial \boldsymbol{\theta}}\frac{\partial C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}^T} =\frac{\partial}{\partial \boldsymbol{\theta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
The Hessian matrix plays an important role and is defined here as
@@ -826,7 +825,7 @@ $$
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
-\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is
+\( \hat{\boldsymbol{\theta}} \). Furthermore, we will see later this week that it is
(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
@@ -843,20 +842,20 @@ v
The residuals \( \boldsymbol{\epsilon} \) are in turn given by
$$
-\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta},
+\boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta},
$$
and with
$$
-\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
+\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
$$
we have
$$
-\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0,
+\boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)= 0,
$$
-
meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals.
+
meaning that the solution for \( \boldsymbol{\theta} \) is the one which minimizes the residuals.
we have five predictors/features. The first is the intercept \( \beta_0 \). The other terms are \( \beta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
+
we have five predictors/features. The first is the intercept \( \theta_0 \). The other terms are \( \theta_i \) with \( i=1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is an
\( n\times p \) matrix \( \boldsymbol{X} \).
Own code for Ordinary Least Squares
-It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
+It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\theta} \). After having defined the matrix \( \boldsymbol{X} \) and the outputs \( \boldsymbol{y} \) we have
@@ -1468,18 +1467,18 @@ plt
.show()
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
-
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
+
We have shown that in ordinary least squares the optimal parameters \( \theta \) are given by
$$
-\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
-
The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
+
The hat over \( \boldsymbol{\theta} \) means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
We now define a matrix
@@ -1489,7 +1488,7 @@ $$
We can rewrite
$$
-\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\theta}} = \boldsymbol{A}\boldsymbol{y}.
$$
The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
@@ -1627,16 +1626,16 @@ This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least a
If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem
$$
\begin{align}
-\boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
+\boldsymbol{\theta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y},
\label{_auto1}
\end{align}
$$
has linearly dependent column vectors, we will not be able to compute the inverse
-of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \).
+of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \theta_i \).
The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits.
This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where
-the regression parameters \( \beta_i \) cannot be estimated.
+the regression parameters \( \theta_i \) cannot be estimated.
A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change
@@ -2033,7 +2032,7 @@ function, that is we have
$$
-\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\partial \boldsymbol{\beta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial^2 C(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}\partial \boldsymbol{\theta}^T} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
@@ -2575,13 +2574,13 @@ values and the column vectors of \( \boldsymbol{V} \).
our optimization problem is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}.
+{\displaystyle \min_{\boldsymbol{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\right)\right\}.
$$
or we can state it as
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2,
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2,
$$
where we have used the definition of a norm-2 vector, that is
@@ -2590,29 +2589,29 @@ $$
$$
By minimizing the above equation with respect to the parameters
-\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
-parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by
+\( \boldsymbol{\theta} \) we could then obtain an analytical expression for the
+parameters \( \boldsymbol{\theta} \). We can add a regularization parameter \( \lambda \) by
defining a new cost function to be optimized, that is
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_2^2
$$
which leads to the Ridge regression minimization problem where we
-require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is
+require that \( \vert\vert \boldsymbol{\theta}\vert\vert_2^2\le t \), where \( t \) is
a finite number larger than zero. By defining
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1,
$$
we have a new optimization equation
$$
-{\displaystyle \min_{\boldsymbol{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1
+{\displaystyle \min_{\boldsymbol{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\theta}\vert\vert_1
$$
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
@@ -2629,37 +2628,37 @@ $$
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
+C(\boldsymbol{X},\boldsymbol{\theta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\boldsymbol{\theta}^T\boldsymbol{\theta},
$$
and
-taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
+taking the derivatives with respect to \( \boldsymbol{\theta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that
$$
-\sum_{i=0}^{p-1} \beta_i^2 \leq t,
+\sum_{i=0}^{p-1} \theta_i^2 \leq t,
$$
with \( t \) a finite positive number.
-
If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to
+
If we keep the \( 1/n \) factor, the equation for the optimal \( \theta \) changes to
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+\hat{\boldsymbol{\theta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.
When we compare this with the ordinary least squares result we have
$$
-\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+\hat{\boldsymbol{\theta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
@@ -2668,20 +2667,20 @@ $$
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
-even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+even reduce the variance of the optimal parameters \( \boldsymbol{\theta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
$$
-\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\theta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
For Ridge regression this becomes
$$
-\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
+\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\theta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).
@@ -2715,13 +2714,13 @@ $$
In this case the standard OLS results in
$$
-\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
+\boldsymbol{\theta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}_i^T\boldsymbol{y},
$$
and
$$
-\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}},
+\boldsymbol{\theta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\theta}^{\mathrm{OLS}},
$$
that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and
@@ -2741,28 +2740,28 @@ Similarly, Mehta et a
Using the matrix-vector expression for Lasso regression, we have the following cost function
$$
-C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
+C(\boldsymbol{X},\boldsymbol{\theta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})\right\}+\lambda\vert\vert\boldsymbol{\theta}\vert\vert_1,
$$
-Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+Taking the derivative with respect to \( \boldsymbol{\theta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
$$
-\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
$$
we have that the derivative of the cost function is
$$
-\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
+\frac{\partial C(\boldsymbol{X},\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}=-\frac{2}{n}\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\theta})+\lambda sgn(\boldsymbol{\theta})=0,
$$
and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\frac{n}{2}\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\frac{n}{2}\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
We can redefine \( \lambda \) to absorb the constant \( n/2 \) and we rewrite the last equation as
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\theta}+\lambda sgn(\boldsymbol{\theta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
index bdcd9a505..61a745084 100644
Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ
diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb
index 95e79be49..36dd263dd 100644
--- a/doc/pub/week35/ipynb/week35.ipynb
+++ b/doc/pub/week35/ipynb/week35.ipynb
@@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
- "id": "f59bef82",
+ "id": "3803481e",
"metadata": {
"editable": true
},
@@ -14,7 +14,7 @@
},
{
"cell_type": "markdown",
- "id": "d23d7811",
+ "id": "90e0b055",
"metadata": {
"editable": true
},
@@ -27,7 +27,7 @@
},
{
"cell_type": "markdown",
- "id": "82fad38e",
+ "id": "0b94b0fb",
"metadata": {
"editable": true
},
@@ -49,7 +49,7 @@
},
{
"cell_type": "markdown",
- "id": "380a8031",
+ "id": "c8ad9a9b",
"metadata": {
"editable": true
},
@@ -69,21 +69,20 @@
},
{
"cell_type": "markdown",
- "id": "fc0a0056",
+ "id": "f5f74962",
"metadata": {
"editable": true
},
"source": [
"## Reminder from last week\n",
"\n",
- "We need first a reminder from last week about linear regression. \n",
+ "We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters $\\boldsymbol{\\theta}$ and our first encounter is ordinary least squares.\n",
"\n",
- "Fitting a continuous function with linear parameterization in terms of the parameters $\\boldsymbol{\\beta}$.\n",
- "* Method of choice for fitting a continuous function!\n",
+ "* It is the method of choice for fitting a continuous function\n",
"\n",
"* Gives an excellent introduction to central Machine Learning features with **understandable pedagogical** links to other methods like **Neural Networks**, **Support Vector Machines** etc\n",
"\n",
- "* Analytical expression for the fitting parameters $\\boldsymbol{\\beta}$\n",
+ "* Analytical expression for the fitting parameters $\\boldsymbol{\\theta}$\n",
"\n",
"* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more\n",
"\n",
@@ -103,7 +102,7 @@
},
{
"cell_type": "markdown",
- "id": "5ff8abac",
+ "id": "d61252cd",
"metadata": {
"editable": true
},
@@ -119,7 +118,7 @@
},
{
"cell_type": "markdown",
- "id": "df0716ae",
+ "id": "3dd45959",
"metadata": {
"editable": true
},
@@ -131,7 +130,7 @@
},
{
"cell_type": "markdown",
- "id": "72d9a463",
+ "id": "0ab40f7c",
"metadata": {
"editable": true
},
@@ -141,7 +140,7 @@
},
{
"cell_type": "markdown",
- "id": "c81a5718",
+ "id": "d219bba3",
"metadata": {
"editable": true
},
@@ -153,7 +152,7 @@
},
{
"cell_type": "markdown",
- "id": "4d357994",
+ "id": "806e921f",
"metadata": {
"editable": true
},
@@ -165,33 +164,33 @@
"In linear regression we approximate the unknown function with another\n",
"continuous function $\\tilde{\\boldsymbol{y}}(\\boldsymbol{x})$ which depends linearly on\n",
"some unknown parameters\n",
- "$\\boldsymbol{\\beta}^T=[\\beta_0,\\beta_1,\\beta_2,\\dots,\\beta_{p-1}]$.\n",
+ "$\\boldsymbol{\\theta}^T=[\\theta_0,\\theta_1,\\theta_2,\\dots,\\theta_{p-1}]$.\n",
"\n",
"Last week we introduced the so-called design matrix in order to define\n",
"the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity\n",
- "$\\boldsymbol{\\beta}$ as"
+ "$\\boldsymbol{\\theta}$ as"
]
},
{
"cell_type": "markdown",
- "id": "cf4d9cee",
+ "id": "5a10e185",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n",
+ "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\theta},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "c5ad4e38",
+ "id": "87b9af07",
"metadata": {
"editable": true
},
"source": [
- "and in order to find the optimal parameters $\\beta_i$ we defined a function which\n",
+ "and in order to find the optimal parameters $\\theta_i$ we defined a function which\n",
"gives a measure of the spread between the values $y_i$ (which\n",
"represent the output values we want to reproduce) and the parametrized\n",
"values $\\tilde{y}_i$, namely the so-called cost/loss function."
@@ -199,7 +198,7 @@
},
{
"cell_type": "markdown",
- "id": "297ced52",
+ "id": "b5e5e170",
"metadata": {
"editable": true
},
@@ -211,19 +210,19 @@
},
{
"cell_type": "markdown",
- "id": "0f7fefee",
+ "id": "bacf306d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "bd8dc4f2",
+ "id": "971edd52",
"metadata": {
"editable": true
},
@@ -233,19 +232,19 @@
},
{
"cell_type": "markdown",
- "id": "da82ea39",
+ "id": "1e764eaf",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "23e19e04",
+ "id": "f2f1e2f4",
"metadata": {
"editable": true
},
@@ -258,29 +257,29 @@
},
{
"cell_type": "markdown",
- "id": "24f8e2c6",
+ "id": "f00007b0",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "2f9e0a7f",
+ "id": "ad127894",
"metadata": {
"editable": true
},
"source": [
- "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out."
+ "since when taking the first derivative with respect to the unknown parameters $\\theta$, the factor of $2$ cancels out."
]
},
{
"cell_type": "markdown",
- "id": "62eff411",
+ "id": "87bea682",
"metadata": {
"editable": true
},
@@ -292,19 +291,19 @@
},
{
"cell_type": "markdown",
- "id": "a70a3f6f",
+ "id": "592cb7cc",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "ca289772",
+ "id": "8ad2c594",
"metadata": {
"editable": true
},
@@ -315,19 +314,19 @@
},
{
"cell_type": "markdown",
- "id": "a8312deb",
+ "id": "0d53b5e7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
+ "y_{i}=\\langle y_i \\rangle = \\theta_0x_{i,0}+\\theta_1x_{i,1}+\\theta_2x_{i,2}+\\dots+\\theta_{n-1}x_{i,n-1}+\\epsilon_i,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "df1f3210",
+ "id": "a3cbbda7",
"metadata": {
"editable": true
},
@@ -341,25 +340,25 @@
"the standard deviation discussed earlier. In the discussion here we\n",
"will treat $y_i$ as our exact value for the response variable.\n",
"\n",
- "In order to find the parameters $\\beta_i$ we will then minimize the spread of $C(\\boldsymbol{\\beta})$, that is we are going to solve the problem"
+ "In order to find the parameters $\\theta_i$ we will then minimize the spread of $C(\\boldsymbol{\\theta})$, that is we are going to solve the problem"
]
},
{
"cell_type": "markdown",
- "id": "20dd8ffe",
+ "id": "169c08ab",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
- "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+ "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+ "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "f5acd1e3",
+ "id": "710a21eb",
"metadata": {
"editable": true
},
@@ -369,19 +368,19 @@
},
{
"cell_type": "markdown",
- "id": "f5f153bf",
+ "id": "f18b4513",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = \\frac{\\partial }{\\partial \\theta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "bc60661c",
+ "id": "20b56c08",
"metadata": {
"editable": true
},
@@ -391,19 +390,19 @@
},
{
"cell_type": "markdown",
- "id": "ad0c51c8",
+ "id": "7f6b7e59",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\theta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\theta_0x_{i,0}-\\theta_1x_{i,1}-\\theta_2x_{i,2}-\\dots-\\theta_{n-1}x_{i,n-1}\\right)\\right]=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "299217eb",
+ "id": "36cdc402",
"metadata": {
"editable": true
},
@@ -413,19 +412,19 @@
},
{
"cell_type": "markdown",
- "id": "37c4f3d2",
+ "id": "cb8648c4",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "0aa3bf78",
+ "id": "9d60b674",
"metadata": {
"editable": true
},
@@ -436,19 +435,19 @@
},
{
"cell_type": "markdown",
- "id": "c7cd2c51",
+ "id": "475a553b",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right),\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "8188af0b",
+ "id": "69814fd7",
"metadata": {
"editable": true
},
@@ -458,19 +457,19 @@
},
{
"cell_type": "markdown",
- "id": "bb587a70",
+ "id": "75c465dc",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+ "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "a0facf6c",
+ "id": "fb3f5818",
"metadata": {
"editable": true
},
@@ -480,19 +479,19 @@
},
{
"cell_type": "markdown",
- "id": "a5128b98",
+ "id": "01c31fce",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\boldsymbol{\\theta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "25915602",
+ "id": "869d8825",
"metadata": {
"editable": true
},
@@ -512,7 +511,7 @@
},
{
"cell_type": "markdown",
- "id": "a47041a7",
+ "id": "86056e8c",
"metadata": {
"editable": true
},
@@ -539,7 +538,7 @@
},
{
"cell_type": "markdown",
- "id": "94f82a14",
+ "id": "d36297d2",
"metadata": {
"editable": true
},
@@ -551,7 +550,7 @@
},
{
"cell_type": "markdown",
- "id": "c10b4b3f",
+ "id": "42a61ab6",
"metadata": {
"editable": true
},
@@ -563,7 +562,7 @@
},
{
"cell_type": "markdown",
- "id": "a2ff2fae",
+ "id": "6216fdfa",
"metadata": {
"editable": true
},
@@ -579,7 +578,7 @@
},
{
"cell_type": "markdown",
- "id": "35369c98",
+ "id": "1d8f89cb",
"metadata": {
"editable": true
},
@@ -597,7 +596,7 @@
},
{
"cell_type": "markdown",
- "id": "c3cb1b39",
+ "id": "4c3d1ea5",
"metadata": {
"editable": true
},
@@ -609,7 +608,7 @@
},
{
"cell_type": "markdown",
- "id": "78fec5b2",
+ "id": "41d96182",
"metadata": {
"editable": true
},
@@ -621,7 +620,7 @@
},
{
"cell_type": "markdown",
- "id": "c2cb8b77",
+ "id": "187e7d47",
"metadata": {
"editable": true
},
@@ -632,7 +631,7 @@
},
{
"cell_type": "markdown",
- "id": "672e3ddc",
+ "id": "fe831c6b",
"metadata": {
"editable": true
},
@@ -644,7 +643,7 @@
},
{
"cell_type": "markdown",
- "id": "3d652847",
+ "id": "5e9942fa",
"metadata": {
"editable": true
},
@@ -654,7 +653,7 @@
},
{
"cell_type": "markdown",
- "id": "e4d1006a",
+ "id": "a1589dc4",
"metadata": {
"editable": true
},
@@ -666,7 +665,7 @@
},
{
"cell_type": "markdown",
- "id": "27e8e9da",
+ "id": "f4980fc7",
"metadata": {
"editable": true
},
@@ -680,7 +679,7 @@
},
{
"cell_type": "markdown",
- "id": "f1b4e3e2",
+ "id": "5e5803ca",
"metadata": {
"editable": true
},
@@ -692,7 +691,7 @@
},
{
"cell_type": "markdown",
- "id": "d247b10c",
+ "id": "8ff2d8a0",
"metadata": {
"editable": true
},
@@ -704,7 +703,7 @@
},
{
"cell_type": "markdown",
- "id": "9d9b5cae",
+ "id": "a14f7d1e",
"metadata": {
"editable": true
},
@@ -716,7 +715,7 @@
},
{
"cell_type": "markdown",
- "id": "28bc97b3",
+ "id": "8d0c272b",
"metadata": {
"editable": true
},
@@ -726,7 +725,7 @@
},
{
"cell_type": "markdown",
- "id": "9bd668ca",
+ "id": "9bfc54a6",
"metadata": {
"editable": true
},
@@ -738,7 +737,7 @@
},
{
"cell_type": "markdown",
- "id": "1f072069",
+ "id": "f68602df",
"metadata": {
"editable": true
},
@@ -750,7 +749,7 @@
},
{
"cell_type": "markdown",
- "id": "468e0aaf",
+ "id": "f3a9b46b",
"metadata": {
"editable": true
},
@@ -762,7 +761,7 @@
},
{
"cell_type": "markdown",
- "id": "df18a7bc",
+ "id": "8b9c9796",
"metadata": {
"editable": true
},
@@ -776,7 +775,7 @@
},
{
"cell_type": "markdown",
- "id": "7e778a20",
+ "id": "e55c0131",
"metadata": {
"editable": true
},
@@ -788,7 +787,7 @@
},
{
"cell_type": "markdown",
- "id": "7b09debd",
+ "id": "0213690c",
"metadata": {
"editable": true
},
@@ -800,7 +799,7 @@
},
{
"cell_type": "markdown",
- "id": "bf4df408",
+ "id": "4bebb2ab",
"metadata": {
"editable": true
},
@@ -812,7 +811,7 @@
},
{
"cell_type": "markdown",
- "id": "9527ed06",
+ "id": "f90da960",
"metadata": {
"editable": true
},
@@ -822,7 +821,7 @@
},
{
"cell_type": "markdown",
- "id": "4b2b0605",
+ "id": "8178ffe4",
"metadata": {
"editable": true
},
@@ -834,7 +833,7 @@
},
{
"cell_type": "markdown",
- "id": "8a355b26",
+ "id": "d937141e",
"metadata": {
"editable": true
},
@@ -844,7 +843,7 @@
},
{
"cell_type": "markdown",
- "id": "ac595cbf",
+ "id": "a5a8213f",
"metadata": {
"editable": true
},
@@ -856,7 +855,7 @@
},
{
"cell_type": "markdown",
- "id": "823f52d0",
+ "id": "3bdb5bee",
"metadata": {
"editable": true
},
@@ -866,7 +865,7 @@
},
{
"cell_type": "markdown",
- "id": "a90148b7",
+ "id": "4a9ff5dc",
"metadata": {
"editable": true
},
@@ -878,7 +877,7 @@
},
{
"cell_type": "markdown",
- "id": "d923e6f1",
+ "id": "97098140",
"metadata": {
"editable": true
},
@@ -890,7 +889,7 @@
},
{
"cell_type": "markdown",
- "id": "1106be29",
+ "id": "76008e78",
"metadata": {
"editable": true
},
@@ -902,7 +901,7 @@
},
{
"cell_type": "markdown",
- "id": "5ed90732",
+ "id": "7e07b9bf",
"metadata": {
"editable": true
},
@@ -917,7 +916,7 @@
},
{
"cell_type": "markdown",
- "id": "d516517b",
+ "id": "a691ed11",
"metadata": {
"editable": true
},
@@ -929,7 +928,7 @@
},
{
"cell_type": "markdown",
- "id": "c1379242",
+ "id": "73cfbba0",
"metadata": {
"editable": true
},
@@ -939,7 +938,7 @@
},
{
"cell_type": "markdown",
- "id": "3724106d",
+ "id": "d78a9df5",
"metadata": {
"editable": true
},
@@ -951,7 +950,7 @@
},
{
"cell_type": "markdown",
- "id": "e085c2ec",
+ "id": "9e1e3a87",
"metadata": {
"editable": true
},
@@ -961,7 +960,7 @@
},
{
"cell_type": "markdown",
- "id": "01fa0478",
+ "id": "aea5a5e4",
"metadata": {
"editable": true
},
@@ -973,7 +972,7 @@
},
{
"cell_type": "markdown",
- "id": "8c75a8cd",
+ "id": "89e6fde8",
"metadata": {
"editable": true
},
@@ -983,7 +982,7 @@
},
{
"cell_type": "markdown",
- "id": "e3e275e6",
+ "id": "6c0f4449",
"metadata": {
"editable": true
},
@@ -995,7 +994,7 @@
},
{
"cell_type": "markdown",
- "id": "ab39269e",
+ "id": "a390ac6e",
"metadata": {
"editable": true
},
@@ -1007,19 +1006,19 @@
},
{
"cell_type": "markdown",
- "id": "ce0b7d4b",
+ "id": "59d6ab1c",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "31fbfca8",
+ "id": "8c99ee21",
"metadata": {
"editable": true
},
@@ -1029,66 +1028,66 @@
},
{
"cell_type": "markdown",
- "id": "f3506f4e",
+ "id": "7ea454e1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "6f709ae5",
+ "id": "5a555af4",
"metadata": {
"editable": true
},
"source": [
- "We note that the design matrix $\\boldsymbol{X}$ does not depend on the unknown parameters defined by the vector $\\boldsymbol{\\beta}$.\n",
- "We are now interested in minimizing the cost function with respect to the unknown parameters $\\boldsymbol{\\beta}$.\n",
+ "We note that the design matrix $\\boldsymbol{X}$ does not depend on the unknown parameters defined by the vector $\\boldsymbol{\\theta}$.\n",
+ "We are now interested in minimizing the cost function with respect to the unknown parameters $\\boldsymbol{\\theta}$.\n",
"\n",
"The mean squared error is a scalar and if we use the results from example three above, we can define a new vector"
]
},
{
"cell_type": "markdown",
- "id": "02fba45f",
+ "id": "a39c02c1",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{w}=\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+ "\\boldsymbol{w}=\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "af072826",
+ "id": "a9804433",
"metadata": {
"editable": true
},
"source": [
- "which depends on $\\boldsymbol{\\beta}$. We rewrite the cost function as"
+ "which depends on $\\boldsymbol{\\theta}$. We rewrite the cost function as"
]
},
{
"cell_type": "markdown",
- "id": "35947153",
+ "id": "8be8de93",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\boldsymbol{w}^T\\boldsymbol{w},\n",
+ "C(\\boldsymbol{\\theta})=\\frac{1}{n}\\boldsymbol{w}^T\\boldsymbol{w},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "84e2032b",
+ "id": "4139dd13",
"metadata": {
"editable": true
},
@@ -1098,19 +1097,19 @@
},
{
"cell_type": "markdown",
- "id": "46895a0e",
+ "id": "6f1c778a",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=\\frac{2}{n}\\boldsymbol{w}^T\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\beta}},\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}}=\\frac{2}{n}\\boldsymbol{w}^T\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\theta}},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "b2ffaf06",
+ "id": "03ae847f",
"metadata": {
"editable": true
},
@@ -1120,19 +1119,19 @@
},
{
"cell_type": "markdown",
- "id": "1f591640",
+ "id": "054f8ba7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\beta}}=-\\boldsymbol{X},\n",
+ "\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\theta}}=-\\boldsymbol{X},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "0322fd59",
+ "id": "61c62efa",
"metadata": {
"editable": true
},
@@ -1142,19 +1141,19 @@
},
{
"cell_type": "markdown",
- "id": "1e03255f",
+ "id": "ac84f4fd",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\boldsymbol{X},\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}}=-\\frac{2}{n}\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\boldsymbol{X},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "0a1403f8",
+ "id": "b2f0a3d0",
"metadata": {
"editable": true
},
@@ -1164,19 +1163,19 @@
},
{
"cell_type": "markdown",
- "id": "41a42e64",
+ "id": "9c6460f2",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T}=-\\frac{2}{n}\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n",
+ "\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T}=-\\frac{2}{n}\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right).\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "e68e4f65",
+ "id": "0c4aebc4",
"metadata": {
"editable": true
},
@@ -1188,7 +1187,7 @@
},
{
"cell_type": "markdown",
- "id": "9f4ecb7d",
+ "id": "67aef36e",
"metadata": {
"editable": true
},
@@ -1200,7 +1199,7 @@
},
{
"cell_type": "markdown",
- "id": "d050fc18",
+ "id": "30c88be7",
"metadata": {
"editable": true
},
@@ -1212,7 +1211,7 @@
},
{
"cell_type": "markdown",
- "id": "67837a5c",
+ "id": "fb84fcda",
"metadata": {
"editable": true
},
@@ -1224,7 +1223,7 @@
},
{
"cell_type": "markdown",
- "id": "562ad5bd",
+ "id": "ebbeeb3b",
"metadata": {
"editable": true
},
@@ -1236,7 +1235,7 @@
},
{
"cell_type": "markdown",
- "id": "bf5751b0",
+ "id": "fb88afbf",
"metadata": {
"editable": true
},
@@ -1245,26 +1244,26 @@
"\n",
"A very important matrix we will meet again and again in machine\n",
"learning is the Hessian. It is given by the second derivative of the\n",
- "cost function with respect to the parameters $\\boldsymbol{\\beta}$. Using the above\n",
+ "cost function with respect to the parameters $\\boldsymbol{\\theta}$. Using the above\n",
"expression for derivatives of vectors and matrices, we find that the\n",
"second derivative of the mean squared error as cost function is,"
]
},
{
"cell_type": "markdown",
- "id": "6f1a224c",
+ "id": "89512329",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+ "\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}\\frac{\\partial C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}^T} =\\frac{\\partial}{\\partial \\boldsymbol{\\theta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "4c9387ca",
+ "id": "a651d788",
"metadata": {
"editable": true
},
@@ -1274,7 +1273,7 @@
},
{
"cell_type": "markdown",
- "id": "e55cd91c",
+ "id": "dca9d68f",
"metadata": {
"editable": true
},
@@ -1286,14 +1285,14 @@
},
{
"cell_type": "markdown",
- "id": "5834e282",
+ "id": "2865ced8",
"metadata": {
"editable": true
},
"source": [
"For ordinary least squares, it is inversely proportional (derivation\n",
"next week) with the variance of the optimal parameters\n",
- "$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that it is\n",
+ "$\\hat{\\boldsymbol{\\theta}}$. Furthermore, we will see later this week that it is\n",
"(aside the factor $1/n$) equal to the covariance matrix. It plays also a very\n",
"important role in optmization algorithms and Principal Component\n",
"Analysis as a way to reduce the dimensionality of a machine learning/data analysis\n",
@@ -1304,7 +1303,7 @@
},
{
"cell_type": "markdown",
- "id": "d79e83c1",
+ "id": "1a7a4045",
"metadata": {
"editable": true
},
@@ -1316,19 +1315,19 @@
},
{
"cell_type": "markdown",
- "id": "435ad3ea",
+ "id": "a483362e",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n",
+ "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "3d0c550b",
+ "id": "dcf896a3",
"metadata": {
"editable": true
},
@@ -1338,19 +1337,19 @@
},
{
"cell_type": "markdown",
- "id": "1317ebae",
+ "id": "13c80a7c",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n",
+ "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "c8c5a477",
+ "id": "582225c6",
"metadata": {
"editable": true
},
@@ -1360,29 +1359,29 @@
},
{
"cell_type": "markdown",
- "id": "93d276b1",
+ "id": "90476b18",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n",
+ "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)= 0,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "b292af90",
+ "id": "267ae1ff",
"metadata": {
"editable": true
},
"source": [
- "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals."
+ "meaning that the solution for $\\boldsymbol{\\theta}$ is the one which minimizes the residuals."
]
},
{
"cell_type": "markdown",
- "id": "e8b71fe3",
+ "id": "8c9cb8c5",
"metadata": {
"editable": true
},
@@ -1396,43 +1395,43 @@
},
{
"cell_type": "markdown",
- "id": "b5b88839",
+ "id": "acbe4114",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\tilde{y}_i = \\beta_0+\\beta_1x_i+\\beta_2x_i^2+\\beta_3x_i^3+\\beta_4x_i^4.\n",
+ "\\tilde{y}_i = \\theta_0+\\theta_1x_i+\\theta_2x_i^2+\\theta_3x_i^3+\\theta_4x_i^4.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "4b65ef7c",
+ "id": "ba945f21",
"metadata": {
"editable": true
},
"source": [
- "we have five predictors/features. The first is the intercept $\\beta_0$. The other terms are $\\beta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an \n",
+ "we have five predictors/features. The first is the intercept $\\theta_0$. The other terms are $\\theta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an \n",
"$n\\times p$ matrix $\\boldsymbol{X}$."
]
},
{
"cell_type": "markdown",
- "id": "36fa0c5d",
+ "id": "dc442b90",
"metadata": {
"editable": true
},
"source": [
"## Own code for Ordinary Least Squares\n",
"\n",
- "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\beta}$. After having defined the matrix $\\boldsymbol{X}$ and the outputs $\\boldsymbol{y}$ we have"
+ "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\theta}$. After having defined the matrix $\\boldsymbol{X}$ and the outputs $\\boldsymbol{y}$ we have"
]
},
{
"cell_type": "code",
"execution_count": 1,
- "id": "5bfc7faa",
+ "id": "883026f0",
"metadata": {
"collapsed": false,
"editable": true
@@ -1459,7 +1458,7 @@
},
{
"cell_type": "markdown",
- "id": "07764fc2",
+ "id": "826466d3",
"metadata": {
"editable": true
},
@@ -1470,7 +1469,7 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "f299da23",
+ "id": "decea3b4",
"metadata": {
"collapsed": false,
"editable": true
@@ -1483,7 +1482,7 @@
},
{
"cell_type": "markdown",
- "id": "839633de",
+ "id": "b52ed870",
"metadata": {
"editable": true
},
@@ -1497,7 +1496,7 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "abfc6c01",
+ "id": "ee4d5722",
"metadata": {
"collapsed": false,
"editable": true
@@ -1510,7 +1509,7 @@
},
{
"cell_type": "markdown",
- "id": "aff33949",
+ "id": "01892aaa",
"metadata": {
"editable": true
},
@@ -1521,7 +1520,7 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "67e83121",
+ "id": "d6ce54e2",
"metadata": {
"collapsed": false,
"editable": true
@@ -1533,7 +1532,7 @@
},
{
"cell_type": "markdown",
- "id": "14a294e9",
+ "id": "91ed5956",
"metadata": {
"editable": true
},
@@ -1544,7 +1543,7 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "8d80a11c",
+ "id": "7ffd63cb",
"metadata": {
"collapsed": false,
"editable": true
@@ -1560,7 +1559,7 @@
},
{
"cell_type": "markdown",
- "id": "fd9c909e",
+ "id": "737370ed",
"metadata": {
"editable": true
},
@@ -1571,7 +1570,7 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "fcd82857",
+ "id": "f1bbe22d",
"metadata": {
"collapsed": false,
"editable": true
@@ -1585,7 +1584,7 @@
},
{
"cell_type": "markdown",
- "id": "a91adf8f",
+ "id": "8e3a976b",
"metadata": {
"editable": true
},
@@ -1606,7 +1605,7 @@
},
{
"cell_type": "markdown",
- "id": "24f5afb4",
+ "id": "8c2dee8c",
"metadata": {
"editable": true
},
@@ -1617,7 +1616,7 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "093bd1f1",
+ "id": "89d0750d",
"metadata": {
"collapsed": false,
"editable": true
@@ -1670,7 +1669,7 @@
},
{
"cell_type": "markdown",
- "id": "08a5d650",
+ "id": "12c44026",
"metadata": {
"editable": true
},
@@ -1681,7 +1680,7 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "26c93a00",
+ "id": "6daa8ad9",
"metadata": {
"collapsed": false,
"editable": true
@@ -1706,7 +1705,7 @@
},
{
"cell_type": "markdown",
- "id": "0b3c6835",
+ "id": "9b4cbdbc",
"metadata": {
"editable": true
},
@@ -1718,7 +1717,7 @@
},
{
"cell_type": "markdown",
- "id": "7eae01b1",
+ "id": "7f685bde",
"metadata": {
"editable": true
},
@@ -1747,7 +1746,7 @@
},
{
"cell_type": "markdown",
- "id": "f4576ab0",
+ "id": "a8e92a0b",
"metadata": {
"editable": true
},
@@ -1772,7 +1771,7 @@
},
{
"cell_type": "markdown",
- "id": "04b757cc",
+ "id": "b0ac1e21",
"metadata": {
"editable": true
},
@@ -1792,7 +1791,7 @@
},
{
"cell_type": "markdown",
- "id": "e96e0ec7",
+ "id": "3c8d9613",
"metadata": {
"editable": true
},
@@ -1819,7 +1818,7 @@
},
{
"cell_type": "markdown",
- "id": "b4e7944d",
+ "id": "1d817e20",
"metadata": {
"editable": true
},
@@ -1832,7 +1831,7 @@
},
{
"cell_type": "markdown",
- "id": "385fbd02",
+ "id": "20673f07",
"metadata": {
"editable": true
},
@@ -1844,7 +1843,7 @@
},
{
"cell_type": "markdown",
- "id": "65176139",
+ "id": "4a36afc7",
"metadata": {
"editable": true
},
@@ -1855,7 +1854,7 @@
},
{
"cell_type": "markdown",
- "id": "bc2bda95",
+ "id": "4db40306",
"metadata": {
"editable": true
},
@@ -1871,7 +1870,7 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "f741746f",
+ "id": "ac782b4e",
"metadata": {
"collapsed": false,
"editable": true
@@ -1905,7 +1904,7 @@
},
{
"cell_type": "markdown",
- "id": "91b66d01",
+ "id": "d60792bf",
"metadata": {
"editable": true
},
@@ -1915,7 +1914,7 @@
},
{
"cell_type": "markdown",
- "id": "376037d2",
+ "id": "6dbeeee3",
"metadata": {
"editable": true
},
@@ -1930,7 +1929,7 @@
},
{
"cell_type": "markdown",
- "id": "3b6fa136",
+ "id": "f1bd9e06",
"metadata": {
"editable": true
},
@@ -1942,7 +1941,7 @@
},
{
"cell_type": "markdown",
- "id": "f1087613",
+ "id": "c4e9c9cc",
"metadata": {
"editable": true
},
@@ -1952,7 +1951,7 @@
},
{
"cell_type": "markdown",
- "id": "be2cd7be",
+ "id": "c682dfb2",
"metadata": {
"editable": true
},
@@ -1968,7 +1967,7 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "0ac0921a",
+ "id": "f95e4477",
"metadata": {
"collapsed": false,
"editable": true
@@ -1985,7 +1984,7 @@
},
{
"cell_type": "markdown",
- "id": "700d2c70",
+ "id": "3eaa6d2e",
"metadata": {
"editable": true
},
@@ -1998,7 +1997,7 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "3556cc72",
+ "id": "fe340fe4",
"metadata": {
"collapsed": false,
"editable": true
@@ -2045,7 +2044,7 @@
},
{
"cell_type": "markdown",
- "id": "88222231",
+ "id": "e2f14a01",
"metadata": {
"editable": true
},
@@ -2054,48 +2053,48 @@
"\n",
"What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n",
"\n",
- "We have shown that in ordinary least squares the optimal parameters $\\beta$ are given by"
+ "We have shown that in ordinary least squares the optimal parameters $\\theta$ are given by"
]
},
{
"cell_type": "markdown",
- "id": "9fa5b4c6",
+ "id": "ad741d5a",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\hat{\\boldsymbol{\\theta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "81bb7765",
+ "id": "05b080a8",
"metadata": {
"editable": true
},
"source": [
- "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n",
+ "The **hat** over $\\boldsymbol{\\theta}$ means we have the optimal parameters after minimization of the cost function.\n",
"\n",
"This means that our best model is defined as"
]
},
{
"cell_type": "markdown",
- "id": "e6d5287b",
+ "id": "5c64cd00",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "c95f3611",
+ "id": "66d8648a",
"metadata": {
"editable": true
},
@@ -2105,7 +2104,7 @@
},
{
"cell_type": "markdown",
- "id": "b688fc2e",
+ "id": "47178854",
"metadata": {
"editable": true
},
@@ -2117,7 +2116,7 @@
},
{
"cell_type": "markdown",
- "id": "3109192b",
+ "id": "9d89dcc7",
"metadata": {
"editable": true
},
@@ -2127,19 +2126,19 @@
},
{
"cell_type": "markdown",
- "id": "833ca3e1",
+ "id": "d037dda7",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
+ "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\theta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "f2f8408c",
+ "id": "4d559b4d",
"metadata": {
"editable": true
},
@@ -2150,7 +2149,7 @@
},
{
"cell_type": "markdown",
- "id": "b93b6e5d",
+ "id": "f8eec117",
"metadata": {
"editable": true
},
@@ -2162,7 +2161,7 @@
},
{
"cell_type": "markdown",
- "id": "71a3aeb6",
+ "id": "ecc2fed3",
"metadata": {
"editable": true
},
@@ -2174,7 +2173,7 @@
},
{
"cell_type": "markdown",
- "id": "cd016113",
+ "id": "8d3ae448",
"metadata": {
"editable": true
},
@@ -2184,7 +2183,7 @@
},
{
"cell_type": "markdown",
- "id": "510e85fb",
+ "id": "0d7267fa",
"metadata": {
"editable": true
},
@@ -2196,7 +2195,7 @@
},
{
"cell_type": "markdown",
- "id": "b00cee57",
+ "id": "06e5716f",
"metadata": {
"editable": true
},
@@ -2208,7 +2207,7 @@
},
{
"cell_type": "markdown",
- "id": "9ee31587",
+ "id": "f8f001a6",
"metadata": {
"editable": true
},
@@ -2218,7 +2217,7 @@
},
{
"cell_type": "markdown",
- "id": "deca37bb",
+ "id": "761f6c51",
"metadata": {
"editable": true
},
@@ -2230,7 +2229,7 @@
},
{
"cell_type": "markdown",
- "id": "836d9d6c",
+ "id": "468bc321",
"metadata": {
"editable": true
},
@@ -2240,7 +2239,7 @@
},
{
"cell_type": "markdown",
- "id": "eee14cd2",
+ "id": "eb61e7ef",
"metadata": {
"editable": true
},
@@ -2252,7 +2251,7 @@
},
{
"cell_type": "markdown",
- "id": "40906c45",
+ "id": "be3586ce",
"metadata": {
"editable": true
},
@@ -2262,7 +2261,7 @@
},
{
"cell_type": "markdown",
- "id": "09e6f2ed",
+ "id": "f79b7c51",
"metadata": {
"editable": true
},
@@ -2302,7 +2301,7 @@
},
{
"cell_type": "markdown",
- "id": "ffd73218",
+ "id": "ea84dc35",
"metadata": {
"editable": true
},
@@ -2319,7 +2318,7 @@
},
{
"cell_type": "markdown",
- "id": "93e66912",
+ "id": "0ad4b3f9",
"metadata": {
"editable": true
},
@@ -2342,7 +2341,7 @@
},
{
"cell_type": "markdown",
- "id": "600d07ab",
+ "id": "671100e8",
"metadata": {
"editable": true
},
@@ -2359,7 +2358,7 @@
},
{
"cell_type": "markdown",
- "id": "46e320a9",
+ "id": "9d07ce4a",
"metadata": {
"editable": true
},
@@ -2378,7 +2377,7 @@
},
{
"cell_type": "markdown",
- "id": "b23338e4",
+ "id": "a26b13de",
"metadata": {
"editable": true
},
@@ -2389,7 +2388,7 @@
},
{
"cell_type": "markdown",
- "id": "bc566714",
+ "id": "2352aaa2",
"metadata": {
"editable": true
},
@@ -2401,7 +2400,7 @@
},
{
"cell_type": "markdown",
- "id": "ea1b8269",
+ "id": "5c201fe7",
"metadata": {
"editable": true
},
@@ -2411,7 +2410,7 @@
"\n",
"$$\n",
"\\begin{equation}\n",
- "\\boldsymbol{\\beta} = (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n",
+ "\\boldsymbol{\\theta} = (\\boldsymbol{X}^{T} \\boldsymbol{X})^{-1} \\boldsymbol{X}^{T} \\boldsymbol{y},\n",
"\\label{_auto1} \\tag{1}\n",
"\\end{equation}\n",
"$$"
@@ -2419,23 +2418,23 @@
},
{
"cell_type": "markdown",
- "id": "9aee9220",
+ "id": "a711f9aa",
"metadata": {
"editable": true
},
"source": [
"has linearly dependent column vectors, we will not be able to compute the inverse\n",
- "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n",
+ "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\theta_i$. \n",
"The estimators are only well-defined if $(\\boldsymbol{X}^{T}\\boldsymbol{X})^{-1}$ exits. \n",
"This is more likely to happen when the matrix $\\boldsymbol{X}$ is high-dimensional. In this case it is likely to encounter a situation where \n",
- "the regression parameters $\\beta_i$ cannot be estimated.\n",
+ "the regression parameters $\\theta_i$ cannot be estimated.\n",
"\n",
"A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change"
]
},
{
"cell_type": "markdown",
- "id": "61e7fbda",
+ "id": "816ee812",
"metadata": {
"editable": true
},
@@ -2447,7 +2446,7 @@
},
{
"cell_type": "markdown",
- "id": "afcc930d",
+ "id": "ca4f2d5d",
"metadata": {
"editable": true
},
@@ -2457,7 +2456,7 @@
},
{
"cell_type": "markdown",
- "id": "6ed4a916",
+ "id": "11b18875",
"metadata": {
"editable": true
},
@@ -2472,7 +2471,7 @@
},
{
"cell_type": "markdown",
- "id": "20949d31",
+ "id": "a6cebe96",
"metadata": {
"editable": true
},
@@ -2484,7 +2483,7 @@
},
{
"cell_type": "markdown",
- "id": "1ad7169f",
+ "id": "4056c51d",
"metadata": {
"editable": true
},
@@ -2494,7 +2493,7 @@
},
{
"cell_type": "markdown",
- "id": "4d8a46e3",
+ "id": "8c6fc251",
"metadata": {
"editable": true
},
@@ -2506,7 +2505,7 @@
},
{
"cell_type": "markdown",
- "id": "79a96264",
+ "id": "85d151ae",
"metadata": {
"editable": true
},
@@ -2516,7 +2515,7 @@
},
{
"cell_type": "markdown",
- "id": "0d30c476",
+ "id": "a7ed48ac",
"metadata": {
"editable": true
},
@@ -2528,7 +2527,7 @@
},
{
"cell_type": "markdown",
- "id": "d8320347",
+ "id": "d61100de",
"metadata": {
"editable": true
},
@@ -2540,7 +2539,7 @@
},
{
"cell_type": "markdown",
- "id": "54e7ef4a",
+ "id": "ae931201",
"metadata": {
"editable": true
},
@@ -2555,7 +2554,7 @@
},
{
"cell_type": "markdown",
- "id": "ea96bab9",
+ "id": "a1a55970",
"metadata": {
"editable": true
},
@@ -2566,7 +2565,7 @@
},
{
"cell_type": "markdown",
- "id": "8f610591",
+ "id": "7aff14ea",
"metadata": {
"editable": true
},
@@ -2586,7 +2585,7 @@
},
{
"cell_type": "markdown",
- "id": "3cbcadcc",
+ "id": "c5bcd441",
"metadata": {
"editable": true
},
@@ -2598,7 +2597,7 @@
},
{
"cell_type": "markdown",
- "id": "3d6fa2f6",
+ "id": "ed33684a",
"metadata": {
"editable": true
},
@@ -2608,7 +2607,7 @@
},
{
"cell_type": "markdown",
- "id": "03adee16",
+ "id": "1d6ca687",
"metadata": {
"editable": true
},
@@ -2620,7 +2619,7 @@
},
{
"cell_type": "markdown",
- "id": "e2f52707",
+ "id": "c0535cd8",
"metadata": {
"editable": true
},
@@ -2649,7 +2648,7 @@
},
{
"cell_type": "markdown",
- "id": "143557e4",
+ "id": "c6c08bf2",
"metadata": {
"editable": true
},
@@ -2676,7 +2675,7 @@
},
{
"cell_type": "markdown",
- "id": "8c5068c0",
+ "id": "487a7ae7",
"metadata": {
"editable": true
},
@@ -2687,7 +2686,7 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "090ea41c",
+ "id": "3ff87df9",
"metadata": {
"collapsed": false,
"editable": true
@@ -2727,7 +2726,7 @@
},
{
"cell_type": "markdown",
- "id": "7d055231",
+ "id": "600f74f5",
"metadata": {
"editable": true
},
@@ -2744,7 +2743,7 @@
},
{
"cell_type": "markdown",
- "id": "5d1a15fe",
+ "id": "da923c99",
"metadata": {
"editable": true
},
@@ -2767,7 +2766,7 @@
},
{
"cell_type": "markdown",
- "id": "ee3105fb",
+ "id": "989e9de3",
"metadata": {
"editable": true
},
@@ -2781,7 +2780,7 @@
},
{
"cell_type": "markdown",
- "id": "b62baed5",
+ "id": "b010904b",
"metadata": {
"editable": true
},
@@ -2800,7 +2799,7 @@
},
{
"cell_type": "markdown",
- "id": "1c4c8788",
+ "id": "1385bdc7",
"metadata": {
"editable": true
},
@@ -2810,7 +2809,7 @@
},
{
"cell_type": "markdown",
- "id": "a6c47646",
+ "id": "1824f1de",
"metadata": {
"editable": true
},
@@ -2822,7 +2821,7 @@
},
{
"cell_type": "markdown",
- "id": "d5903c9f",
+ "id": "50fb0869",
"metadata": {
"editable": true
},
@@ -2836,7 +2835,7 @@
},
{
"cell_type": "markdown",
- "id": "64e17d36",
+ "id": "242e0186",
"metadata": {
"editable": true
},
@@ -2848,7 +2847,7 @@
},
{
"cell_type": "markdown",
- "id": "572c9d3f",
+ "id": "b3d8ff35",
"metadata": {
"editable": true
},
@@ -2858,7 +2857,7 @@
},
{
"cell_type": "markdown",
- "id": "3f1a2cb9",
+ "id": "a8799cf8",
"metadata": {
"editable": true
},
@@ -2870,7 +2869,7 @@
},
{
"cell_type": "markdown",
- "id": "41708552",
+ "id": "69ecbf25",
"metadata": {
"editable": true
},
@@ -2887,7 +2886,7 @@
},
{
"cell_type": "markdown",
- "id": "97003366",
+ "id": "accaad01",
"metadata": {
"editable": true
},
@@ -2897,7 +2896,7 @@
},
{
"cell_type": "markdown",
- "id": "3bb5d954",
+ "id": "7c6d6458",
"metadata": {
"editable": true
},
@@ -2913,7 +2912,7 @@
},
{
"cell_type": "markdown",
- "id": "0c60d5d7",
+ "id": "92943a13",
"metadata": {
"editable": true
},
@@ -2923,7 +2922,7 @@
},
{
"cell_type": "markdown",
- "id": "bcb087ce",
+ "id": "27a56b68",
"metadata": {
"editable": true
},
@@ -2939,7 +2938,7 @@
},
{
"cell_type": "markdown",
- "id": "b718bf55",
+ "id": "7fab099a",
"metadata": {
"editable": true
},
@@ -2949,7 +2948,7 @@
},
{
"cell_type": "markdown",
- "id": "3d4eef3e",
+ "id": "86ef772d",
"metadata": {
"editable": true
},
@@ -2965,7 +2964,7 @@
},
{
"cell_type": "markdown",
- "id": "9a61fbcc",
+ "id": "a9fb58a3",
"metadata": {
"editable": true
},
@@ -2975,7 +2974,7 @@
},
{
"cell_type": "markdown",
- "id": "bc19b5f2",
+ "id": "218d36d5",
"metadata": {
"editable": true
},
@@ -2992,7 +2991,7 @@
},
{
"cell_type": "markdown",
- "id": "4449fbb0",
+ "id": "3c358960",
"metadata": {
"editable": true
},
@@ -3004,7 +3003,7 @@
},
{
"cell_type": "markdown",
- "id": "31bcf6bb",
+ "id": "384a8da4",
"metadata": {
"editable": true
},
@@ -3016,7 +3015,7 @@
},
{
"cell_type": "markdown",
- "id": "844ac1e6",
+ "id": "506802df",
"metadata": {
"editable": true
},
@@ -3028,7 +3027,7 @@
},
{
"cell_type": "markdown",
- "id": "22f3a41d",
+ "id": "d160546b",
"metadata": {
"editable": true
},
@@ -3038,7 +3037,7 @@
},
{
"cell_type": "markdown",
- "id": "15d8fff5",
+ "id": "22fa787b",
"metadata": {
"editable": true
},
@@ -3050,7 +3049,7 @@
},
{
"cell_type": "markdown",
- "id": "430bf7ee",
+ "id": "4f9cc0fa",
"metadata": {
"editable": true
},
@@ -3062,7 +3061,7 @@
},
{
"cell_type": "markdown",
- "id": "720d4fb8",
+ "id": "5228fab9",
"metadata": {
"editable": true
},
@@ -3074,7 +3073,7 @@
},
{
"cell_type": "markdown",
- "id": "8deee9d7",
+ "id": "803c129e",
"metadata": {
"editable": true
},
@@ -3084,7 +3083,7 @@
},
{
"cell_type": "markdown",
- "id": "897ea593",
+ "id": "a0229d06",
"metadata": {
"editable": true
},
@@ -3096,7 +3095,7 @@
},
{
"cell_type": "markdown",
- "id": "5321f718",
+ "id": "927c0677",
"metadata": {
"editable": true
},
@@ -3106,7 +3105,7 @@
},
{
"cell_type": "markdown",
- "id": "ba34da1a",
+ "id": "9b1faff4",
"metadata": {
"editable": true
},
@@ -3118,7 +3117,7 @@
},
{
"cell_type": "markdown",
- "id": "0e8a413f",
+ "id": "2a9ba785",
"metadata": {
"editable": true
},
@@ -3134,7 +3133,7 @@
},
{
"cell_type": "markdown",
- "id": "dd064267",
+ "id": "13dd8452",
"metadata": {
"editable": true
},
@@ -3146,7 +3145,7 @@
},
{
"cell_type": "markdown",
- "id": "3a1bc23a",
+ "id": "db18ae22",
"metadata": {
"editable": true
},
@@ -3158,7 +3157,7 @@
},
{
"cell_type": "markdown",
- "id": "3bc120c3",
+ "id": "cc002ef4",
"metadata": {
"editable": true
},
@@ -3168,7 +3167,7 @@
},
{
"cell_type": "markdown",
- "id": "fa44e331",
+ "id": "7808b2cd",
"metadata": {
"editable": true
},
@@ -3180,7 +3179,7 @@
},
{
"cell_type": "markdown",
- "id": "2f7d8c59",
+ "id": "f6a8e671",
"metadata": {
"editable": true
},
@@ -3191,7 +3190,7 @@
},
{
"cell_type": "markdown",
- "id": "e5c786e1",
+ "id": "e6298500",
"metadata": {
"editable": true
},
@@ -3203,7 +3202,7 @@
},
{
"cell_type": "markdown",
- "id": "d4c7d4c6",
+ "id": "1a70e57a",
"metadata": {
"editable": true
},
@@ -3213,7 +3212,7 @@
},
{
"cell_type": "markdown",
- "id": "c8bf0455",
+ "id": "f2091227",
"metadata": {
"editable": true
},
@@ -3225,7 +3224,7 @@
},
{
"cell_type": "markdown",
- "id": "b8a2bdb3",
+ "id": "029a7fce",
"metadata": {
"editable": true
},
@@ -3235,7 +3234,7 @@
},
{
"cell_type": "markdown",
- "id": "20e30049",
+ "id": "62dd4376",
"metadata": {
"editable": true
},
@@ -3247,7 +3246,7 @@
},
{
"cell_type": "markdown",
- "id": "801ec139",
+ "id": "00865f17",
"metadata": {
"editable": true
},
@@ -3258,7 +3257,7 @@
},
{
"cell_type": "markdown",
- "id": "66a71bf6",
+ "id": "64bd2f73",
"metadata": {
"editable": true
},
@@ -3270,7 +3269,7 @@
},
{
"cell_type": "markdown",
- "id": "975d23d9",
+ "id": "ae49067e",
"metadata": {
"editable": true
},
@@ -3288,7 +3287,7 @@
},
{
"cell_type": "markdown",
- "id": "93c0159c",
+ "id": "9ea73c78",
"metadata": {
"editable": true
},
@@ -3304,19 +3303,19 @@
},
{
"cell_type": "markdown",
- "id": "29add97b",
+ "id": "6dcf8fff",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+ "\\frac{\\partial^2 C(\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}\\partial \\boldsymbol{\\theta}^T} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "b17c198b",
+ "id": "8f12f6ca",
"metadata": {
"editable": true
},
@@ -3328,7 +3327,7 @@
},
{
"cell_type": "markdown",
- "id": "ea1d3d0b",
+ "id": "03362376",
"metadata": {
"editable": true
},
@@ -3340,7 +3339,7 @@
},
{
"cell_type": "markdown",
- "id": "a169ccb3",
+ "id": "76d4b169",
"metadata": {
"editable": true
},
@@ -3353,7 +3352,7 @@
},
{
"cell_type": "markdown",
- "id": "c90bce62",
+ "id": "6775daac",
"metadata": {
"editable": true
},
@@ -3369,7 +3368,7 @@
},
{
"cell_type": "markdown",
- "id": "3b26baba",
+ "id": "dc1392b1",
"metadata": {
"editable": true
},
@@ -3383,7 +3382,7 @@
},
{
"cell_type": "markdown",
- "id": "8331409e",
+ "id": "7faf1d79",
"metadata": {
"editable": true
},
@@ -3393,7 +3392,7 @@
},
{
"cell_type": "markdown",
- "id": "4d391ba3",
+ "id": "4c148d48",
"metadata": {
"editable": true
},
@@ -3405,7 +3404,7 @@
},
{
"cell_type": "markdown",
- "id": "46947320",
+ "id": "d2549195",
"metadata": {
"editable": true
},
@@ -3415,7 +3414,7 @@
},
{
"cell_type": "markdown",
- "id": "acf0ace2",
+ "id": "4d7f14be",
"metadata": {
"editable": true
},
@@ -3427,7 +3426,7 @@
},
{
"cell_type": "markdown",
- "id": "02563a13",
+ "id": "3d795b82",
"metadata": {
"editable": true
},
@@ -3437,7 +3436,7 @@
},
{
"cell_type": "markdown",
- "id": "349de4e6",
+ "id": "033e1b21",
"metadata": {
"editable": true
},
@@ -3451,7 +3450,7 @@
},
{
"cell_type": "markdown",
- "id": "39e116df",
+ "id": "5d172bfd",
"metadata": {
"editable": true
},
@@ -3468,7 +3467,7 @@
},
{
"cell_type": "markdown",
- "id": "30d2f2c3",
+ "id": "672de375",
"metadata": {
"editable": true
},
@@ -3484,7 +3483,7 @@
},
{
"cell_type": "markdown",
- "id": "7e5a597b",
+ "id": "d0ed234f",
"metadata": {
"editable": true
},
@@ -3496,7 +3495,7 @@
},
{
"cell_type": "markdown",
- "id": "4514d138",
+ "id": "7a2bde58",
"metadata": {
"editable": true
},
@@ -3509,7 +3508,7 @@
},
{
"cell_type": "markdown",
- "id": "581de71b",
+ "id": "88344390",
"metadata": {
"editable": true
},
@@ -3523,7 +3522,7 @@
},
{
"cell_type": "markdown",
- "id": "7eade20d",
+ "id": "4019cf50",
"metadata": {
"editable": true
},
@@ -3533,7 +3532,7 @@
},
{
"cell_type": "markdown",
- "id": "164ca354",
+ "id": "95d6e396",
"metadata": {
"editable": true
},
@@ -3546,7 +3545,7 @@
},
{
"cell_type": "markdown",
- "id": "f82f15ec",
+ "id": "d5baa57d",
"metadata": {
"editable": true
},
@@ -3565,7 +3564,7 @@
},
{
"cell_type": "markdown",
- "id": "e9ecfbe0",
+ "id": "f310ebda",
"metadata": {
"editable": true
},
@@ -3577,7 +3576,7 @@
},
{
"cell_type": "markdown",
- "id": "69aac1cd",
+ "id": "a8366c66",
"metadata": {
"editable": true
},
@@ -3589,7 +3588,7 @@
},
{
"cell_type": "markdown",
- "id": "2055a5b8",
+ "id": "9983b7fb",
"metadata": {
"editable": true
},
@@ -3599,7 +3598,7 @@
},
{
"cell_type": "markdown",
- "id": "927516d0",
+ "id": "196e9dff",
"metadata": {
"editable": true
},
@@ -3611,7 +3610,7 @@
},
{
"cell_type": "markdown",
- "id": "a58a52b4",
+ "id": "fd64e6a2",
"metadata": {
"editable": true
},
@@ -3624,7 +3623,7 @@
},
{
"cell_type": "markdown",
- "id": "819a1ff5",
+ "id": "46a1e1da",
"metadata": {
"editable": true
},
@@ -3643,7 +3642,7 @@
},
{
"cell_type": "markdown",
- "id": "24d43e16",
+ "id": "6ea1ae21",
"metadata": {
"editable": true
},
@@ -3653,7 +3652,7 @@
},
{
"cell_type": "markdown",
- "id": "7eca0f06",
+ "id": "fccc043c",
"metadata": {
"editable": true
},
@@ -3672,7 +3671,7 @@
},
{
"cell_type": "markdown",
- "id": "04bf3889",
+ "id": "5d65cf6e",
"metadata": {
"editable": true
},
@@ -3690,7 +3689,7 @@
},
{
"cell_type": "markdown",
- "id": "c1f805e1",
+ "id": "f6c00582",
"metadata": {
"editable": true
},
@@ -3704,7 +3703,7 @@
},
{
"cell_type": "markdown",
- "id": "c696df78",
+ "id": "fd88cfa1",
"metadata": {
"editable": true
},
@@ -3719,7 +3718,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "5f05dc7e",
+ "id": "c6da7058",
"metadata": {
"collapsed": false,
"editable": true
@@ -3740,7 +3739,7 @@
},
{
"cell_type": "markdown",
- "id": "2d99bd39",
+ "id": "bd31ca44",
"metadata": {
"editable": true
},
@@ -3757,7 +3756,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "a40890e3",
+ "id": "d809cece",
"metadata": {
"collapsed": false,
"editable": true
@@ -3789,7 +3788,7 @@
},
{
"cell_type": "markdown",
- "id": "c031063b",
+ "id": "8253074c",
"metadata": {
"editable": true
},
@@ -3803,7 +3802,7 @@
},
{
"cell_type": "markdown",
- "id": "f441b1cc",
+ "id": "7a74656a",
"metadata": {
"editable": true
},
@@ -3816,7 +3815,7 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "e1485588",
+ "id": "804a8ee1",
"metadata": {
"collapsed": false,
"editable": true
@@ -3841,7 +3840,7 @@
},
{
"cell_type": "markdown",
- "id": "e5210041",
+ "id": "3871673a",
"metadata": {
"editable": true
},
@@ -3851,7 +3850,7 @@
},
{
"cell_type": "markdown",
- "id": "d97cf3fc",
+ "id": "5e75414e",
"metadata": {
"editable": true
},
@@ -3862,7 +3861,7 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "19805d36",
+ "id": "b3efcdb9",
"metadata": {
"collapsed": false,
"editable": true
@@ -3916,7 +3915,7 @@
},
{
"cell_type": "markdown",
- "id": "f7c176cb",
+ "id": "03523168",
"metadata": {
"editable": true
},
@@ -3933,7 +3932,7 @@
},
{
"cell_type": "markdown",
- "id": "99748b66",
+ "id": "ab0fcb46",
"metadata": {
"editable": true
},
@@ -3945,7 +3944,7 @@
},
{
"cell_type": "markdown",
- "id": "146a709a",
+ "id": "9e11454d",
"metadata": {
"editable": true
},
@@ -3957,7 +3956,7 @@
},
{
"cell_type": "markdown",
- "id": "7bb735a1",
+ "id": "35eaf4ef",
"metadata": {
"editable": true
},
@@ -3967,7 +3966,7 @@
},
{
"cell_type": "markdown",
- "id": "f28204bc",
+ "id": "a426a48c",
"metadata": {
"editable": true
},
@@ -3984,7 +3983,7 @@
},
{
"cell_type": "markdown",
- "id": "d7632836",
+ "id": "f0ac2bfa",
"metadata": {
"editable": true
},
@@ -3994,7 +3993,7 @@
},
{
"cell_type": "markdown",
- "id": "3b66313c",
+ "id": "c6d8c9b4",
"metadata": {
"editable": true
},
@@ -4009,7 +4008,7 @@
},
{
"cell_type": "markdown",
- "id": "7970a23a",
+ "id": "a87b1d66",
"metadata": {
"editable": true
},
@@ -4019,7 +4018,7 @@
},
{
"cell_type": "markdown",
- "id": "20f1bfc3",
+ "id": "cb950b85",
"metadata": {
"editable": true
},
@@ -4033,7 +4032,7 @@
},
{
"cell_type": "markdown",
- "id": "5314cc68",
+ "id": "c79e96c6",
"metadata": {
"editable": true
},
@@ -4045,7 +4044,7 @@
},
{
"cell_type": "markdown",
- "id": "df38126e",
+ "id": "1fceefdc",
"metadata": {
"editable": true
},
@@ -4057,7 +4056,7 @@
},
{
"cell_type": "markdown",
- "id": "c0b851ee",
+ "id": "a8db1029",
"metadata": {
"editable": true
},
@@ -4069,7 +4068,7 @@
},
{
"cell_type": "markdown",
- "id": "1f7f4a30",
+ "id": "03d9d476",
"metadata": {
"editable": true
},
@@ -4079,7 +4078,7 @@
},
{
"cell_type": "markdown",
- "id": "096824a1",
+ "id": "9ccc8461",
"metadata": {
"editable": true
},
@@ -4091,7 +4090,7 @@
},
{
"cell_type": "markdown",
- "id": "54e04c69",
+ "id": "c6183bd4",
"metadata": {
"editable": true
},
@@ -4101,7 +4100,7 @@
},
{
"cell_type": "markdown",
- "id": "7e8d11af",
+ "id": "5cc84134",
"metadata": {
"editable": true
},
@@ -4118,7 +4117,7 @@
},
{
"cell_type": "markdown",
- "id": "54c1dc98",
+ "id": "788d9c68",
"metadata": {
"editable": true
},
@@ -4128,7 +4127,7 @@
},
{
"cell_type": "markdown",
- "id": "db51e12a",
+ "id": "bd0e5aee",
"metadata": {
"editable": true
},
@@ -4140,7 +4139,7 @@
},
{
"cell_type": "markdown",
- "id": "683fad63",
+ "id": "dc52b90c",
"metadata": {
"editable": true
},
@@ -4150,7 +4149,7 @@
},
{
"cell_type": "markdown",
- "id": "1b731b71",
+ "id": "b593db6e",
"metadata": {
"editable": true
},
@@ -4162,7 +4161,7 @@
},
{
"cell_type": "markdown",
- "id": "2e3d21a5",
+ "id": "8dcd6cc8",
"metadata": {
"editable": true
},
@@ -4176,7 +4175,7 @@
},
{
"cell_type": "markdown",
- "id": "5c62c361",
+ "id": "29312718",
"metadata": {
"editable": true
},
@@ -4188,7 +4187,7 @@
},
{
"cell_type": "markdown",
- "id": "64b8d908",
+ "id": "9818fcb1",
"metadata": {
"editable": true
},
@@ -4210,7 +4209,7 @@
},
{
"cell_type": "markdown",
- "id": "fd0e56a2",
+ "id": "56fbca85",
"metadata": {
"editable": true
},
@@ -4222,7 +4221,7 @@
},
{
"cell_type": "markdown",
- "id": "1b0aa1e6",
+ "id": "d9d152fe",
"metadata": {
"editable": true
},
@@ -4237,7 +4236,7 @@
},
{
"cell_type": "markdown",
- "id": "6924b66b",
+ "id": "b82dd67d",
"metadata": {
"editable": true
},
@@ -4249,7 +4248,7 @@
},
{
"cell_type": "markdown",
- "id": "9996cc20",
+ "id": "7dab117e",
"metadata": {
"editable": true
},
@@ -4261,7 +4260,7 @@
},
{
"cell_type": "markdown",
- "id": "586c291e",
+ "id": "3649c008",
"metadata": {
"editable": true
},
@@ -4271,7 +4270,7 @@
},
{
"cell_type": "markdown",
- "id": "0b9dd539",
+ "id": "665e42ab",
"metadata": {
"editable": true
},
@@ -4283,7 +4282,7 @@
},
{
"cell_type": "markdown",
- "id": "a8794b80",
+ "id": "254d4fd4",
"metadata": {
"editable": true
},
@@ -4293,7 +4292,7 @@
},
{
"cell_type": "markdown",
- "id": "045c8882",
+ "id": "4b84bb18",
"metadata": {
"editable": true
},
@@ -4305,7 +4304,7 @@
},
{
"cell_type": "markdown",
- "id": "e28bdf2f",
+ "id": "d84538dd",
"metadata": {
"editable": true
},
@@ -4315,7 +4314,7 @@
},
{
"cell_type": "markdown",
- "id": "f3c04273",
+ "id": "f9a8f033",
"metadata": {
"editable": true
},
@@ -4327,7 +4326,7 @@
},
{
"cell_type": "markdown",
- "id": "9a0b68ea",
+ "id": "0c900c57",
"metadata": {
"editable": true
},
@@ -4344,7 +4343,7 @@
},
{
"cell_type": "markdown",
- "id": "5eed2425",
+ "id": "ef8aafcf",
"metadata": {
"editable": true
},
@@ -4357,19 +4356,19 @@
},
{
"cell_type": "markdown",
- "id": "9dab0c45",
+ "id": "efc29850",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
+ "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\right)\\right\\}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "5644b71a",
+ "id": "586af0fa",
"metadata": {
"editable": true
},
@@ -4379,20 +4378,20 @@
},
{
"cell_type": "markdown",
- "id": "e4639fb2",
+ "id": "116d6e8b",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
- "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
+ "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+ "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "5fee1177",
+ "id": "90567753",
"metadata": {
"editable": true
},
@@ -4402,7 +4401,7 @@
},
{
"cell_type": "markdown",
- "id": "d536a108",
+ "id": "aa3053f1",
"metadata": {
"editable": true
},
@@ -4414,57 +4413,57 @@
},
{
"cell_type": "markdown",
- "id": "33e4ced5",
+ "id": "b2382074",
"metadata": {
"editable": true
},
"source": [
"By minimizing the above equation with respect to the parameters\n",
- "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
- "parameters $\\boldsymbol{\\beta}$. We can add a regularization parameter $\\lambda$ by\n",
+ "$\\boldsymbol{\\theta}$ we could then obtain an analytical expression for the\n",
+ "parameters $\\boldsymbol{\\theta}$. We can add a regularization parameter $\\lambda$ by\n",
"defining a new cost function to be optimized, that is"
]
},
{
"cell_type": "markdown",
- "id": "88d83772",
+ "id": "716a3c2a",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
- "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\n",
+ "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+ "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "ec444bdc",
+ "id": "dfd4ebf1",
"metadata": {
"editable": true
},
"source": [
"which leads to the Ridge regression minimization problem where we\n",
- "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
+ "require that $\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
"a finite number larger than zero. By defining"
]
},
{
"cell_type": "markdown",
- "id": "968537e5",
+ "id": "d782ab23",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n",
+ "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "fd16ef5d",
+ "id": "f16d590d",
"metadata": {
"editable": true
},
@@ -4474,20 +4473,20 @@
},
{
"cell_type": "markdown",
- "id": "ee9bafbd",
+ "id": "a373fb74",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
- "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1\n",
+ "{\\displaystyle \\min_{\\boldsymbol{\\theta}\\in\n",
+ "{\\mathbb{R}}^{p}}}\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\theta}\\vert\\vert_1\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "cf86de9f",
+ "id": "c4c2a6db",
"metadata": {
"editable": true
},
@@ -4499,7 +4498,7 @@
},
{
"cell_type": "markdown",
- "id": "d65614ad",
+ "id": "f14edde4",
"metadata": {
"editable": true
},
@@ -4511,7 +4510,7 @@
},
{
"cell_type": "markdown",
- "id": "ca3686ce",
+ "id": "899bc3ae",
"metadata": {
"editable": true
},
@@ -4523,25 +4522,25 @@
},
{
"cell_type": "markdown",
- "id": "aeec861a",
+ "id": "0968804d",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n",
+ "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\boldsymbol{\\theta}^T\\boldsymbol{\\theta},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "c0cbdf16",
+ "id": "87643992",
"metadata": {
"editable": true
},
"source": [
"and \n",
- "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
+ "taking the derivatives with respect to $\\boldsymbol{\\theta}$ we obtain then\n",
"a slightly modified matrix inversion problem which for finite values\n",
"of $\\lambda$ does not suffer from singularity problems. We obtain\n",
"the optimal parameters"
@@ -4549,19 +4548,19 @@
},
{
"cell_type": "markdown",
- "id": "f3ce9172",
+ "id": "900e9e55",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+ "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "63fb91f4",
+ "id": "6d3a5c46",
"metadata": {
"editable": true
},
@@ -4571,43 +4570,43 @@
},
{
"cell_type": "markdown",
- "id": "e2e09d2c",
+ "id": "db3c68ed",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n",
+ "\\sum_{i=0}^{p-1} \\theta_i^2 \\leq t,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "51621114",
+ "id": "ab6d0410",
"metadata": {
"editable": true
},
"source": [
"with $t$ a finite positive number. \n",
"\n",
- "If we keep the $1/n$ factor, the equation for the optimal $\\beta$ changes to"
+ "If we keep the $1/n$ factor, the equation for the optimal $\\theta$ changes to"
]
},
{
"cell_type": "markdown",
- "id": "78f80a63",
+ "id": "ae454f20",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "ac75ee51",
+ "id": "2a508398",
"metadata": {
"editable": true
},
@@ -4619,19 +4618,19 @@
},
{
"cell_type": "markdown",
- "id": "474894b7",
+ "id": "6df6cf96",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+ "\\hat{\\boldsymbol{\\theta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "254701bf",
+ "id": "4ab38d70",
"metadata": {
"editable": true
},
@@ -4642,7 +4641,7 @@
"modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n",
"particular for our discussion of the bias-variance tradeoff are rather\n",
"interesting. We will see that for specific values of $\\lambda$, we may\n",
- "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
+ "even reduce the variance of the optimal parameters $\\boldsymbol{\\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
"\n",
"Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n",
"We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as"
@@ -4650,19 +4649,19 @@
},
{
"cell_type": "markdown",
- "id": "31fe4fa7",
+ "id": "cfa1a762",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
+ "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\theta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "3a9c5112",
+ "id": "7394c1c2",
"metadata": {
"editable": true
},
@@ -4672,19 +4671,19 @@
},
{
"cell_type": "markdown",
- "id": "cd0353f6",
+ "id": "c67e8228",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
+ "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\theta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "dc9874e9",
+ "id": "95c698ae",
"metadata": {
"editable": true
},
@@ -4694,7 +4693,7 @@
},
{
"cell_type": "markdown",
- "id": "11386feb",
+ "id": "372c4fae",
"metadata": {
"editable": true
},
@@ -4706,7 +4705,7 @@
},
{
"cell_type": "markdown",
- "id": "affaaaf6",
+ "id": "826e010b",
"metadata": {
"editable": true
},
@@ -4718,7 +4717,7 @@
},
{
"cell_type": "markdown",
- "id": "1a888182",
+ "id": "b3c2a178",
"metadata": {
"editable": true
},
@@ -4734,7 +4733,7 @@
},
{
"cell_type": "markdown",
- "id": "86273ea0",
+ "id": "2ec82017",
"metadata": {
"editable": true
},
@@ -4746,7 +4745,7 @@
},
{
"cell_type": "markdown",
- "id": "2d4aaac8",
+ "id": "dacb08f8",
"metadata": {
"editable": true
},
@@ -4758,7 +4757,7 @@
},
{
"cell_type": "markdown",
- "id": "2bfc281a",
+ "id": "af53eaa0",
"metadata": {
"editable": true
},
@@ -4768,19 +4767,19 @@
},
{
"cell_type": "markdown",
- "id": "2040777d",
+ "id": "128f95fc",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
+ "\\boldsymbol{\\theta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "a63d3173",
+ "id": "bd5b4b5d",
"metadata": {
"editable": true
},
@@ -4790,19 +4789,19 @@
},
{
"cell_type": "markdown",
- "id": "271e2323",
+ "id": "d316a1ae",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n",
+ "\\boldsymbol{\\theta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\theta}^{\\mathrm{OLS}},\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "9e2436e2",
+ "id": "1198b684",
"metadata": {
"editable": true
},
@@ -4819,7 +4818,7 @@
},
{
"cell_type": "markdown",
- "id": "17a55a55",
+ "id": "bb24826b",
"metadata": {
"editable": true
},
@@ -4831,41 +4830,41 @@
},
{
"cell_type": "markdown",
- "id": "c313de2e",
+ "id": "b9e921e5",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
+ "C(\\boldsymbol{X},\\boldsymbol{\\theta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\theta}\\vert\\vert_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "a8f03e7f",
+ "id": "06ad064c",
"metadata": {
"editable": true
},
"source": [
- "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
+ "Taking the derivative with respect to $\\boldsymbol{\\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
]
},
{
"cell_type": "markdown",
- "id": "359569a7",
+ "id": "ab63b0d3",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{d \\vert \\beta\\vert}{d \\beta}=\\mathrm{sgn}(\\beta)=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n",
+ "\\frac{d \\vert \\theta\\vert}{d \\theta}=\\mathrm{sgn}(\\theta)=\\left\\{\\begin{array}{cc} 1 & \\theta > 0 \\\\-1 & \\theta < 0, \\end{array}\\right.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "76d42b3d",
+ "id": "49285a6a",
"metadata": {
"editable": true
},
@@ -4875,19 +4874,19 @@
},
{
"cell_type": "markdown",
- "id": "4bb63551",
+ "id": "46be0f62",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n",
+ "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\theta})}{\\partial \\boldsymbol{\\theta}}=-\\frac{2}{n}\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\theta})+\\lambda sgn(\\boldsymbol{\\theta})=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "d5eb25ad",
+ "id": "1ad7dd7c",
"metadata": {
"editable": true
},
@@ -4897,19 +4896,19 @@
},
{
"cell_type": "markdown",
- "id": "38afb347",
+ "id": "3d0b7ab5",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\frac{n}{2}\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\frac{n}{2}\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "687738a0",
+ "id": "b360d642",
"metadata": {
"editable": true
},
@@ -4919,19 +4918,19 @@
},
{
"cell_type": "markdown",
- "id": "06c513a7",
+ "id": "8677a588",
"metadata": {
"editable": true
},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\theta}+\\lambda sgn(\\boldsymbol{\\theta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
- "id": "88794c16",
+ "id": "d4f0eccb",
"metadata": {
"editable": true
},
diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt
index e8ffa19c5..bfde18397 100644
--- a/doc/src/week35/week35.do.txt
+++ b/doc/src/week35/week35.do.txt
@@ -30,12 +30,11 @@ o For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Mar
!split
===== Reminder from last week =====
-We need first a reminder from last week about linear regression.
+We need first a reminder from last week about linear regression. We are going to fit a continuous function with linear parameterization in terms of the parameters $\bm{\theta}$ and our first encounter is ordinary least squares.
-Fitting a continuous function with linear parameterization in terms of the parameters $\bm{\beta}$.
-* Method of choice for fitting a continuous function!
+* It is the method of choice for fitting a continuous function
* Gives an excellent introduction to central Machine Learning features with _understandable pedagogical_ links to other methods like _Neural Networks_, _Support Vector Machines_ etc
-* Analytical expression for the fitting parameters $\bm{\beta}$
+* Analytical expression for the fitting parameters $\bm{\theta}$
* Analytical expressions for statistical propertiers like mean values, variances, confidence intervals and more
* Analytical relation with probabilistic interpretations
* Easy to introduce basic concepts like bias-variance tradeoff, cross-validation, resampling and regularization techniques and many other ML topics
@@ -76,20 +75,20 @@ value and a variance $\sigma^2$.
In linear regression we approximate the unknown function with another
continuous function $\tilde{\bm{y}}(\bm{x})$ which depends linearly on
some unknown parameters
-$\bm{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}]$.
+$\bm{\theta}^T=[\theta_0,\theta_1,\theta_2,\dots,\theta_{p-1}]$.
Last week we introduced the so-called design matrix in order to define
the approximation $\bm{\tilde{y}}$ via the unknown quantity
-$\bm{\beta}$ as
+$\bm{\theta}$ as
!bt
\[
-\bm{\tilde{y}}= \bm{X}\bm{\beta},
+\bm{\tilde{y}}= \bm{X}\bm{\theta},
\]
!et
-and in order to find the optimal parameters $\beta_i$ we defined a function which
+and in order to find the optimal parameters $\theta_i$ we defined a function which
gives a measure of the spread between the values $y_i$ (which
represent the output values we want to reproduce) and the parametrized
values $\tilde{y}_i$, namely the so-called cost/loss function.
@@ -101,13 +100,13 @@ values $\tilde{y}_i$, namely the so-called cost/loss function.
We used the mean squared error to define the way we measure the quality of our model
!bt
\[
-C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
+C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
\]
!et
or using the matrix $\bm{X}$ and in a more compact matrix-vector notation as
!bt
\[
-C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
+C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
This function represents one of many possible ways to define the so-called cost function.
@@ -119,10 +118,10 @@ the function $C$ as
!bt
\[
-C(\bm{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
+C(\bm{\theta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2,
\]
!et
-since when taking the first derivative with respect to the unknown parameters $\beta$, the factor of $2$ cancels out.
+since when taking the first derivative with respect to the unknown parameters $\theta$, the factor of $2$ cancels out.
@@ -133,14 +132,14 @@ since when taking the first derivative with respect to the unknown parameters $\
The function
!bt
\[
-C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\},
+C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\},
\]
!et
can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value.
When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value
!bt
\[
-y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i,
+y_{i}=\langle y_i \rangle = \theta_0x_{i,0}+\theta_1x_{i,1}+\theta_2x_{i,2}+\dots+\theta_{n-1}x_{i,n-1}+\epsilon_i,
\]
!et
@@ -153,29 +152,29 @@ error estimate, often limited to a statistical error estimate given by
the standard deviation discussed earlier. In the discussion here we
will treat $y_i$ as our exact value for the response variable.
-In order to find the parameters $\beta_i$ we will then minimize the spread of $C(\bm{\beta})$, that is we are going to solve the problem
+In order to find the parameters $\theta_i$ we will then minimize the spread of $C(\bm{\theta})$, that is we are going to solve the problem
!bt
\[
-{\displaystyle \min_{\bm{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
+{\displaystyle \min_{\bm{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
In practical terms it means we will require
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0,
+\frac{\partial C(\bm{\theta})}{\partial \theta_j} = \frac{\partial }{\partial \theta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)^2\right]=0,
\]
!et
which results in
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0,
+\frac{\partial C(\bm{\theta})}{\partial \theta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\theta_0x_{i,0}-\theta_1x_{i,1}-\theta_2x_{i,2}-\dots-\theta_{n-1}x_{i,n-1}\right)\right]=0,
\]
!et
or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivation below)
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right).
+\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right).
\]
!et
@@ -186,19 +185,19 @@ or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivatio
We can rewrite, see the derivations below,
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right),
+\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} = 0 = \bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right),
\]
!et
as
!bt
\[
-\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\beta},
+\bm{X}^T\bm{y} = \bm{X}^T\bm{X}\bm{\theta},
\]
!et
and if the matrix $\bm{X}^T\bm{X}$ is invertible we have the solution
!bt
\[
-\bm{\beta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
+\bm{\theta} =\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
@@ -420,53 +419,53 @@ and if $\bm{y}=\bm{x}$ we have
We defined earlier a possible cost function using the mean squared error
!bt
\[
-C(\bm{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
+C(\bm{\theta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\bm{y}-\bm{\tilde{y}}\right)^T\left(\bm{y}-\bm{\tilde{y}}\right)\right\},
\]
!et
or using the design/feature matrix $\bm{X}$ we have the more compact matrix-vector
!bt
\[
-C(\bm{\beta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
+C(\bm{\theta})=\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
-We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\beta}$.
-We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\beta}$.
+We note that the design matrix $\bm{X}$ does not depend on the unknown parameters defined by the vector $\bm{\theta}$.
+We are now interested in minimizing the cost function with respect to the unknown parameters $\bm{\theta}$.
The mean squared error is a scalar and if we use the results from example three above, we can define a new vector
!bt
\[
-\bm{w}=\bm{y}-\bm{X}\bm{\beta},
+\bm{w}=\bm{y}-\bm{X}\bm{\theta},
\]
!et
-which depends on $\bm{\beta}$. We rewrite the cost function as
+which depends on $\bm{\theta}$. We rewrite the cost function as
!bt
\[
-C(\bm{\beta})=\frac{1}{n}\bm{w}^T\bm{w},
+C(\bm{\theta})=\frac{1}{n}\bm{w}^T\bm{w},
\]
!et
with partial derivative
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}}=\frac{2}{n}\bm{w}^T\frac{\partial \bm{w}}{\partial \bm{\beta}},
+\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}}=\frac{2}{n}\bm{w}^T\frac{\partial \bm{w}}{\partial \bm{\theta}},
\]
!et
and using that
!bt
\[
-\frac{\partial \bm{w}}{\partial \bm{\beta}}=-\bm{X},
+\frac{\partial \bm{w}}{\partial \bm{\theta}}=-\bm{X},
\]
!et
where we used the result from example two above. Inserting the last expression we obtain
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\bm{X},
+\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}}=-\frac{2}{n}\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\bm{X},
\]
!et
or as
!bt
\[
-\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T}=-\frac{2}{n}\bm{X}^T\left(\bm{y}-\bm{X}\bm{\beta}\right).
+\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T}=-\frac{2}{n}\bm{X}^T\left(\bm{y}-\bm{X}\bm{\theta}\right).
\]
!et
@@ -505,13 +504,13 @@ We list here some other useful relations we may encounter (recall that vectors a
A very important matrix we will meet again and again in machine
learning is the Hessian. It is given by the second derivative of the
-cost function with respect to the parameters $\bm{\beta}$. Using the above
+cost function with respect to the parameters $\bm{\theta}$. Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the mean squared error as cost function is,
!bt
\[
-\frac{\partial}{\partial \bm{\beta}}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}^T} =\frac{\partial}{\partial \bm{\beta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
+\frac{\partial}{\partial \bm{\theta}}\frac{\partial C(\bm{\theta})}{\partial \bm{\theta}^T} =\frac{\partial}{\partial \bm{\theta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
@@ -527,7 +526,7 @@ The Hessian matrix plays an important role and is defined here as
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
-$\hat{\bm{\beta}}$. Furthermore, we will see later this week that it is
+$\hat{\bm{\theta}}$. Furthermore, we will see later this week that it is
(aside the factor $1/n$) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning/data analysis
@@ -542,22 +541,22 @@ _Linear algebra question:_ Can we use the Hessian matrix to say something about
The residuals $\bm{\epsilon}$ are in turn given by
!bt
\[
-\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\beta},
+\bm{\epsilon} = \bm{y}-\bm{\tilde{y}} = \bm{y}-\bm{X}\bm{\theta},
\]
!et
and with
!bt
\[
-\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0,
+\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0,
\]
!et
we have
!bt
\[
-\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)= 0,
+\bm{X}^T\bm{\epsilon}=\bm{X}^T\left( \bm{y}-\bm{X}\bm{\theta}\right)= 0,
\]
!et
-meaning that the solution for $\bm{\beta}$ is the one which minimizes the residuals.
+meaning that the solution for $\bm{\theta}$ is the one which minimizes the residuals.
!eblock
@@ -571,17 +570,17 @@ we condiser a simple polynomial fit.
We assume our data can represented by a fourth-order polynomial. For the $i$th component we have
!bt
\[
-\tilde{y}_i = \beta_0+\beta_1x_i+\beta_2x_i^2+\beta_3x_i^3+\beta_4x_i^4.
+\tilde{y}_i = \theta_0+\theta_1x_i+\theta_2x_i^2+\theta_3x_i^3+\theta_4x_i^4.
\]
!et
-we have five predictors/features. The first is the intercept $\beta_0$. The other terms are $\beta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an
+we have five predictors/features. The first is the intercept $\theta_0$. The other terms are $\theta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is an
$n\times p$ matrix $\bm{X}$.
!split
===== Own code for Ordinary Least Squares =====
-It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\beta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
+It is rather straightforward to implement the matrix inversion and obtain the parameters $\bm{\theta}$. After having defined the matrix $\bm{X}$ and the outputs $\bm{y}$ we have
!bc pycod
# matrix inversion to find beta
# First we set up the data
@@ -952,21 +951,21 @@ plt.show()
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
-We have shown that in ordinary least squares the optimal parameters $\beta$ are given by
+We have shown that in ordinary least squares the optimal parameters $\theta$ are given by
!bt
\[
-\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
+\hat{\bm{\theta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
-The _hat_ over $\bm{\beta}$ means we have the optimal parameters after minimization of the cost function.
+The _hat_ over $\bm{\theta}$ means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
!bt
\[
-\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
+\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
@@ -980,7 +979,7 @@ We now define a matrix
We can rewrite
!bt
\[
-\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}.
+\tilde{\bm{y}}=\bm{X}\hat{\bm{\theta}} = \bm{A}\bm{y}.
\]
!et
@@ -1125,14 +1124,14 @@ This is equivalent to saying that the matrix $\bm{X}$ has at least an eigenvalue
If our design matrix $\bm{X}$ which enters the linear regression problem
!bt
\begin{align}
-\bm{\beta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y},
+\bm{\theta} & = (\bm{X}^{T} \bm{X})^{-1} \bm{X}^{T} \bm{y},
\end{align}
!et
has linearly dependent column vectors, we will not be able to compute the inverse
-of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\beta_i$.
+of $\bm{X}^T\bm{X}$ and we cannot find the parameters (estimators) $\theta_i$.
The estimators are only well-defined if $(\bm{X}^{T}\bm{X})^{-1}$ exits.
This is more likely to happen when the matrix $\bm{X}$ is high-dimensional. In this case it is likely to encounter a situation where
-the regression parameters $\beta_i$ cannot be estimated.
+the regression parameters $\theta_i$ cannot be estimated.
A cheap *ad hoc* approach is simply to add a small diagonal component to the matrix to invert, that is we change
!bt
@@ -1541,7 +1540,7 @@ function, that is we have
!bt
\[
-\frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}\partial \bm{\beta}^T} =\frac{2}{n}\bm{X}^T\bm{X}.
+\frac{\partial^2 C(\bm{\theta})}{\partial \bm{\theta}\partial \bm{\theta}^T} =\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
@@ -2034,14 +2033,14 @@ Let us remind ourselves about the expression for the standard Mean Squared Error
our optimization problem is
!bt
\[
-{\displaystyle \min_{\bm{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\beta}\right)^T\left(\bm{y}-\bm{X}\bm{\beta}\right)\right\}.
+{\displaystyle \min_{\bm{\theta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\bm{y}-\bm{X}\bm{\theta}\right)^T\left(\bm{y}-\bm{X}\bm{\theta}\right)\right\}.
\]
!et
or we can state it as
!bt
\[
-{\displaystyle \min_{\bm{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2,
+{\displaystyle \min_{\bm{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2,
\]
!et
where we have used the definition of a norm-2 vector, that is
@@ -2052,32 +2051,32 @@ where we have used the definition of a norm-2 vector, that is
!et
By minimizing the above equation with respect to the parameters
-$\bm{\beta}$ we could then obtain an analytical expression for the
-parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
+$\bm{\theta}$ we could then obtain an analytical expression for the
+parameters $\bm{\theta}$. We can add a regularization parameter $\lambda$ by
defining a new cost function to be optimized, that is
!bt
\[
-{\displaystyle \min_{\bm{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_2^2
+{\displaystyle \min_{\bm{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_2^2
\]
!et
which leads to the Ridge regression minimization problem where we
-require that $\vert\vert \bm{\beta}\vert\vert_2^2\le t$, where $t$ is
+require that $\vert\vert \bm{\theta}\vert\vert_2^2\le t$, where $t$ is
a finite number larger than zero. By defining
!bt
\[
-C(\bm{X},\bm{\beta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1,
+C(\bm{X},\bm{\theta})=\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1,
\]
!et
we have a new optimization equation
!bt
\[
-{\displaystyle \min_{\bm{\beta}\in
-{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\beta}\vert\vert_2^2+\lambda\vert\vert \bm{\beta}\vert\vert_1
+{\displaystyle \min_{\bm{\theta}\in
+{\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \bm{y}-\bm{X}\bm{\theta}\vert\vert_2^2+\lambda\vert\vert \bm{\theta}\vert\vert_1
\]
!et
which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.
@@ -2097,17 +2096,17 @@ Using the matrix-vector expression for Ridge regression and dropping the paramet
!bt
\[
-C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
+C(\bm{X},\bm{\theta})=\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\bm{\theta}^T\bm{\theta},
\]
!et
and
-taking the derivatives with respect to $\bm{\beta}$ we obtain then
+taking the derivatives with respect to $\bm{\theta}$ we obtain then
a slightly modified matrix inversion problem which for finite values
of $\lambda$ does not suffer from singularity problems. We obtain
the optimal parameters
!bt
\[
-\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
+\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
@@ -2115,16 +2114,16 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that
!bt
\[
-\sum_{i=0}^{p-1} \beta_i^2 \leq t,
+\sum_{i=0}^{p-1} \theta_i^2 \leq t,
\]
!et
with $t$ a finite positive number.
-If we keep the $1/n$ factor, the equation for the optimal $\beta$ changes to
+If we keep the $1/n$ factor, the equation for the optimal $\theta$ changes to
!bt
\[
-\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
+\hat{\bm{\theta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
@@ -2133,7 +2132,7 @@ In many textbooks the $1/n$ term is often omitted. Note that a library like _Sci
When we compare this with the ordinary least squares result we have
!bt
\[
-\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
+\hat{\bm{\theta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\bm{X}^T\bm{X}$.
@@ -2143,13 +2142,13 @@ We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to $\bm{X}^T\bm{X}$. The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of $\lambda$, we may
-even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+even reduce the variance of the optimal parameters $\bm{\theta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix $\bm{X}$
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
!bt
\[
-\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} =\bm{U}\bm{U}^T\bm{y}.
+\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\theta} =\bm{U}\bm{U}^T\bm{y}.
\]
!et
@@ -2158,7 +2157,7 @@ For Ridge regression this becomes
!bt
\[
-\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\beta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
+\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\theta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
\]
!et
@@ -2198,7 +2197,7 @@ For the sake of simplicity, let us assume that the design matrix is orthonormal,
In this case the standard OLS results in
!bt
\[
-\bm{\beta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}_i^T\bm{y},
+\bm{\theta}^{\mathrm{OLS}} = \bm{X}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}_i^T\bm{y},
\]
!et
@@ -2206,7 +2205,7 @@ and
!bt
\[
-\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\beta}^{\mathrm{OLS}},
+\bm{\theta}^{\mathrm{Ridge}} = \left(\bm{I}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}=\left(1+\lambda\right)^{-1}\bm{\theta}^{\mathrm{OLS}},
\]
!et
@@ -2226,33 +2225,33 @@ Using the matrix-vector expression for Lasso regression, we have the following _
!bt
\[
-C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
+C(\bm{X},\bm{\theta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\theta})^T(\bm{y}-\bm{X}\bm{\theta})\right\}+\lambda\vert\vert\bm{\theta}\vert\vert_1,
\]
!et
-Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
+Taking the derivative with respect to $\bm{\theta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
!bt
\[
-\frac{d \vert \beta\vert}{d \beta}=\mathrm{sgn}(\beta)=\left\{\begin{array}{cc} 1 & \beta > 0 \\-1 & \beta < 0, \end{array}\right.
+\frac{d \vert \theta\vert}{d \theta}=\mathrm{sgn}(\theta)=\left\{\begin{array}{cc} 1 & \theta > 0 \\-1 & \theta < 0, \end{array}\right.
\]
!et
we have that the derivative of the cost function is
!bt
\[
-\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
+\frac{\partial C(\bm{X},\bm{\theta})}{\partial \bm{\theta}}=-\frac{2}{n}\bm{X}^T(\bm{y}-\bm{X}\bm{\theta})+\lambda sgn(\bm{\theta})=0,
\]
!et
and reordering we have
!bt
\[
-\bm{X}^T\bm{X}\bm{\beta}+\frac{n}{2}\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
+\bm{X}^T\bm{X}\bm{\theta}+\frac{n}{2}\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}.
\]
!et
We can redefine $\lambda$ to absorb the constant $n/2$ and we rewrite the last equation as
!bt
\[
-\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
+\bm{X}^T\bm{X}\bm{\theta}+\lambda sgn(\bm{\theta})=2\bm{X}^T\bm{y}.
\]
!et