diff --git a/doc/pub/week35/html/._week35-bs000.html b/doc/pub/week35/html/._week35-bs000.html index 4ce3129de..f6f977e68 100644 --- a/doc/pub/week35/html/._week35-bs000.html +++ b/doc/pub/week35/html/._week35-bs000.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
@@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
-
-
-

 

 

 

- - - -
-

Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

+
+

Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

+
-

-

Morten Hjorth-Jensen [1, 2]
- -

+

+[1] Department of Physics, University of Oslo +
+
+[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
+
+
+

Nov 3, 2021

+
+
-
[1] Department of Physics, University of Oslo
-
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
-
-

-

Sep 28, 2021

-
-

Read »

@@ -434,25 +422,18 @@ MathJax.Hub.Config({
  • »
    • -
    - - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week35/html/._week35-bs001.html b/doc/pub/week35/html/._week35-bs001.html index 40dad0cee..2af7c186e 100644 --- a/doc/pub/week35/html/._week35-bs001.html +++ b/doc/pub/week35/html/._week35-bs001.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,14 +371,10 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Plans for week 35, August 30 -September 3

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs002.html b/doc/pub/week35/html/._week35-bs002.html index d3cf64d04..af1a23a8d 100644 --- a/doc/pub/week35/html/._week35-bs002.html +++ b/doc/pub/week35/html/._week35-bs002.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    -
    -

     

     

     

    - -

    Thursday September 2

    -

    -The main topics on Thursday are: - +

    The main topics on Thursday are:

    1. Repetition from last week on linear regression
    2. Discussion of how to prepare data and examples of applications of linear regression
    3. Mathematical interpretations of Linear Regression
    4. Start discussing Ridge and Lasso regression and Singular Value Decomposition
    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs003.html b/doc/pub/week35/html/._week35-bs003.html index 7d55cf37b..a80d2fd80 100644 --- a/doc/pub/week35/html/._week35-bs003.html +++ b/doc/pub/week35/html/._week35-bs003.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    -
    -

     

     

     

    - -

    Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week

    -

    -We need first a reminder from last week about linear regression. - -

    -Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \). +

    We need first a reminder from last week about linear regression.

    +

    Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).

    - -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs004.html b/doc/pub/week35/html/._week35-bs004.html index a6cf527f8..ca6eaffaa 100644 --- a/doc/pub/week35/html/._week35-bs004.html +++ b/doc/pub/week35/html/._week35-bs004.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    -
    -

     

     

     

    - -

    Regression analysis, overarching aims

    -

    + -

    -Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). -The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. - -

    -A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with +

    Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). +The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. +

    +

    A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with

    • \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    • Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    • \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    - - The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +

    The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs005.html b/doc/pub/week35/html/._week35-bs005.html index a8eb2ce10..a9948d15a 100644 --- a/doc/pub/week35/html/._week35-bs005.html +++ b/doc/pub/week35/html/._week35-bs005.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    -
    -

     

     

     

    - -

    Regression analysis, overarching aims II

    -

    + -

    -Consider an experiment in which \( p \) characteristics of \( n \) samples are +

    Consider an experiment in which \( p \) characteristics of \( n \) samples are measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix \( \mathbf{X} \). +

    -

    -The matrix \( \mathbf{X} \) is called the design +

    The matrix \( \mathbf{X} \) is called the design matrix. Additional information of the samples is available in the form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is generally referred to as the response variable. The aim of @@ -408,17 +395,14 @@ f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of \( f(\cdot) \) is available, it is common to assume a linear relationship between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, -\beta_{p-1}]^{T} \) are the regression parameters. +\beta_{p-1}]^{T} \) are the regression parameters. +

    -

    -Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). - -

    +

    Linear regression gives us a set of analytical equations for the parameters \( \beta_j \).

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs006.html b/doc/pub/week35/html/._week35-bs006.html index 788eb6068..97bbb235c 100644 --- a/doc/pub/week35/html/._week35-bs006.html +++ b/doc/pub/week35/html/._week35-bs006.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,42 +371,37 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Examples

    -

    -In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), -consider the model we discussed for describing nuclear binding energies. + +

    In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. +

    -

    -There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +

    There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. Assuming +

    $$ BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, $$ -we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +

    we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a \( p\times n \) matrix \( \boldsymbol{X} \). +

    -

    -Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the -so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \). This is however a classification problem and we will come back to it when we discuss Logistic Regression. - -

    +

    Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \). This is however a classification problem and we will come back to it when we discuss Logistic Regression. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs007.html b/doc/pub/week35/html/._week35-bs007.html index 1d17a9f91..a74a0c4d0 100644 --- a/doc/pub/week35/html/._week35-bs007.html +++ b/doc/pub/week35/html/._week35-bs007.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,34 +371,26 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    General linear models

    -

    -Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. + +

    Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function.

    -

    -Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +

    Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is

    $$ y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, $$ -where \( \epsilon_i \) is the error in our approximation. - -

    +

    where \( \epsilon_i \) is the error in our approximation.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs008.html b/doc/pub/week35/html/._week35-bs008.html index 998e361fc..3ae96ab97 100644 --- a/doc/pub/week35/html/._week35-bs008.html +++ b/doc/pub/week35/html/._week35-bs008.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,19 +371,15 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Rewriting the fitting procedure as a linear algebra problem

    -

    -For every set of values \( y_i,x_i \) we have thus the corresponding set of equations + +

    For every set of values \( y_i,x_i \) we have thus the corresponding set of equations

    $$ \begin{align*} y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ @@ -405,7 +393,6 @@ $$
    -

      @@ -433,25 +420,18 @@ $$
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs009.html b/doc/pub/week35/html/._week35-bs009.html index 1b8e650a3..a2327d1af 100644 --- a/doc/pub/week35/html/._week35-bs009.html +++ b/doc/pub/week35/html/._week35-bs009.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,34 +371,30 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Rewriting the fitting procedure as a linear algebra problem, more details

    -

    -Defining the vectors + +

    Defining the vectors

    $$ \boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, $$ -and +

    and

    $$ \boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, $$ -and +

    and

    $$ \boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, $$ -and the design matrix +

    and the design matrix

    $$ \boldsymbol{X}= \begin{bmatrix} @@ -418,17 +406,16 @@ $$ \end{bmatrix} $$ -we can rewrite our equations as +

    we can rewrite our equations as

    $$ \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. $$ -The above design matrix is called a Vandermonde matrix. +

    The above design matrix is called a Vandermonde matrix.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs010.html b/doc/pub/week35/html/._week35-bs010.html index 5509a62e5..7a91e79b7 100644 --- a/doc/pub/week35/html/._week35-bs010.html +++ b/doc/pub/week35/html/._week35-bs010.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,25 +371,21 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Generalizing the fitting procedure as a linear algebra problem

    -

    + -

    -We are obviously not limited to the above polynomial expansions. We +

    We are obviously not limited to the above polynomial expansions. We could replace the various powers of \( x \) with elements of Fourier series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions. For every set of values \( y_i,x_i \) we can then generalize the equations to +

    $$ \begin{align*} @@ -411,13 +399,12 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{ \end{align*} $$ -

    + Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case!

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs011.html b/doc/pub/week35/html/._week35-bs011.html index f6a655a06..20dd4910e 100644 --- a/doc/pub/week35/html/._week35-bs011.html +++ b/doc/pub/week35/html/._week35-bs011.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,19 +371,15 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Generalizing the fitting procedure as a linear algebra problem

    -

    -We redefine in turn the matrix \( \boldsymbol{X} \) as + +

    We redefine in turn the matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{X}= \begin{bmatrix} @@ -403,17 +391,16 @@ x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ \end{bmatrix} $$ -and without loss of generality we rewrite again our equations as +

    and without loss of generality we rewrite again our equations as

    $$ \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. $$ -The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

    The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs012.html b/doc/pub/week35/html/._week35-bs012.html index 71c7e5f9b..e70f9e3bf 100644 --- a/doc/pub/week35/html/._week35-bs012.html +++ b/doc/pub/week35/html/._week35-bs012.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,19 +371,15 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Optimizing our parameters

    -

    -We have defined the matrix \( \boldsymbol{X} \) via the equations + +

    We have defined the matrix \( \boldsymbol{X} \) via the equations

    $$ \begin{align*} y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ @@ -404,17 +392,14 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{ \end{align*} $$ -

    -As we noted above, we stayed with a system with the design matrix +

    As we noted above, we stayed with a system with the design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. - -

    +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs013.html b/doc/pub/week35/html/._week35-bs013.html index 1cf3811e5..f377b4488 100644 --- a/doc/pub/week35/html/._week35-bs013.html +++ b/doc/pub/week35/html/._week35-bs013.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,25 +371,23 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Our model for the nuclear binding energies

    -

    -In our introductory notes we looked at the so-called liquid drop model. Let us remind ourselves about what we did by looking at the code. +

    In our introductory notes we looked at the so-called liquid drop model. Let us remind ourselves about what we did by looking at the code.

    -

    -We restate the parts of the code we are most interested in. -

    +

    We restate the parts of the code we are most interested in.

    -
    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -466,16 +456,28 @@ DesignMatrix = pd.index = A
 DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
 display(DesignMatrix)
-
    
-
    
-With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as

    $$ \boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, $$ -throughout these lectures. +

    throughout these lectures.

    -

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs014.html b/doc/pub/week35/html/._week35-bs014.html index 058df6ab7..4dc2b9b3b 100644 --- a/doc/pub/week35/html/._week35-bs014.html +++ b/doc/pub/week35/html/._week35-bs014.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,49 +371,44 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Optimizing our parameters, more details

    -

    -With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as + +

    With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as

    $$ \boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, $$ -and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +

    and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, $$ -or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +

    or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -This function is one possible way to define the so-called cost function. +

    This function is one possible way to define the so-called cost function.

    -

    -It is also common to define +

    It is also common to define the function \( C \) as +

    $$ C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, $$ -since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

    since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs015.html b/doc/pub/week35/html/._week35-bs015.html index b2bfcc61f..d770d9588 100644 --- a/doc/pub/week35/html/._week35-bs015.html +++ b/doc/pub/week35/html/._week35-bs015.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,33 +371,28 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Interpretations and optimizing our parameters

    -

    + -

    -The function +

    The function

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, $$ -can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +

    can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value +

    $$ y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, $$ -

    -where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +

    where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an @@ -413,35 +400,32 @@ approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable. +

    -

    -In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +

    In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -In practical terms it means we will require +

    In practical terms it means we will require

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, $$ -which results in +

    which results in

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ -or in a matrix-vector form as +

    or in a matrix-vector form as

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$ - -

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs016.html b/doc/pub/week35/html/._week35-bs016.html index 7c5c643b2..505ca8579 100644 --- a/doc/pub/week35/html/._week35-bs016.html +++ b/doc/pub/week35/html/._week35-bs016.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,35 +371,30 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Interpretations and optimizing our parameters

    -

    -We can rewrite + +

    We can rewrite

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$ -as +

    as

    $$ \boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, $$ -and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +

    and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution

    $$ \boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -We note also that since our design matrix is defined as \( \boldsymbol{X}\in +

    We note also that since our design matrix is defined as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in {\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), in our case \( p=5 \) meaning that we end up with inverting a small @@ -416,21 +403,20 @@ matrices to invert. The methods discussed here and for many other supervised learning algorithms like classification with logistic regression or support vector machines, exhibit dimensionalities which allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix -\( \boldsymbol{X}^T\boldsymbol{X} \). +\( \boldsymbol{X}^T\boldsymbol{X} \). +

    -

    -

    -Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)? What kind of problems can we expect? + +

    Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)? What kind of problems can we expect?

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs017.html b/doc/pub/week35/html/._week35-bs017.html index d29decc17..814c190eb 100644 --- a/doc/pub/week35/html/._week35-bs017.html +++ b/doc/pub/week35/html/._week35-bs017.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,19 +371,15 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Some useful matrix and vector expressions

    -

    -The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and +

    The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. +

    $$ \frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b}, @@ -409,7 +397,7 @@ $$ \frac{\partial \log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}} = (\boldsymbol{A}^{-1})^T. $$ -

    +

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs018.html b/doc/pub/week35/html/._week35-bs018.html index e643774b0..3001a47e8 100644 --- a/doc/pub/week35/html/._week35-bs018.html +++ b/doc/pub/week35/html/._week35-bs018.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,47 +371,40 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Meet the Hessian Matrix

    -

    -A very important matrix we will meet again and again in Machine +

    A very important matrix we will meet again and again in Machine Learning is the Hessian. It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, +

    $$ \frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -The Hessian matrix plays an important role and is defined here as +

    The Hessian matrix plays an important role and is defined here as

    $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -For ordinary least squares, it is inversely proportional (derivation +

    For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. +

    -

    -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). +

    Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs019.html b/doc/pub/week35/html/._week35-bs019.html index 87cb4f698..097daf45c 100644 --- a/doc/pub/week35/html/._week35-bs019.html +++ b/doc/pub/week35/html/._week35-bs019.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,41 +371,34 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Interpretations and optimizing our parameters

    -

    -The residuals \( \boldsymbol{\epsilon} \) are in turn given by + +

    The residuals \( \boldsymbol{\epsilon} \) are in turn given by

    $$ \boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, $$ -and with +

    and with

    $$ \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, $$ -we have +

    we have

    $$ \boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, $$ -meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. - -

    +

    meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs020.html b/doc/pub/week35/html/._week35-bs020.html index 8a59b55b2..96b4ecaf7 100644 --- a/doc/pub/week35/html/._week35-bs020.html +++ b/doc/pub/week35/html/._week35-bs020.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,41 +371,75 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Own code for Ordinary Least Squares

    -

    -It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +

    It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to write -

    +

    -
    # matrix inversion to find beta
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # matrix inversion to find beta
 beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
 # and then make the prediction
 ytilde = X @ beta
-
    
-
    
-Alternatively, you can use the least squares functionality in Numpy as
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Alternatively, you can use the least squares functionality in Numpy as

    -
    fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+
    
+  
    
+    
    
+      
    
+        
    
+  
    fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
 ytildenp = np.dot(fit,X.T)
-
    
-
    
-And finally we plot our fit with and compare with data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    And finally we plot our fit with and compare with data

    -
    Masses['Eapprox']  = ytilde
+
    
+  
    
+    
    
+      
    
+        
    
+  
    Masses['Eapprox']  = ytilde
 # Generate a plot comparing the experimental with the fitted values values.
 fig, ax = plt.subplots()
 ax.set_xlabel(r'$A = N + Z$')
@@ -425,8 +451,22 @@ ax.plot(Masses[
 ax.legend()
 save_fig("Masses2016OLS")
 plt.show()
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs021.html b/doc/pub/week35/html/._week35-bs021.html index 68b046921..62f43b27b 100644 --- a/doc/pub/week35/html/._week35-bs021.html +++ b/doc/pub/week35/html/._week35-bs021.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,53 +371,118 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Adding error analysis and training set up

    -

    -We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of Scikit-Learn in the introductory slides. +

    We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of Scikit-Learn in the introductory slides. Since we are not using Scikit-Learn here we can define our own \( R2 \) function as -

    +

    -
    def R2(y_data, y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def R2(y_data, y_model):
     return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-
    
-
    
-and we would be using it as 
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and we would be using it as

    -
    print(R2(Energies,ytilde))
-
    -

    -We can easily add our MSE score as -

    +

    +
    +
    +
    +
    + 
    print(R2(Energies,ytilde))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We can easily add our MSE score as

    -
    def MSE(y_data,y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def MSE(y_data,y_model):
     n = np.size(y_model)
     return np.sum((y_data-y_model)**2)/n
 
 print(MSE(Energies,ytilde))
-
    
-
    
-and finally the relative error as
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and finally the relative error as

    -
    def RelativeError(y_data,y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def RelativeError(y_data,y_model):
     return abs((y_data-y_model)/y_data)
 print(RelativeError(Energies, ytilde))
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs022.html b/doc/pub/week35/html/._week35-bs022.html index a2c0752dd..1d3d493bb 100644 --- a/doc/pub/week35/html/._week35-bs022.html +++ b/doc/pub/week35/html/._week35-bs022.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,23 +371,17 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Splitting our Data in Training and Test data

    -

    -

    + -

    -It is normal in essentially all Machine Learning studies to split the +

    It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (sometimes also an additional validation set). Scikit-Learn has an own function for this. There is no explicit recipe for how much data should be included as training @@ -405,13 +391,11 @@ postpone a discussion of this splitting to the end of these notes and our discussion of the so-called bias-variance tradeoff. Here we limit ourselves to repeat the above equation of state fitting example but now splitting the data into a training set and a test set. - -

    +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs023.html b/doc/pub/week35/html/._week35-bs023.html index 23ac0efee..f40ad4122 100644 --- a/doc/pub/week35/html/._week35-bs023.html +++ b/doc/pub/week35/html/._week35-bs023.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,20 +371,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Examples

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -430,8 +422,22 @@ ypredict = X_test print(R2(y_test,ypredict))
 print("Test MSE")
 print(MSE(y_test,ypredict))
-
    
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs024.html b/doc/pub/week35/html/._week35-bs024.html index feb16e1ab..ba79d8c9e 100644 --- a/doc/pub/week35/html/._week35-bs024.html +++ b/doc/pub/week35/html/._week35-bs024.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,20 +371,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Making your own test-train splitting

    -

    -

    # equivalently in numpy
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # equivalently in numpy
 def train_test_split_numpy(inputs, labels, train_size, test_size):
     n_inputs = len(inputs)
     inputs_shuffled = inputs.copy()
@@ -406,13 +398,26 @@ MathJax.Hub.Config({
     Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]
 
     return X_train, X_test, Y_train, Y_test
-
    
-
    
-But since scikit-learn has its own function for doing this and since
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    But since scikit-learn has its own function for doing this and since it interfaces easily with tensorflow and other libraries, we normally recommend using the latter functionality. +

    -

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs025.html b/doc/pub/week35/html/._week35-bs025.html index 8d522c052..e8aa574d3 100644 --- a/doc/pub/week35/html/._week35-bs025.html +++ b/doc/pub/week35/html/._week35-bs025.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,27 +371,21 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    The Boston housing data example

    -

    -The Boston housing +

    The Boston housing data set was originally a part of UCI Machine Learning Repository and has been removed now. The data set is now included in Scikit-Learn's library. There are 506 samples and 13 feature (predictor) variables in this data set. The objective is to predict the value of prices of the house using the features (predictors) listed here. +

    -

    -The features/predictors are - +

    The features/predictors are

    1. CRIM: Per capita crime rate by town
    2. ZN: Proportion of residential land zoned for lots over 25000 square feet
      3. @@ -415,7 +401,6 @@ The features/predictors are
    3. LSTAT: Percentage of lower status of the population
    4. MEDV: Median value of owner-occupied homes in USD 1000s
    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs026.html b/doc/pub/week35/html/._week35-bs026.html index 9badf1799..af958211e 100644 --- a/doc/pub/week35/html/._week35-bs026.html +++ b/doc/pub/week35/html/._week35-bs026.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,86 +371,188 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Housing data, the code

    -We start by importing the libraries -

    +

    We start by importing the libraries

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import matplotlib.pyplot as plt 
 
 import pandas as pd  
 import seaborn as sns 
-
    
-
    
-and load the Boston Housing DataSet from Scikit-Learn
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and load the Boston Housing DataSet from Scikit-Learn

    -

    -

    from sklearn.datasets import load_boston
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.datasets import load_boston
 
 boston_dataset = load_boston()
 
 # boston_dataset is a dictionary
 # let's check what it contains
 boston_dataset.keys()
-
    
-
    
-Then we invoke Pandas
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Then we invoke Pandas

    -
    boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
 boston.head()
 boston['MEDV'] = boston_dataset.target
-
    
-
    
-and preprocess the data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and preprocess the data

    -
    # check for missing values in all the columns
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # check for missing values in all the columns
 boston.isnull().sum()
-
    
-
    
-We can then visualize the data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We can then visualize the data

    -
    # set the size of the figure
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # set the size of the figure
 sns.set(rc={'figure.figsize':(11.7,8.27)})
 
 # plot a histogram showing the distribution of the target values
 sns.distplot(boston['MEDV'], bins=30)
 plt.show()
-
    
-
    
-It is now useful to look at the correlation matrix
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    It is now useful to look at the correlation matrix

    -
    # compute the pair wise correlation for all columns  
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # compute the pair wise correlation for all columns  
 correlation_matrix = boston.corr().round(2)
 # use the heatmap function from seaborn to plot the correlation matrix
 # annot = True to print the values inside the square
 sns.heatmap(data=correlation_matrix, annot=True)
-
    
-
    
-From the above coorelation plot we can see that MEDV is strongly correlated to LSTAT and  RM. We see also that RAD and TAX are stronly correlated, but we don't include this in our features together to avoid multi-colinearity
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    From the above coorelation plot we can see that MEDV is strongly correlated to LSTAT and RM. We see also that RAD and TAX are stronly correlated, but we don't include this in our features together to avoid multi-colinearity

    -

    -

    plt.figure(figsize=(20, 5))
+
    
+  
    
+    
    
+      
    
+        
    
+  
    plt.figure(figsize=(20, 5))
 
 features = ['LSTAT', 'RM']
 target = boston['MEDV']
@@ -471,22 +565,56 @@ target = boston[.title(col)
     plt.xlabel(col)
     plt.ylabel('MEDV')
-
    
-
    
-Now we start training our model
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Now we start training our model

    -
    X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
+
    
+  
    
+    
    
+      
    
+        
    
+  
    X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
 Y = boston['MEDV']
-
    
-
    
-We split the data into training and test sets
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We split the data into training and test sets

    -

    -

    from sklearn.model_selection import train_test_split
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.model_selection import train_test_split
 
 # splits the training and test data set in 80% : 20%
 # assign random_state to any value.This ensures consistency.
@@ -495,13 +623,30 @@ X_train, X_test, Y_train, Y_test = train_tes
 print(X_test.shape)
 print(Y_train.shape)
 print(Y_test.shape)
-
    
-
    
-Then we use the linear regression functionality from Scikit-Learn
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Then we use the linear regression functionality from Scikit-Learn

    -
    from sklearn.linear_model import LinearRegression
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.linear_model import LinearRegression
 from sklearn.metrics import mean_squared_error, r2_score
 
 lin_model = LinearRegression()
@@ -532,16 +677,45 @@ r2 = r2_score(Y_test, y_test_predict)
 print("--------------------------------------")
 print('RMSE is {}'.format(rmse))
 print('R2 score is {}'.format(r2))
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    -
    # plotting the y_test vs y_pred
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # plotting the y_test vs y_pred
 # ideally should have been a straight line
 plt.scatter(Y_test, y_test_predict)
 plt.show()
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs027.html b/doc/pub/week35/html/._week35-bs027.html index b487508ae..4fe3f287f 100644 --- a/doc/pub/week35/html/._week35-bs027.html +++ b/doc/pub/week35/html/._week35-bs027.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,47 +371,40 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Reducing the number of degrees of freedom, overarching view

    -

    + -

    -Many Machine Learning problems involve thousands or even millions of +

    Many Machine Learning problems involve thousands or even millions of features for each training instance. Not only does this make training extremely slow, it can also make it much harder to find a good solution, as we will see. This problem is often referred to as the curse of dimensionality. Fortunately, in real-world problems, it is often possible to reduce the number of features considerably, turning an intractable problem into a tractable one. +

    -

    -Later we will discuss some of the most popular dimensionality reduction +

    Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). +Locally Linear Embedding (LLE). +

    -

    -Principal component analysis and its various variants deal with the +

    Principal component analysis and its various variants deal with the problem of fitting a low-dimensional affine subspace to a set of of data points in a high-dimensional space. With its family of methods it is one of the most used tools in data modeling, compression and visualization. - -

    +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs028.html b/doc/pub/week35/html/._week35-bs028.html index 8ecf956ec..76acc5ef5 100644 --- a/doc/pub/week35/html/._week35-bs028.html +++ b/doc/pub/week35/html/._week35-bs028.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,42 +371,35 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Preprocessing our data

    -

    + -

    -Before we proceed however, we will discuss how to preprocess our +

    Before we proceed however, we will discuss how to preprocess our data. Till now and in connection with our previous examples we have not met so many cases where we are too sensitive to the scaling of our data. Normally the data may need a rescaling and/or may be sensitive to extreme values. Scaling the data renders our inputs much more suitable for the algorithms we want to employ. +

    -

    -For data sets gathered for real world applications, it is rather normal that +

    For data sets gathered for real world applications, it is rather normal that different features have very different units and numerical scales. For example, a data set detailing health habits may include features such as age in the range \( 0-80 \), and caloric intake of order \( 2000 \). Many machine learning methods sensitive to the scales of the features and may perform poorly if they are very different scales. Therefore, it is typical to scale the features in a way to avoid such outlier values. - -

    +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs029.html b/doc/pub/week35/html/._week35-bs029.html index cbdc2688d..3e9f20fc8 100644 --- a/doc/pub/week35/html/._week35-bs029.html +++ b/doc/pub/week35/html/._week35-bs029.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,18 +371,13 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Functionality in Scikit-Learn

    -

    -Scikit-Learn has several functions which allow us to rescale the +

    Scikit-Learn has several functions which allow us to rescale the data, normally resulting in much better results in terms of various accuracy scores. The StandardScaler function in Scikit-Learn ensures that for each feature/predictor we study the mean value is @@ -399,8 +386,8 @@ matrix). This scaling has the drawback that it does not ensure that we have a particular maximum or minimum in our data set. Another function included in Scikit-Learn is the MinMaxScaler which ensures that all features are exactly between \( 0 \) and \( 1 \). The +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs030.html b/doc/pub/week35/html/._week35-bs030.html index 99229ca9c..032375288 100644 --- a/doc/pub/week35/html/._week35-bs030.html +++ b/doc/pub/week35/html/._week35-bs030.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,30 +371,25 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    More preprocessing

    -

    -

    -The Normalizer scales each data + +

    The Normalizer scales each data point such that the feature vector has a euclidean length of one. In other words, it projects a data point on the circle (or sphere in the case of higher dimensions) with a radius of 1. This means every data point is scaled by a different number (by the inverse of it’s length). This normalization is often used when only the direction (or angle) of the data matters, not the length of the feature vector. +

    -

    -The RobustScaler works similarly to the StandardScaler in that it +

    The RobustScaler works similarly to the StandardScaler in that it ensures statistical properties for each feature that guarantee that they are on the same scale. However, the RobustScaler uses the median and quartiles, instead of mean and variance. This makes the @@ -410,13 +397,11 @@ RobustScaler ignore data points that are very different from the rest (like measurement errors). These odd data points are also called outliers, and might often lead to trouble for other scaling techniques. - -

    +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs031.html b/doc/pub/week35/html/._week35-bs031.html index 052a5b9d0..44dd180dc 100644 --- a/doc/pub/week35/html/._week35-bs031.html +++ b/doc/pub/week35/html/._week35-bs031.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,28 +371,24 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Frequently used scaling functions

    -

    -Many features are often scaled using standardization to improve performance. In Scikit-Learn this is given by the StandardScaler function as discussed above. It is easy however to write your own. +

    Many features are often scaled using standardization to improve performance. In Scikit-Learn this is given by the StandardScaler function as discussed above. It is easy however to write your own. Mathematically, this involves subtracting the mean and divide by the standard deviation over the data set, for each feature: +

    $$ x_j^{(i)} \rightarrow \frac{x_j^{(i)} - \overline{x}_j}{\sigma(x_j)}, $$ -where \( \overline{x}_j \) and \( \sigma(x_j) \) are the mean and standard deviation, respectively, of the feature \( x_j \). +

    where \( \overline{x}_j \) and \( \sigma(x_j) \) are the mean and standard deviation, respectively, of the feature \( x_j \). This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs032.html b/doc/pub/week35/html/._week35-bs032.html index b9011ece2..36b8eb255 100644 --- a/doc/pub/week35/html/._week35-bs032.html +++ b/doc/pub/week35/html/._week35-bs032.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,26 +371,26 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Example of own Standard scaling

    -

    -Let us consider the following vanilla example where we use both +

    Let us consider the following vanilla example where we use both Scikit-Learn and write our own function as well. We produce a simple test design matrix with random numbers. Each column could then represent a specific feature whose mean value is subracted. +

    -

    -

    import sklearn.linear_model as skl
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import sklearn.linear_model as skl
 from sklearn.metrics import mean_squared_error
 from sklearn.model_selection import  train_test_split
 from sklearn.preprocessing import MinMaxScaler, StandardScaler, Normalizer
@@ -421,11 +413,23 @@ scaler = StandardScaler(with_std.fit(X)
 Xscaled = scaler.transform(X)
 display(XPandas-Xscaled)
-
    
-
    
-Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.

    -

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs033.html b/doc/pub/week35/html/._week35-bs033.html index 2f641852d..8fd9e9ed1 100644 --- a/doc/pub/week35/html/._week35-bs033.html +++ b/doc/pub/week35/html/._week35-bs033.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,29 +371,24 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Min-Max Scaling

    -

    -Another commonly used scaling method is min-max scaling. This is very +

    Another commonly used scaling method is min-max scaling. This is very useful for when we want the features to lie in a certain interval. To scale the feature \( x_j \) to the interval \( [a, b] \), we can apply the transformation +

    $$ x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} - a $$ -where \( \min(x_j) \) and \( \max(x_j) \) return the minimum and maximum value of \( x_j \) over the data set, respectively. +

    where \( \min(x_j) \) and \( \max(x_j) \) return the minimum and maximum value of \( x_j \) over the data set, respectively.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs034.html b/doc/pub/week35/html/._week35-bs034.html index 436998e5e..ea05a5100 100644 --- a/doc/pub/week35/html/._week35-bs034.html +++ b/doc/pub/week35/html/._week35-bs034.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,40 +371,56 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Testing the Means Squared Error as function of Complexity

    -One of +

    One of the aims is to reproduce Figure 2.11 of Hastie et al. -We will also use Ridge and Lasso regression. +We will also use Ridge and Lasso regression. +

    -

    -Our data is defined by \( x\in [-3,3] \) with a total of for example \( 100 \) data points. -

    +

    Our data is defined by \( x\in [-3,3] \) with a total of for example \( 100 \) data points.

    -
    np.random.seed()
+
    
+  
    
+    
    
+      
    
+        
    
+  
    np.random.seed()
 n = 100
 maxdegree = 14
 # Make data set.
 x = np.linspace(-3, 3, n).reshape(-1, 1)
 y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
    
-
    
-where \( y \) is the function we want to fit with a given polynomial.
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + -

    -Write a first code which sets up a design matrix \( X \) defined by a fifth-order polynomial. Scale your data and split it in training and test data. -

    +

    where \( y \) is the function we want to fit with a given polynomial.

    + +

    Write a first code which sets up a design matrix \( X \) defined by a fifth-order polynomial. Scale your data and split it in training and test data.

    -
    import matplotlib.pyplot as plt
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import matplotlib.pyplot as plt
 import numpy as np
 from sklearn.linear_model import LinearRegression, Ridge, Lasso
 from sklearn.preprocessing import PolynomialFeatures
@@ -448,8 +456,22 @@ plt.plot(polydegree, TestError, label.plot(polydegree, TrainError, label='Train Error')
 plt.legend()
 plt.show()
-
    
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs035.html b/doc/pub/week35/html/._week35-bs035.html index b8ab58356..1e38b42c5 100644 --- a/doc/pub/week35/html/._week35-bs035.html +++ b/doc/pub/week35/html/._week35-bs035.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,20 +371,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    More preprocessing examples, Franke function and regression

    -

    -

    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import os
 import numpy as np
 import pandas as pd
@@ -484,8 +476,22 @@ clf = skl.print("MSE after  scaling: {:.2f}".format(mean_squared_error(clf.predict(X_test_scaled), y_test)))
 print("R2 score for  scaled data: {:.2f}".format(clf.score(X_test_scaled,y_test)))
-
    
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs036.html b/doc/pub/week35/html/._week35-bs036.html index 4c11d7b80..8a1590040 100644 --- a/doc/pub/week35/html/._week35-bs036.html +++ b/doc/pub/week35/html/._week35-bs036.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,53 +371,42 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Mathematical Interpretation of Ordinary Least Squares

    -

    -What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). +

    What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

    -

    -We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by +

    We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by

    $$ \hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

    The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.

    -

    -This means that our best model is defined as +

    This means that our best model is defined as

    $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -We now define a matrix +

    We now define a matrix

    $$ \boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$ -

    -We can rewrite +

    We can rewrite

    $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$ -

    -The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. +

    The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs037.html b/doc/pub/week35/html/._week35-bs037.html index 5e37eeae0..4817e7bd5 100644 --- a/doc/pub/week35/html/._week35-bs037.html +++ b/doc/pub/week35/html/._week35-bs037.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,26 +371,19 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Residual Error

    -

    -We have defined the residual error as +

    We have defined the residual error as

    $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}. $$ -

    -The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \). +

    The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs038.html b/doc/pub/week35/html/._week35-bs038.html index c2f147ceb..9fed27ee2 100644 --- a/doc/pub/week35/html/._week35-bs038.html +++ b/doc/pub/week35/html/._week35-bs038.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,38 +371,30 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Simple case

    -

    -If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have +

    If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}. $$ -

    -In this case the matrix \( \boldsymbol{A} \) becomes +

    In this case the matrix \( \boldsymbol{A} \) becomes

    $$ \boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ -and we have the obvious case +

    and we have the obvious case

    $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ -

    -This serves also as a useful test of our codes. +

    This serves also as a useful test of our codes.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs039.html b/doc/pub/week35/html/._week35-bs039.html index 043422126..39c486f19 100644 --- a/doc/pub/week35/html/._week35-bs039.html +++ b/doc/pub/week35/html/._week35-bs039.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,62 +371,54 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    The singular value decomposition

    -

    -

    + -

    -The examples we have looked at so far are cases where we normally can +

    The examples we have looked at so far are cases where we normally can invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky decomposition. +

    -

    -As we will also see in the first project, +

    As we will also see in the first project, this may however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +

    -

    -There is however a way to circumvent this problem and also +

    There is however a way to circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +

    -

    -This is given by the Singular Value Decomposition (SVD) algorithm, +

    This is given by the Singular Value Decomposition (SVD) algorithm, perhaps the most powerful linear algebra algorithm. The SVD provides a numerically stable matrix decomposition that is used in a large swath oc applications and the decomposition is always stable numerically. +

    -

    -In machine learning it plays a central role in dealing with for +

    In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - -

    +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs040.html b/doc/pub/week35/html/._week35-bs040.html index 23ce5aec9..b3534289e 100644 --- a/doc/pub/week35/html/._week35-bs040.html +++ b/doc/pub/week35/html/._week35-bs040.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,23 +371,19 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Linear Regression Problems

    -

    -One of the typical problems we encounter with linear regression, in particular +

    One of the typical problems we encounter with linear regression, in particular when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) may be linearly dependent, normally referred to as super-collinearity. This means that the matrix may be rank deficient and it is basically impossible to to model the data using linear regression. As an example, consider the matrix +

    $$ \begin{align*} \mathbf{X} & = \left[ @@ -411,16 +399,16 @@ $$ \end{align*} $$ -

    -The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +

    The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the the first column is the row-wise sum of the other two columns. The rank (more correct, the column rank) of a matrix is the dimension of the space spanned by the column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number of linearly independent columns. In this particular case the matrix has rank 2. +

    -

    -Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +

    Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +

    $$ \begin{align*} \boldsymbol{X} & = \left[ @@ -432,10 +420,10 @@ $$ \end{align*} $$ -We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +

    We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs041.html b/doc/pub/week35/html/._week35-bs041.html index fc1b01818..b6d2dd85e 100644 --- a/doc/pub/week35/html/._week35-bs041.html +++ b/doc/pub/week35/html/._week35-bs041.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,18 +371,13 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Fixing the singularity

    -

    -If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +

    If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem

    $$ \begin{align} \boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, @@ -398,21 +385,20 @@ $$ \end{align} $$ -has linearly dependent column vectors, we will not be able to compute the inverse +

    has linearly dependent column vectors, we will not be able to compute the inverse of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where the regression parameters \( \beta_i \) cannot be estimated. +

    -

    -A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +

    A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change

    $$ \boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, $$ -where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. +

    where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs042.html b/doc/pub/week35/html/._week35-bs042.html index 4813ccc42..9c5c42553 100644 --- a/doc/pub/week35/html/._week35-bs042.html +++ b/doc/pub/week35/html/._week35-bs042.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,40 +371,35 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Basic math of the SVD

    -

    -From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +

    From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). The matrix has then a set of eigenpairs +

    $$ (\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), $$ -and the eigenvalues are given by the diagonal matrix +

    and the eigenvalues are given by the diagonal matrix

    $$ \boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). $$ -The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +

    The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \)

    $$ \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). +

    with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \).

    -

    -Not all square matrices are diagonalizable. A matrix like the one discussed above +

    Not all square matrices are diagonalizable. A matrix like the one discussed above

    $$ \boldsymbol{X} = \begin{bmatrix} 1& -1 \\ @@ -420,10 +407,10 @@ $$ \end{bmatrix} $$ -is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition -\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs043.html b/doc/pub/week35/html/._week35-bs043.html index 5b321c1dd..6ae948f18 100644 --- a/doc/pub/week35/html/._week35-bs043.html +++ b/doc/pub/week35/html/._week35-bs043.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,18 +371,13 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    The SVD, a Fantastic Algorithm

    -

    -However, and this is the strength of the SVD algorithm, any general +

    However, and this is the strength of the SVD algorithm, any general matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and two orthogonal/unitary matrices. The Singular Value Decompostion (SVD) theorem @@ -399,30 +386,29 @@ terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( m\time and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has dimensionality \( m \times m \) and the last dimensionality \( n\times n \). We have then +

    $$ \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T $$ -

    -As an example, the above defective matrix can be decomposed as +

    As an example, the above defective matrix can be decomposed as

    $$ \boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -

    -with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). -The SVD exits always! +

    with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! +

    -

    -The SVD +

    The SVD decomposition (singular values) gives eigenvalues \( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=p \), the eigenvalues (singular values) are zero. +

    -

    -In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +

    In the general case, where our design matrix \( \boldsymbol{X} \) has dimension \( n\times p \), the matrix is thus decomposed into an \( n\times n \) orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) @@ -432,11 +418,10 @@ assuming that \( n > p \). In our regression examples for the nuclear masses and the equation of state this is indeed the case, while for the Ising model we have \( p > n \). These are often cases that lead to near singular or singular matrices. +

    -

    -The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. +

    The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs044.html b/doc/pub/week35/html/._week35-bs044.html index 52f9c4fad..ce5f5ea90 100644 --- a/doc/pub/week35/html/._week35-bs044.html +++ b/doc/pub/week35/html/._week35-bs044.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,37 +371,32 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Economy-size SVD

    -

    -If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +

    If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n \times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however irrelevant in our calculations since they are multiplied with the zeros in \( \boldsymbol{\Sigma} \). +

    -

    -The economy-size decomposition removes extra rows or columns of zeros +

    The economy-size decomposition removes extra rows or columns of zeros from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. Removing these zeros and columns can improve execution time and reduce storage requirements without compromising the accuracy of the decomposition. +

    -

    -If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +

    If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). The \( n=p \) case is obvious, we retain the full SVD. In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs045.html b/doc/pub/week35/html/._week35-bs045.html index 79a976c09..6e83a9f37 100644 --- a/doc/pub/week35/html/._week35-bs045.html +++ b/doc/pub/week35/html/._week35-bs045.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,20 +371,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Codes for the SVD

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
@@ -421,9 +413,22 @@ X = np.a
 C = SVD(X)
 # Print the difference between the original matrix and the SVD one
 print(C-X)
-
    
-
    
-The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first column is the row-wise sum of the other two columns. The rank of a matrix (the column rank) is the dimension of space spanned by the column vectors. The rank of the matrix is the number of linearly @@ -431,8 +436,8 @@ independent columns, in this case just \( 2 \). We see this from the singular values when running the above code. Running the standard inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results in the program terminating due to a singular matrix. +

    -

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs046.html b/doc/pub/week35/html/._week35-bs046.html index 0c35a1028..800e39296 100644 --- a/doc/pub/week35/html/._week35-bs046.html +++ b/doc/pub/week35/html/._week35-bs046.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,33 +371,28 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Note about SVD Calculations

    -

    -The \( U \), \( S \), and \( V \) matrices returned from the svd() function +

    The \( U \), \( S \), and \( V \) matrices returned from the svd() function cannot be multiplied directly. +

    -

    -As you can see from the code, the \( S \) vector must be converted into a +

    As you can see from the code, the \( S \) vector must be converted into a diagonal matrix. This may cause a problem as the size of the matrices do not fit the rules of matrix multiplication, where the number of columns in a matrix must match the number of rows in the subsequent matrix. +

    -

    -If you wish to include the zero singular values, you will need to +

    If you wish to include the zero singular values, you will need to resize the matrices and set up a diagonal matrix as done in the above example +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs047.html b/doc/pub/week35/html/._week35-bs047.html index a82021dfd..c278aa85d 100644 --- a/doc/pub/week35/html/._week35-bs047.html +++ b/doc/pub/week35/html/._week35-bs047.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,23 +371,16 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Friday September 3

    -

    -Video of Lecture from 2020 and handwritten notes +

    Video of Lecture from 2020 and handwritten notes

    -

    -More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021. +

    More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs048.html b/doc/pub/week35/html/._week35-bs048.html index ecfb4eb31..91f5fc815 100644 --- a/doc/pub/week35/html/._week35-bs048.html +++ b/doc/pub/week35/html/._week35-bs048.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,21 +371,15 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Mathematics of the SVD and implications

    -

    -Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. +

    Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.

    -

    -Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \) +

    Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)

    $$ \boldsymbol{X}=\begin{bmatrix} x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ @@ -405,28 +391,23 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ \end{bmatrix}. $$ -

    -We can SVD decompose our matrix as +

    We can SVD decompose our matrix as

    $$ \boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \). +

    where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).

    -

    -Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \). +

    Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).

    -

    -Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is +

    Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is

    $$ \sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. $$ -

    -All values beyond \( p-1 \) are all zero. +

    All values beyond \( p-1 \) are all zero.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs049.html b/doc/pub/week35/html/._week35-bs049.html index d5a9bf5cb..417930650 100644 --- a/doc/pub/week35/html/._week35-bs049.html +++ b/doc/pub/week35/html/._week35-bs049.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,18 +371,13 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Example Matrix

    -

    -As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \) +

    As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)

    $$ \boldsymbol{\Sigma}= @@ -401,8 +388,7 @@ $$ \end{bmatrix} $$ -

    -The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as +

    The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as

    $$ \boldsymbol{\Sigma}= @@ -412,8 +398,7 @@ $$ \end{bmatrix}, $$ -

    -where +

    where

    $$ \boldsymbol{\tilde{\Sigma}}= \begin{bmatrix} @@ -422,7 +407,7 @@ $$ \end{bmatrix}, $$ -contains only the singular values. Note also (and we will use this below) that +

    contains only the singular values. Note also (and we will use this below) that

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= @@ -432,7 +417,7 @@ $$ \end{bmatrix}, $$ -which is a \( 2\times 2 \) matrix while +

    which is a \( 2\times 2 \) matrix while

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= \begin{bmatrix} @@ -442,12 +427,11 @@ $$ \end{bmatrix}, $$ -

    -is a \( 3\times 3 \) matrix. The last row and column of this last matrix +

    is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs050.html b/doc/pub/week35/html/._week35-bs050.html index f274153b0..efb51d14f 100644 --- a/doc/pub/week35/html/._week35-bs050.html +++ b/doc/pub/week35/html/._week35-bs050.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,69 +371,59 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Setting up the Matrix to be inverted

    -

    -The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as +

    The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -and using the orthogonality of the matrix \( \boldsymbol{U} \) we have +

    and using the orthogonality of the matrix \( \boldsymbol{U} \) we have

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \). +

    We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).

    -

    -This means, using the orthogonality of \( \boldsymbol{V} \), that we get +

    This means, using the orthogonality of \( \boldsymbol{V} \), that we get

    $$ \boldsymbol{X}^T\boldsymbol{X}=\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where +

    We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -and using our SVD decomposition of \( \boldsymbol{X} \) we have +

    and using our SVD decomposition of \( \boldsymbol{X} \) we have

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\tilde{\boldsymbol{\Sigma}}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, $$ -which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), +

    which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y}, $$ -

    -Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \) +

    Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)

    $$ \boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}], $$ -that belong to \( i>p-1 \), result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero). +

    that belong to \( i>p-1 \), result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).

    -

    -It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \). +

    It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs051.html b/doc/pub/week35/html/._week35-bs051.html index 1333ac3c8..9692ca676 100644 --- a/doc/pub/week35/html/._week35-bs051.html +++ b/doc/pub/week35/html/._week35-bs051.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,65 +371,58 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Further properties (important for our analyses later)

    -

    -Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, +

    Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -

    -If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. $$ -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have +

    Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. $$ -

    -If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get +

    If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get

    $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. $$ -This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +

    This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. $$ -

    -Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an +

    Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an \( n\times p \) matrix. In most supervised learning cases we have that \( n \ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). +

    -

    -In our lectures, the number of columns will +

    In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences for the definition of for example the covariance matrix and its relation to the SVD. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs052.html b/doc/pub/week35/html/._week35-bs052.html index b81ca182d..841981e05 100644 --- a/doc/pub/week35/html/._week35-bs052.html +++ b/doc/pub/week35/html/._week35-bs052.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,44 +371,37 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Meet the Covariance Matrix

    -

    -Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above. +

    Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.

    -

    -We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +

    We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have +

    $$ \frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize). +

    This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).

    -

    -The Hessian matrix plays an important role and is defined in this course as +

    The Hessian matrix plays an important role and is defined in this course as

    $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -The Hessian matrix for ordinary least squares is also proportional to +

    The Hessian matrix for ordinary least squares is also proportional to the covariance matrix. This means also that we can use the SVD to find the eigenvalues of the covariance matrix and the Hessian matrix in terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs053.html b/doc/pub/week35/html/._week35-bs053.html index 26eab7d24..12ab8d2f1 100644 --- a/doc/pub/week35/html/._week35-bs053.html +++ b/doc/pub/week35/html/._week35-bs053.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,48 +371,43 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Introducing the Covariance and Correlation functions

    -

    -Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about +

    Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods. +

    -

    -Suppose we have defined two vectors +

    Suppose we have defined two vectors \( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    $$ \boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ \end{bmatrix}, $$ -where for example +

    where for example

    $$ \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). $$ -With this definition and recalling that the variance is defined as +

    With this definition and recalling that the variance is defined as

    $$ \mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, $$ -we can rewrite the covariance matrix as +

    we can rewrite the covariance matrix as

    $$ \boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ \end{bmatrix}. $$ -

    -Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +

    Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. What you will find in essentially all statistics texts are equations with a factor \( 1/(n-1) \). This is called Bessel's correction. This method corrects the bias in the estimation of the population variance @@ -428,8 +415,8 @@ and covariance. It also partially corrects the bias in the estimation of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs054.html b/doc/pub/week35/html/._week35-bs054.html index af9600ba6..f7c64d982 100644 --- a/doc/pub/week35/html/._week35-bs054.html +++ b/doc/pub/week35/html/._week35-bs054.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,32 +371,28 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Covariance and Correlation Matrix

    -

    -The covariance takes values between zero and infinity and may thus +

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +

    $$ \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. $$ -

    -The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as +

    $$ \boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ @@ -412,10 +400,8 @@ $$ \end{bmatrix}, $$ -

    -In the above example this is the function we constructed using pandas. +

    In the above example this is the function we constructed using pandas.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs055.html b/doc/pub/week35/html/._week35-bs055.html index d669aa053..0c84e8e04 100644 --- a/doc/pub/week35/html/._week35-bs055.html +++ b/doc/pub/week35/html/._week35-bs055.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,19 +371,15 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Correlation Function and Design/Feature Matrix

    -

    -In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as +

    $$ \boldsymbol{X}=\begin{bmatrix} @@ -404,23 +392,24 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ \end{bmatrix}, $$ -with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the entries \( n \) being the row elements. We can rewrite the design/feature matrix in terms of its column vectors as +

    $$ \boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, $$ -with a given vector +

    with a given vector

    $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$ -

    -With these definitions, we can now rewrite our \( 2\times 2 \) +

    With these definitions, we can now rewrite our \( 2\times 2 \) correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    $$ \boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} @@ -433,7 +422,7 @@ $$ \end{bmatrix}, $$ -and the correlation matrix +

    and the correlation matrix

    $$ \boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ @@ -445,7 +434,7 @@ $$ \end{bmatrix}, $$ -

    +

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs056.html b/doc/pub/week35/html/._week35-bs056.html index e0bbf6598..f4f487e95 100644 --- a/doc/pub/week35/html/._week35-bs056.html +++ b/doc/pub/week35/html/._week35-bs056.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,42 +371,41 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Covariance Matrix Examples

    -

    -The Numpy function np.cov calculates the covariance elements using +

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    -

    -Note that this assumes you have the features as the rows, and the inputs as columns, that is +

    Note that this assumes you have the features as the rows, and the inputs as columns, that is

    $$ \boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ -

    -which in turn is converted into into the \( 2\times 2 \) covariance matrix +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix \( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy function np.mean(x). We can also extract the eigenvalues of the covariance matrix through the np.linalg.eig() function. +

    -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 import numpy as np
 n = 100
 x = np.random.normal(size=n)
@@ -424,8 +415,22 @@ y = 4+3*
 W = np.vstack((x, y))
 C = np.cov(W)
 print(C)
-
    
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs057.html b/doc/pub/week35/html/._week35-bs057.html index 1357f43e3..1bf34488c 100644 --- a/doc/pub/week35/html/._week35-bs057.html +++ b/doc/pub/week35/html/._week35-bs057.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,27 +371,27 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Correlation Matrix

    -

    -The previous example can be converted into the correlation matrix by +

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also subtract the mean values for each column. This leads to the following code which sets up the correlations matrix for the previous example in -a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 n = 100
 # define two vectors                                                                                           
 x = np.random.random(size=n)
@@ -420,16 +412,28 @@ C[1,1]0,1]= cov_xy/np.sqrt(variance_y*variance_x)
 C[1,0]= C[0,1]
 print(C)
-
    
-
    
-We see that the matrix elements along the diagonal are one as they
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We see that the matrix elements along the diagonal are one as they should be and that the matrix is symmetric. Furthermore, diagonalizing this matrix we easily see that it is a positive definite matrix. +

    -

    -The above procedure with numpy can be made more compact if we use pandas. +

    The above procedure with numpy can be made more compact if we use pandas.

    -

    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs058.html b/doc/pub/week35/html/._week35-bs058.html index ebeaa6b31..e0a360ae8 100644 --- a/doc/pub/week35/html/._week35-bs058.html +++ b/doc/pub/week35/html/._week35-bs058.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,22 +371,21 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Correlation Matrix with Pandas

    -

    -We whow here how we can set up the correlation matrix using pandas, as done in this simple code -

    +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import pandas as pd
 n = 10
 x = np.random.normal(size=n)
@@ -408,11 +399,23 @@ Xpd = pd.print(Xpd)
 correlation_matrix = Xpd.corr()
 print(correlation_matrix)
-
    
-
    
-We expand this model to the Franke function discussed above.
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We expand this model to the Franke function discussed above.

    -

      @@ -442,25 +445,18 @@ We expand this model to the Franke function discussed above.
    • »
    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs059.html b/doc/pub/week35/html/._week35-bs059.html index 45b089c86..403f33a25 100644 --- a/doc/pub/week35/html/._week35-bs059.html +++ b/doc/pub/week35/html/._week35-bs059.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,20 +371,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Correlation Matrix with Pandas and the Franke function

    -

    -

    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import numpy as np
 import pandas as pd
 
@@ -435,19 +427,32 @@ Xpd = pd.= Xpd - Xpd.mean()
 covariance_matrix = Xpd.cov()
 print(covariance_matrix)
-
    
-
    
-We note here that the covariance is zero for the first rows and
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We note here that the covariance is zero for the first rows and columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). +

    -

    -This means that the variance for these elements will be zero and will +

    This means that the variance for these elements will be zero and will cause problems when we set up the correlation matrix. We can simply drop these elements and construct a correlation -matrix without these elements. +matrix without these elements. +

    -

      @@ -477,25 +482,18 @@ matrix without these elements.
    • »
    - - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs060.html b/doc/pub/week35/html/._week35-bs060.html index 91c082eb0..4c1a4cf00 100644 --- a/doc/pub/week35/html/._week35-bs060.html +++ b/doc/pub/week35/html/._week35-bs060.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,24 +371,18 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Rewriting the Covariance and/or Correlation Matrix

    -

    -We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$ -

    -To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \) +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    $$ \boldsymbol{X}=\begin{bmatrix} x_{00} & x_{01}\\ @@ -406,8 +392,7 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$ -

    -If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \)) +

    If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \))

    $$ \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\frac{1}{n}\begin{bmatrix} x_{00}^2+x_{10}^2 & x_{00}x_{01}+x_{10}x_{11}\\ @@ -415,19 +400,17 @@ x_{01}x_{00}+x_{11}x_{10} & x_{01}^2+x_{11}^2\\ \end{bmatrix}, $$ -which is just +

    which is just

    $$ \boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ \end{bmatrix}, $$ -where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \). +

    where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    -

    -It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    -

      @@ -455,25 +438,18 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs061.html b/doc/pub/week35/html/._week35-bs061.html index cd0033439..882a589e3 100644 --- a/doc/pub/week35/html/._week35-bs061.html +++ b/doc/pub/week35/html/._week35-bs061.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,28 +371,23 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Linking with the SVD

    -

    -We saw earlier that +

    We saw earlier that

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +

    Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is +

    where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

    $$ \tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ @@ -411,17 +398,17 @@ $$ \end{bmatrix}, $$ -meaning we can write +

    meaning we can write

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -

    +

      @@ -448,25 +435,18 @@ $$
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs062.html b/doc/pub/week35/html/._week35-bs062.html index 7cda55c54..8ecdb0b6e 100644 --- a/doc/pub/week35/html/._week35-bs062.html +++ b/doc/pub/week35/html/._week35-bs062.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,55 +371,50 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    What does it mean?

    -

    -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +

    In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that the columns of \( \boldsymbol{V} \) are the eigenvectors of \( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of \( \boldsymbol{X} \) in a descending order, it means that the column vectors \( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). +encode from the columns of \( \boldsymbol{X} \). +

    -

    -Note that these are also the eigenvectors and eigenvalues of the +

    Note that these are also the eigenvectors and eigenvalues of the Hessian matrix. +

    -

    -If we now recall the definition of the covariance matrix (not using +

    If we now recall the definition of the covariance matrix (not using Bessel's correction) we have +

    $$ \boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ -

    -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +

    meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    -

      @@ -453,25 +440,18 @@ absolute value of the eigenvalues of \( \boldsymbol{X} \).
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs063.html b/doc/pub/week35/html/._week35-bs063.html index ef73e2446..8a34331dc 100644 --- a/doc/pub/week35/html/._week35-bs063.html +++ b/doc/pub/week35/html/._week35-bs063.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,51 +371,44 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    -

    -For \( \boldsymbol{X}\boldsymbol{X}^T \) we found +

    For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -Since the matrices here have dimension \( n\times n \), we have +

    Since the matrices here have dimension \( n\times n \), we have

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -leading to +

    leading to

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -

    -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +

    Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

    $$ (\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$ -

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +

    It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

    -

    -Since we will mainly be interested in the correlations among the features +

    Since we will mainly be interested in the correlations among the features of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +

    -

      @@ -448,25 +433,18 @@ values and the column vectors of \( \boldsymbol{V} \).
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs064.html b/doc/pub/week35/html/._week35-bs064.html index 22ae7ccbf..ad6fc6a47 100644 --- a/doc/pub/week35/html/._week35-bs064.html +++ b/doc/pub/week35/html/._week35-bs064.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,70 +371,64 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Ridge and LASSO Regression

    -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -or we can state it as +

    or we can state it as

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, $$ -where we have used the definition of a norm-2 vector, that is +

    where we have used the definition of a norm-2 vector, that is

    $$ \vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -

    -By minimizing the above equation with respect to the parameters +

    By minimizing the above equation with respect to the parameters \( \boldsymbol{\beta} \) we could then obtain an analytical expression for the parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by defining a new cost function to be optimized, that is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

    -which leads to the Ridge regression minimization problem where we +

    which leads to the Ridge regression minimization problem where we require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is a finite number larger than zero. By defining +

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, $$ -

    -we have a new optimization equation +

    we have a new optimization equation

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 $$ -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    -

    -Here we have defined the norm-1 as +

    Here we have defined the norm-1 as

    $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ -

    +

      @@ -466,25 +452,18 @@ $$
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs065.html b/doc/pub/week35/html/._week35-bs065.html index aa769f0bb..bbf82b1ac 100644 --- a/doc/pub/week35/html/._week35-bs065.html +++ b/doc/pub/week35/html/._week35-bs065.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,75 +371,72 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Deriving the Ridge Regression Equations

    -

    -Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -and +

    and taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters +

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that +

    with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

    $$ \sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ -

    -with \( t \) a finite positive number. +

    with \( t \) a finite positive number.

    -

    -When we compare this with the ordinary least squares result we have +

    If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to

    +$$ +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

    In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.

    + +

    When we compare this with the ordinary least squares result we have

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

    -

    -We see that Ridge regression is nothing but the standard OLS with a +

    We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    -

    -Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

    $$ \tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$ -

    -For Ridge regression this becomes +

    For Ridge regression this becomes

    $$ \tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$ -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). +

    with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

    -

      @@ -470,25 +459,18 @@ with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs066.html b/doc/pub/week35/html/._week35-bs066.html index 6fd75b053..9bdcb712d 100644 --- a/doc/pub/week35/html/._week35-bs066.html +++ b/doc/pub/week35/html/._week35-bs066.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,34 +371,27 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Interpreting the Ridge results

    -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have +

    Since \( \lambda \geq 0 \), it means that compared to OLS, we have

    $$ \frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. $$ -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +

    Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by \( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has eigenvalues ordered in a descending way, that is \( \sigma_i \geq \sigma_{i+1} \). +

    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods. +

    For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

    -

      @@ -428,25 +413,18 @@ For small eigenvalues \( \sigma_i \) it means that their contributions become le
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs067.html b/doc/pub/week35/html/._week35-bs067.html index 0dd6aafb2..09d18adea 100644 --- a/doc/pub/week35/html/._week35-bs067.html +++ b/doc/pub/week35/html/._week35-bs067.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,49 +371,40 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    More interpretations

    -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is +

    For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

    $$ \boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -

    -In this case the standard OLS results in +

    In this case the standard OLS results in

    $$ \boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -

    -and +

    and

    $$ \boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, $$ -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +

    that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and the Ridge estimator converges to zero when the hyperparameter goes to infinity. +

    -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. +

    We will come back to more interpreations after we have gone through some of the statistical analysis part.

    -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -

    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs068.html b/doc/pub/week35/html/._week35-bs068.html index a07d68a26..085cd6c92 100644 --- a/doc/pub/week35/html/._week35-bs068.html +++ b/doc/pub/week35/html/._week35-bs068.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,43 +371,36 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Deriving the Lasso Regression Equations

    -

    -Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function +

    Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

    -Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +

    Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

    $$ \frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. $$ -we have that the derivative of the cost function is +

    we have that the derivative of the cost function is

    $$ \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, $$ -and reordering we have +

    and reordering we have

    $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. $$ -This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. +

    This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    -

      @@ -435,25 +420,18 @@ This equation does not lead to a nice analytical equation as in either Ridge reg
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week35/html/._week35-bs069.html b/doc/pub/week35/html/._week35-bs069.html index 315f94652..a06aabd22 100644 --- a/doc/pub/week35/html/._week35-bs069.html +++ b/doc/pub/week35/html/._week35-bs069.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,57 +371,70 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Exercises for week 36, September 6-10

    -

    -The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3). +

    The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3).

    -

    -

    Exercise 1: Adding Ridge and Lasso Regression

    -

    -This exercise is a continuation of exercise 2 from exercise set 1 +

    This exercise is a continuation of exercise 2 from exercise set 1 (week 35, August 30-September 3). We will use the same function to generate our data set, still staying with a simple function \( y(x) \) which we want to fit using linear regression, but now extending the analysis to include the Ridge and the Lasso regression methods. +

    -

    -We will thus again generate our own dataset for a function \( y(x) \) where +

    We will thus again generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). +

    -

    -The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). -

    +

    The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).

    -
    x = np.random.rand(100)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = np.random.rand(100)
 y = 2.0+5*x*x+0.1*np.random.randn(100)
-
    
-
    
-Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 3. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + -

    -The code here allows you to perform your own Ridge calculation and +

    Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 3. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).

    + +

    The code here allows you to perform your own Ridge calculation and perform calculations for various values of the regularization parameter \( \lambda \). This program can easily be extended upon. +

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -498,49 +503,61 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-Repeat the above but using the functionality of
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Repeat the above but using the functionality of Scikit-Learn. Compare your code with the results from Scikit-Learn. Remember to run with the same random numbers for generating \( x \) and \( y \). Observe also that when you compare with Scikit-Learn, you need to pay attention to how the intercept is dealt with. +

    -

    -Finally, using Scikit-Learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +

    Finally, using Scikit-Learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as

    $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$ -and the \( R^2 \) score function. +

    and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

    $$ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$ -where we have defined the mean value of \( \hat{y} \) as +

    where we have defined the mean value of \( \hat{y} \) as

    $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. - +

    Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.

    Exercise: Linear Regression for a two-dimensional function

    -

    -This is a longer exercise and the aim is to study in more detail various +

    This is a longer exercise and the aim is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, Ridge regression and finally Lasso regression. This exercise forms a part of project 1. +

    -

    -We will study how to fit polynomials to a specific +

    We will study how to fit polynomials to a specific two-dimensional function called Franke's function. This is a function which has been widely used when testing various -interpolation and fitting algorithms. +interpolation and fitting algorithms. +

    -

    -The Franke function, which is a weighted sum of four exponentials reads as follows +

    The Franke function, which is a weighted sum of four exponentials reads as follows

    $$ \begin{align*} f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ @@ -548,8 +565,7 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)} \end{align*} $$ -

    -The function will be defined for \( x,y\in [0,1] \). Our first step will +

    The function will be defined for \( x,y\in [0,1] \). Our first step will be to perform an OLS regression analysis of this function, trying out a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, x^2, y^2, xy, \dots] \). We will fit a @@ -557,13 +573,17 @@ function (for example a polynomial) of \( x \) and \( y \). Thereafter we will repeat much of the same procedure using the Ridge and Lasso regression methods, introducing thus a dependence on the bias (penalty) \( \lambda \). +

    -

    -The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) -

    +

    The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)

    -
    from mpl_toolkits.mplot3d import Axes3D
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from mpl_toolkits.mplot3d import Axes3D
 import matplotlib.pyplot as plt
 from matplotlib import cm
 from matplotlib.ticker import LinearLocator, FormatStrFormatter
@@ -602,51 +622,65 @@ ax.zaxis..colorbar(surf, shrink=0.5, aspect=5)
 
 plt.show()
-
    
-
    
-We will generate our own dataset for a function
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We will generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) with \( x,y \in [0,1] \). The function \( f(x,y) \) is the Franke function. You should explore also the addition an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). +

    -

    -Write your own code (using either a matrix inversion or a singular +

    Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. You can use scikit-learn as well. +

    -

    -Evaluate the Mean Squared error (MSE) +

    Evaluate the Mean Squared error (MSE)

    $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$ -

    -and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted +

    and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

    $$ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$ -

    -where we have defined the mean value of \( \hat{y} \) as +

    where we have defined the mean value of \( \hat{y} \) as

    $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -

    -You should split your data in train and test and also consider scaling the data. +

    You should split your data in train and test and also consider scaling the data.

    -

    -To set up the design matrix, the following code can be used -

    +

    To set up the design matrix, the following code can be used

    -
    def FrankeFunction(x,y):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def FrankeFunction(x,y):
 	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
 	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
 	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
@@ -678,24 +712,36 @@ x = np.s
 y = np.sort(np.random.uniform(0, 1, N))
 z = FrankeFunction(x, y)
 X = create_X(x, y, n=n)    
-
    
-
    
-Write then your own code for the Ridge method or use Scikit-Learn.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Write then your own code for the Ridge method or use Scikit-Learn. Perform the same analysis as you did for ordinary Least Squares (for the same polynomials) but now for different values of \( \lambda \). Compare and analyze your results with those obtained with ordinary Least Squares. Study the dependence on \( \lambda \). +

    -

    -This part is essentially a repeat of the previous ones, but now +

    This part is essentially a repeat of the previous ones, but now with Lasso regression. Write either your own code or use the functionalities of Scikit-Learn (recommended). Give a critical discussion of the three methods and a judgement of which model fits the data best. +

    -

    -

      @@ -713,25 +759,18 @@ model fits the data best.
    • 70
    - - - -
    - - - diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 4ce3129de..f6f977e68 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - + - - -
    @@ -295,7 +288,6 @@ MathJax.Hub.Config({ Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression
    -
    • @@ -379,36 +371,32 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - - - -
    -

    Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

    +
    +

    Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

    +
    -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 28, 2021

    -
    -

    Read »

    @@ -434,25 +422,18 @@ MathJax.Hub.Config({
  • »
    • -
    - - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index 3d5a63425..de9c10b7e 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -1,18 +1,17 @@ + - + + - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - - - - - - @@ -55,36 +54,81 @@ document.getElementsByTagName( 'head' )[0].appendChild( link ); - - - +
    +

    Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

    +
    - - -

    Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 28, 2021

    -
    -











    -

    Plans for week 35, August 30 -September 3

    -









    -

    Thursday September 2

    -

    -The main topics on Thursday are: - +

    The main topics on Thursday are:

    1. Repetition from last week on linear regression
    2. Discussion of how to prepare data and examples of applications of linear regression
    3. Mathematical interpretations of Linear Regression
    4. Start discussing Ridge and Lasso regression and Singular Value Decomposition
    -









    -

    Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week

    -

    -We need first a reminder from last week about linear regression. - -

    -Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \). +

    We need first a reminder from last week about linear regression.

    +

    Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).

    • Method of choice for fitting a continuous function!
    • Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
      • @@ -375,50 +364,42 @@ Fitting a continuous function with linear parameterization in terms of the param
    • Allows for easy hands-on understanding of gradient descent methods
    • and many more features
    - -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -











    -

    Regression analysis, overarching aims

    -

    -Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). -The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. - -

    -A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with +

    Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). +The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. +

    +

    A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with

    • \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    • Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    • \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    - - The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +

    The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.

    -











    -

    Regression analysis, overarching aims II

    -

    -Consider an experiment in which \( p \) characteristics of \( n \) samples are +

    Consider an experiment in which \( p \) characteristics of \( n \) samples are measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix \( \mathbf{X} \). +

    -

    -The matrix \( \mathbf{X} \) is called the design +

    The matrix \( \mathbf{X} \) is called the design matrix. Additional information of the samples is available in the form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is generally referred to as the response variable. The aim of @@ -428,73 +409,62 @@ f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of \( f(\cdot) \) is available, it is common to assume a linear relationship between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, -\beta_{p-1}]^{T} \) are the regression parameters. - -

    -Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). - +\beta_{p-1}]^{T} \) are the regression parameters. +

    +

    Linear regression gives us a set of analytical equations for the parameters \( \beta_j \).

    -











    -

    Examples

    -In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), -consider the model we discussed for describing nuclear binding energies. +

    In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. +

    -

    -There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +

    There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. Assuming +

    $$ BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, $$ -we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +

    we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a \( p\times n \) matrix \( \boldsymbol{X} \). +

    -

    -Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the -so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \). This is however a classification problem and we will come back to it when we discuss Logistic Regression. - - +

    Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \). This is however a classification problem and we will come back to it when we discuss Logistic Regression. +

    -











    -

    General linear models

    -Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. +

    Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function.

    -

    -Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +

    Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is

    $$ y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, $$ -where \( \epsilon_i \) is the error in our approximation. - - +

    where \( \epsilon_i \) is the error in our approximation.

    -











    -

    Rewriting the fitting procedure as a linear algebra problem

    -For every set of values \( y_i,x_i \) we have thus the corresponding set of equations +

    For every set of values \( y_i,x_i \) we have thus the corresponding set of equations

    $$ \begin{align*} y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ @@ -507,29 +477,27 @@ $$
    -











    -

    Rewriting the fitting procedure as a linear algebra problem, more details

    -Defining the vectors +

    Defining the vectors

    $$ \boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, $$ -and +

    and

    $$ \boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, $$ -and +

    and

    $$ \boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, $$ -and the design matrix +

    and the design matrix

    $$ \boldsymbol{X}= \begin{bmatrix} @@ -541,29 +509,27 @@ $$ \end{bmatrix} $$ -we can rewrite our equations as +

    we can rewrite our equations as

    $$ \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. $$ -The above design matrix is called a Vandermonde matrix. +

    The above design matrix is called a Vandermonde matrix.

    -











    -

    Generalizing the fitting procedure as a linear algebra problem

    -

    -We are obviously not limited to the above polynomial expansions. We +

    We are obviously not limited to the above polynomial expansions. We could replace the various powers of \( x \) with elements of Fourier series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions. For every set of values \( y_i,x_i \) we can then generalize the equations to +

    $$ \begin{align*} @@ -577,19 +543,17 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{ \end{align*} $$ -

    + Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case!

    -











    -

    Generalizing the fitting procedure as a linear algebra problem

    -We redefine in turn the matrix \( \boldsymbol{X} \) as +

    We redefine in turn the matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{X}= \begin{bmatrix} @@ -601,23 +565,21 @@ x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ \end{bmatrix} $$ -and without loss of generality we rewrite again our equations as +

    and without loss of generality we rewrite again our equations as

    $$ \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. $$ -The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

    The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?

    -











    -

    Optimizing our parameters

    -We have defined the matrix \( \boldsymbol{X} \) via the equations +

    We have defined the matrix \( \boldsymbol{X} \) via the equations

    $$ \begin{align*} y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ @@ -630,29 +592,27 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{ \end{align*} $$ -

    -As we noted above, we stayed with a system with the design matrix +

    As we noted above, we stayed with a system with the design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. - - +

    -











    -

    Our model for the nuclear binding energies

    -

    -In our introductory notes we looked at the so-called liquid drop model. Let us remind ourselves about what we did by looking at the code. +

    In our introductory notes we looked at the so-called liquid drop model. Let us remind ourselves about what we did by looking at the code.

    -

    -We restate the parts of the code we are most interested in. -

    +

    We restate the parts of the code we are most interested in.

    -
    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -721,73 +681,81 @@ DesignMatrix = pd.DataFrame(X)
 DesignMatrix.index = A
 DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
 display(DesignMatrix)
-
    
-
    
-With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as

    $$ \boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, $$ -throughout these lectures. +

    throughout these lectures.

    -











    -

    Optimizing our parameters, more details

    -With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as +

    With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as

    $$ \boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, $$ -and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +

    and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, $$ -or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +

    or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -This function is one possible way to define the so-called cost function. +

    This function is one possible way to define the so-called cost function.

    -

    -It is also common to define +

    It is also common to define the function \( C \) as +

    $$ C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, $$ -since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

    since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.

    -











    -

    Interpretations and optimizing our parameters

    -

    -The function +

    The function

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, $$ -can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +

    can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value +

    $$ y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, $$ -

    -where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +

    where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an @@ -795,57 +763,52 @@ approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable. +

    -

    -In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +

    In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -In practical terms it means we will require +

    In practical terms it means we will require

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, $$ -which results in +

    which results in

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ -or in a matrix-vector form as +

    or in a matrix-vector form as

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$ - -
    -











    -

    Interpretations and optimizing our parameters

    -We can rewrite +

    We can rewrite

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$ -as +

    as

    $$ \boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, $$ -and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +

    and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution

    $$ \boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -We note also that since our design matrix is defined as \( \boldsymbol{X}\in +

    We note also that since our design matrix is defined as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in {\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), in our case \( p=5 \) meaning that we end up with inverting a small @@ -854,26 +817,24 @@ matrices to invert. The methods discussed here and for many other supervised learning algorithms like classification with logistic regression or support vector machines, exhibit dimensionalities which allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix -\( \boldsymbol{X}^T\boldsymbol{X} \). +\( \boldsymbol{X}^T\boldsymbol{X} \). +

    -

    -Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)? What kind of problems can we expect? +

    Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)? What kind of problems can we expect?

    -











    -

    Some useful matrix and vector expressions

    -

    -The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and +

    The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. +

    $$ \frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b}, @@ -891,99 +852,128 @@ $$ \frac{\partial \log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}} = (\boldsymbol{A}^{-1})^T. $$ -

    -









    +









    Meet the Hessian Matrix

    -

    -A very important matrix we will meet again and again in Machine +

    A very important matrix we will meet again and again in Machine Learning is the Hessian. It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, +

    $$ \frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -The Hessian matrix plays an important role and is defined here as +

    The Hessian matrix plays an important role and is defined here as

    $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -For ordinary least squares, it is inversely proportional (derivation +

    For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. +

    -

    -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). +

    Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).

    -











    -

    Interpretations and optimizing our parameters

    -The residuals \( \boldsymbol{\epsilon} \) are in turn given by +

    The residuals \( \boldsymbol{\epsilon} \) are in turn given by

    $$ \boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, $$ -and with +

    and with

    $$ \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, $$ -we have +

    we have

    $$ \boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, $$ -meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. - - +

    meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.

    -











    -

    Own code for Ordinary Least Squares

    -

    -It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +

    It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to write -

    +

    -
    # matrix inversion to find beta
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # matrix inversion to find beta
 beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
 # and then make the prediction
 ytilde = X @ beta
-
    
-
    
-Alternatively, you can use the least squares functionality in Numpy as
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Alternatively, you can use the least squares functionality in Numpy as

    -
    fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+
    
+  
    
+    
    
+      
    
+        
    
+  
    fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
 ytildenp = np.dot(fit,X.T)
-
    
-
    
-And finally we plot our fit with and compare with data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    And finally we plot our fit with and compare with data

    -
    Masses['Eapprox']  = ytilde
+
    
+  
    
+    
    
+      
    
+        
    
+  
    Masses['Eapprox']  = ytilde
 # Generate a plot comparing the experimental with the fitted values values.
 fig, ax = plt.subplots()
 ax.set_xlabel(r'$A = N + Z$')
@@ -995,60 +985,139 @@ ax.plot(Masses['A'], Masses["Masses2016OLS")
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Adding error analysis and training set up

    -

    -We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of Scikit-Learn in the introductory slides. +

    We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of Scikit-Learn in the introductory slides. Since we are not using Scikit-Learn here we can define our own \( R2 \) function as -

    +

    -
    def R2(y_data, y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def R2(y_data, y_model):
     return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-
    
-
    
-and we would be using it as 
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and we would be using it as

    -
    print(R2(Energies,ytilde))
-
    -

    -We can easily add our MSE score as -

    +

    +
    +
    +
    +
    + 
    print(R2(Energies,ytilde))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We can easily add our MSE score as

    -
    def MSE(y_data,y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def MSE(y_data,y_model):
     n = np.size(y_model)
     return np.sum((y_data-y_model)**2)/n
 
 print(MSE(Energies,ytilde))
-
    
-
    
-and finally the relative error as
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and finally the relative error as

    -
    def RelativeError(y_data,y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def RelativeError(y_data,y_model):
     return abs((y_data-y_model)/y_data)
 print(RelativeError(Energies, ytilde))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Splitting our Data in Training and Test data

    -

    -

    -It is normal in essentially all Machine Learning studies to split the +

    It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (sometimes also an additional validation set). Scikit-Learn has an own function for this. There is no explicit recipe for how much data should be included as training @@ -1058,20 +1127,21 @@ postpone a discussion of this splitting to the end of these notes and our discussion of the so-called bias-variance tradeoff. Here we limit ourselves to repeat the above equation of state fitting example but now splitting the data into a training set and a test set. - - +

    -











    -

    Examples

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -1109,16 +1179,33 @@ ypredict = X_test @ beta
 print(R2(y_test,ypredict))
 print("Test MSE")
 print(MSE(y_test,ypredict))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Making your own test-train splitting

    -

    -

    # equivalently in numpy
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # equivalently in numpy
 def train_test_split_numpy(inputs, labels, train_size, test_size):
     n_inputs = len(inputs)
     inputs_shuffled = inputs.copy()
@@ -1132,28 +1219,38 @@ ypredict = X_test @ beta
     Y_train, Y_test = labels_shuffled[:train_end], labels_shuffled[train_end:]
 
     return X_train, X_test, Y_train, Y_test
-
    
-
    
-But since scikit-learn has its own function for doing this and since
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    But since scikit-learn has its own function for doing this and since it interfaces easily with tensorflow and other libraries, we normally recommend using the latter functionality. +

    -

    -

    The Boston housing data example

    -

    -The Boston housing +

    The Boston housing data set was originally a part of UCI Machine Learning Repository and has been removed now. The data set is now included in Scikit-Learn's library. There are 506 samples and 13 feature (predictor) variables in this data set. The objective is to predict the value of prices of the house using the features (predictors) listed here. +

    -

    -The features/predictors are - +

    The features/predictors are

    1. CRIM: Per capita crime rate by town
    2. ZN: Proportion of residential land zoned for lots over 25000 square feet
      3. @@ -1169,81 +1266,185 @@ The features/predictors are
    3. LSTAT: Percentage of lower status of the population
    4. MEDV: Median value of owner-occupied homes in USD 1000s
    -









    -

    Housing data, the code

    -We start by importing the libraries -

    +

    We start by importing the libraries

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import matplotlib.pyplot as plt 
 
 import pandas as pd  
 import seaborn as sns 
-
    
-
    
-and load the Boston Housing DataSet from Scikit-Learn
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and load the Boston Housing DataSet from Scikit-Learn

    -

    -

    from sklearn.datasets import load_boston
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.datasets import load_boston
 
 boston_dataset = load_boston()
 
 # boston_dataset is a dictionary
 # let's check what it contains
 boston_dataset.keys()
-
    
-
    
-Then we invoke Pandas
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Then we invoke Pandas

    -
    boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
 boston.head()
 boston['MEDV'] = boston_dataset.target
-
    
-
    
-and preprocess the data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and preprocess the data

    -
    # check for missing values in all the columns
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # check for missing values in all the columns
 boston.isnull().sum()
-
    
-
    
-We can then visualize the data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We can then visualize the data

    -
    # set the size of the figure
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # set the size of the figure
 sns.set(rc={'figure.figsize':(11.7,8.27)})
 
 # plot a histogram showing the distribution of the target values
 sns.distplot(boston['MEDV'], bins=30)
 plt.show()
-
    
-
    
-It is now useful to look at the correlation matrix
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    It is now useful to look at the correlation matrix

    -
    # compute the pair wise correlation for all columns  
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # compute the pair wise correlation for all columns  
 correlation_matrix = boston.corr().round(2)
 # use the heatmap function from seaborn to plot the correlation matrix
 # annot = True to print the values inside the square
 sns.heatmap(data=correlation_matrix, annot=True)
-
    
-
    
-From the above coorelation plot we can see that MEDV is strongly correlated to LSTAT and  RM. We see also that RAD and TAX are stronly correlated, but we don't include this in our features together to avoid multi-colinearity
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    From the above coorelation plot we can see that MEDV is strongly correlated to LSTAT and RM. We see also that RAD and TAX are stronly correlated, but we don't include this in our features together to avoid multi-colinearity

    -

    -

    plt.figure(figsize=(20, 5))
+
    
+  
    
+    
    
+      
    
+        
    
+  
    plt.figure(figsize=(20, 5))
 
 features = ['LSTAT', 'RM']
 target = boston['MEDV']
@@ -1256,22 +1457,56 @@ target = boston['MEDV']
     plt.title(col)
     plt.xlabel(col)
     plt.ylabel('MEDV')
-
    
-
    
-Now we start training our model
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Now we start training our model

    -
    X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
+
    
+  
    
+    
    
+      
    
+        
    
+  
    X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
 Y = boston['MEDV']
-
    
-
    
-We split the data into training and test sets
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We split the data into training and test sets

    -

    -

    from sklearn.model_selection import train_test_split
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.model_selection import train_test_split
 
 # splits the training and test data set in 80% : 20%
 # assign random_state to any value.This ensures consistency.
@@ -1280,13 +1515,30 @@ X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = print(X_test.shape)
 print(Y_train.shape)
 print(Y_test.shape)
-
    
-
    
-Then we use the linear regression functionality from Scikit-Learn
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Then we use the linear regression functionality from Scikit-Learn

    -
    from sklearn.linear_model import LinearRegression
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.linear_model import LinearRegression
 from sklearn.metrics import mean_squared_error, r2_score
 
 lin_model = LinearRegression()
@@ -1317,85 +1569,104 @@ r2 = r2_score(Y_test, y_test_predict)
 print("--------------------------------------")
 print('RMSE is {}'.format(rmse))
 print('R2 score is {}'.format(r2))
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    -
    # plotting the y_test vs y_pred
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # plotting the y_test vs y_pred
 # ideally should have been a straight line
 plt.scatter(Y_test, y_test_predict)
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Reducing the number of degrees of freedom, overarching view

    -

    -Many Machine Learning problems involve thousands or even millions of +

    Many Machine Learning problems involve thousands or even millions of features for each training instance. Not only does this make training extremely slow, it can also make it much harder to find a good solution, as we will see. This problem is often referred to as the curse of dimensionality. Fortunately, in real-world problems, it is often possible to reduce the number of features considerably, turning an intractable problem into a tractable one. +

    -

    -Later we will discuss some of the most popular dimensionality reduction +

    Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). +Locally Linear Embedding (LLE). +

    -

    -Principal component analysis and its various variants deal with the +

    Principal component analysis and its various variants deal with the problem of fitting a low-dimensional affine subspace to a set of of data points in a high-dimensional space. With its family of methods it is one of the most used tools in data modeling, compression and visualization. - - +

    -











    -

    Preprocessing our data

    -

    -Before we proceed however, we will discuss how to preprocess our +

    Before we proceed however, we will discuss how to preprocess our data. Till now and in connection with our previous examples we have not met so many cases where we are too sensitive to the scaling of our data. Normally the data may need a rescaling and/or may be sensitive to extreme values. Scaling the data renders our inputs much more suitable for the algorithms we want to employ. +

    -

    -For data sets gathered for real world applications, it is rather normal that +

    For data sets gathered for real world applications, it is rather normal that different features have very different units and numerical scales. For example, a data set detailing health habits may include features such as age in the range \( 0-80 \), and caloric intake of order \( 2000 \). Many machine learning methods sensitive to the scales of the features and may perform poorly if they are very different scales. Therefore, it is typical to scale the features in a way to avoid such outlier values. - - +

    -











    -

    Functionality in Scikit-Learn

    -

    -Scikit-Learn has several functions which allow us to rescale the +

    Scikit-Learn has several functions which allow us to rescale the data, normally resulting in much better results in terms of various accuracy scores. The StandardScaler function in Scikit-Learn ensures that for each feature/predictor we study the mean value is @@ -1404,26 +1675,24 @@ matrix). This scaling has the drawback that it does not ensure that we have a particular maximum or minimum in our data set. Another function included in Scikit-Learn is the MinMaxScaler which ensures that all features are exactly between \( 0 \) and \( 1 \). The +

    -











    -

    More preprocessing

    -

    -The Normalizer scales each data +

    The Normalizer scales each data point such that the feature vector has a euclidean length of one. In other words, it projects a data point on the circle (or sphere in the case of higher dimensions) with a radius of 1. This means every data point is scaled by a different number (by the inverse of it’s length). This normalization is often used when only the direction (or angle) of the data matters, not the length of the feature vector. +

    -

    -The RobustScaler works similarly to the StandardScaler in that it +

    The RobustScaler works similarly to the StandardScaler in that it ensures statistical properties for each feature that guarantee that they are on the same scale. However, the RobustScaler uses the median and quartiles, instead of mean and variance. This makes the @@ -1431,42 +1700,42 @@ RobustScaler ignore data points that are very different from the rest (like measurement errors). These odd data points are also called outliers, and might often lead to trouble for other scaling techniques. - - +

    -











    -

    Frequently used scaling functions

    -

    -Many features are often scaled using standardization to improve performance. In Scikit-Learn this is given by the StandardScaler function as discussed above. It is easy however to write your own. +

    Many features are often scaled using standardization to improve performance. In Scikit-Learn this is given by the StandardScaler function as discussed above. It is easy however to write your own. Mathematically, this involves subtracting the mean and divide by the standard deviation over the data set, for each feature: +

    $$ x_j^{(i)} \rightarrow \frac{x_j^{(i)} - \overline{x}_j}{\sigma(x_j)}, $$ -where \( \overline{x}_j \) and \( \sigma(x_j) \) are the mean and standard deviation, respectively, of the feature \( x_j \). +

    where \( \overline{x}_j \) and \( \sigma(x_j) \) are the mean and standard deviation, respectively, of the feature \( x_j \). This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one. +

    -











    -

    Example of own Standard scaling

    -

    -Let us consider the following vanilla example where we use both +

    Let us consider the following vanilla example where we use both Scikit-Learn and write our own function as well. We produce a simple test design matrix with random numbers. Each column could then represent a specific feature whose mean value is subracted. +

    -

    -

    import sklearn.linear_model as skl
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import sklearn.linear_model as skl
 from sklearn.metrics import mean_squared_error
 from sklearn.model_selection import  train_test_split
 from sklearn.preprocessing import MinMaxScaler, StandardScaler, Normalizer
@@ -1489,56 +1758,85 @@ scaler = StandardScaler(with_std=
    
-
    
-Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.

    -











    -

    Min-Max Scaling

    -

    -Another commonly used scaling method is min-max scaling. This is very +

    Another commonly used scaling method is min-max scaling. This is very useful for when we want the features to lie in a certain interval. To scale the feature \( x_j \) to the interval \( [a, b] \), we can apply the transformation +

    $$ x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} - a $$ -where \( \min(x_j) \) and \( \max(x_j) \) return the minimum and maximum value of \( x_j \) over the data set, respectively. +

    where \( \min(x_j) \) and \( \max(x_j) \) return the minimum and maximum value of \( x_j \) over the data set, respectively.

    -











    -

    Testing the Means Squared Error as function of Complexity

    -One of +

    One of the aims is to reproduce Figure 2.11 of Hastie et al. -We will also use Ridge and Lasso regression. +We will also use Ridge and Lasso regression. +

    -

    -Our data is defined by \( x\in [-3,3] \) with a total of for example \( 100 \) data points. -

    +

    Our data is defined by \( x\in [-3,3] \) with a total of for example \( 100 \) data points.

    -
    np.random.seed()
+
    
+  
    
+    
    
+      
    
+        
    
+  
    np.random.seed()
 n = 100
 maxdegree = 14
 # Make data set.
 x = np.linspace(-3, 3, n).reshape(-1, 1)
 y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
    
-
    
-where \( y \) is the function we want to fit with a given polynomial.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + -

    -Write a first code which sets up a design matrix \( X \) defined by a fifth-order polynomial. Scale your data and split it in training and test data. -

    +

    where \( y \) is the function we want to fit with a given polynomial.

    + +

    Write a first code which sets up a design matrix \( X \) defined by a fifth-order polynomial. Scale your data and split it in training and test data.

    -
    import matplotlib.pyplot as plt
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import matplotlib.pyplot as plt
 import numpy as np
 from sklearn.linear_model import LinearRegression, Ridge, Lasso
 from sklearn.preprocessing import PolynomialFeatures
@@ -1574,16 +1872,33 @@ plt.plot(polydegree, TestError, label='Test Err
 plt.plot(polydegree, TrainError, label='Train Error')
 plt.legend()
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    More preprocessing examples, Franke function and regression

    -

    -

    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import os
 import numpy as np
 import pandas as pd
@@ -1675,149 +1990,140 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train)
 
 print("MSE after  scaling: {:.2f}".format(mean_squared_error(clf.predict(X_test_scaled), y_test)))
 print("R2 score for  scaled data: {:.2f}".format(clf.score(X_test_scaled,y_test)))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Mathematical Interpretation of Ordinary Least Squares

    -

    -What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). +

    What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

    -

    -We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by +

    We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by

    $$ \hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

    The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.

    -

    -This means that our best model is defined as +

    This means that our best model is defined as

    $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -We now define a matrix +

    We now define a matrix

    $$ \boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$ -

    -We can rewrite +

    We can rewrite

    $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$ -

    -The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. +

    The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix. +

    -











    -

    Residual Error

    -

    -We have defined the residual error as +

    We have defined the residual error as

    $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}. $$ -

    -The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \). +

    The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).

    -











    -

    Simple case

    -

    -If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have +

    If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}. $$ -

    -In this case the matrix \( \boldsymbol{A} \) becomes +

    In this case the matrix \( \boldsymbol{A} \) becomes

    $$ \boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ -and we have the obvious case +

    and we have the obvious case

    $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ -

    -This serves also as a useful test of our codes. +

    This serves also as a useful test of our codes.

    -











    -

    The singular value decomposition

    -

    -

    -The examples we have looked at so far are cases where we normally can +

    The examples we have looked at so far are cases where we normally can invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky decomposition. +

    -

    -As we will also see in the first project, +

    As we will also see in the first project, this may however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +

    -

    -There is however a way to circumvent this problem and also +

    There is however a way to circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +

    -

    -This is given by the Singular Value Decomposition (SVD) algorithm, +

    This is given by the Singular Value Decomposition (SVD) algorithm, perhaps the most powerful linear algebra algorithm. The SVD provides a numerically stable matrix decomposition that is used in a large swath oc applications and the decomposition is always stable numerically. +

    -

    -In machine learning it plays a central role in dealing with for +

    In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - - +

    -











    -

    Linear Regression Problems

    -

    -One of the typical problems we encounter with linear regression, in particular +

    One of the typical problems we encounter with linear regression, in particular when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) may be linearly dependent, normally referred to as super-collinearity. This means that the matrix may be rank deficient and it is basically impossible to to model the data using linear regression. As an example, consider the matrix +

    $$ \begin{align*} \mathbf{X} & = \left[ @@ -1833,16 +2139,16 @@ $$ \end{align*} $$ -

    -The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +

    The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the the first column is the row-wise sum of the other two columns. The rank (more correct, the column rank) of a matrix is the dimension of the space spanned by the column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number of linearly independent columns. In this particular case the matrix has rank 2. +

    -

    -Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +

    Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +

    $$ \begin{align*} \boldsymbol{X} & = \left[ @@ -1854,16 +2160,14 @@ $$ \end{align*} $$ -We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +

    We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. +

    -











    -

    Fixing the singularity

    -

    -If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +

    If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem

    $$ \begin{align} \boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, @@ -1871,49 +2175,46 @@ $$ \end{align} $$ -has linearly dependent column vectors, we will not be able to compute the inverse +

    has linearly dependent column vectors, we will not be able to compute the inverse of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where the regression parameters \( \beta_i \) cannot be estimated. +

    -

    -A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +

    A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change

    $$ \boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, $$ -where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. +

    where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later.

    -











    -

    Basic math of the SVD

    -

    -From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +

    From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). The matrix has then a set of eigenpairs +

    $$ (\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), $$ -and the eigenvalues are given by the diagonal matrix +

    and the eigenvalues are given by the diagonal matrix

    $$ \boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). $$ -The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +

    The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \)

    $$ \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). +

    with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \).

    -

    -Not all square matrices are diagonalizable. A matrix like the one discussed above +

    Not all square matrices are diagonalizable. A matrix like the one discussed above

    $$ \boldsymbol{X} = \begin{bmatrix} 1& -1 \\ @@ -1921,16 +2222,14 @@ $$ \end{bmatrix} $$ -is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition -\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    -











    -

    The SVD, a Fantastic Algorithm

    -

    -However, and this is the strength of the SVD algorithm, any general +

    However, and this is the strength of the SVD algorithm, any general matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and two orthogonal/unitary matrices. The Singular Value Decompostion (SVD) theorem @@ -1939,30 +2238,29 @@ terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( m\time and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has dimensionality \( m \times m \) and the last dimensionality \( n\times n \). We have then +

    $$ \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T $$ -

    -As an example, the above defective matrix can be decomposed as +

    As an example, the above defective matrix can be decomposed as

    $$ \boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -

    -with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). -The SVD exits always! +

    with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! +

    -

    -The SVD +

    The SVD decomposition (singular values) gives eigenvalues \( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=p \), the eigenvalues (singular values) are zero. +

    -

    -In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +

    In the general case, where our design matrix \( \boldsymbol{X} \) has dimension \( n\times p \), the matrix is thus decomposed into an \( n\times n \) orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) @@ -1972,44 +2270,44 @@ assuming that \( n > p \). In our regression examples for the nuclear masses and the equation of state this is indeed the case, while for the Ising model we have \( p > n \). These are often cases that lead to near singular or singular matrices. +

    -

    -The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. +

    The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.

    -











    -

    Economy-size SVD

    -

    -If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +

    If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n \times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however irrelevant in our calculations since they are multiplied with the zeros in \( \boldsymbol{\Sigma} \). +

    -

    -The economy-size decomposition removes extra rows or columns of zeros +

    The economy-size decomposition removes extra rows or columns of zeros from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. Removing these zeros and columns can improve execution time and reduce storage requirements without compromising the accuracy of the decomposition. +

    -

    -If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +

    If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). The \( n=p \) case is obvious, we retain the full SVD. In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. +

    -











    -

    Codes for the SVD

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
@@ -2038,9 +2336,22 @@ X = np.array([ [1.0,-# Print the difference between the original matrix and the SVD one
 print(C-X)
-
    
-
    
-The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first column is the row-wise sum of the other two columns. The rank of a matrix (the column rank) is the dimension of space spanned by the column vectors. The rank of the matrix is the number of linearly @@ -2048,49 +2359,40 @@ independent columns, in this case just \( 2 \). We see this from the singular values when running the above code. Running the standard inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results in the program terminating due to a singular matrix. +

    -











    -

    Note about SVD Calculations

    -

    -The \( U \), \( S \), and \( V \) matrices returned from the svd() function +

    The \( U \), \( S \), and \( V \) matrices returned from the svd() function cannot be multiplied directly. +

    -

    -As you can see from the code, the \( S \) vector must be converted into a +

    As you can see from the code, the \( S \) vector must be converted into a diagonal matrix. This may cause a problem as the size of the matrices do not fit the rules of matrix multiplication, where the number of columns in a matrix must match the number of rows in the subsequent matrix. +

    -

    -If you wish to include the zero singular values, you will need to +

    If you wish to include the zero singular values, you will need to resize the matrices and set up a diagonal matrix as done in the above example +

    -











    -

    Friday September 3

    -

    -Video of Lecture from 2020 and handwritten notes +

    Video of Lecture from 2020 and handwritten notes

    -

    -More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021. +

    More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021.

    -











    -

    Mathematics of the SVD and implications

    -

    -Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. +

    Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.

    -

    -Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \) +

    Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)

    $$ \boldsymbol{X}=\begin{bmatrix} x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ @@ -2102,34 +2404,27 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ \end{bmatrix}. $$ -

    -We can SVD decompose our matrix as +

    We can SVD decompose our matrix as

    $$ \boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \). +

    where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).

    -

    -Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \). +

    Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).

    -

    -Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is +

    Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is

    $$ \sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. $$ -

    -All values beyond \( p-1 \) are all zero. +

    All values beyond \( p-1 \) are all zero.

    -











    -

    Example Matrix

    -

    -As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \) +

    As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)

    $$ \boldsymbol{\Sigma}= @@ -2140,8 +2435,7 @@ $$ \end{bmatrix} $$ -

    -The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as +

    The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as

    $$ \boldsymbol{\Sigma}= @@ -2151,8 +2445,7 @@ $$ \end{bmatrix}, $$ -

    -where +

    where

    $$ \boldsymbol{\tilde{\Sigma}}= \begin{bmatrix} @@ -2161,7 +2454,7 @@ $$ \end{bmatrix}, $$ -contains only the singular values. Note also (and we will use this below) that +

    contains only the singular values. Note also (and we will use this below) that

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= @@ -2171,7 +2464,7 @@ $$ \end{bmatrix}, $$ -which is a \( 2\times 2 \) matrix while +

    which is a \( 2\times 2 \) matrix while

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= \begin{bmatrix} @@ -2181,190 +2474,172 @@ $$ \end{bmatrix}, $$ -

    -is a \( 3\times 3 \) matrix. The last row and column of this last matrix +

    is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. +

    -











    -

    Setting up the Matrix to be inverted

    -

    -The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as +

    The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -and using the orthogonality of the matrix \( \boldsymbol{U} \) we have +

    and using the orthogonality of the matrix \( \boldsymbol{U} \) we have

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \). +

    We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).

    -

    -This means, using the orthogonality of \( \boldsymbol{V} \), that we get +

    This means, using the orthogonality of \( \boldsymbol{V} \), that we get

    $$ \boldsymbol{X}^T\boldsymbol{X}=\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where +

    We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -and using our SVD decomposition of \( \boldsymbol{X} \) we have +

    and using our SVD decomposition of \( \boldsymbol{X} \) we have

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\tilde{\boldsymbol{\Sigma}}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, $$ -which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), +

    which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y}, $$ -

    -Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \) +

    Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)

    $$ \boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}], $$ -that belong to \( i>p-1 \), result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero). +

    that belong to \( i>p-1 \), result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).

    -

    -It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \). +

    It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

    -











    -

    Further properties (important for our analyses later)

    -

    -Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, +

    Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -

    -If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. $$ -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have +

    Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. $$ -

    -If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get +

    If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get

    $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. $$ -This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +

    This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. $$ -

    -Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an +

    Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an \( n\times p \) matrix. In most supervised learning cases we have that \( n \ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). +

    -

    -In our lectures, the number of columns will +

    In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences for the definition of for example the covariance matrix and its relation to the SVD. +

    -











    -

    Meet the Covariance Matrix

    -

    -Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above. +

    Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.

    -

    -We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +

    We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have +

    $$ \frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize). +

    This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).

    -

    -The Hessian matrix plays an important role and is defined in this course as +

    The Hessian matrix plays an important role and is defined in this course as

    $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -The Hessian matrix for ordinary least squares is also proportional to +

    The Hessian matrix for ordinary least squares is also proportional to the covariance matrix. This means also that we can use the SVD to find the eigenvalues of the covariance matrix and the Hessian matrix in terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies. +

    -











    -

    Introducing the Covariance and Correlation functions

    -

    -Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about +

    Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods. +

    -

    -Suppose we have defined two vectors +

    Suppose we have defined two vectors \( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    $$ \boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ \end{bmatrix}, $$ -where for example +

    where for example

    $$ \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). $$ -With this definition and recalling that the variance is defined as +

    With this definition and recalling that the variance is defined as

    $$ \mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, $$ -we can rewrite the covariance matrix as +

    we can rewrite the covariance matrix as

    $$ \boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ \end{bmatrix}. $$ -

    -Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +

    Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. What you will find in essentially all statistics texts are equations with a factor \( 1/(n-1) \). This is called Bessel's correction. This method corrects the bias in the estimation of the population variance @@ -2372,28 +2647,27 @@ and covariance. It also partially corrects the bias in the estimation of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). +

    -











    -

    Covariance and Correlation Matrix

    -

    -The covariance takes values between zero and infinity and may thus +

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +

    $$ \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. $$ -

    -The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as +

    $$ \boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ @@ -2401,17 +2675,14 @@ $$ \end{bmatrix}, $$ -

    -In the above example this is the function we constructed using pandas. +

    In the above example this is the function we constructed using pandas.

    -











    -

    Correlation Function and Design/Feature Matrix

    -

    -In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as +

    $$ \boldsymbol{X}=\begin{bmatrix} @@ -2424,23 +2695,24 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ \end{bmatrix}, $$ -with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the entries \( n \) being the row elements. We can rewrite the design/feature matrix in terms of its column vectors as +

    $$ \boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, $$ -with a given vector +

    with a given vector

    $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$ -

    -With these definitions, we can now rewrite our \( 2\times 2 \) +

    With these definitions, we can now rewrite our \( 2\times 2 \) correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    $$ \boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} @@ -2453,7 +2725,7 @@ $$ \end{bmatrix}, $$ -and the correlation matrix +

    and the correlation matrix

    $$ \boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ @@ -2465,37 +2737,39 @@ $$ \end{bmatrix}, $$ -

    -









    +









    Covariance Matrix Examples

    -

    -The Numpy function np.cov calculates the covariance elements using +

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    -

    -Note that this assumes you have the features as the rows, and the inputs as columns, that is +

    Note that this assumes you have the features as the rows, and the inputs as columns, that is

    $$ \boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ -

    -which in turn is converted into into the \( 2\times 2 \) covariance matrix +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix \( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy function np.mean(x). We can also extract the eigenvalues of the covariance matrix through the np.linalg.eig() function. +

    -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 import numpy as np
 n = 100
 x = np.random.normal(size=n)
@@ -2505,23 +2779,40 @@ y = 4+3*
 W = np.vstack((x, y))
 C = np.cov(W)
 print(C)
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Correlation Matrix

    -

    -The previous example can be converted into the correlation matrix by +

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also subtract the mean values for each column. This leads to the following code which sets up the correlations matrix for the previous example in -a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 n = 100
 # define two vectors                                                                                           
 x = np.random.random(size=n)
@@ -2542,26 +2833,40 @@ C[1,1]=
 C[0,1]= cov_xy/np.sqrt(variance_y*variance_x)
 C[1,0]= C[0,1]
 print(C)
-
    
-
    
-We see that the matrix elements along the diagonal are one as they
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We see that the matrix elements along the diagonal are one as they should be and that the matrix is symmetric. Furthermore, diagonalizing this matrix we easily see that it is a positive definite matrix. +

    -

    -The above procedure with numpy can be made more compact if we use pandas. +

    The above procedure with numpy can be made more compact if we use pandas.

    -











    -

    Correlation Matrix with Pandas

    -

    -We whow here how we can set up the correlation matrix using pandas, as done in this simple code -

    +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import pandas as pd
 n = 10
 x = np.random.normal(size=n)
@@ -2575,19 +2880,34 @@ Xpd = pd.DataFrame(X)
 print(Xpd)
 correlation_matrix = Xpd.corr()
 print(correlation_matrix)
-
    
-
    
-We expand this model to the Franke function discussed above.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We expand this model to the Franke function discussed above.

    -











    -

    Correlation Matrix with Pandas and the Franke function

    -

    -

    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import numpy as np
 import pandas as pd
 
@@ -2630,31 +2950,41 @@ Xpd = pd.DataFrame(X)
 Xpd = Xpd - Xpd.mean()
 covariance_matrix = Xpd.cov()
 print(covariance_matrix)
-
    
-
    
-We note here that the covariance is zero for the first rows and
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We note here that the covariance is zero for the first rows and columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). +

    -

    -This means that the variance for these elements will be zero and will +

    This means that the variance for these elements will be zero and will cause problems when we set up the correlation matrix. We can simply drop these elements and construct a correlation -matrix without these elements. +matrix without these elements. +

    -











    -

    Rewriting the Covariance and/or Correlation Matrix

    -

    -We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$ -

    -To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \) +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    $$ \boldsymbol{X}=\begin{bmatrix} x_{00} & x_{01}\\ @@ -2664,8 +2994,7 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$ -

    -If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \)) +

    If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \))

    $$ \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\frac{1}{n}\begin{bmatrix} x_{00}^2+x_{10}^2 & x_{00}x_{01}+x_{10}x_{11}\\ @@ -2673,35 +3002,31 @@ x_{01}x_{00}+x_{11}x_{10} & x_{01}^2+x_{11}^2\\ \end{bmatrix}, $$ -which is just +

    which is just

    $$ \boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ \end{bmatrix}, $$ -where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \). +

    where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    -

    -It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    -











    -

    Linking with the SVD

    -

    -We saw earlier that +

    We saw earlier that

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +

    Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is +

    where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

    $$ \tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ @@ -2712,355 +3037,348 @@ $$ \end{bmatrix}, $$ -meaning we can write +

    meaning we can write

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -









    +









    What does it mean?

    -

    -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +

    In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that the columns of \( \boldsymbol{V} \) are the eigenvectors of \( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of \( \boldsymbol{X} \) in a descending order, it means that the column vectors \( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). +encode from the columns of \( \boldsymbol{X} \). +

    -

    -Note that these are also the eigenvectors and eigenvalues of the +

    Note that these are also the eigenvectors and eigenvalues of the Hessian matrix. +

    -

    -If we now recall the definition of the covariance matrix (not using +

    If we now recall the definition of the covariance matrix (not using Bessel's correction) we have +

    $$ \boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ -

    -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +

    meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    -











    -

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    -

    -For \( \boldsymbol{X}\boldsymbol{X}^T \) we found +

    For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -Since the matrices here have dimension \( n\times n \), we have +

    Since the matrices here have dimension \( n\times n \), we have

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -leading to +

    leading to

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -

    -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +

    Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

    $$ (\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$ -

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +

    It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

    -

    -Since we will mainly be interested in the correlations among the features +

    Since we will mainly be interested in the correlations among the features of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +

    -











    -

    Ridge and LASSO Regression

    -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -or we can state it as +

    or we can state it as

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, $$ -where we have used the definition of a norm-2 vector, that is +

    where we have used the definition of a norm-2 vector, that is

    $$ \vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -

    -By minimizing the above equation with respect to the parameters +

    By minimizing the above equation with respect to the parameters \( \boldsymbol{\beta} \) we could then obtain an analytical expression for the parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by defining a new cost function to be optimized, that is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

    -which leads to the Ridge regression minimization problem where we +

    which leads to the Ridge regression minimization problem where we require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is a finite number larger than zero. By defining +

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, $$ -

    -we have a new optimization equation +

    we have a new optimization equation

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 $$ -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    -

    -Here we have defined the norm-1 as +

    Here we have defined the norm-1 as

    $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ -

    -









    +









    Deriving the Ridge Regression Equations

    -

    -Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -and +

    and taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters +

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that +

    with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

    $$ \sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ -

    -with \( t \) a finite positive number. +

    with \( t \) a finite positive number.

    -

    -When we compare this with the ordinary least squares result we have +

    If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to

    +$$ +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

    In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.

    + +

    When we compare this with the ordinary least squares result we have

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

    -

    -We see that Ridge regression is nothing but the standard OLS with a +

    We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    -

    -Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

    $$ \tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$ -

    -For Ridge regression this becomes +

    For Ridge regression this becomes

    $$ \tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$ -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). +

    with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

    -











    -

    Interpreting the Ridge results

    -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have +

    Since \( \lambda \geq 0 \), it means that compared to OLS, we have

    $$ \frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. $$ -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +

    Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by \( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has eigenvalues ordered in a descending way, that is \( \sigma_i \geq \sigma_{i+1} \). +

    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods. +

    For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

    -











    -

    More interpretations

    -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is +

    For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

    $$ \boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -

    -In this case the standard OLS results in +

    In this case the standard OLS results in

    $$ \boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -

    -and +

    and

    $$ \boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, $$ -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +

    that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and the Ridge estimator converges to zero when the hyperparameter goes to infinity. +

    -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. +

    We will come back to more interpreations after we have gone through some of the statistical analysis part.

    -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -











    -

    Deriving the Lasso Regression Equations

    -

    -Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function +

    Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

    -Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +

    Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

    $$ \frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. $$ -we have that the derivative of the cost function is +

    we have that the derivative of the cost function is

    $$ \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, $$ -and reordering we have +

    and reordering we have

    $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. $$ -This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. +

    This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    -











    -

    Exercises for week 36, September 6-10

    -

    -The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3). +

    The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3).

    -

    -

    Exercise 1: Adding Ridge and Lasso Regression

    -

    -This exercise is a continuation of exercise 2 from exercise set 1 +

    This exercise is a continuation of exercise 2 from exercise set 1 (week 35, August 30-September 3). We will use the same function to generate our data set, still staying with a simple function \( y(x) \) which we want to fit using linear regression, but now extending the analysis to include the Ridge and the Lasso regression methods. +

    -

    -We will thus again generate our own dataset for a function \( y(x) \) where +

    We will thus again generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). +

    -

    -The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). -

    +

    The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).

    -
    x = np.random.rand(100)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = np.random.rand(100)
 y = 2.0+5*x*x+0.1*np.random.randn(100)
-
    
-
    
-Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 3. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + -

    -The code here allows you to perform your own Ridge calculation and +

    Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 3. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).

    + +

    The code here allows you to perform your own Ridge calculation and perform calculations for various values of the regularization parameter \( \lambda \). This program can easily be extended upon. +

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -3129,49 +3447,61 @@ plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-Repeat the above but using the functionality of
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Repeat the above but using the functionality of Scikit-Learn. Compare your code with the results from Scikit-Learn. Remember to run with the same random numbers for generating \( x \) and \( y \). Observe also that when you compare with Scikit-Learn, you need to pay attention to how the intercept is dealt with. +

    -

    -Finally, using Scikit-Learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +

    Finally, using Scikit-Learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as

    $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$ -and the \( R^2 \) score function. +

    and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

    $$ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$ -where we have defined the mean value of \( \hat{y} \) as +

    where we have defined the mean value of \( \hat{y} \) as

    $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. - +

    Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.

    Exercise: Linear Regression for a two-dimensional function

    -

    -This is a longer exercise and the aim is to study in more detail various +

    This is a longer exercise and the aim is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, Ridge regression and finally Lasso regression. This exercise forms a part of project 1. +

    -

    -We will study how to fit polynomials to a specific +

    We will study how to fit polynomials to a specific two-dimensional function called Franke's function. This is a function which has been widely used when testing various -interpolation and fitting algorithms. +interpolation and fitting algorithms. +

    -

    -The Franke function, which is a weighted sum of four exponentials reads as follows +

    The Franke function, which is a weighted sum of four exponentials reads as follows

    $$ \begin{align*} f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ @@ -3179,8 +3509,7 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)} \end{align*} $$ -

    -The function will be defined for \( x,y\in [0,1] \). Our first step will +

    The function will be defined for \( x,y\in [0,1] \). Our first step will be to perform an OLS regression analysis of this function, trying out a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, x^2, y^2, xy, \dots] \). We will fit a @@ -3188,13 +3517,17 @@ function (for example a polynomial) of \( x \) and \( y \). Thereafter we will repeat much of the same procedure using the Ridge and Lasso regression methods, introducing thus a dependence on the bias (penalty) \( \lambda \). +

    -

    -The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) -

    +

    The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)

    -
    from mpl_toolkits.mplot3d import Axes3D
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from mpl_toolkits.mplot3d import Axes3D
 import matplotlib.pyplot as plt
 from matplotlib import cm
 from matplotlib.ticker import LinearLocator, FormatStrFormatter
@@ -3233,51 +3566,65 @@ ax.zaxis.set_major_formatter(FormatStrFormatter(
 fig.colorbar(surf, shrink=0.5, aspect=5)
 
 plt.show()
-
    
-
    
-We will generate our own dataset for a function
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We will generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) with \( x,y \in [0,1] \). The function \( f(x,y) \) is the Franke function. You should explore also the addition an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). +

    -

    -Write your own code (using either a matrix inversion or a singular +

    Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. You can use scikit-learn as well. +

    -

    -Evaluate the Mean Squared error (MSE) +

    Evaluate the Mean Squared error (MSE)

    $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$ -

    -and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted +

    and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

    $$ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$ -

    -where we have defined the mean value of \( \hat{y} \) as +

    where we have defined the mean value of \( \hat{y} \) as

    $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -

    -You should split your data in train and test and also consider scaling the data. +

    You should split your data in train and test and also consider scaling the data.

    -

    -To set up the design matrix, the following code can be used -

    +

    To set up the design matrix, the following code can be used

    -
    def FrankeFunction(x,y):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def FrankeFunction(x,y):
 	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
 	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
 	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
@@ -3309,33 +3656,40 @@ x = np.sort(np.random.uniform(0, 0, 1, N))
 z = FrankeFunction(x, y)
 X = create_X(x, y, n=n)    
-
    
-
    
-Write then your own code for the Ridge method or use Scikit-Learn.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Write then your own code for the Ridge method or use Scikit-Learn. Perform the same analysis as you did for ordinary Least Squares (for the same polynomials) but now for different values of \( \lambda \). Compare and analyze your results with those obtained with ordinary Least Squares. Study the dependence on \( \lambda \). +

    -

    -This part is essentially a repeat of the previous ones, but now +

    This part is essentially a repeat of the previous ones, but now with Lasso regression. Write either your own code or use the functionalities of Scikit-Learn (recommended). Give a critical discussion of the three methods and a judgement of which model fits the data best. +

    -

    - - -

    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 6a7d46867..c058a7206 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -1,6 +1,7 @@ @@ -8,29 +9,97 @@ Automatically generated HTML file from DocOnce source - Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression - - - - +
    +

    Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

    +
    - - -

    Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 28, 2021

    -
    -











    -

    Plans for week 35, August 30 -September 3

    -









    -

    Thursday September 2

    -

    -The main topics on Thursday are: - +

    The main topics on Thursday are:

    1. Repetition from last week on linear regression
    2. Discussion of how to prepare data and examples of applications of linear regression
    3. Mathematical interpretations of Linear Regression
    4. Start discussing Ridge and Lasso regression and Singular Value Decomposition
    -









    -

    Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week

    -

    -We need first a reminder from last week about linear regression. - -

    -Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \). +

    We need first a reminder from last week about linear regression.

    +

    Fitting a continuous function with linear parameterization in terms of the parameters \( \boldsymbol{\beta} \).

    • Method of choice for fitting a continuous function!
    • Gives an excellent introduction to central Machine Learning features with understandable pedagogical links to other methods like Neural Networks, Support Vector Machines etc
      • @@ -380,50 +441,42 @@ Fitting a continuous function with linear parameterization in terms of the param
    • Allows for easy hands-on understanding of gradient descent methods
    • and many more features
    - -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -











    -

    Regression analysis, overarching aims

    -

    -Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). -The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. - -

    -A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with +

    Regression modeling deals with the description of the sampling distribution of a given random variable \( y \) and how it varies as function of another variable or a set of such variables \( \boldsymbol{x} =[x_0, x_1,\dots, x_{n-1}]^T \). +The first variable is called the dependent, the outcome or the response variable while the set of variables \( \boldsymbol{x} \) is called the independent variable, or the predictor variable or the explanatory variable. +

    +

    A regression model aims at finding a likelihood function \( p(\boldsymbol{y}\vert \boldsymbol{x}) \), that is the conditional distribution for \( \boldsymbol{y} \) with a given \( \boldsymbol{x} \). The estimation of \( p(\boldsymbol{y}\vert \boldsymbol{x}) \) is made using a data set with

    • \( n \) cases \( i = 0, 1, 2, \dots, n-1 \)
    • Response (target, dependent or outcome) variable \( y_i \) with \( i = 0, 1, 2, \dots, n-1 \)
    • \( p \) so-called explanatory (independent or predictor) variables \( \boldsymbol{x}_i=[x_{i0}, x_{i1}, \dots, x_{ip-1}] \) with \( i = 0, 1, 2, \dots, n-1 \) and explanatory variables running from \( 0 \) to \( p-1 \). See below for more explicit examples.
    - - The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things. +

    The goal of the regression analysis is to extract/exploit relationship between \( \boldsymbol{y} \) and \( \boldsymbol{x} \) in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.

    -











    -

    Regression analysis, overarching aims II

    -

    -Consider an experiment in which \( p \) characteristics of \( n \) samples are +

    Consider an experiment in which \( p \) characteristics of \( n \) samples are measured. The data from this experiment, for various explanatory variables \( p \) are normally represented by a matrix \( \mathbf{X} \). +

    -

    -The matrix \( \mathbf{X} \) is called the design +

    The matrix \( \mathbf{X} \) is called the design matrix. Additional information of the samples is available in the form of \( \boldsymbol{y} \) (also as above). The variable \( \boldsymbol{y} \) is generally referred to as the response variable. The aim of @@ -433,73 +486,62 @@ f(\mathbf{X}_{i,\ast}) \). When no prior knowledge on the form of \( f(\cdot) \) is available, it is common to assume a linear relationship between \( \boldsymbol{X} \) and \( \boldsymbol{y} \). This assumption gives rise to the linear regression model where \( \boldsymbol{\beta} = [\beta_0, \ldots, -\beta_{p-1}]^{T} \) are the regression parameters. - -

    -Linear regression gives us a set of analytical equations for the parameters \( \beta_j \). - +\beta_{p-1}]^{T} \) are the regression parameters. +

    +

    Linear regression gives us a set of analytical equations for the parameters \( \beta_j \).

    -











    -

    Examples

    -In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), -consider the model we discussed for describing nuclear binding energies. +

    In order to understand the relation among the predictors \( p \), the set of data \( n \) and the target (outcome, output etc) \( \boldsymbol{y} \), +consider the model we discussed for describing nuclear binding energies. +

    -

    -There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. +

    There we assumed that we could parametrize the data using a polynomial approximation based on the liquid drop model. Assuming +

    $$ BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1}, $$ -we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. +

    we have five predictors, that is the intercept, the \( A \) dependent term, the \( A^{2/3} \) term and the \( A^{-1/3} \) and \( A^{-1} \) terms. This gives \( p=0,1,2,3,4 \). Furthermore we have \( n \) entries for each predictor. It means that our design matrix is a \( p\times n \) matrix \( \boldsymbol{X} \). +

    -

    -Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the -so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \). This is however a classification problem and we will come back to it when we discuss Logistic Regression. - - +

    Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the +so-called credit card default data from Taiwan. The data set contains data on \( n=30000 \) credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are \( 24 \) such predictors or attributes leading to a design matrix of dimensionality \( 24 \times 30000 \). This is however a classification problem and we will come back to it when we discuss Logistic Regression. +

    -











    -

    General linear models

    -Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function. +

    Before we proceed let us study a case from linear algebra where we aim at fitting a set of data \( \boldsymbol{y}=[y_0,y_1,\dots,y_{n-1}] \). We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables \( \boldsymbol{x}=[x_0,x_1,\dots,x_{n-1}] \), that is \( y_i = y(x_i) \) with \( i=0,1,2,\dots,n-1 \). The variables \( x_i \) could represent physical quantities like time, temperature, position etc. We assume that \( y(x) \) is a smooth function.

    -

    -Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is +

    Since obtaining these data points may not be trivial, we want to use these data to fit a function which can allow us to make predictions for values of \( y \) which are not in the present set. The perhaps simplest approach is to assume we can parametrize our function in terms of a polynomial of degree \( n-1 \) with \( n \) points, that is

    $$ y=y(x) \rightarrow y(x_i)=\tilde{y}_i+\epsilon_i=\sum_{j=0}^{n-1} \beta_j x_i^j+\epsilon_i, $$ -where \( \epsilon_i \) is the error in our approximation. - - +

    where \( \epsilon_i \) is the error in our approximation.

    -











    -

    Rewriting the fitting procedure as a linear algebra problem

    -For every set of values \( y_i,x_i \) we have thus the corresponding set of equations +

    For every set of values \( y_i,x_i \) we have thus the corresponding set of equations

    $$ \begin{align*} y_0&=\beta_0+\beta_1x_0^1+\beta_2x_0^2+\dots+\beta_{n-1}x_0^{n-1}+\epsilon_0\\ @@ -512,29 +554,27 @@ $$
    -











    -

    Rewriting the fitting procedure as a linear algebra problem, more details

    -Defining the vectors +

    Defining the vectors

    $$ \boldsymbol{y} = [y_0,y_1, y_2,\dots, y_{n-1}]^T, $$ -and +

    and

    $$ \boldsymbol{\beta} = [\beta_0,\beta_1, \beta_2,\dots, \beta_{n-1}]^T, $$ -and +

    and

    $$ \boldsymbol{\epsilon} = [\epsilon_0,\epsilon_1, \epsilon_2,\dots, \epsilon_{n-1}]^T, $$ -and the design matrix +

    and the design matrix

    $$ \boldsymbol{X}= \begin{bmatrix} @@ -546,29 +586,27 @@ $$ \end{bmatrix} $$ -we can rewrite our equations as +

    we can rewrite our equations as

    $$ \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. $$ -The above design matrix is called a Vandermonde matrix. +

    The above design matrix is called a Vandermonde matrix.

    -











    -

    Generalizing the fitting procedure as a linear algebra problem

    -

    -We are obviously not limited to the above polynomial expansions. We +

    We are obviously not limited to the above polynomial expansions. We could replace the various powers of \( x \) with elements of Fourier series or instead of \( x_i^j \) we could have \( \cos{(j x_i)} \) or \( \sin{(j x_i)} \), or time series or other orthogonal functions. For every set of values \( y_i,x_i \) we can then generalize the equations to +

    $$ \begin{align*} @@ -582,19 +620,17 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{ \end{align*} $$ -

    + Note that we have \( p=n \) here. The matrix is symmetric. This is generally not the case!

    -











    -

    Generalizing the fitting procedure as a linear algebra problem

    -We redefine in turn the matrix \( \boldsymbol{X} \) as +

    We redefine in turn the matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{X}= \begin{bmatrix} @@ -606,23 +642,21 @@ x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,n-1}\\ \end{bmatrix} $$ -and without loss of generality we rewrite again our equations as +

    and without loss of generality we rewrite again our equations as

    $$ \boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}. $$ -The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values? +

    The left-hand side of this equation is kwown. Our error vector \( \boldsymbol{\epsilon} \) and the parameter vector \( \boldsymbol{\beta} \) are our unknow quantities. How can we obtain the optimal set of \( \beta_i \) values?

    -











    -

    Optimizing our parameters

    -We have defined the matrix \( \boldsymbol{X} \) via the equations +

    We have defined the matrix \( \boldsymbol{X} \) via the equations

    $$ \begin{align*} y_0&=\beta_0x_{00}+\beta_1x_{01}+\beta_2x_{02}+\dots+\beta_{n-1}x_{0n-1}+\epsilon_0\\ @@ -635,29 +669,27 @@ y_{n-1}&=\beta_0x_{n-1,0}+\beta_1x_{n-1,2}+\beta_2x_{n-1,2}+\dots+\beta_{n-1}x_{ \end{align*} $$ -

    -As we noted above, we stayed with a system with the design matrix +

    As we noted above, we stayed with a system with the design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \), that is we have \( p=n \). For reasons to come later (algorithmic arguments) we will hereafter define our matrix as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors refering to the column numbers and the entries \( n \) being the row elements. - - +

    -











    -

    Our model for the nuclear binding energies

    -

    -In our introductory notes we looked at the so-called liquid drop model. Let us remind ourselves about what we did by looking at the code. +

    In our introductory notes we looked at the so-called liquid drop model. Let us remind ourselves about what we did by looking at the code.

    -

    -We restate the parts of the code we are most interested in. -

    +

    We restate the parts of the code we are most interested in.

    -
    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -726,73 +758,81 @@ DesignMatrix = pd.index = A
 DesignMatrix.columns = ['1', 'A', 'A^(2/3)', 'A^(-1/3)', '1/A']
 display(DesignMatrix)
-
    
-
    
-With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    With \( \boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1} \), it means that we will hereafter write our equations for the approximation as

    $$ \boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, $$ -throughout these lectures. +

    throughout these lectures.

    -











    -

    Optimizing our parameters, more details

    -With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as +

    With the above we use the design matrix to define the approximation \( \boldsymbol{\tilde{y}} \) via the unknown quantity \( \boldsymbol{\beta} \) as

    $$ \boldsymbol{\tilde{y}}= \boldsymbol{X}\boldsymbol{\beta}, $$ -and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely +

    and in order to find the optimal parameters \( \beta_i \) instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values \( y_i \) (which represent hopefully the exact values) and the parameterized values \( \tilde{y}_i \), namely

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)^T\left(\boldsymbol{y}-\boldsymbol{\tilde{y}}\right)\right\}, $$ -or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as +

    or using the matrix \( \boldsymbol{X} \) and in a more compact matrix-vector notation as

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -This function is one possible way to define the so-called cost function. +

    This function is one possible way to define the so-called cost function.

    -

    -It is also common to define +

    It is also common to define the function \( C \) as +

    $$ C(\boldsymbol{\beta})=\frac{1}{2n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2, $$ -since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out. +

    since when taking the first derivative with respect to the unknown parameters \( \beta \), the factor of \( 2 \) cancels out.

    -











    -

    Interpretations and optimizing our parameters

    -

    -The function +

    The function

    $$ C(\boldsymbol{\beta})=\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}, $$ -can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. +

    can be linked to the variance of the quantity \( y_i \) if we interpret the latter as the mean value. When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret \( y_i \) as a mean value +

    $$ y_{i}=\langle y_i \rangle = \beta_0x_{i,0}+\beta_1x_{i,1}+\beta_2x_{i,2}+\dots+\beta_{n-1}x_{i,n-1}+\epsilon_i, $$ -

    -where \( \langle y_i \rangle \) is the mean value. Keep in mind also that +

    where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the response (dependent or outcome) variable \( y_i \) the outcome of a numerical experiment or another type of experiment and is thus only an @@ -800,57 +840,52 @@ approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we will treat \( y_i \) as our exact value for the response variable. +

    -

    -In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem +

    In order to find the parameters \( \beta_i \) we will then minimize the spread of \( C(\boldsymbol{\beta}) \), that is we are going to solve the problem

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -In practical terms it means we will require +

    In practical terms it means we will require

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = \frac{\partial }{\partial \beta_j}\left[ \frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)^2\right]=0, $$ -which results in +

    which results in

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ -or in a matrix-vector form as +

    or in a matrix-vector form as

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$ - -
    -











    -

    Interpretations and optimizing our parameters

    -We can rewrite +

    We can rewrite

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$ -as +

    as

    $$ \boldsymbol{X}^T\boldsymbol{y} = \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}, $$ -and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution +

    and if the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) is invertible we have the solution

    $$ \boldsymbol{\beta} =\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -We note also that since our design matrix is defined as \( \boldsymbol{X}\in +

    We note also that since our design matrix is defined as \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), the product \( \boldsymbol{X}^T\boldsymbol{X} \in {\mathbb{R}}^{p\times p} \). In the above case we have that \( p \ll n \), in our case \( p=5 \) meaning that we end up with inverting a small @@ -859,26 +894,24 @@ matrices to invert. The methods discussed here and for many other supervised learning algorithms like classification with logistic regression or support vector machines, exhibit dimensionalities which allow for the usage of direct linear algebra methods such as LU decomposition or Singular Value Decomposition (SVD) for finding the inverse of the matrix -\( \boldsymbol{X}^T\boldsymbol{X} \). +\( \boldsymbol{X}^T\boldsymbol{X} \). +

    -

    -Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)? What kind of problems can we expect? +

    Small question: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)? What kind of problems can we expect?

    -











    -

    Some useful matrix and vector expressions

    -

    -The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and +

    The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and matrices as upper case boldfaced letters. +

    $$ \frac{\partial (\boldsymbol{b}^T\boldsymbol{a})}{\partial \boldsymbol{a}} = \boldsymbol{b}, @@ -896,99 +929,128 @@ $$ \frac{\partial \log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}} = (\boldsymbol{A}^{-1})^T. $$ -

    -









    +









    Meet the Hessian Matrix

    -

    -A very important matrix we will meet again and again in Machine +

    A very important matrix we will meet again and again in Machine Learning is the Hessian. It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, +

    $$ \frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -The Hessian matrix plays an important role and is defined here as +

    The Hessian matrix plays an important role and is defined here as

    $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -For ordinary least squares, it is inversely proportional (derivation +

    For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. +

    -

    -Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). +

    Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).

    -











    -

    Interpretations and optimizing our parameters

    -The residuals \( \boldsymbol{\epsilon} \) are in turn given by +

    The residuals \( \boldsymbol{\epsilon} \) are in turn given by

    $$ \boldsymbol{\epsilon} = \boldsymbol{y}-\boldsymbol{\tilde{y}} = \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, $$ -and with +

    and with

    $$ \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, $$ -we have +

    we have

    $$ \boldsymbol{X}^T\boldsymbol{\epsilon}=\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)= 0, $$ -meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach. - - +

    meaning that the solution for \( \boldsymbol{\beta} \) is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.

    -











    -

    Own code for Ordinary Least Squares

    -

    -It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to +

    It is rather straightforward to implement the matrix inversion and obtain the parameters \( \boldsymbol{\beta} \). After having defined the matrix \( \boldsymbol{X} \) we simply need to write -

    +

    -
    # matrix inversion to find beta
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # matrix inversion to find beta
 beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(Energies)
 # and then make the prediction
 ytilde = X @ beta
-
    
-
    
-Alternatively, you can use the least squares functionality in Numpy as
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Alternatively, you can use the least squares functionality in Numpy as

    -
    fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
+
    
+  
    
+    
    
+      
    
+        
    
+  
    fit = np.linalg.lstsq(X, Energies, rcond =None)[0]
 ytildenp = np.dot(fit,X.T)
-
    
-
    
-And finally we plot our fit with and compare with data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    And finally we plot our fit with and compare with data

    -
    Masses['Eapprox']  = ytilde
+
    
+  
    
+    
    
+      
    
+        
    
+  
    Masses['Eapprox']  = ytilde
 # Generate a plot comparing the experimental with the fitted values values.
 fig, ax = plt.subplots()
 ax.set_xlabel(r'$A = N + Z$')
@@ -1000,60 +1062,139 @@ ax.plot(Masses[
 ax.legend()
 save_fig("Masses2016OLS")
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Adding error analysis and training set up

    -

    -We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of Scikit-Learn in the introductory slides. +

    We can easily test our fit by computing the \( R2 \) score that we discussed in connection with the functionality of Scikit-Learn in the introductory slides. Since we are not using Scikit-Learn here we can define our own \( R2 \) function as -

    +

    -
    def R2(y_data, y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def R2(y_data, y_model):
     return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-
    
-
    
-and we would be using it as 
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and we would be using it as

    -
    print(R2(Energies,ytilde))
-
    -

    -We can easily add our MSE score as -

    +

    +
    +
    +
    +
    + 
    print(R2(Energies,ytilde))
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    We can easily add our MSE score as

    -
    def MSE(y_data,y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def MSE(y_data,y_model):
     n = np.size(y_model)
     return np.sum((y_data-y_model)**2)/n
 
 print(MSE(Energies,ytilde))
-
    
-
    
-and finally the relative error as
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and finally the relative error as

    -
    def RelativeError(y_data,y_model):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def RelativeError(y_data,y_model):
     return abs((y_data-y_model)/y_data)
 print(RelativeError(Energies, ytilde))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Splitting our Data in Training and Test data

    -

    -

    -It is normal in essentially all Machine Learning studies to split the +

    It is normal in essentially all Machine Learning studies to split the data in a training set and a test set (sometimes also an additional validation set). Scikit-Learn has an own function for this. There is no explicit recipe for how much data should be included as training @@ -1063,20 +1204,21 @@ postpone a discussion of this splitting to the end of these notes and our discussion of the so-called bias-variance tradeoff. Here we limit ourselves to repeat the above equation of state fitting example but now splitting the data into a training set and a test set. - - +

    -











    -

    Examples

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -1114,16 +1256,33 @@ ypredict = X_test print(R2(y_test,ypredict))
 print("Test MSE")
 print(MSE(y_test,ypredict))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Making your own test-train splitting

    -

    -

    # equivalently in numpy
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # equivalently in numpy
 def train_test_split_numpy(inputs, labels, train_size, test_size):
     n_inputs = len(inputs)
     inputs_shuffled = inputs.copy()
@@ -1137,28 +1296,38 @@ ypredict = X_test = labels_shuffled[:train_end], labels_shuffled[train_end:]
 
     return X_train, X_test, Y_train, Y_test
-
    
-
    
-But since scikit-learn has its own function for doing this and since
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    But since scikit-learn has its own function for doing this and since it interfaces easily with tensorflow and other libraries, we normally recommend using the latter functionality. +

    -

    -

    The Boston housing data example

    -

    -The Boston housing +

    The Boston housing data set was originally a part of UCI Machine Learning Repository and has been removed now. The data set is now included in Scikit-Learn's library. There are 506 samples and 13 feature (predictor) variables in this data set. The objective is to predict the value of prices of the house using the features (predictors) listed here. +

    -

    -The features/predictors are - +

    The features/predictors are

    1. CRIM: Per capita crime rate by town
    2. ZN: Proportion of residential land zoned for lots over 25000 square feet
      3. @@ -1174,81 +1343,185 @@ The features/predictors are
    3. LSTAT: Percentage of lower status of the population
    4. MEDV: Median value of owner-occupied homes in USD 1000s
    -









    -

    Housing data, the code

    -We start by importing the libraries -

    +

    We start by importing the libraries

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import matplotlib.pyplot as plt 
 
 import pandas as pd  
 import seaborn as sns 
-
    
-
    
-and load the Boston Housing DataSet from Scikit-Learn
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and load the Boston Housing DataSet from Scikit-Learn

    -

    -

    from sklearn.datasets import load_boston
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.datasets import load_boston
 
 boston_dataset = load_boston()
 
 # boston_dataset is a dictionary
 # let's check what it contains
 boston_dataset.keys()
-
    
-
    
-Then we invoke Pandas
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Then we invoke Pandas

    -
    boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)
 boston.head()
 boston['MEDV'] = boston_dataset.target
-
    
-
    
-and preprocess the data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    and preprocess the data

    -
    # check for missing values in all the columns
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # check for missing values in all the columns
 boston.isnull().sum()
-
    
-
    
-We can then visualize the data
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We can then visualize the data

    -
    # set the size of the figure
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # set the size of the figure
 sns.set(rc={'figure.figsize':(11.7,8.27)})
 
 # plot a histogram showing the distribution of the target values
 sns.distplot(boston['MEDV'], bins=30)
 plt.show()
-
    
-
    
-It is now useful to look at the correlation matrix
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    It is now useful to look at the correlation matrix

    -
    # compute the pair wise correlation for all columns  
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # compute the pair wise correlation for all columns  
 correlation_matrix = boston.corr().round(2)
 # use the heatmap function from seaborn to plot the correlation matrix
 # annot = True to print the values inside the square
 sns.heatmap(data=correlation_matrix, annot=True)
-
    
-
    
-From the above coorelation plot we can see that MEDV is strongly correlated to LSTAT and  RM. We see also that RAD and TAX are stronly correlated, but we don't include this in our features together to avoid multi-colinearity
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    From the above coorelation plot we can see that MEDV is strongly correlated to LSTAT and RM. We see also that RAD and TAX are stronly correlated, but we don't include this in our features together to avoid multi-colinearity

    -

    -

    plt.figure(figsize=(20, 5))
+
    
+  
    
+    
    
+      
    
+        
    
+  
    plt.figure(figsize=(20, 5))
 
 features = ['LSTAT', 'RM']
 target = boston['MEDV']
@@ -1261,22 +1534,56 @@ target = boston[.title(col)
     plt.xlabel(col)
     plt.ylabel('MEDV')
-
    
-
    
-Now we start training our model
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Now we start training our model

    -
    X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
+
    
+  
    
+    
    
+      
    
+        
    
+  
    X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])
 Y = boston['MEDV']
-
    
-
    
-We split the data into training and test sets
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We split the data into training and test sets

    -

    -

    from sklearn.model_selection import train_test_split
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.model_selection import train_test_split
 
 # splits the training and test data set in 80% : 20%
 # assign random_state to any value.This ensures consistency.
@@ -1285,13 +1592,30 @@ X_train, X_test, Y_train, Y_test = train_tes
 print(X_test.shape)
 print(Y_train.shape)
 print(Y_test.shape)
-
    
-
    
-Then we use the linear regression functionality from Scikit-Learn
-
    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Then we use the linear regression functionality from Scikit-Learn

    -
    from sklearn.linear_model import LinearRegression
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from sklearn.linear_model import LinearRegression
 from sklearn.metrics import mean_squared_error, r2_score
 
 lin_model = LinearRegression()
@@ -1322,85 +1646,104 @@ r2 = r2_score(Y_test, y_test_predict)
 print("--------------------------------------")
 print('RMSE is {}'.format(rmse))
 print('R2 score is {}'.format(r2))
-
    
-
    
-
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    -
    # plotting the y_test vs y_pred
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # plotting the y_test vs y_pred
 # ideally should have been a straight line
 plt.scatter(Y_test, y_test_predict)
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Reducing the number of degrees of freedom, overarching view

    -

    -Many Machine Learning problems involve thousands or even millions of +

    Many Machine Learning problems involve thousands or even millions of features for each training instance. Not only does this make training extremely slow, it can also make it much harder to find a good solution, as we will see. This problem is often referred to as the curse of dimensionality. Fortunately, in real-world problems, it is often possible to reduce the number of features considerably, turning an intractable problem into a tractable one. +

    -

    -Later we will discuss some of the most popular dimensionality reduction +

    Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). +Locally Linear Embedding (LLE). +

    -

    -Principal component analysis and its various variants deal with the +

    Principal component analysis and its various variants deal with the problem of fitting a low-dimensional affine subspace to a set of of data points in a high-dimensional space. With its family of methods it is one of the most used tools in data modeling, compression and visualization. - - +

    -











    -

    Preprocessing our data

    -

    -Before we proceed however, we will discuss how to preprocess our +

    Before we proceed however, we will discuss how to preprocess our data. Till now and in connection with our previous examples we have not met so many cases where we are too sensitive to the scaling of our data. Normally the data may need a rescaling and/or may be sensitive to extreme values. Scaling the data renders our inputs much more suitable for the algorithms we want to employ. +

    -

    -For data sets gathered for real world applications, it is rather normal that +

    For data sets gathered for real world applications, it is rather normal that different features have very different units and numerical scales. For example, a data set detailing health habits may include features such as age in the range \( 0-80 \), and caloric intake of order \( 2000 \). Many machine learning methods sensitive to the scales of the features and may perform poorly if they are very different scales. Therefore, it is typical to scale the features in a way to avoid such outlier values. - - +

    -











    -

    Functionality in Scikit-Learn

    -

    -Scikit-Learn has several functions which allow us to rescale the +

    Scikit-Learn has several functions which allow us to rescale the data, normally resulting in much better results in terms of various accuracy scores. The StandardScaler function in Scikit-Learn ensures that for each feature/predictor we study the mean value is @@ -1409,26 +1752,24 @@ matrix). This scaling has the drawback that it does not ensure that we have a particular maximum or minimum in our data set. Another function included in Scikit-Learn is the MinMaxScaler which ensures that all features are exactly between \( 0 \) and \( 1 \). The +

    -











    -

    More preprocessing

    -

    -The Normalizer scales each data +

    The Normalizer scales each data point such that the feature vector has a euclidean length of one. In other words, it projects a data point on the circle (or sphere in the case of higher dimensions) with a radius of 1. This means every data point is scaled by a different number (by the inverse of it’s length). This normalization is often used when only the direction (or angle) of the data matters, not the length of the feature vector. +

    -

    -The RobustScaler works similarly to the StandardScaler in that it +

    The RobustScaler works similarly to the StandardScaler in that it ensures statistical properties for each feature that guarantee that they are on the same scale. However, the RobustScaler uses the median and quartiles, instead of mean and variance. This makes the @@ -1436,42 +1777,42 @@ RobustScaler ignore data points that are very different from the rest (like measurement errors). These odd data points are also called outliers, and might often lead to trouble for other scaling techniques. - - +

    -











    -

    Frequently used scaling functions

    -

    -Many features are often scaled using standardization to improve performance. In Scikit-Learn this is given by the StandardScaler function as discussed above. It is easy however to write your own. +

    Many features are often scaled using standardization to improve performance. In Scikit-Learn this is given by the StandardScaler function as discussed above. It is easy however to write your own. Mathematically, this involves subtracting the mean and divide by the standard deviation over the data set, for each feature: +

    $$ x_j^{(i)} \rightarrow \frac{x_j^{(i)} - \overline{x}_j}{\sigma(x_j)}, $$ -where \( \overline{x}_j \) and \( \sigma(x_j) \) are the mean and standard deviation, respectively, of the feature \( x_j \). +

    where \( \overline{x}_j \) and \( \sigma(x_j) \) are the mean and standard deviation, respectively, of the feature \( x_j \). This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one. +

    -











    -

    Example of own Standard scaling

    -

    -Let us consider the following vanilla example where we use both +

    Let us consider the following vanilla example where we use both Scikit-Learn and write our own function as well. We produce a simple test design matrix with random numbers. Each column could then represent a specific feature whose mean value is subracted. +

    -

    -

    import sklearn.linear_model as skl
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import sklearn.linear_model as skl
 from sklearn.metrics import mean_squared_error
 from sklearn.model_selection import  train_test_split
 from sklearn.preprocessing import MinMaxScaler, StandardScaler, Normalizer
@@ -1494,56 +1835,85 @@ scaler = StandardScaler(with_std.fit(X)
 Xscaled = scaler.transform(X)
 display(XPandas-Xscaled)
-
    
-
    
-Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.

    -











    -

    Min-Max Scaling

    -

    -Another commonly used scaling method is min-max scaling. This is very +

    Another commonly used scaling method is min-max scaling. This is very useful for when we want the features to lie in a certain interval. To scale the feature \( x_j \) to the interval \( [a, b] \), we can apply the transformation +

    $$ x_j^{(i)} \rightarrow (b-a)\frac{x_j^{(i)} - \min(x_j)}{\max(x_j) - \min(x_j)} - a $$ -where \( \min(x_j) \) and \( \max(x_j) \) return the minimum and maximum value of \( x_j \) over the data set, respectively. +

    where \( \min(x_j) \) and \( \max(x_j) \) return the minimum and maximum value of \( x_j \) over the data set, respectively.

    -











    -

    Testing the Means Squared Error as function of Complexity

    -One of +

    One of the aims is to reproduce Figure 2.11 of Hastie et al. -We will also use Ridge and Lasso regression. +We will also use Ridge and Lasso regression. +

    -

    -Our data is defined by \( x\in [-3,3] \) with a total of for example \( 100 \) data points. -

    +

    Our data is defined by \( x\in [-3,3] \) with a total of for example \( 100 \) data points.

    -
    np.random.seed()
+
    
+  
    
+    
    
+      
    
+        
    
+  
    np.random.seed()
 n = 100
 maxdegree = 14
 # Make data set.
 x = np.linspace(-3, 3, n).reshape(-1, 1)
 y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)
-
    
-
    
-where \( y \) is the function we want to fit with a given polynomial.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + -

    -Write a first code which sets up a design matrix \( X \) defined by a fifth-order polynomial. Scale your data and split it in training and test data. -

    +

    where \( y \) is the function we want to fit with a given polynomial.

    + +

    Write a first code which sets up a design matrix \( X \) defined by a fifth-order polynomial. Scale your data and split it in training and test data.

    -
    import matplotlib.pyplot as plt
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import matplotlib.pyplot as plt
 import numpy as np
 from sklearn.linear_model import LinearRegression, Ridge, Lasso
 from sklearn.preprocessing import PolynomialFeatures
@@ -1579,16 +1949,33 @@ plt.plot(polydegree, TestError, label.plot(polydegree, TrainError, label='Train Error')
 plt.legend()
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    More preprocessing examples, Franke function and regression

    -

    -

    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import os
 import numpy as np
 import pandas as pd
@@ -1680,149 +2067,140 @@ clf = skl.print("MSE after  scaling: {:.2f}".format(mean_squared_error(clf.predict(X_test_scaled), y_test)))
 print("R2 score for  scaled data: {:.2f}".format(clf.score(X_test_scaled,y_test)))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Mathematical Interpretation of Ordinary Least Squares

    -

    -What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). +

    What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).

    -

    -We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by +

    We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by

    $$ \hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function. +

    The hat over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.

    -

    -This means that our best model is defined as +

    This means that our best model is defined as

    $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -We now define a matrix +

    We now define a matrix

    $$ \boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T. $$ -

    -We can rewrite +

    We can rewrite

    $$ \tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. $$ -

    -The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. +

    The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix. +

    -











    -

    Residual Error

    -

    -We have defined the residual error as +

    We have defined the residual error as

    $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}. $$ -

    -The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \). +

    The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).

    -











    -

    Simple case

    -

    -If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have +

    If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}. $$ -

    -In this case the matrix \( \boldsymbol{A} \) becomes +

    In this case the matrix \( \boldsymbol{A} \) becomes

    $$ \boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, $$ -and we have the obvious case +

    and we have the obvious case

    $$ \boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. $$ -

    -This serves also as a useful test of our codes. +

    This serves also as a useful test of our codes.

    -











    -

    The singular value decomposition

    -

    -

    -The examples we have looked at so far are cases where we normally can +

    The examples we have looked at so far are cases where we normally can invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky decomposition. +

    -

    -As we will also see in the first project, +

    As we will also see in the first project, this may however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below. +

    -

    -There is however a way to circumvent this problem and also +

    There is however a way to circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +

    -

    -This is given by the Singular Value Decomposition (SVD) algorithm, +

    This is given by the Singular Value Decomposition (SVD) algorithm, perhaps the most powerful linear algebra algorithm. The SVD provides a numerically stable matrix decomposition that is used in a large swath oc applications and the decomposition is always stable numerically. +

    -

    -In machine learning it plays a central role in dealing with for +

    In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - - +

    -











    -

    Linear Regression Problems

    -

    -One of the typical problems we encounter with linear regression, in particular +

    One of the typical problems we encounter with linear regression, in particular when the matrix \( \boldsymbol{X} \) (our so-called design matrix) is high-dimensional, are problems with near singular or singular matrices. The column vectors of \( \boldsymbol{X} \) may be linearly dependent, normally referred to as super-collinearity. This means that the matrix may be rank deficient and it is basically impossible to to model the data using linear regression. As an example, consider the matrix +

    $$ \begin{align*} \mathbf{X} & = \left[ @@ -1838,16 +2216,16 @@ $$ \end{align*} $$ -

    -The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the +

    The columns of \( \boldsymbol{X} \) are linearly dependent. We see this easily since the the first column is the row-wise sum of the other two columns. The rank (more correct, the column rank) of a matrix is the dimension of the space spanned by the column vectors. Hence, the rank of \( \mathbf{X} \) is equal to the number of linearly independent columns. In this particular case the matrix has rank 2. +

    -

    -Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies +

    Super-collinearity of an \( (n \times p) \)-dimensional design matrix \( \mathbf{X} \) implies that the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) (the matrix we need to invert to solve the linear regression equations) is non-invertible. If we have a square matrix that does not have an inverse, we say this matrix singular. The example here demonstrates this +

    $$ \begin{align*} \boldsymbol{X} & = \left[ @@ -1859,16 +2237,14 @@ $$ \end{align*} $$ -We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. +

    We see easily that \( \mbox{det}(\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \times (-1) - 1 \times (-1) = 0 \). Hence, \( \mathbf{X} \) is singular and its inverse is undefined. This is equivalent to saying that the matrix \( \boldsymbol{X} \) has at least an eigenvalue which is zero. +

    -











    -

    Fixing the singularity

    -

    -If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem +

    If our design matrix \( \boldsymbol{X} \) which enters the linear regression problem

    $$ \begin{align} \boldsymbol{\beta} & = (\boldsymbol{X}^{T} \boldsymbol{X})^{-1} \boldsymbol{X}^{T} \boldsymbol{y}, @@ -1876,49 +2252,46 @@ $$ \end{align} $$ -has linearly dependent column vectors, we will not be able to compute the inverse +

    has linearly dependent column vectors, we will not be able to compute the inverse of \( \boldsymbol{X}^T\boldsymbol{X} \) and we cannot find the parameters (estimators) \( \beta_i \). The estimators are only well-defined if \( (\boldsymbol{X}^{T}\boldsymbol{X})^{-1} \) exits. This is more likely to happen when the matrix \( \boldsymbol{X} \) is high-dimensional. In this case it is likely to encounter a situation where the regression parameters \( \beta_i \) cannot be estimated. +

    -

    -A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change +

    A cheap ad hoc approach is simply to add a small diagonal component to the matrix to invert, that is we change

    $$ \boldsymbol{X}^{T} \boldsymbol{X} \rightarrow \boldsymbol{X}^{T} \boldsymbol{X}+\lambda \boldsymbol{I}, $$ -where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later. +

    where \( \boldsymbol{I} \) is the identity matrix. When we discuss Ridge regression this is actually what we end up evaluating. The parameter \( \lambda \) is called a hyperparameter. More about this later.

    -











    -

    Basic math of the SVD

    -

    -From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is +

    From standard linear algebra we know that a square matrix \( \boldsymbol{X} \) can be diagonalized if and only it is a so-called normal matrix, that is if \( \boldsymbol{X}\in {\mathbb{R}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) or if \( \boldsymbol{X}\in {\mathbb{C}}^{n\times n} \) we have \( \boldsymbol{X}\boldsymbol{X}^{\dagger}=\boldsymbol{X}^{\dagger}\boldsymbol{X} \). The matrix has then a set of eigenpairs +

    $$ (\lambda_1,\boldsymbol{u}_1),\dots, (\lambda_n,\boldsymbol{u}_n), $$ -and the eigenvalues are given by the diagonal matrix +

    and the eigenvalues are given by the diagonal matrix

    $$ \boldsymbol{\Sigma}=\mathrm{Diag}(\lambda_1, \dots,\lambda_n). $$ -The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \) +

    The matrix \( \boldsymbol{X} \) can be written in terms of an orthogonal/unitary transformation \( \boldsymbol{U} \)

    $$ \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \). +

    with \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) or \( \boldsymbol{U}\boldsymbol{U}^{\dagger}=\boldsymbol{I} \).

    -

    -Not all square matrices are diagonalizable. A matrix like the one discussed above +

    Not all square matrices are diagonalizable. A matrix like the one discussed above

    $$ \boldsymbol{X} = \begin{bmatrix} 1& -1 \\ @@ -1926,16 +2299,14 @@ $$ \end{bmatrix} $$ -is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition -\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    is not diagonalizable, it is a so-called defective matrix. It is easy to see that the condition +\( \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{X}^T\boldsymbol{X} \) is not fulfilled. +

    -











    -

    The SVD, a Fantastic Algorithm

    -

    -However, and this is the strength of the SVD algorithm, any general +

    However, and this is the strength of the SVD algorithm, any general matrix \( \boldsymbol{X} \) can be decomposed in terms of a diagonal matrix and two orthogonal/unitary matrices. The Singular Value Decompostion (SVD) theorem @@ -1944,30 +2315,29 @@ terms of a diagonal matrix \( \boldsymbol{\Sigma} \) of dimensionality \( m\time and two orthognal matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), where the first has dimensionality \( m \times m \) and the last dimensionality \( n\times n \). We have then +

    $$ \boldsymbol{X} = \boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T $$ -

    -As an example, the above defective matrix can be decomposed as +

    As an example, the above defective matrix can be decomposed as

    $$ \boldsymbol{X} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1& 1 \\ 1& -1\\ \end{bmatrix} \begin{bmatrix} 2& 0 \\ 0& 0\\ \end{bmatrix} \frac{1}{\sqrt{2}}\begin{bmatrix} 1& -1 \\ 1& 1\\ \end{bmatrix}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -

    -with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). -The SVD exits always! +

    with eigenvalues \( \sigma_1=2 \) and \( \sigma_2=0 \). +The SVD exits always! +

    -

    -The SVD +

    The SVD decomposition (singular values) gives eigenvalues \( \sigma_i\geq\sigma_{i+1} \) for all \( i \) and for dimensions larger than \( i=p \), the eigenvalues (singular values) are zero. +

    -

    -In the general case, where our design matrix \( \boldsymbol{X} \) has dimension +

    In the general case, where our design matrix \( \boldsymbol{X} \) has dimension \( n\times p \), the matrix is thus decomposed into an \( n\times n \) orthogonal matrix \( \boldsymbol{U} \), a \( p\times p \) orthogonal matrix \( \boldsymbol{V} \) and a diagonal matrix \( \boldsymbol{\Sigma} \) with \( r=\mathrm{min}(n,p) \) @@ -1977,44 +2347,44 @@ assuming that \( n > p \). In our regression examples for the nuclear masses and the equation of state this is indeed the case, while for the Ising model we have \( p > n \). These are often cases that lead to near singular or singular matrices. +

    -

    -The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors. +

    The columns of \( \boldsymbol{U} \) are called the left singular vectors while the columns of \( \boldsymbol{V} \) are the right singular vectors.

    -











    -

    Economy-size SVD

    -

    -If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n +

    If we assume that \( n > p \), then our matrix \( \boldsymbol{U} \) has dimension \( n \times n \). The last \( n-p \) columns of \( \boldsymbol{U} \) become however irrelevant in our calculations since they are multiplied with the zeros in \( \boldsymbol{\Sigma} \). +

    -

    -The economy-size decomposition removes extra rows or columns of zeros +

    The economy-size decomposition removes extra rows or columns of zeros from the diagonal matrix of singular values, \( \boldsymbol{\Sigma} \), along with the columns in either \( \boldsymbol{U} \) or \( \boldsymbol{V} \) that multiply those zeros in the expression. Removing these zeros and columns can improve execution time and reduce storage requirements without compromising the accuracy of the decomposition. +

    -

    -If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). +

    If \( n > p \), we keep only the first \( p \) columns of \( \boldsymbol{U} \) and \( \boldsymbol{\Sigma} \) has dimension \( p\times p \). If \( p > n \), then only the first \( n \) columns of \( \boldsymbol{V} \) are computed and \( \boldsymbol{\Sigma} \) has dimension \( n\times n \). The \( n=p \) case is obvious, we retain the full SVD. In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy. +

    -











    -

    Codes for the SVD

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
@@ -2043,9 +2413,22 @@ X = np.a
 C = SVD(X)
 # Print the difference between the original matrix and the SVD one
 print(C-X)
-
    
-
    
-The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    The matrix \( \boldsymbol{X} \) has columns that are linearly dependent. The first column is the row-wise sum of the other two columns. The rank of a matrix (the column rank) is the dimension of space spanned by the column vectors. The rank of the matrix is the number of linearly @@ -2053,49 +2436,40 @@ independent columns, in this case just \( 2 \). We see this from the singular values when running the above code. Running the standard inversion algorithm for matrix inversion with \( \boldsymbol{X}^T\boldsymbol{X} \) results in the program terminating due to a singular matrix. +

    -











    -

    Note about SVD Calculations

    -

    -The \( U \), \( S \), and \( V \) matrices returned from the svd() function +

    The \( U \), \( S \), and \( V \) matrices returned from the svd() function cannot be multiplied directly. +

    -

    -As you can see from the code, the \( S \) vector must be converted into a +

    As you can see from the code, the \( S \) vector must be converted into a diagonal matrix. This may cause a problem as the size of the matrices do not fit the rules of matrix multiplication, where the number of columns in a matrix must match the number of rows in the subsequent matrix. +

    -

    -If you wish to include the zero singular values, you will need to +

    If you wish to include the zero singular values, you will need to resize the matrices and set up a diagonal matrix as done in the above example +

    -











    -

    Friday September 3

    -

    -Video of Lecture from 2020 and handwritten notes +

    Video of Lecture from 2020 and handwritten notes

    -

    -More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021. +

    More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021.

    -











    -

    Mathematics of the SVD and implications

    -

    -Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies. +

    Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.

    -

    -Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \) +

    Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)

    $$ \boldsymbol{X}=\begin{bmatrix} x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\ @@ -2107,34 +2481,27 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ \end{bmatrix}. $$ -

    -We can SVD decompose our matrix as +

    We can SVD decompose our matrix as

    $$ \boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \). +

    where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).

    -

    -Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \). +

    Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).

    -

    -Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is +

    Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is

    $$ \sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. $$ -

    -All values beyond \( p-1 \) are all zero. +

    All values beyond \( p-1 \) are all zero.

    -











    -

    Example Matrix

    -

    -As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \) +

    As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)

    $$ \boldsymbol{\Sigma}= @@ -2145,8 +2512,7 @@ $$ \end{bmatrix} $$ -

    -The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as +

    The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as

    $$ \boldsymbol{\Sigma}= @@ -2156,8 +2522,7 @@ $$ \end{bmatrix}, $$ -

    -where +

    where

    $$ \boldsymbol{\tilde{\Sigma}}= \begin{bmatrix} @@ -2166,7 +2531,7 @@ $$ \end{bmatrix}, $$ -contains only the singular values. Note also (and we will use this below) that +

    contains only the singular values. Note also (and we will use this below) that

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= @@ -2176,7 +2541,7 @@ $$ \end{bmatrix}, $$ -which is a \( 2\times 2 \) matrix while +

    which is a \( 2\times 2 \) matrix while

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= \begin{bmatrix} @@ -2186,190 +2551,172 @@ $$ \end{bmatrix}, $$ -

    -is a \( 3\times 3 \) matrix. The last row and column of this last matrix +

    is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix. +

    -











    -

    Setting up the Matrix to be inverted

    -

    -The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as +

    The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, $$ -and using the orthogonality of the matrix \( \boldsymbol{U} \) we have +

    and using the orthogonality of the matrix \( \boldsymbol{U} \) we have

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \). +

    We define \( \boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=\tilde{\boldsymbol{\Sigma}}^2 \) which is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).

    -

    -This means, using the orthogonality of \( \boldsymbol{V} \), that we get +

    This means, using the orthogonality of \( \boldsymbol{V} \), that we get

    $$ \boldsymbol{X}^T\boldsymbol{X}=\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where +

    We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -and using our SVD decomposition of \( \boldsymbol{X} \) we have +

    and using our SVD decomposition of \( \boldsymbol{X} \) we have

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\tilde{\boldsymbol{\Sigma}}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, $$ -which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), +

    which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

    $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y}, $$ -

    -Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \) +

    Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)

    $$ \boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}], $$ -that belong to \( i>p-1 \), result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero). +

    that belong to \( i>p-1 \), result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).

    -

    -It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \). +

    It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

    -











    -

    Further properties (important for our analyses later)

    -

    -Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, +

    Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -

    -If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}. $$ -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have +

    Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T. $$ -

    -If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get +

    If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get

    $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T. $$ -This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) +

    This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. $$ -

    -Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an +

    Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an \( n\times p \) matrix. In most supervised learning cases we have that \( n \ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). +

    -

    -In our lectures, the number of columns will +

    In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences for the definition of for example the covariance matrix and its relation to the SVD. +

    -











    -

    Meet the Covariance Matrix

    -

    -Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above. +

    Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.

    -

    -We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +

    We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have +

    $$ \frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ -This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize). +

    This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).

    -

    -The Hessian matrix plays an important role and is defined in this course as +

    The Hessian matrix plays an important role and is defined in this course as

    $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. $$ -

    -The Hessian matrix for ordinary least squares is also proportional to +

    The Hessian matrix for ordinary least squares is also proportional to the covariance matrix. This means also that we can use the SVD to find the eigenvalues of the covariance matrix and the Hessian matrix in terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies. +

    -











    -

    Introducing the Covariance and Correlation functions

    -

    -Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about +

    Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about the definition of the covariance and the correlation function. These are quantities that play a central role in machine learning methods. +

    -

    -Suppose we have defined two vectors +

    Suppose we have defined two vectors \( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as +

    $$ \boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ \mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\ \end{bmatrix}, $$ -where for example +

    where for example

    $$ \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}). $$ -With this definition and recalling that the variance is defined as +

    With this definition and recalling that the variance is defined as

    $$ \mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2, $$ -we can rewrite the covariance matrix as +

    we can rewrite the covariance matrix as

    $$ \boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\ \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\ \end{bmatrix}. $$ -

    -Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +

    Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. What you will find in essentially all statistics texts are equations with a factor \( 1/(n-1) \). This is called Bessel's correction. This method corrects the bias in the estimation of the population variance @@ -2377,28 +2724,27 @@ and covariance. It also partially corrects the bias in the estimation of the population standard deviation. If you use a library like Scikit-Learn or nunmpy's function calculate the covariance, this quantity will be computed with a factor \( 1/(n-1) \). +

    -











    -

    Covariance and Correlation Matrix

    -

    -The covariance takes values between zero and infinity and may thus +

    The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function +

    $$ \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}. $$ -

    -The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] +

    The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as +

    $$ \boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\ @@ -2406,17 +2752,14 @@ $$ \end{bmatrix}, $$ -

    -In the above example this is the function we constructed using pandas. +

    In the above example this is the function we constructed using pandas.

    -











    -

    Correlation Function and Design/Feature Matrix

    -

    -In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression +

    In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as +

    $$ \boldsymbol{X}=\begin{bmatrix} @@ -2429,23 +2772,24 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\ \end{bmatrix}, $$ -with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the +

    with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the entries \( n \) being the row elements. We can rewrite the design/feature matrix in terms of its column vectors as +

    $$ \boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_1 & \boldsymbol{x}_2 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix}, $$ -with a given vector +

    with a given vector

    $$ \boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}. $$ -

    -With these definitions, we can now rewrite our \( 2\times 2 \) +

    With these definitions, we can now rewrite our \( 2\times 2 \) correlation/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \) +

    $$ \boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} @@ -2458,7 +2802,7 @@ $$ \end{bmatrix}, $$ -and the correlation matrix +

    and the correlation matrix

    $$ \boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\ @@ -2470,37 +2814,39 @@ $$ \end{bmatrix}, $$ -

    -









    +









    Covariance Matrix Examples

    -

    -The Numpy function np.cov calculates the covariance elements using +

    The Numpy function np.cov calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values. The following simple function uses the np.vstack function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \boldsymbol{W} \) +

    -

    -Note that this assumes you have the features as the rows, and the inputs as columns, that is +

    Note that this assumes you have the features as the rows, and the inputs as columns, that is

    $$ \boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\ y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\ \end{bmatrix}, $$ -

    -which in turn is converted into into the \( 2\times 2 \) covariance matrix +

    which in turn is converted into into the \( 2\times 2 \) covariance matrix \( \boldsymbol{C} \) via the Numpy function np.cov(). We note that we can also calculate the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy function np.mean(x). We can also extract the eigenvalues of the covariance matrix through the np.linalg.eig() function. +

    -

    -

    # Importing various packages
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Importing various packages
 import numpy as np
 n = 100
 x = np.random.normal(size=n)
@@ -2510,23 +2856,40 @@ y = 4+3*
 W = np.vstack((x, y))
 C = np.cov(W)
 print(C)
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Correlation Matrix

    -

    -The previous example can be converted into the correlation matrix by +

    The previous example can be converted into the correlation matrix by simply scaling the matrix elements with the variances. We should also subtract the mean values for each column. This leads to the following code which sets up the correlations matrix for the previous example in -a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors). +

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 n = 100
 # define two vectors                                                                                           
 x = np.random.random(size=n)
@@ -2547,26 +2910,40 @@ C[1,1]0,1]= cov_xy/np.sqrt(variance_y*variance_x)
 C[1,0]= C[0,1]
 print(C)
-
    
-
    
-We see that the matrix elements along the diagonal are one as they
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We see that the matrix elements along the diagonal are one as they should be and that the matrix is symmetric. Furthermore, diagonalizing this matrix we easily see that it is a positive definite matrix. +

    -

    -The above procedure with numpy can be made more compact if we use pandas. +

    The above procedure with numpy can be made more compact if we use pandas.

    -











    -

    Correlation Matrix with Pandas

    -

    -We whow here how we can set up the correlation matrix using pandas, as done in this simple code -

    +

    We whow here how we can set up the correlation matrix using pandas, as done in this simple code

    -
    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import pandas as pd
 n = 10
 x = np.random.normal(size=n)
@@ -2580,19 +2957,34 @@ Xpd = pd.print(Xpd)
 correlation_matrix = Xpd.corr()
 print(correlation_matrix)
-
    
-
    
-We expand this model to the Franke function discussed above.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We expand this model to the Franke function discussed above.

    -











    -

    Correlation Matrix with Pandas and the Franke function

    -

    -

    # Common imports
+
    
+  
    
+    
    
+      
    
+        
    
+  
    # Common imports
 import numpy as np
 import pandas as pd
 
@@ -2635,31 +3027,41 @@ Xpd = pd.= Xpd - Xpd.mean()
 covariance_matrix = Xpd.cov()
 print(covariance_matrix)
-
    
-
    
-We note here that the covariance is zero for the first rows and
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We note here that the covariance is zero for the first rows and columns since all matrix elements in the design matrix were set to one (we are fitting the function in terms of a polynomial of degree \( n \)). +

    -

    -This means that the variance for these elements will be zero and will +

    This means that the variance for these elements will be zero and will cause problems when we set up the correlation matrix. We can simply drop these elements and construct a correlation -matrix without these elements. +matrix without these elements. +

    -











    -

    Rewriting the Covariance and/or Correlation Matrix

    -

    -We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as +

    We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

    $$ \boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$ -

    -To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \) +

    To see this let us simply look at a design matrix \( \boldsymbol{X}\in {\mathbb{R}}^{2\times 2} \)

    $$ \boldsymbol{X}=\begin{bmatrix} x_{00} & x_{01}\\ @@ -2669,8 +3071,7 @@ x_{10} & x_{11}\\ \end{bmatrix}. $$ -

    -If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \)) +

    If we then compute the expectation value (note the \( 1/n \) factor instead of \( 1/(n-1) \))

    $$ \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\frac{1}{n}\begin{bmatrix} x_{00}^2+x_{10}^2 & x_{00}x_{01}+x_{10}x_{11}\\ @@ -2678,35 +3079,31 @@ x_{01}x_{00}+x_{11}x_{10} & x_{01}^2+x_{11}^2\\ \end{bmatrix}, $$ -which is just +

    which is just

    $$ \boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]=\begin{bmatrix} \mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] \\ \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] \\ \end{bmatrix}, $$ -where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \). +

    where we wrote $$\boldsymbol{C}[\boldsymbol{x}_0,\boldsymbol{x}_1] = \boldsymbol{C}[\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors \( \boldsymbol{x} \) of the design/feature matrix \( \boldsymbol{X} \).

    -

    -It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). +

    It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \).

    -











    -

    Linking with the SVD

    -

    -We saw earlier that +

    We saw earlier that

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix +

    Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}, $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is +

    where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

    $$ \tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ @@ -2717,355 +3114,348 @@ $$ \end{bmatrix}, $$ -meaning we can write +

    meaning we can write

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -









    +









    What does it mean?

    -

    -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +

    In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that the columns of \( \boldsymbol{V} \) are the eigenvectors of \( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of \( \boldsymbol{X} \) in a descending order, it means that the column vectors \( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). +encode from the columns of \( \boldsymbol{X} \). +

    -

    -Note that these are also the eigenvectors and eigenvalues of the +

    Note that these are also the eigenvectors and eigenvalues of the Hessian matrix. +

    -

    -If we now recall the definition of the covariance matrix (not using +

    If we now recall the definition of the covariance matrix (not using Bessel's correction) we have +

    $$ \boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ -

    -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +

    meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    -











    -

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    -

    -For \( \boldsymbol{X}\boldsymbol{X}^T \) we found +

    For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -Since the matrices here have dimension \( n\times n \), we have +

    Since the matrices here have dimension \( n\times n \), we have

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -leading to +

    leading to

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -

    -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +

    Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

    $$ (\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$ -

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +

    It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

    -

    -Since we will mainly be interested in the correlations among the features +

    Since we will mainly be interested in the correlations among the features of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +

    -











    -

    Ridge and LASSO Regression

    -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -or we can state it as +

    or we can state it as

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, $$ -where we have used the definition of a norm-2 vector, that is +

    where we have used the definition of a norm-2 vector, that is

    $$ \vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -

    -By minimizing the above equation with respect to the parameters +

    By minimizing the above equation with respect to the parameters \( \boldsymbol{\beta} \) we could then obtain an analytical expression for the parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by defining a new cost function to be optimized, that is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

    -which leads to the Ridge regression minimization problem where we +

    which leads to the Ridge regression minimization problem where we require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is a finite number larger than zero. By defining +

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, $$ -

    -we have a new optimization equation +

    we have a new optimization equation

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 $$ -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    -

    -Here we have defined the norm-1 as +

    Here we have defined the norm-1 as

    $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ -

    -









    +









    Deriving the Ridge Regression Equations

    -

    -Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -and +

    and taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters +

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that +

    with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

    $$ \sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ -

    -with \( t \) a finite positive number. +

    with \( t \) a finite positive number.

    -

    -When we compare this with the ordinary least squares result we have +

    If we keep the \( 1/n \) factor, the equation for the optimal \( \beta \) changes to

    +$$ +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

    In many textbooks the \( 1/n \) term is often omitted. Note that a library like Scikit-Learn does not include the \( 1/n \) factor in the setup of the cost function.

    + +

    When we compare this with the ordinary least squares result we have

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

    -

    -We see that Ridge regression is nothing but the standard OLS with a +

    We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    -

    -Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

    $$ \tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$ -

    -For Ridge regression this becomes +

    For Ridge regression this becomes

    $$ \tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$ -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). +

    with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

    -











    -

    Interpreting the Ridge results

    -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have +

    Since \( \lambda \geq 0 \), it means that compared to OLS, we have

    $$ \frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. $$ -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +

    Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by \( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has eigenvalues ordered in a descending way, that is \( \sigma_i \geq \sigma_{i+1} \). +

    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods. +

    For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

    -











    -

    More interpretations

    -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is +

    For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

    $$ \boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -

    -In this case the standard OLS results in +

    In this case the standard OLS results in

    $$ \boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -

    -and +

    and

    $$ \boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, $$ -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +

    that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and the Ridge estimator converges to zero when the hyperparameter goes to infinity. +

    -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. +

    We will come back to more interpreations after we have gone through some of the statistical analysis part.

    -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -











    -

    Deriving the Lasso Regression Equations

    -

    -Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function +

    Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

    -Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +

    Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

    $$ \frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. $$ -we have that the derivative of the cost function is +

    we have that the derivative of the cost function is

    $$ \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, $$ -and reordering we have +

    and reordering we have

    $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}. $$ -This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. +

    This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    -











    -

    Exercises for week 36, September 6-10

    -

    -The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3). +

    The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3).

    -

    -

    Exercise 1: Adding Ridge and Lasso Regression

    -

    -This exercise is a continuation of exercise 2 from exercise set 1 +

    This exercise is a continuation of exercise 2 from exercise set 1 (week 35, August 30-September 3). We will use the same function to generate our data set, still staying with a simple function \( y(x) \) which we want to fit using linear regression, but now extending the analysis to include the Ridge and the Lasso regression methods. +

    -

    -We will thus again generate our own dataset for a function \( y(x) \) where +

    We will thus again generate our own dataset for a function \( y(x) \) where \( x \in [0,1] \) and defined by random numbers computed with the uniform distribution. The function \( y \) is a quadratic polynomial in \( x \) with added stochastic noise according to the normal distribution \( \cal{N}(0,1) \). +

    -

    -The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points). -

    +

    The following simple Python instructions define our \( x \) and \( y \) values (with 100 data points).

    -
    x = np.random.rand(100)
+
    
+  
    
+    
    
+      
    
+        
    
+  
    x = np.random.rand(100)
 y = 2.0+5*x*x+0.1*np.random.randn(100)
-
    
-
    
-Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 3. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + -

    -The code here allows you to perform your own Ridge calculation and +

    Write your own code for the Ridge method (see chapter 3.4 of Hastie et al., equations (3.43) and (3.44)) and compute the parametrization for different values of \( \lambda \). Compare and analyze your results with those from exercise 3. Study the dependence on \( \lambda \) while also varying the strength of the noise in your expression for \( y(x) \).

    + +

    The code here allows you to perform your own Ridge calculation and perform calculations for various values of the regularization parameter \( \lambda \). This program can easily be extended upon. +

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -3134,49 +3524,61 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-Repeat the above but using the functionality of
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Repeat the above but using the functionality of Scikit-Learn. Compare your code with the results from Scikit-Learn. Remember to run with the same random numbers for generating \( x \) and \( y \). Observe also that when you compare with Scikit-Learn, you need to pay attention to how the intercept is dealt with. +

    -

    -Finally, using Scikit-Learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as +

    Finally, using Scikit-Learn or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as

    $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$ -and the \( R^2 \) score function. +

    and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

    $$ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$ -where we have defined the mean value of \( \hat{y} \) as +

    where we have defined the mean value of \( \hat{y} \) as

    $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods. - +

    Discuss these quantities as functions of the variable \( \lambda \) in the Ridge and Lasso regression methods.

    Exercise: Linear Regression for a two-dimensional function

    -

    -This is a longer exercise and the aim is to study in more detail various +

    This is a longer exercise and the aim is to study in more detail various regression methods, including the Ordinary Least Squares (OLS) method, Ridge regression and finally Lasso regression. This exercise forms a part of project 1. +

    -

    -We will study how to fit polynomials to a specific +

    We will study how to fit polynomials to a specific two-dimensional function called Franke's function. This is a function which has been widely used when testing various -interpolation and fitting algorithms. +interpolation and fitting algorithms. +

    -

    -The Franke function, which is a weighted sum of four exponentials reads as follows +

    The Franke function, which is a weighted sum of four exponentials reads as follows

    $$ \begin{align*} f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)}+\frac{3}{4}\exp{\left(-\frac{(9x+1)^2}{49}- \frac{(9y+1)}{10}\right)} \\ @@ -3184,8 +3586,7 @@ f(x,y) &= \frac{3}{4}\exp{\left(-\frac{(9x-2)^2}{4} - \frac{(9y-2)^2}{4}\right)} \end{align*} $$ -

    -The function will be defined for \( x,y\in [0,1] \). Our first step will +

    The function will be defined for \( x,y\in [0,1] \). Our first step will be to perform an OLS regression analysis of this function, trying out a polynomial fit with an \( x \) and \( y \) dependence of the form \( [x, y, x^2, y^2, xy, \dots] \). We will fit a @@ -3193,13 +3594,17 @@ function (for example a polynomial) of \( x \) and \( y \). Thereafter we will repeat much of the same procedure using the Ridge and Lasso regression methods, introducing thus a dependence on the bias (penalty) \( \lambda \). +

    -

    -The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it) -

    +

    The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)

    -
    from mpl_toolkits.mplot3d import Axes3D
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from mpl_toolkits.mplot3d import Axes3D
 import matplotlib.pyplot as plt
 from matplotlib import cm
 from matplotlib.ticker import LinearLocator, FormatStrFormatter
@@ -3238,51 +3643,65 @@ ax.zaxis..colorbar(surf, shrink=0.5, aspect=5)
 
 plt.show()
-
    
-
    
-We will generate our own dataset for a function
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We will generate our own dataset for a function \( \mathrm{FrankeFunction}(x,y) \) with \( x,y \in [0,1] \). The function \( f(x,y) \) is the Franke function. You should explore also the addition an added stochastic noise to this function using the normal distribution \( \cal{N}(0,1) \). +

    -

    -Write your own code (using either a matrix inversion or a singular +

    Write your own code (using either a matrix inversion or a singular value decomposition from e.g., numpy ) or use your code and perform a standard least square regression analysis using polynomials in \( x \) and \( y \) up to fifth order. You can use scikit-learn as well. +

    -

    -Evaluate the Mean Squared error (MSE) +

    Evaluate the Mean Squared error (MSE)

    $$ MSE(\hat{y},\hat{\tilde{y}}) = \frac{1}{n} \sum_{i=0}^{n-1}(y_i-\tilde{y}_i)^2, $$ -

    -and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted +

    and the \( R^2 \) score function. If \( \tilde{\hat{y}}_i \) is the predicted value of the \( i-th \) sample and \( y_i \) is the corresponding true value, then the score \( R^2 \) is defined as +

    $$ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)^2}{\sum_{i=0}^{n - 1} (y_i - \bar{y})^2}, $$ -

    -where we have defined the mean value of \( \hat{y} \) as +

    where we have defined the mean value of \( \hat{y} \) as

    $$ \bar{y} = \frac{1}{n} \sum_{i=0}^{n - 1} y_i. $$ -

    -You should split your data in train and test and also consider scaling the data. +

    You should split your data in train and test and also consider scaling the data.

    -

    -To set up the design matrix, the following code can be used -

    +

    To set up the design matrix, the following code can be used

    -
    def FrankeFunction(x,y):
+
    
+  
    
+    
    
+      
    
+        
    
+  
    def FrankeFunction(x,y):
 	term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
 	term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
 	term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
@@ -3314,33 +3733,40 @@ x = np.s
 y = np.sort(np.random.uniform(0, 1, N))
 z = FrankeFunction(x, y)
 X = create_X(x, y, n=n)    
-
    
-
    
-Write then your own code for the Ridge method or use Scikit-Learn.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    Write then your own code for the Ridge method or use Scikit-Learn. Perform the same analysis as you did for ordinary Least Squares (for the same polynomials) but now for different values of \( \lambda \). Compare and analyze your results with those obtained with ordinary Least Squares. Study the dependence on \( \lambda \). +

    -

    -This part is essentially a repeat of the previous ones, but now +

    This part is essentially a repeat of the previous ones, but now with Lasso regression. Write either your own code or use the functionalities of Scikit-Learn (recommended). Give a critical discussion of the three methods and a judgement of which model fits the data best. +

    -

    - - -

    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 496a64a27..f2a68d3a0 100644 Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 56dc72629..cb0902c63 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2,21 +2,38 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "id": "26ae76a1", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "890d8431", + "metadata": { + "editable": true + }, "source": [ - "\n", "# Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression\n", - "\n", - " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 28, 2021**\n", - "\n", - "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", - "\n", - "\n", - "\n", + "Date: **Nov 3, 2021**\n", "\n", + "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" + ] + }, + { + "cell_type": "markdown", + "id": "434881ac", + "metadata": { + "editable": true + }, + "source": [ "## Plans for week 35, August 30 -September 3\n", "\n", "* Thursday: Review of ordinary Least Squares with applications and discussion of Ridge Regression and Singular Value Decomposition\n", @@ -27,8 +44,16 @@ "\n", "* [Video of lecture Friday](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember3.mp4?vrtx=view-as-webpage)\n", "\n", - "* [Video series on the SVD](http://databookuw.com/page-2/page-4/). Highly recommended.\n", - "\n", + "* [Video series on the SVD](http://databookuw.com/page-2/page-4/). Highly recommended." + ] + }, + { + "cell_type": "markdown", + "id": "797fbc70", + "metadata": { + "editable": true + }, + "source": [ "## Thursday September 2\n", "\n", "The main topics on Thursday are:\n", @@ -38,8 +63,16 @@ "\n", "3. Mathematical interpretations of Linear Regression\n", "\n", - "4. Start discussing Ridge and Lasso regression and Singular Value Decomposition\n", - "\n", + "4. Start discussing Ridge and Lasso regression and Singular Value Decomposition" + ] + }, + { + "cell_type": "markdown", + "id": "a28f0549", + "metadata": { + "editable": true + }, + "source": [ "## Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week\n", "\n", "We need first a reminder from last week about linear regression. \n", @@ -64,9 +97,16 @@ "* and many more features\n", "\n", "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", - "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", - "\n", - "\n", + "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended." + ] + }, + { + "cell_type": "markdown", + "id": "9f12cdfb", + "metadata": { + "editable": true + }, + "source": [ "## Regression analysis, overarching aims\n", "\n", "Regression modeling deals with the description of the sampling distribution of a given random variable $y$ and how it varies as function of another variable or a set of such variables $\\boldsymbol{x} =[x_0, x_1,\\dots, x_{n-1}]^T$. \n", @@ -79,13 +119,18 @@ "\n", "* $p$ so-called explanatory (independent or predictor) variables $\\boldsymbol{x}_i=[x_{i0}, x_{i1}, \\dots, x_{ip-1}]$ with $i = 0, 1, 2, \\dots, n-1$ and explanatory variables running from $0$ to $p-1$. See below for more explicit examples. \n", "\n", - " The goal of the regression analysis is to extract/exploit relationship between $\\boldsymbol{y}$ and $\\boldsymbol{x}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things.\n", - "\n", - "\n", - "\n", + " The goal of the regression analysis is to extract/exploit relationship between $\\boldsymbol{y}$ and $\\boldsymbol{x}$ in or to infer causal dependencies, approximations to the likelihood functions, functional relationships and to make predictions, making fits and many other things." + ] + }, + { + "cell_type": "markdown", + "id": "fe22bbd9", + "metadata": { + "editable": true + }, + "source": [ "## Regression analysis, overarching aims II\n", "\n", - "\n", "Consider an experiment in which $p$ characteristics of $n$ samples are\n", "measured. The data from this experiment, for various explanatory variables $p$ are normally represented by a matrix \n", "$\\mathbf{X}$.\n", @@ -102,12 +147,16 @@ "the *linear regression model* where $\\boldsymbol{\\beta} = [\\beta_0, \\ldots,\n", "\\beta_{p-1}]^{T}$ are the *regression parameters*. \n", "\n", - "Linear regression gives us a set of analytical equations for the parameters $\\beta_j$.\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "Linear regression gives us a set of analytical equations for the parameters $\\beta_j$." + ] + }, + { + "cell_type": "markdown", + "id": "88accdb4", + "metadata": { + "editable": true + }, + "source": [ "## Examples\n", "In order to understand the relation among the predictors $p$, the set of data $n$ and the target (outcome, output etc) $\\boldsymbol{y}$,\n", "consider the model we discussed for describing nuclear binding energies. \n", @@ -118,7 +167,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9e3bb9b4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "BE(A) = a_0+a_1A+a_2A^{2/3}+a_3A^{-1/3}+a_4A^{-1},\n", @@ -127,21 +179,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e7cf3568", + "metadata": { + "editable": true + }, "source": [ "we have five predictors, that is the intercept, the $A$ dependent term, the $A^{2/3}$ term and the $A^{-1/3}$ and $A^{-1}$ terms.\n", "This gives $p=0,1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is a \n", "$p\\times n$ matrix $\\boldsymbol{X}$.\n", "\n", "Here the predictors are based on a model we have made. A popular data set which is widely encountered in ML applications is the\n", - "so-called [credit card default data from Taiwan](https://www.sciencedirect.com/science/article/pii/S0957417407006719?via%3Dihub). The data set contains data on $n=30000$ credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are $24$ such predictors or attributes leading to a design matrix of dimensionality $24 \\times 30000$. This is however a classification problem and we will come back to it when we discuss Logistic Regression.\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "so-called [credit card default data from Taiwan](https://www.sciencedirect.com/science/article/pii/S0957417407006719?via%3Dihub). The data set contains data on $n=30000$ credit card holders with predictors like gender, marital status, age, profession, education, etc. In total there are $24$ such predictors or attributes leading to a design matrix of dimensionality $24 \\times 30000$. This is however a classification problem and we will come back to it when we discuss Logistic Regression." + ] + }, + { + "cell_type": "markdown", + "id": "28d154ab", + "metadata": { + "editable": true + }, + "source": [ "## General linear models\n", "Before we proceed let us study a case from linear algebra where we aim at fitting a set of data $\\boldsymbol{y}=[y_0,y_1,\\dots,y_{n-1}]$. We could think of these data as a result of an experiment or a complicated numerical experiment. These data are functions of a series of variables $\\boldsymbol{x}=[x_0,x_1,\\dots,x_{n-1}]$, that is $y_i = y(x_i)$ with $i=0,1,2,\\dots,n-1$. The variables $x_i$ could represent physical quantities like time, temperature, position etc. We assume that $y(x)$ is a smooth function. \n", "\n", @@ -150,7 +207,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "471571ac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y=y(x) \\rightarrow y(x_i)=\\tilde{y}_i+\\epsilon_i=\\sum_{j=0}^{n-1} \\beta_j x_i^j+\\epsilon_i,\n", @@ -159,20 +219,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "55f3ec65", + "metadata": { + "editable": true + }, + "source": [ + "where $\\epsilon_i$ is the error in our approximation." + ] + }, + { + "cell_type": "markdown", + "id": "a8364a31", + "metadata": { + "editable": true + }, "source": [ - "where $\\epsilon_i$ is the error in our approximation.\n", - "\n", - "\n", - "\n", - "\n", "## Rewriting the fitting procedure as a linear algebra problem\n", "For every set of values $y_i,x_i$ we have thus the corresponding set of equations" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4a97ef95", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -187,7 +258,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f6fa4c72", + "metadata": { + "editable": true + }, "source": [ "## Rewriting the fitting procedure as a linear algebra problem, more details\n", "Defining the vectors" @@ -195,7 +269,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c9ca1dc8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y} = [y_0,y_1, y_2,\\dots, y_{n-1}]^T,\n", @@ -204,14 +281,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8b769bbc", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1bb26ed0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta} = [\\beta_0,\\beta_1, \\beta_2,\\dots, \\beta_{n-1}]^T,\n", @@ -220,14 +303,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9dda37f7", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "01376567", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon} = [\\epsilon_0,\\epsilon_1, \\epsilon_2,\\dots, \\epsilon_{n-1}]^T,\n", @@ -236,14 +325,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6d470e53", + "metadata": { + "editable": true + }, "source": [ "and the design matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "923ff5eb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\n", @@ -259,14 +354,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "15ccdf9e", + "metadata": { + "editable": true + }, "source": [ "we can rewrite our equations as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b7d4c290", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n", @@ -275,13 +376,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "27af50a7", + "metadata": { + "editable": true + }, + "source": [ + "The above design matrix is called a [Vandermonde matrix](https://en.wikipedia.org/wiki/Vandermonde_matrix)." + ] + }, + { + "cell_type": "markdown", + "id": "7dd2cd83", + "metadata": { + "editable": true + }, "source": [ - "The above design matrix is called a [Vandermonde matrix](https://en.wikipedia.org/wiki/Vandermonde_matrix).\n", - "\n", - "\n", - "\n", - "\n", "## Generalizing the fitting procedure as a linear algebra problem\n", "\n", "We are obviously not limited to the above polynomial expansions. We\n", @@ -293,7 +402,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "635ecdc8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -310,20 +422,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "24e2fd35", + "metadata": { + "editable": true + }, + "source": [ + "**Note that we have $p=n$ here. The matrix is symmetric. This is generally not the case!**" + ] + }, + { + "cell_type": "markdown", + "id": "480d9d03", + "metadata": { + "editable": true + }, "source": [ - "**Note that we have $p=n$ here. The matrix is symmetric. This is generally not the case!**\n", - "\n", - "\n", - "\n", - "\n", "## Generalizing the fitting procedure as a linear algebra problem\n", "We redefine in turn the matrix $\\boldsymbol{X}$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3d9b2af3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\n", @@ -339,14 +462,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a3ced18b", + "metadata": { + "editable": true + }, "source": [ "and without loss of generality we rewrite again our equations as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1f72e12d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y} = \\boldsymbol{X}\\boldsymbol{\\beta}+\\boldsymbol{\\epsilon}.\n", @@ -355,20 +484,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "47eda213", + "metadata": { + "editable": true + }, + "source": [ + "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values?" + ] + }, + { + "cell_type": "markdown", + "id": "169cfe9d", + "metadata": { + "editable": true + }, "source": [ - "The left-hand side of this equation is kwown. Our error vector $\\boldsymbol{\\epsilon}$ and the parameter vector $\\boldsymbol{\\beta}$ are our unknow quantities. How can we obtain the optimal set of $\\beta_i$ values?\n", - "\n", - "\n", - "\n", - "\n", "## Optimizing our parameters\n", "We have defined the matrix $\\boldsymbol{X}$ via the equations" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a2180298", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -385,15 +525,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d515ebce", + "metadata": { + "editable": true + }, "source": [ "As we noted above, we stayed with a system with the design matrix \n", " $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$, that is we have $p=n$. For reasons to come later (algorithmic arguments) we will hereafter define \n", - "our matrix as $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors refering to the column numbers and the entries $n$ being the row elements.\n", - "\n", - "\n", - "\n", - "\n", + "our matrix as $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors refering to the column numbers and the entries $n$ being the row elements." + ] + }, + { + "cell_type": "markdown", + "id": "7d67b025", + "metadata": { + "editable": true + }, + "source": [ "## Our model for the nuclear binding energies\n", "\n", "In our [introductory notes](https://compphysics.github.io/MachineLearning/doc/pub/How2ReadData/html/How2ReadData.html) we looked at the so-called [liquid drop model](https://en.wikipedia.org/wiki/Semi-empirical_mass_formula). Let us remind ourselves about what we did by looking at the code.\n", @@ -403,7 +551,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, + "id": "be62aa5c", "metadata": { "collapsed": false, "editable": true @@ -485,14 +634,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3ae8b693", + "metadata": { + "editable": true + }, "source": [ "With $\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p\\times 1}$, it means that we will hereafter write our equations for the approximation as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "bbb85729", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -501,18 +656,31 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5acf922c", + "metadata": { + "editable": true + }, + "source": [ + "throughout these lectures." + ] + }, + { + "cell_type": "markdown", + "id": "919a84f2", + "metadata": { + "editable": true + }, "source": [ - "throughout these lectures. \n", - "\n", - "\n", "## Optimizing our parameters, more details\n", "With the above we use the design matrix to define the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity $\\boldsymbol{\\beta}$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1b793777", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -521,14 +689,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7556c68c", + "metadata": { + "editable": true + }, "source": [ "and in order to find the optimal parameters $\\beta_i$ instead of solving the above linear algebra problem, we define a function which gives a measure of the spread between the values $y_i$ (which represent hopefully the exact values) and the parameterized values $\\tilde{y}_i$, namely" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "aa335b7c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", @@ -537,14 +711,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b33853bc", + "metadata": { + "editable": true + }, "source": [ "or using the matrix $\\boldsymbol{X}$ and in a more compact matrix-vector notation as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b6930121", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -553,19 +733,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6fd23574", + "metadata": { + "editable": true + }, "source": [ "This function is one possible way to define the so-called cost function.\n", "\n", - "\n", - "\n", "It is also common to define\n", "the function $C$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "e1214ba2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n", @@ -574,13 +758,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c51c03b5", + "metadata": { + "editable": true + }, + "source": [ + "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out." + ] + }, + { + "cell_type": "markdown", + "id": "fbeeb5c1", + "metadata": { + "editable": true + }, "source": [ - "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out.\n", - "\n", - "\n", - "\n", - "\n", "## Interpretations and optimizing our parameters\n", "\n", "The function" @@ -588,7 +780,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bbf3d893", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n", @@ -597,7 +792,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0ed621c2", + "metadata": { + "editable": true + }, "source": [ "can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. \n", "When linking (see the discussion below) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value" @@ -605,7 +803,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9be95ee2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n", @@ -614,7 +815,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "234ddf49", + "metadata": { + "editable": true + }, "source": [ "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", "till now we have treated $y_i$ as the exact value. Normally, the\n", @@ -630,7 +834,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ef177e0d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -640,14 +847,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "066c1c10", + "metadata": { + "editable": true + }, "source": [ "In practical terms it means we will require" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "e0e0d740", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n", @@ -656,14 +869,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "623edac9", + "metadata": { + "editable": true + }, "source": [ "which results in" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "787d2117", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n", @@ -672,14 +891,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "877597cf", + "metadata": { + "editable": true + }, "source": [ "or in a matrix-vector form as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8d770e9d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", @@ -688,7 +913,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b1dfe8bf", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "We can rewrite" @@ -696,7 +924,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0c3505c9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n", @@ -705,14 +936,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6096739c", + "metadata": { + "editable": true + }, "source": [ "as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "435f4a55", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -721,14 +958,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dbf7ae4d", + "metadata": { + "editable": true + }, "source": [ "and if the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ is invertible we have the solution" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "6745642f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -737,7 +980,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ddda6f20", + "metadata": { + "editable": true + }, "source": [ "We note also that since our design matrix is defined as $\\boldsymbol{X}\\in\n", "{\\mathbb{R}}^{n\\times p}$, the product $\\boldsymbol{X}^T\\boldsymbol{X} \\in\n", @@ -750,12 +996,16 @@ "allow for the usage of direct linear algebra methods such as **LU** decomposition or **Singular Value Decomposition** (SVD) for finding the inverse of the matrix\n", "$\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", - "\n", - "\n", - "**Small question**: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$? What kind of problems can we expect?\n", - "\n", - "\n", - "\n", + "**Small question**: Do you think the example we have at hand here (the nuclear binding energies) can lead to problems in inverting the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$? What kind of problems can we expect?" + ] + }, + { + "cell_type": "markdown", + "id": "b9cf4eae", + "metadata": { + "editable": true + }, + "source": [ "## Some useful matrix and vector expressions\n", "\n", "The following matrix and vector relation will be useful here and for the rest of the course. Vectors are always written as boldfaced lower case letters and \n", @@ -764,79 +1014,46 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "be7471e0", + "metadata": { + "editable": true + }, "source": [ - "2\n", - "6\n", - " \n", - "<\n", - "<\n", - "<\n", - "!\n", - "!\n", - "M\n", - "A\n", - "T\n", - "H\n", - "_\n", - "B\n", - "L\n", - "O\n", - "C\n", - "K" + "$$\n", + "\\frac{\\partial (\\boldsymbol{b}^T\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{b},\n", + "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "345c6f7c", + "metadata": { + "editable": true + }, "source": [ - "2\n", - "7\n", - " \n", - "<\n", - "<\n", - "<\n", - "!\n", - "!\n", - "M\n", - "A\n", - "T\n", - "H\n", - "_\n", - "B\n", - "L\n", - "O\n", - "C\n", - "K" + "$$\n", + "\\frac{\\partial (\\boldsymbol{a}^T\\boldsymbol{A}\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = (\\boldsymbol{A}+\\boldsymbol{A}^T)\\boldsymbol{a},\n", + "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "9fd2aec7", + "metadata": { + "editable": true + }, "source": [ - "2\n", - "8\n", - " \n", - "<\n", - "<\n", - "<\n", - "!\n", - "!\n", - "M\n", - "A\n", - "T\n", - "H\n", - "_\n", - "B\n", - "L\n", - "O\n", - "C\n", - "K" + "$$\n", + "\\frac{\\partial tr(\\boldsymbol{B}\\boldsymbol{A})}{\\partial \\boldsymbol{A}} = \\boldsymbol{B}^T,\n", + "$$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8f327bae", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\log{\\vert\\boldsymbol{A}\\vert}}{\\partial \\boldsymbol{A}} = (\\boldsymbol{A}^{-1})^T.\n", @@ -845,7 +1062,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "37b8427f", + "metadata": { + "editable": true + }, "source": [ "## Meet the Hessian Matrix\n", "\n", @@ -858,7 +1078,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "84f05dc0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -867,14 +1090,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f6b69e17", + "metadata": { + "editable": true + }, "source": [ "The Hessian matrix plays an important role and is defined here as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5ad43036", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -883,7 +1112,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e0697e67", + "metadata": { + "editable": true + }, "source": [ "For ordinary least squares, it is inversely proportional (derivation\n", "next week) with the variance of the optimal parameters\n", @@ -893,15 +1125,26 @@ "Analysis as a way to reduce the dimensionality of a machine learning\n", "problem.\n", "\n", - "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). \n", - "\n", + "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!)." + ] + }, + { + "cell_type": "markdown", + "id": "318716a4", + "metadata": { + "editable": true + }, + "source": [ "## Interpretations and optimizing our parameters\n", "The residuals $\\boldsymbol{\\epsilon}$ are in turn given by" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "9bf30c19", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -910,14 +1153,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c6505dc1", + "metadata": { + "editable": true + }, "source": [ "and with" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1271eed1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", @@ -926,14 +1175,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d38159b8", + "metadata": { + "editable": true + }, "source": [ "we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d7bb1d6d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", @@ -942,13 +1197,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b7dd96c9", + "metadata": { + "editable": true + }, + "source": [ + "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach." + ] + }, + { + "cell_type": "markdown", + "id": "4b6e485f", + "metadata": { + "editable": true + }, "source": [ - "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals. Later we will link this with the maximum likelihood approach.\n", - "\n", - "\n", - "\n", - "\n", "## Own code for Ordinary Least Squares\n", "\n", "It is rather straightforward to implement the matrix inversion and obtain the parameters $\\boldsymbol{\\beta}$. After having defined the matrix $\\boldsymbol{X}$ we simply need to \n", @@ -957,7 +1220,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, + "id": "20e8cb33", "metadata": { "collapsed": false, "editable": true @@ -972,14 +1236,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "77ef22e3", + "metadata": { + "editable": true + }, "source": [ "Alternatively, you can use the least squares functionality in **Numpy** as" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, + "id": "0c7e2e85", "metadata": { "collapsed": false, "editable": true @@ -992,14 +1260,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "776d3539", + "metadata": { + "editable": true + }, "source": [ "And finally we plot our fit with and compare with data" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, + "id": "fe76261f", "metadata": { "collapsed": false, "editable": true @@ -1022,7 +1294,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "34d28ed3", + "metadata": { + "editable": true + }, "source": [ "## Adding error analysis and training set up\n", "\n", @@ -1032,7 +1307,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, + "id": "46c39015", "metadata": { "collapsed": false, "editable": true @@ -1045,14 +1321,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "09150482", + "metadata": { + "editable": true + }, "source": [ "and we would be using it as" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, + "id": "c60fc160", "metadata": { "collapsed": false, "editable": true @@ -1064,14 +1344,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ea9d7abe", + "metadata": { + "editable": true + }, "source": [ "We can easily add our **MSE** score as" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, + "id": "7399848c", "metadata": { "collapsed": false, "editable": true @@ -1087,14 +1371,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0ca43616", + "metadata": { + "editable": true + }, "source": [ "and finally the relative error as" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, + "id": "602b9969", "metadata": { "collapsed": false, "editable": true @@ -1108,11 +1396,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c6c9d23d", + "metadata": { + "editable": true + }, "source": [ "## Splitting our Data in Training and Test data\n", "\n", - "\n", "It is normal in essentially all Machine Learning studies to split the\n", "data in a training set and a test set (sometimes also an additional\n", "validation set). **Scikit-Learn** has an own function for this. There\n", @@ -1122,17 +1412,23 @@ "postpone a discussion of this splitting to the end of these notes and\n", "our discussion of the so-called **bias-variance** tradeoff. Here we\n", "limit ourselves to repeat the above equation of state fitting example\n", - "but now splitting the data into a training set and a test set.\n", - "\n", - "\n", - "\n", - "\n", + "but now splitting the data into a training set and a test set." + ] + }, + { + "cell_type": "markdown", + "id": "46f46d43", + "metadata": { + "editable": true + }, + "source": [ "## Examples" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, + "id": "dd175e90", "metadata": { "collapsed": false, "editable": true @@ -1181,14 +1477,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1afc59cc", + "metadata": { + "editable": true + }, "source": [ "## Making your own test-train splitting" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 10, + "id": "ec006afe", "metadata": { "collapsed": false, "editable": true @@ -1213,14 +1513,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a7ec2fdb", + "metadata": { + "editable": true + }, "source": [ "But since **scikit-learn** has its own function for doing this and since\n", "it interfaces easily with **tensorflow** and other libraries, we\n", - "normally recommend using the latter functionality.\n", - "\n", - "\n", - "\n", + "normally recommend using the latter functionality." + ] + }, + { + "cell_type": "markdown", + "id": "875d7248", + "metadata": { + "editable": true + }, + "source": [ "## The Boston housing data example\n", "\n", "The Boston housing \n", @@ -1255,15 +1564,24 @@ "\n", "12. LSTAT: Percentage of lower status of the population\n", "\n", - "13. MEDV: Median value of owner-occupied homes in USD 1000s\n", - "\n", + "13. MEDV: Median value of owner-occupied homes in USD 1000s" + ] + }, + { + "cell_type": "markdown", + "id": "c0f8545d", + "metadata": { + "editable": true + }, + "source": [ "## Housing data, the code\n", "We start by importing the libraries" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 11, + "id": "81237355", "metadata": { "collapsed": false, "editable": true @@ -1279,14 +1597,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f1962254", + "metadata": { + "editable": true + }, "source": [ "and load the Boston Housing DataSet from **Scikit-Learn**" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 12, + "id": "dee9e300", "metadata": { "collapsed": false, "editable": true @@ -1304,14 +1626,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bef0eb82", + "metadata": { + "editable": true + }, "source": [ "Then we invoke Pandas" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 13, + "id": "a29ff8af", "metadata": { "collapsed": false, "editable": true @@ -1325,14 +1651,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8396e85f", + "metadata": { + "editable": true + }, "source": [ "and preprocess the data" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 14, + "id": "16d25b94", "metadata": { "collapsed": false, "editable": true @@ -1345,14 +1675,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2e90d9ef", + "metadata": { + "editable": true + }, "source": [ "We can then visualize the data" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 15, + "id": "ac2fac03", "metadata": { "collapsed": false, "editable": true @@ -1369,14 +1703,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b5cdd9d1", + "metadata": { + "editable": true + }, "source": [ "It is now useful to look at the correlation matrix" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 16, + "id": "37a352c6", "metadata": { "collapsed": false, "editable": true @@ -1392,14 +1730,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d43fa5cd", + "metadata": { + "editable": true + }, "source": [ "From the above coorelation plot we can see that **MEDV** is strongly correlated to **LSTAT** and **RM**. We see also that **RAD** and **TAX** are stronly correlated, but we don't include this in our features together to avoid multi-colinearity" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 17, + "id": "1fc2b42e", "metadata": { "collapsed": false, "editable": true @@ -1423,14 +1765,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "69cafd8a", + "metadata": { + "editable": true + }, "source": [ "Now we start training our model" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 18, + "id": "48a04777", "metadata": { "collapsed": false, "editable": true @@ -1443,14 +1789,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e21c7f03", + "metadata": { + "editable": true + }, "source": [ "We split the data into training and test sets" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 19, + "id": "fa0eca85", "metadata": { "collapsed": false, "editable": true @@ -1470,14 +1820,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "598473cc", + "metadata": { + "editable": true + }, "source": [ "Then we use the linear regression functionality from **Scikit-Learn**" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 20, + "id": "adc72931", "metadata": { "collapsed": false, "editable": true @@ -1519,7 +1873,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 21, + "id": "5bf0e806", "metadata": { "collapsed": false, "editable": true @@ -1534,7 +1889,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "115406d0", + "metadata": { + "editable": true + }, "source": [ "## Reducing the number of degrees of freedom, overarching view\n", "\n", @@ -1550,17 +1908,21 @@ "techniques: the principal component analysis (PCA), Kernel PCA, and\n", "Locally Linear Embedding (LLE). \n", "\n", - "\n", "Principal component analysis and its various variants deal with the\n", "problem of fitting a low-dimensional [affine\n", "subspace](https://en.wikipedia.org/wiki/Affine_space) to a set of of\n", "data points in a high-dimensional space. With its family of methods it\n", "is one of the most used tools in data modeling, compression and\n", - "visualization.\n", - "\n", - "\n", - "\n", - "\n", + "visualization." + ] + }, + { + "cell_type": "markdown", + "id": "98df56e6", + "metadata": { + "editable": true + }, + "source": [ "## Preprocessing our data\n", "\n", "Before we proceed however, we will discuss how to preprocess our\n", @@ -1576,11 +1938,16 @@ "features such as **age** in the range $0-80$, and **caloric intake** of order $2000$.\n", "Many machine learning methods sensitive to the scales of the features and may perform poorly if they\n", "are very different scales. Therefore, it is typical to scale\n", - "the features in a way to avoid such outlier values.\n", - "\n", - "\n", - "\n", - "\n", + "the features in a way to avoid such outlier values." + ] + }, + { + "cell_type": "markdown", + "id": "1de59215", + "metadata": { + "editable": true + }, + "source": [ "## Functionality in Scikit-Learn\n", "\n", "**Scikit-Learn** has several functions which allow us to rescale the\n", @@ -1591,8 +1958,16 @@ "matrix). This scaling has the drawback that it does not ensure that\n", "we have a particular maximum or minimum in our data set. Another\n", "function included in **Scikit-Learn** is the **MinMaxScaler** which\n", - "ensures that all features are exactly between $0$ and $1$. The\n", - "\n", + "ensures that all features are exactly between $0$ and $1$. The" + ] + }, + { + "cell_type": "markdown", + "id": "e1c1b7ee", + "metadata": { + "editable": true + }, + "source": [ "## More preprocessing\n", "\n", "The **Normalizer** scales each data\n", @@ -1610,21 +1985,28 @@ "RobustScaler ignore data points that are very different from the rest\n", "(like measurement errors). These odd data points are also called\n", "outliers, and might often lead to trouble for other scaling\n", - "techniques.\n", - "\n", - "\n", - "\n", - "\n", + "techniques." + ] + }, + { + "cell_type": "markdown", + "id": "a35f95ed", + "metadata": { + "editable": true + }, + "source": [ "## Frequently used scaling functions\n", "\n", - "\n", "Many features are often scaled using standardization to improve performance. In **Scikit-Learn** this is given by the **StandardScaler** function as discussed above. It is easy however to write your own. \n", "Mathematically, this involves subtracting the mean and divide by the standard deviation over the data set, for each feature:" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "804df448", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_j^{(i)} \\rightarrow \\frac{x_j^{(i)} - \\overline{x}_j}{\\sigma(x_j)},\n", @@ -1633,13 +2015,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8495ae4c", + "metadata": { + "editable": true + }, "source": [ "where $\\overline{x}_j$ and $\\sigma(x_j)$ are the mean and standard deviation, respectively, of the feature $x_j$.\n", - "This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one.\n", - "\n", - "\n", - "\n", + "This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one." + ] + }, + { + "cell_type": "markdown", + "id": "9ebd107f", + "metadata": { + "editable": true + }, + "source": [ "## Example of own Standard scaling\n", "\n", "Let us consider the following vanilla example where we use both\n", @@ -1650,7 +2041,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 22, + "id": "8b62f35a", "metadata": { "collapsed": false, "editable": true @@ -1684,10 +2076,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bbaf614f", + "metadata": { + "editable": true + }, + "source": [ + "Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives." + ] + }, + { + "cell_type": "markdown", + "id": "e297e64a", + "metadata": { + "editable": true + }, "source": [ - "Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.\n", - "\n", "## Min-Max Scaling\n", "\n", "Another commonly used scaling method is min-max scaling. This is very\n", @@ -1698,7 +2101,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3a48d095", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_j^{(i)} \\rightarrow (b-a)\\frac{x_j^{(i)} - \\min(x_j)}{\\max(x_j) - \\min(x_j)} - a\n", @@ -1707,23 +2113,33 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b836623d", + "metadata": { + "editable": true + }, + "source": [ + "where $\\min(x_j)$ and $\\max(x_j)$ return the minimum and maximum value of $x_j$ over the data set, respectively." + ] + }, + { + "cell_type": "markdown", + "id": "0cf9ca3e", + "metadata": { + "editable": true + }, "source": [ - "where $\\min(x_j)$ and $\\max(x_j)$ return the minimum and maximum value of $x_j$ over the data set, respectively.\n", - "\n", - "\n", "## Testing the Means Squared Error as function of Complexity\n", "One of \n", "the aims is to reproduce Figure 2.11 of [Hastie et al](https://github.com/CompPhysics/MLErasmus/blob/master/doc/Textbooks/elementsstat.pdf).\n", "We will also use Ridge and Lasso regression. \n", "\n", - "\n", "Our data is defined by $x\\in [-3,3]$ with a total of for example $100$ data points." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 23, + "id": "409e549f", "metadata": { "collapsed": false, "editable": true @@ -1740,7 +2156,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ed762bab", + "metadata": { + "editable": true + }, "source": [ "where $y$ is the function we want to fit with a given polynomial.\n", "\n", @@ -1749,7 +2168,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 24, + "id": "a2d186a5", "metadata": { "collapsed": false, "editable": true @@ -1796,14 +2216,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8f34fae7", + "metadata": { + "editable": true + }, "source": [ "## More preprocessing examples, Franke function and regression" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 25, + "id": "5fc132f0", "metadata": { "collapsed": false, "editable": true @@ -1906,19 +2330,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ad5b41a1", + "metadata": { + "editable": true + }, "source": [ "## Mathematical Interpretation of Ordinary Least Squares\n", "\n", "What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n", "\n", - "\n", "We have shown that in ordinary least squares the optimal parameters $\\beta$ are given by" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a93ce01c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1927,7 +2356,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2f89cf96", + "metadata": { + "editable": true + }, "source": [ "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n", "\n", @@ -1936,7 +2368,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1bdec2cd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1945,14 +2380,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d6090b0f", + "metadata": { + "editable": true + }, "source": [ "We now define a matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "71edebd0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T.\n", @@ -1961,14 +2402,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fc1b8e60", + "metadata": { + "editable": true + }, "source": [ "We can rewrite" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "c407cc78", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n", @@ -1977,13 +2424,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e674f439", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n", - "We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.\n", - "\n", - "\n", - "\n", + "We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix." + ] + }, + { + "cell_type": "markdown", + "id": "6b5a2484", + "metadata": { + "editable": true + }, + "source": [ "## Residual Error\n", "\n", "We have defined the residual error as" @@ -1991,7 +2447,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "631571fe", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=\\left[\\boldsymbol{I}-\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\right]\\boldsymbol{y}.\n", @@ -2000,10 +2459,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "03e7938e", + "metadata": { + "editable": true + }, + "source": [ + "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal component of the space defined by the column vectors of $\\boldsymbol{X}$." + ] + }, + { + "cell_type": "markdown", + "id": "89c1cbc1", + "metadata": { + "editable": true + }, "source": [ - "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal component of the space defined by the column vectors of $\\boldsymbol{X}$.\n", - "\n", "## Simple case\n", "\n", "If the matrix $\\boldsymbol{X}$ is an orthogonal (or unitary in case of complex values) matrix, we have" @@ -2011,7 +2481,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "997c8992", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{X}\\boldsymbol{X}^T = \\boldsymbol{I}.\n", @@ -2020,14 +2493,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7c3ad56e", + "metadata": { + "editable": true + }, "source": [ "In this case the matrix $\\boldsymbol{A}$ becomes" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4cc0e3f1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n", @@ -2036,14 +2515,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7304c055", + "metadata": { + "editable": true + }, "source": [ "and we have the obvious case" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0a8db981", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=0.\n", @@ -2052,15 +2537,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d04ddf92", + "metadata": { + "editable": true + }, + "source": [ + "This serves also as a useful test of our codes." + ] + }, + { + "cell_type": "markdown", + "id": "044bd0ca", + "metadata": { + "editable": true + }, "source": [ - "This serves also as a useful test of our codes. \n", - "\n", - "\n", - "\n", "## The singular value decomposition\n", "\n", - "\n", "The examples we have looked at so far are cases where we normally can\n", "invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion where we fit of various functions leads to\n", "row vectors of the design matrix which are essentially orthogonal due\n", @@ -2068,7 +2561,6 @@ "design matrix is then often done via a so-called LU, QR or Cholesky\n", "decomposition.\n", "\n", - "\n", "As we will also see in the first project, \n", "this may\n", "however not the be case in general and a standard matrix inversion\n", @@ -2090,12 +2582,16 @@ "to the covariance matrix (and thereby the correlation matrix) and in\n", "turn the variance of a given quantity. It plays also an important role\n", "in the principal component analysis where high-dimensional data can be\n", - "reduced to the statistically relevant features.\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "reduced to the statistically relevant features." + ] + }, + { + "cell_type": "markdown", + "id": "ef9e05c1", + "metadata": { + "editable": true + }, + "source": [ "## Linear Regression Problems\n", "\n", "One of the typical problems we encounter with linear regression, in particular \n", @@ -2108,7 +2604,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d63adf14", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -2128,7 +2627,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5e78a531", + "metadata": { + "editable": true + }, "source": [ "The columns of $\\boldsymbol{X}$ are linearly dependent. We see this easily since the \n", "the first column is the row-wise sum of the other two columns. The rank (more correct,\n", @@ -2142,7 +2644,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2744881a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -2158,12 +2663,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "037e81db", + "metadata": { + "editable": true + }, "source": [ "We see easily that $\\mbox{det}(\\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \\times (-1) - 1 \\times (-1) = 0$. Hence, $\\mathbf{X}$ is singular and its inverse is undefined.\n", - "This is equivalent to saying that the matrix $\\boldsymbol{X}$ has at least an eigenvalue which is zero.\n", - "\n", - "\n", + "This is equivalent to saying that the matrix $\\boldsymbol{X}$ has at least an eigenvalue which is zero." + ] + }, + { + "cell_type": "markdown", + "id": "f21e5f16", + "metadata": { + "editable": true + }, + "source": [ "## Fixing the singularity\n", "\n", "If our design matrix $\\boldsymbol{X}$ which enters the linear regression problem" @@ -2171,7 +2686,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "46f92b4a", + "metadata": { + "editable": true + }, "source": [ "\n", "
    \n", @@ -2186,7 +2704,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "30476b84", + "metadata": { + "editable": true + }, "source": [ "has linearly dependent column vectors, we will not be able to compute the inverse\n", "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n", @@ -2199,7 +2720,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "36716f3b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^{T} \\boldsymbol{X} \\rightarrow \\boldsymbol{X}^{T} \\boldsymbol{X}+\\lambda \\boldsymbol{I},\n", @@ -2208,15 +2732,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a56a9662", + "metadata": { + "editable": true + }, + "source": [ + "where $\\boldsymbol{I}$ is the identity matrix. When we discuss **Ridge** regression this is actually what we end up evaluating. The parameter $\\lambda$ is called a hyperparameter. More about this later." + ] + }, + { + "cell_type": "markdown", + "id": "9484e1b3", + "metadata": { + "editable": true + }, "source": [ - "where $\\boldsymbol{I}$ is the identity matrix. When we discuss **Ridge** regression this is actually what we end up evaluating. The parameter $\\lambda$ is called a hyperparameter. More about this later. \n", - "\n", - "\n", - "\n", "## Basic math of the SVD\n", "\n", - "\n", "From standard linear algebra we know that a square matrix $\\boldsymbol{X}$ can be diagonalized if and only it is \n", "a so-called [normal matrix](https://en.wikipedia.org/wiki/Normal_matrix), that is if $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times n}$\n", "we have $\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ or if $\\boldsymbol{X}\\in {\\mathbb{C}}^{n\\times n}$ we have $\\boldsymbol{X}\\boldsymbol{X}^{\\dagger}=\\boldsymbol{X}^{\\dagger}\\boldsymbol{X}$.\n", @@ -2225,7 +2757,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7d0e79fd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\lambda_1,\\boldsymbol{u}_1),\\dots, (\\lambda_n,\\boldsymbol{u}_n),\n", @@ -2234,14 +2769,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "957d1974", + "metadata": { + "editable": true + }, "source": [ "and the eigenvalues are given by the diagonal matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4da24ddd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\\mathrm{Diag}(\\lambda_1, \\dots,\\lambda_n).\n", @@ -2250,14 +2791,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "63934409", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ can be written in terms of an orthogonal/unitary transformation $\\boldsymbol{U}$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3710e8a3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -2266,7 +2813,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6fc2de6c", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ or $\\boldsymbol{U}\\boldsymbol{U}^{\\dagger}=\\boldsymbol{I}$.\n", "\n", @@ -2275,7 +2825,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5c7cbfc4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\begin{bmatrix} \n", @@ -2287,15 +2840,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "30e83681", + "metadata": { + "editable": true + }, "source": [ "is not diagonalizable, it is a so-called [defective matrix](https://en.wikipedia.org/wiki/Defective_matrix). It is easy to see that the condition\n", - "$\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ is not fulfilled. \n", - "\n", - "\n", + "$\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ is not fulfilled." + ] + }, + { + "cell_type": "markdown", + "id": "10f85da7", + "metadata": { + "editable": true + }, + "source": [ "## The SVD, a Fantastic Algorithm\n", "\n", - "\n", "However, and this is the strength of the SVD algorithm, any general\n", "matrix $\\boldsymbol{X}$ can be decomposed in terms of a diagonal matrix and\n", "two orthogonal/unitary matrices. The [Singular Value Decompostion\n", @@ -2309,7 +2871,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f9269daa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\n", @@ -2318,14 +2883,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "558c53b6", + "metadata": { + "editable": true + }, "source": [ "As an example, the above defective matrix can be decomposed as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5f32a26d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& 1 \\\\ 1& -1\\\\ \\end{bmatrix} \\begin{bmatrix} 2& 0 \\\\ 0& 0\\\\ \\end{bmatrix} \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& -1 \\\\ 1& 1\\\\ \\end{bmatrix}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -2334,7 +2905,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "42617846", + "metadata": { + "editable": true + }, "source": [ "with eigenvalues $\\sigma_1=2$ and $\\sigma_2=0$. \n", "The SVD exits always! \n", @@ -2355,8 +2929,16 @@ "the Ising model we have $p > n$. These are often cases that lead to\n", "near singular or singular matrices.\n", "\n", - "The columns of $\\boldsymbol{U}$ are called the left singular vectors while the columns of $\\boldsymbol{V}$ are the right singular vectors.\n", - "\n", + "The columns of $\\boldsymbol{U}$ are called the left singular vectors while the columns of $\\boldsymbol{V}$ are the right singular vectors." + ] + }, + { + "cell_type": "markdown", + "id": "cc45bdee", + "metadata": { + "editable": true + }, + "source": [ "## Economy-size SVD\n", "\n", "If we assume that $n > p$, then our matrix $\\boldsymbol{U}$ has dimension $n\n", @@ -2374,14 +2956,23 @@ "If $n > p$, we keep only the first $p$ columns of $\\boldsymbol{U}$ and $\\boldsymbol{\\Sigma}$ has dimension $p\\times p$. \n", "If $p > n$, then only the first $n$ columns of $\\boldsymbol{V}$ are computed and $\\boldsymbol{\\Sigma}$ has dimension $n\\times n$.\n", "The $n=p$ case is obvious, we retain the full SVD. \n", - "In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy.\n", - "\n", + "In general the economy-size SVD leads to less FLOPS and still conserving the desired accuracy." + ] + }, + { + "cell_type": "markdown", + "id": "f77eda9e", + "metadata": { + "editable": true + }, + "source": [ "## Codes for the SVD" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 26, + "id": "87edcbe8", "metadata": { "collapsed": false, "editable": true @@ -2421,7 +3012,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9594d07b", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ has columns that are linearly dependent. The first\n", "column is the row-wise sum of the other two columns. The rank of a\n", @@ -2430,9 +3024,16 @@ "independent columns, in this case just $2$. We see this from the\n", "singular values when running the above code. Running the standard\n", "inversion algorithm for matrix inversion with $\\boldsymbol{X}^T\\boldsymbol{X}$ results\n", - "in the program terminating due to a singular matrix.\n", - "\n", - "\n", + "in the program terminating due to a singular matrix." + ] + }, + { + "cell_type": "markdown", + "id": "83495077", + "metadata": { + "editable": true + }, + "source": [ "## Note about SVD Calculations\n", "\n", "The $U$, $S$, and $V$ matrices returned from the **svd()** function\n", @@ -2446,19 +3047,30 @@ "\n", "If you wish to include the zero singular values, you will need to\n", "resize the matrices and set up a diagonal matrix as done in the above\n", - "example\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "example" + ] + }, + { + "cell_type": "markdown", + "id": "1f811c23", + "metadata": { + "editable": true + }, + "source": [ "## Friday September 3\n", "\n", "[Video of Lecture from 2020](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember11.mp4?vrtx=view-as-webpage) and [handwritten notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember11.pdf)\n", "\n", - "More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at .\n", - "\n", - "\n", + "More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at ." + ] + }, + { + "cell_type": "markdown", + "id": "4a266ed5", + "metadata": { + "editable": true + }, + "source": [ "## Mathematics of the SVD and implications\n", "\n", "Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.\n", @@ -2468,7 +3080,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "175eb163", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -2484,14 +3099,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "19a5e14c", + "metadata": { + "editable": true + }, "source": [ "We can SVD decompose our matrix as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "11cf192a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -2500,7 +3121,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f3e550d5", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{U}$ is an orthogonal matrix of dimension $n\\times n$, meaning that $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{I}_n$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $n \\times n$.\n", "\n", @@ -2511,7 +3135,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d53bb578", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sigma_0 > \\sigma_1 > \\sigma_2 > \\dots > \\sigma_{p-1} > 0.\n", @@ -2520,10 +3147,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "57f2f89f", + "metadata": { + "editable": true + }, + "source": [ + "All values beyond $p-1$ are all zero." + ] + }, + { + "cell_type": "markdown", + "id": "0458c51f", + "metadata": { + "editable": true + }, "source": [ - "All values beyond $p-1$ are all zero.\n", - "\n", "## Example Matrix\n", "\n", "As an example, consider the following $3\\times 2$ example for the matrix $\\boldsymbol{\\Sigma}$" @@ -2531,7 +3169,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "58155af6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\n", @@ -2545,14 +3186,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a1996d9b", + "metadata": { + "editable": true + }, "source": [ "The singular values are $\\sigma_0=2$ and $\\sigma_1=1$. It is common to rewrite the matrix $\\boldsymbol{\\Sigma}$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "583c5e45", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\n", @@ -2565,14 +3212,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "988f38ae", + "metadata": { + "editable": true + }, "source": [ "where" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "542eb473", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{\\Sigma}}=\n", @@ -2585,14 +3238,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "81dfd087", + "metadata": { + "editable": true + }, "source": [ "contains only the singular values. Note also (and we will use this below) that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "98e874a5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\n", @@ -2605,14 +3264,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "431542eb", + "metadata": { + "editable": true + }, "source": [ "which is a $2\\times 2 $ matrix while" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3b821038", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T=\n", @@ -2626,13 +3291,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0279e1cb", + "metadata": { + "editable": true + }, "source": [ "is a $3\\times 3 $ matrix. The last row and column of this last matrix\n", "contain only zeros. This will have important consequences for our SVD\n", - "decomposition of the design matrix.\n", - "\n", - "\n", + "decomposition of the design matrix." + ] + }, + { + "cell_type": "markdown", + "id": "9d85ed00", + "metadata": { + "editable": true + }, + "source": [ "## Setting up the Matrix to be inverted\n", "\n", "The matrix that may cause problems for us is $\\boldsymbol{X}^T\\boldsymbol{X}$. Using the SVD we can rewrite this matrix as" @@ -2640,7 +3315,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4224af8b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -2649,14 +3327,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8312e008", + "metadata": { + "editable": true + }, "source": [ "and using the orthogonality of the matrix $\\boldsymbol{U}$ we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "9674e71f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -2665,7 +3349,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1dc69f6e", + "metadata": { + "editable": true + }, "source": [ "We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\\tilde{\\boldsymbol{\\Sigma}}^2$ which is a diagonal matrix containing only the singular values squared. It has dimensionality $p \\times p$.\n", "\n", @@ -2674,7 +3361,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ad5b0673", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -2683,14 +3373,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a955f323", + "metadata": { + "editable": true + }, "source": [ "We can now insert the result for the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ into our equation for ordinary least squares where" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "bb180718", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -2699,14 +3395,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d869817e", + "metadata": { + "editable": true + }, "source": [ "and using our SVD decomposition of $\\boldsymbol{X}$ we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f89bdf6f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\tilde{\\boldsymbol{\\Sigma}}^{-2}\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{y},\n", @@ -2715,14 +3417,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a0b18249", + "metadata": { + "editable": true + }, "source": [ "which gives us, using the orthogonality of the matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$," ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "638cd1cd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}^T_j\\boldsymbol{y},\n", @@ -2731,14 +3439,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3f302527", + "metadata": { + "editable": true + }, "source": [ "Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix $\\boldsymbol{U}$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8db82136", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{U}=[\\boldsymbol{u}_0,\\boldsymbol{u}_1,\\dots,\\boldsymbol{u}_{n-1}],\n", @@ -2747,12 +3461,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "cf0ac2c9", + "metadata": { + "editable": true + }, "source": [ "that belong to $i>p-1$, result in only zeros when we perform the multiplications. This means that the sum above has non-zero elements only up to $i=p-1$. This corresponds also to the number of singular values (these are all non-zero).\n", "\n", - "It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n", - "\n", + "It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$." + ] + }, + { + "cell_type": "markdown", + "id": "cfa84fdc", + "metadata": { + "editable": true + }, + "source": [ "## Further properties (important for our analyses later)\n", "\n", "Let us study again $\\boldsymbol{X}^T\\boldsymbol{X}$ in terms of our SVD," @@ -2760,7 +3485,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f308d879", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -2769,14 +3497,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "cfc8d50e", + "metadata": { + "editable": true + }, "source": [ "If we now multiply from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4578a076", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}.\n", @@ -2785,7 +3519,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ceaea02f", + "metadata": { + "editable": true + }, "source": [ "This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n", "with eigenvalues given by the singular values squared, that is" @@ -2793,7 +3530,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5595d033", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -2802,14 +3542,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a2b4b94e", + "metadata": { + "editable": true + }, "source": [ "Similarly, if we use the SVD decomposition for the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$, we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "af283d4c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T.\n", @@ -2818,14 +3564,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2bdfba47", + "metadata": { + "editable": true + }, "source": [ "If we now multiply from the right with $\\boldsymbol{U}$ (using the orthogonality of $\\boldsymbol{U}$) we get" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "bb3c855d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T.\n", @@ -2834,7 +3586,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f4d2ed1b", + "metadata": { + "editable": true + }, "source": [ "This means the vectors $\\boldsymbol{u}_i$ of the orthogonal matrix $\\boldsymbol{U}$ are the eigenvectors of the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "with eigenvalues given by the singular values squared, that is" @@ -2842,7 +3597,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3250e943", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{u}_i=\\boldsymbol{u}_i\\sigma_i^2.\n", @@ -2851,7 +3609,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "122744d7", + "metadata": { + "editable": true + }, "source": [ "**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n", "$n\\times p$ matrix. In most supervised learning cases we have that $n\n", @@ -2861,12 +3622,18 @@ "always refer to the number of features in our data set, while the\n", "number of rows represents the number of data inputs. Note that in\n", "other texts you may find the opposite notation. This has consequences\n", - "for the definition of for example the covariance matrix and its relation to the SVD.\n", - "\n", - "\n", + "for the definition of for example the covariance matrix and its relation to the SVD." + ] + }, + { + "cell_type": "markdown", + "id": "c970d800", + "metadata": { + "editable": true + }, + "source": [ "## Meet the Covariance Matrix\n", "\n", - "\n", "Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.\n", "\n", "We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary\n", @@ -2876,7 +3643,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1b3fd7ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -2885,7 +3655,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "622b35d2", + "metadata": { + "editable": true + }, "source": [ "This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n", "\n", @@ -2894,7 +3667,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bb65ea69", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -2903,14 +3679,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dd5b2301", + "metadata": { + "editable": true + }, "source": [ "The Hessian matrix for ordinary least squares is also proportional to\n", "the covariance matrix. This means also that we can use the SVD to find\n", "the eigenvalues of the covariance matrix and the Hessian matrix in\n", - "terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.\n", - "\n", - "\n", + "terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies." + ] + }, + { + "cell_type": "markdown", + "id": "1518b5a6", + "metadata": { + "editable": true + }, + "source": [ "## Introducing the Covariance and Correlation functions\n", "\n", "Before we discuss the link between for example Ridge regression and the singular value decomposition, we need to remind ourselves about\n", @@ -2922,7 +3708,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "32731b28", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -2933,14 +3722,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1546b8ab", + "metadata": { + "editable": true + }, "source": [ "where for example" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "7b3b0e00", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n", @@ -2949,14 +3744,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "404a1659", + "metadata": { + "editable": true + }, "source": [ "With this definition and recalling that the variance is defined as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3a1ff8da", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{var}[\\boldsymbol{x}]=\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})^2,\n", @@ -2965,14 +3766,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9e5fe3fe", + "metadata": { + "editable": true + }, "source": [ "we can rewrite the covariance matrix as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "e476e478", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -2983,7 +3790,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "035beba2", + "metadata": { + "editable": true + }, "source": [ "**Note:** we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. \n", "What you will find in essentially all statistics texts are equations\n", @@ -2992,8 +3802,16 @@ "and covariance. It also partially corrects the bias in the estimation\n", "of the population standard deviation. If you use a library like\n", "**Scikit-Learn** or **nunmpy's** function calculate the covariance, this\n", - "quantity will be computed with a factor $1/(n-1)$.\n", - "\n", + "quantity will be computed with a factor $1/(n-1)$." + ] + }, + { + "cell_type": "markdown", + "id": "ce0c8788", + "metadata": { + "editable": true + }, + "source": [ "## Covariance and Correlation Matrix\n", "\n", "The covariance takes values between zero and infinity and may thus\n", @@ -3005,7 +3823,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8ee8be1b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]=\\frac{\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}]}{\\sqrt{\\mathrm{var}[\\boldsymbol{x}] \\mathrm{var}[\\boldsymbol{y}]}}.\n", @@ -3014,7 +3835,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e05c764a", + "metadata": { + "editable": true + }, "source": [ "The correlation function is then given by values $\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]\n", "\\in [-1,1]$. This avoids eventual problems with too large values. We\n", @@ -3024,7 +3848,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e351ae83", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{K}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} 1 & \\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -3035,10 +3862,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "00e81faf", + "metadata": { + "editable": true + }, + "source": [ + "In the above example this is the function we constructed using **pandas**." + ] + }, + { + "cell_type": "markdown", + "id": "c2209483", + "metadata": { + "editable": true + }, "source": [ - "In the above example this is the function we constructed using **pandas**.\n", - "\n", "## Correlation Function and Design/Feature Matrix\n", "\n", "In our derivation of the various regression algorithms like **Ordinary Least Squares** or **Ridge regression**\n", @@ -3047,7 +3885,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fd757d09", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -3063,7 +3904,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d3ddca6a", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors/features $p$ refering to the column numbers and the\n", "entries $n$ being the row elements.\n", @@ -3072,7 +3916,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0819a630", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} \\boldsymbol{x}_0 & \\boldsymbol{x}_1 & \\boldsymbol{x}_2 & \\dots & \\dots & \\boldsymbol{x}_{p-1}\\end{bmatrix},\n", @@ -3081,14 +3928,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8d70123f", + "metadata": { + "editable": true + }, "source": [ "with a given vector" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "39032237", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T = \\begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \\dots & \\dots x_{n-1,i}\\end{bmatrix}.\n", @@ -3097,7 +3950,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "616bfec3", + "metadata": { + "editable": true + }, "source": [ "With these definitions, we can now rewrite our $2\\times 2$\n", "correlation/covariance matrix in terms of a moe general design/feature\n", @@ -3107,7 +3963,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3e0872f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix}\n", @@ -3123,14 +3982,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a95cb740", + "metadata": { + "editable": true + }, "source": [ "and the correlation matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "97b7477a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{K}[\\boldsymbol{x}] = \\begin{bmatrix}\n", @@ -3146,11 +4011,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "089f07ac", + "metadata": { + "editable": true + }, "source": [ "## Covariance Matrix Examples\n", "\n", - "\n", "The Numpy function **np.cov** calculates the covariance elements using\n", "the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have\n", "the exact mean values. The following simple function uses the\n", @@ -3162,7 +4029,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "25fd911d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{W} = \\begin{bmatrix} x_0 & x_1 & x_2 & \\dots & x_{n-2} & x_{n-1} \\\\\n", @@ -3173,7 +4043,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f03f377c", + "metadata": { + "editable": true + }, "source": [ "which in turn is converted into into the $2\\times 2$ covariance matrix\n", "$\\boldsymbol{C}$ via the Numpy function **np.cov()**. We note that we can also calculate\n", @@ -3184,7 +4057,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 27, + "id": "b1661961", "metadata": { "collapsed": false, "editable": true @@ -3205,7 +4079,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f91ad28e", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix\n", "\n", @@ -3218,7 +4095,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 28, + "id": "3ff08e48", "metadata": { "collapsed": false, "editable": true @@ -3250,14 +4128,25 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "eed40eec", + "metadata": { + "editable": true + }, "source": [ "We see that the matrix elements along the diagonal are one as they\n", "should be and that the matrix is symmetric. Furthermore, diagonalizing\n", "this matrix we easily see that it is a positive definite matrix.\n", "\n", - "The above procedure with **numpy** can be made more compact if we use **pandas**.\n", - "\n", + "The above procedure with **numpy** can be made more compact if we use **pandas**." + ] + }, + { + "cell_type": "markdown", + "id": "4aff4d1d", + "metadata": { + "editable": true + }, + "source": [ "## Correlation Matrix with Pandas\n", "\n", "We whow here how we can set up the correlation matrix using **pandas**, as done in this simple code" @@ -3265,7 +4154,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 29, + "id": "a3347660", "metadata": { "collapsed": false, "editable": true @@ -3290,16 +4180,28 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "48a0cf23", + "metadata": { + "editable": true + }, + "source": [ + "We expand this model to the Franke function discussed above." + ] + }, + { + "cell_type": "markdown", + "id": "ee4d2fa0", + "metadata": { + "editable": true + }, "source": [ - "We expand this model to the Franke function discussed above.\n", - "\n", "## Correlation Matrix with Pandas and the Franke function" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 30, + "id": "f370e6da", "metadata": { "collapsed": false, "editable": true @@ -3353,7 +4255,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "71b41cc3", + "metadata": { + "editable": true + }, "source": [ "We note here that the covariance is zero for the first rows and\n", "columns since all matrix elements in the design matrix were set to one\n", @@ -3362,9 +4267,16 @@ "This means that the variance for these elements will be zero and will\n", "cause problems when we set up the correlation matrix. We can simply\n", "drop these elements and construct a correlation\n", - "matrix without these elements. \n", - "\n", - "\n", + "matrix without these elements." + ] + }, + { + "cell_type": "markdown", + "id": "ec1bbf8b", + "metadata": { + "editable": true + }, + "source": [ "## Rewriting the Covariance and/or Correlation Matrix\n", "\n", "We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix $\\boldsymbol{X}$ as" @@ -3372,7 +4284,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "90bac5f3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}] = \\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X}= \\mathbb{E}[\\boldsymbol{X}^T\\boldsymbol{X}].\n", @@ -3381,14 +4296,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8d0e3e56", + "metadata": { + "editable": true + }, "source": [ "To see this let us simply look at a design matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{2\\times 2}$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a796bd05", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -3402,14 +4323,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "860b78db", + "metadata": { + "editable": true + }, "source": [ "If we then compute the expectation value (note the $1/n$ factor instead of $1/(n-1)$)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "03e8fc84", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E}[\\boldsymbol{X}^T\\boldsymbol{X}] = \\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X}=\\frac{1}{n}\\begin{bmatrix}\n", @@ -3421,14 +4348,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "007e9517", + "metadata": { + "editable": true + }, "source": [ "which is just" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4f59cf88", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] = \\boldsymbol{C}[\\boldsymbol{x}]=\\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}_0] & \\mathrm{cov}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] \\\\\n", @@ -3439,13 +4372,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1ffd6dae", + "metadata": { + "editable": true + }, "source": [ "where we wrote $$\\boldsymbol{C}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] = \\boldsymbol{C}[\\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors $\\boldsymbol{x}$ of the design/feature matrix $\\boldsymbol{X}$.\n", "\n", - "It is easy to generalize this to a matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$.\n", - "\n", - "\n", + "It is easy to generalize this to a matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$." + ] + }, + { + "cell_type": "markdown", + "id": "48e43aec", + "metadata": { + "editable": true + }, + "source": [ "## Linking with the SVD\n", "\n", "We saw earlier that" @@ -3453,7 +4396,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a9e79c63", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -3462,14 +4408,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "53c6be82", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f338f425", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -3478,14 +4430,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "99d983a1", + "metadata": { + "editable": true + }, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "71c5eed8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -3499,14 +4457,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f15223b4", + "metadata": { + "editable": true + }, "source": [ "meaning we can write" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "413194f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -3515,14 +4479,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dd296b3f", + "metadata": { + "editable": true + }, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "9688049c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -3531,7 +4501,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fe0ed0fb", + "metadata": { + "editable": true + }, "source": [ "## What does it mean?\n", "\n", @@ -3542,7 +4515,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "49bcec1c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -3551,7 +4527,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d217e1c2", + "metadata": { + "editable": true + }, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -3561,7 +4540,6 @@ "$\\boldsymbol{v}_i$ are hierarchically ordered by how much correlation they\n", "encode from the columns of $\\boldsymbol{X}$. \n", "\n", - "\n", "Note that these are also the eigenvectors and eigenvalues of the\n", "Hessian matrix.\n", "\n", @@ -3571,7 +4549,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3a1e3aaf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -3580,15 +4561,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ced3b70f", + "metadata": { + "editable": true + }, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", "matrix. Every singular value of $\\boldsymbol{X}$ is thus a positive square\n", "root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. If the matrix $\\boldsymbol{X}$ is\n", "self-adjoint, the singular values of $\\boldsymbol{X}$ are equal to the\n", - "absolute value of the eigenvalues of $\\boldsymbol{X}$.\n", - "\n", + "absolute value of the eigenvalues of $\\boldsymbol{X}$." + ] + }, + { + "cell_type": "markdown", + "id": "f2589fdd", + "metadata": { + "editable": true + }, + "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", "For $\\boldsymbol{X}\\boldsymbol{X}^T$ we found" @@ -3596,7 +4588,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "02248c4a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -3605,14 +4600,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "96d09d6e", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4b99c87c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -3621,14 +4622,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d8faf331", + "metadata": { + "editable": true + }, "source": [ "leading to" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0bb641d6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -3637,14 +4644,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "25ebcc39", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4b2f3c5f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -3653,7 +4666,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e9837512", + "metadata": { + "editable": true + }, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -3662,9 +4678,16 @@ "\n", "Since we will mainly be interested in the correlations among the features\n", "of our data (the columns of $\\boldsymbol{X}$, the quantity of interest for us are the non-zero singular\n", - "values and the column vectors of $\\boldsymbol{V}$.\n", - "\n", - "\n", + "values and the column vectors of $\\boldsymbol{V}$." + ] + }, + { + "cell_type": "markdown", + "id": "4d55630a", + "metadata": { + "editable": true + }, + "source": [ "## Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", @@ -3673,7 +4696,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2a3f0995", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -3682,14 +4708,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "417b607c", + "metadata": { + "editable": true + }, "source": [ "or we can state it as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "aef4e4c9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -3699,14 +4731,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "720d64d9", + "metadata": { + "editable": true + }, "source": [ "where we have used the definition of a norm-2 vector, that is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "580095be", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -3715,7 +4753,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "699717e7", + "metadata": { + "editable": true + }, "source": [ "By minimizing the above equation with respect to the parameters\n", "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", @@ -3725,7 +4766,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ceaebf06", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -3735,7 +4779,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2c7e22af", + "metadata": { + "editable": true + }, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -3744,7 +4791,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "def3adb2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -3753,14 +4803,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "73687e75", + "metadata": { + "editable": true + }, "source": [ "we have a new optimization equation" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8e15f07f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -3770,7 +4826,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6c53998d", + "metadata": { + "editable": true + }, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -3779,7 +4838,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "89b9cbc4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -3788,7 +4850,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "357f8421", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -3797,7 +4862,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ef7abfcd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -3806,7 +4874,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "db09d607", + "metadata": { + "editable": true + }, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -3817,7 +4888,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1fb685f5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -3826,14 +4900,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "25fedebe", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5de8bfbf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -3842,16 +4922,46 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3cdcc769", + "metadata": { + "editable": true + }, "source": [ "with $t$ a finite positive number. \n", "\n", + "If we keep the $1/n$ factor, the equation for the optimal $\\beta$ changes to" + ] + }, + { + "cell_type": "markdown", + "id": "88b1d12e", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "187cbef2", + "metadata": { + "editable": true + }, + "source": [ + "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", + "\n", "When we compare this with the ordinary least squares result we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "397694ac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -3860,11 +4970,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b02cd961", + "metadata": { + "editable": true + }, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", - "\n", "We see that Ridge regression is nothing but the standard OLS with a\n", "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", "particular for our discussion of the bias-variance tradeoff are rather\n", @@ -3877,7 +4989,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f80fdb0c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -3886,14 +5001,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b8421783", + "metadata": { + "editable": true + }, "source": [ "For Ridge regression this becomes" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "615c1cb8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -3902,10 +5023,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dafe3b7e", + "metadata": { + "editable": true + }, + "source": [ + "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." + ] + }, + { + "cell_type": "markdown", + "id": "51dea34a", + "metadata": { + "editable": true + }, "source": [ - "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$. \n", - "\n", "## Interpreting the Ridge results\n", "\n", "Since $\\lambda \\geq 0$, it means that compared to OLS, we have" @@ -3913,7 +5045,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5dd44f1b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -3922,7 +5057,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2f45c598", + "metadata": { + "editable": true + }, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -3930,9 +5068,16 @@ "eigenvalues ordered in a descending way, that is $\\sigma_i \\geq\n", "\\sigma_{i+1}$.\n", "\n", - "For small eigenvalues $\\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.\n", - "\n", - "\n", + "For small eigenvalues $\\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods." + ] + }, + { + "cell_type": "markdown", + "id": "17f9867f", + "metadata": { + "editable": true + }, + "source": [ "## More interpretations\n", "\n", "For the sake of simplicity, let us assume that the design matrix is orthonormal, that is" @@ -3940,7 +5085,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "583738d0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -3949,14 +5097,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "20d77427", + "metadata": { + "editable": true + }, "source": [ "In this case the standard OLS results in" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "9e7706c5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\boldsymbol{y},\n", @@ -3965,14 +5119,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4da844c2", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "525db3d4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -3981,7 +5141,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "cd528cd6", + "metadata": { + "editable": true + }, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -3990,8 +5153,16 @@ "We will come back to more interpreations after we have gone through some of the statistical analysis part. \n", "\n", "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", - "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", - "\n", + "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended." + ] + }, + { + "cell_type": "markdown", + "id": "efbcf3aa", + "metadata": { + "editable": true + }, + "source": [ "## Deriving the Lasso Regression Equations\n", "\n", "Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following **cost** function" @@ -3999,7 +5170,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "41c8f3cb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4008,14 +5182,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "896f0e84", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "515a212f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\ 0 & \\beta =0\\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -4024,14 +5204,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "138183e2", + "metadata": { + "editable": true + }, "source": [ "we have that the derivative of the cost function is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4e475fea", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -4040,14 +5226,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4ed75094", + "metadata": { + "editable": true + }, "source": [ "and reordering we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d4bafc4a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T(\\boldsymbol{y}.\n", @@ -4056,19 +5248,33 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f1ba8e36", + "metadata": { + "editable": true + }, + "source": [ + "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." + ] + }, + { + "cell_type": "markdown", + "id": "8efb3578", + "metadata": { + "editable": true + }, "source": [ - "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n", - "\n", "## Exercises for week 36, September 6-10\n", "\n", - "The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3).\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", + "The exercises here are meant to prepare you for work with project 1. The first exercise is a follow-up of exercise 2 from week 35 August 30-September 3)." + ] + }, + { + "cell_type": "markdown", + "id": "c413d900", + "metadata": { + "editable": true + }, + "source": [ "## Exercise 1: Adding Ridge and Lasso Regression\n", "\n", "This exercise is a continuation of exercise 2 from exercise set 1\n", @@ -4087,7 +5293,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 31, + "id": "afad0eab", "metadata": { "collapsed": false, "editable": true @@ -4100,11 +5307,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3f02a117", + "metadata": { + "editable": true + }, "source": [ "Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\\lambda$. Compare and analyze your results with those from exercise 3. Study the dependence on $\\lambda$ while also varying the strength of the noise in your expression for $y(x)$. \n", "\n", - "\n", "The code here allows you to perform your own Ridge calculation and\n", "perform calculations for various values of the regularization\n", "parameter $\\lambda$. This program can easily be extended upon." @@ -4112,7 +5321,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 32, + "id": "b2c58e37", "metadata": { "collapsed": false, "editable": true @@ -4192,21 +5402,25 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "dc07b126", + "metadata": { + "editable": true + }, "source": [ "Repeat the above but using the functionality of\n", "**Scikit-Learn**. Compare your code with the results from\n", "**Scikit-Learn**. Remember to run with the same random numbers for\n", "generating $x$ and $y$. Observe also that when you compare with **Scikit-Learn**, you need to pay attention to how the intercept is dealt with.\n", "\n", - "\n", - "\n", "Finally, using **Scikit-Learn** or your own code, compute also the mean square error, a risk metric corresponding to the expected value of the squared (quadratic) error defined as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f4d85b30", + "metadata": { + "editable": true + }, "source": [ "$$\n", "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", @@ -4216,7 +5430,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e08d2eda", + "metadata": { + "editable": true + }, "source": [ "and the $R^2$ score function.\n", "If $\\tilde{\\hat{y}}_i$ is the predicted value of the $i-th$ sample and $y_i$ is the corresponding true value, then the score $R^2$ is defined as" @@ -4224,7 +5441,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fd872c6f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", @@ -4233,14 +5453,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "977bd2c8", + "metadata": { + "editable": true + }, "source": [ "where we have defined the mean value of $\\hat{y}$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "eeee30ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n", @@ -4249,16 +5475,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5bf6c9bc", + "metadata": { + "editable": true + }, + "source": [ + "Discuss these quantities as functions of the variable $\\lambda$ in the Ridge and Lasso regression methods." + ] + }, + { + "cell_type": "markdown", + "id": "c223eb1b", + "metadata": { + "editable": true + }, "source": [ - "Discuss these quantities as functions of the variable $\\lambda$ in the Ridge and Lasso regression methods.\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", "### Exercise: Linear Regression for a two-dimensional function\n", "\n", "This is a longer exercise and the aim is to study in more detail various\n", @@ -4277,7 +5508,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "45b1e7c9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -4289,7 +5523,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "425075fd", + "metadata": { + "editable": true + }, "source": [ "The function will be defined for $x,y\\in [0,1]$. Our first step will\n", "be to perform an OLS regression analysis of this function, trying out\n", @@ -4300,13 +5537,13 @@ "regression methods, introducing thus a dependence on the bias\n", "(penalty) $\\lambda$.\n", "\n", - "\n", "The Python fucntion for the Franke function is included here (it performs also a three-dimensional plot of it)" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 33, + "id": "05e1fa34", "metadata": { "collapsed": false, "editable": true @@ -4356,7 +5593,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b1e5347d", + "metadata": { + "editable": true + }, "source": [ "We will generate our own dataset for a function\n", "$\\mathrm{FrankeFunction}(x,y)$ with $x,y \\in [0,1]$. The function\n", @@ -4368,13 +5608,15 @@ "value decomposition from e.g., **numpy** ) or use your code and perform a standard least square regression\n", "analysis using polynomials in $x$ and $y$ up to fifth order. You can use **scikit-learn** as well.\n", "\n", - "\n", "Evaluate the Mean Squared error (MSE)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f60027f0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "MSE(\\hat{y},\\hat{\\tilde{y}}) = \\frac{1}{n}\n", @@ -4384,7 +5626,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0c95b9d2", + "metadata": { + "editable": true + }, "source": [ "and the $R^2$ score function. If $\\tilde{\\hat{y}}_i$ is the predicted\n", "value of the $i-th$ sample and $y_i$ is the corresponding true value,\n", @@ -4393,7 +5638,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "baf15958", + "metadata": { + "editable": true + }, "source": [ "$$\n", "R^2(\\hat{y}, \\tilde{\\hat{y}}) = 1 - \\frac{\\sum_{i=0}^{n - 1} (y_i - \\tilde{y}_i)^2}{\\sum_{i=0}^{n - 1} (y_i - \\bar{y})^2},\n", @@ -4402,14 +5650,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d7f9b2c0", + "metadata": { + "editable": true + }, "source": [ "where we have defined the mean value of $\\hat{y}$ as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "93125b69", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\bar{y} = \\frac{1}{n} \\sum_{i=0}^{n - 1} y_i.\n", @@ -4418,7 +5672,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "973a4d1a", + "metadata": { + "editable": true + }, "source": [ "You should split your data in train and test and also consider scaling the data.\n", "\n", @@ -4427,7 +5684,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 34, + "id": "b173966c", "metadata": { "collapsed": false, "editable": true @@ -4470,26 +5728,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "59c6410f", + "metadata": { + "editable": true + }, "source": [ "Write then your own code for the Ridge method or use **Scikit-Learn**.\n", "Perform the same analysis as you did for ordinary Least Squares (for the same polynomials) but now for different values of $\\lambda$. Compare and\n", "analyze your results with those obtained with ordinary Least Squares. Study the\n", "dependence on $\\lambda$.\n", "\n", - "\n", "This part is essentially a repeat of the previous ones, but now\n", "with Lasso regression. Write either your own code or\n", "use the functionalities of **Scikit-Learn** (recommended). \n", "Give a\n", "critical discussion of the three methods and a judgement of which\n", - "model fits the data best.\n", - "\n", - "" + "model fits the data best." ] } ], "metadata": {}, "nbformat": 4, - "nbformat_minor": 4 + "nbformat_minor": 5 } diff --git a/doc/pub/week36/html/._week36-bs000.html b/doc/pub/week36/html/._week36-bs000.html index 869b857dc..7cbe28630 100644 --- a/doc/pub/week36/html/._week36-bs000.html +++ b/doc/pub/week36/html/._week36-bs000.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,36 +258,32 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - - - -
    -

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    +
    +

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    +
    -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 16, 2021

    -
    -

    Read »

    @@ -313,29 +305,22 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 51
    • +
  • 53
  • »
    • -
    - - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week36/html/._week36-bs001.html b/doc/pub/week36/html/._week36-bs001.html index 063c5a2da..986c891b8 100644 --- a/doc/pub/week36/html/._week36-bs001.html +++ b/doc/pub/week36/html/._week36-bs001.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,29 +258,22 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Plans for week 36

    • Thursday: Summary from last week on SVD, Statistics, probability theory and linear regression. Video of Lecture.
    • Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project. Video of Lecture
    - -Recommended Reading: - +

    Recommended Reading:

    1. Lectures on Regression
    2. Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1
    3. Hastie et al chapter 3
    -

      @@ -301,29 +290,22 @@ Recommended Reading:
    • 10
    • 11
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs002.html b/doc/pub/week36/html/._week36-bs002.html index ab5a65b28..ae16c3d08 100644 --- a/doc/pub/week36/html/._week36-bs002.html +++ b/doc/pub/week36/html/._week36-bs002.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,17 +258,12 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Thursday September 9

    -

      @@ -290,29 +281,22 @@ MathJax.Hub.Config({
    • 11
    • 12
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs003.html b/doc/pub/week36/html/._week36-bs003.html index 06bb28ecb..9b30309ef 100644 --- a/doc/pub/week36/html/._week36-bs003.html +++ b/doc/pub/week36/html/._week36-bs003.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,17 +258,12 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Summary from last Week and Examples

    -

      @@ -291,29 +282,22 @@ MathJax.Hub.Config({
    • 12
    • 13
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs004.html b/doc/pub/week36/html/._week36-bs004.html index ad956480c..3c9dfcf22 100644 --- a/doc/pub/week36/html/._week36-bs004.html +++ b/doc/pub/week36/html/._week36-bs004.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,28 +258,23 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Linear Regression and the SVD

    -

    -We used the SVD to analyse the matrix to invert in ordinary lineat regression +

    We used the SVD to analyse the matrix to invert in ordinary lineat regression

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix +

    Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix}, $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is +

    where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

    $$ \tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ @@ -294,17 +285,17 @@ $$ \end{bmatrix}, $$ -meaning we can write +

    meaning we can write

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs005.html b/doc/pub/week36/html/._week36-bs005.html index 16d7be9bc..4ccc73e8f 100644 --- a/doc/pub/week36/html/._week36-bs005.html +++ b/doc/pub/week36/html/._week36-bs005.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,55 +258,50 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    What does it mean?

    -

    -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +

    In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that the columns of \( \boldsymbol{V} \) are the eigenvectors of \( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of \( \boldsymbol{X} \) in a descending order, it means that the column vectors \( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). +encode from the columns of \( \boldsymbol{X} \). +

    -

    -Note that these are also the eigenvectors and eigenvalues of the +

    Note that these are also the eigenvectors and eigenvalues of the Hessian matrix. +

    -

    -If we now recall the definition of the covariance matrix (not using +

    If we now recall the definition of the covariance matrix (not using Bessel's correction) we have +

    $$ \boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ -

    -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +

    meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    -

      @@ -331,29 +322,22 @@ absolute value of the eigenvalues of \( \boldsymbol{X} \).
    • 14
    • 15
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs006.html b/doc/pub/week36/html/._week36-bs006.html index 70125e4e9..ebdb0eabe 100644 --- a/doc/pub/week36/html/._week36-bs006.html +++ b/doc/pub/week36/html/._week36-bs006.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,51 +258,44 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    -

    -For \( \boldsymbol{X}\boldsymbol{X}^T \) we found +

    For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -Since the matrices here have dimension \( n\times n \), we have +

    Since the matrices here have dimension \( n\times n \), we have

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -leading to +

    leading to

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -

    -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +

    Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

    $$ (\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$ -

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +

    It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

    -

    -Since we will mainly be interested in the correlations among the features +

    Since we will mainly be interested in the correlations among the features of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +

    -

      @@ -328,29 +317,22 @@ values and the column vectors of \( \boldsymbol{V} \).
    • 15
    • 16
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs007.html b/doc/pub/week36/html/._week36-bs007.html index de3a97d20..c803079c2 100644 --- a/doc/pub/week36/html/._week36-bs007.html +++ b/doc/pub/week36/html/._week36-bs007.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,31 +258,48 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Code for SVD and Inversion of Matrices

    -

    -How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? +

    How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? The simple answer is to use the linear algebra function for pseudoinvers, that is -

    +

    -
    Ainv = np.linlag.pinv(A)
-
    -

    -Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD. +

    +
    +
    +
    +
    + 
    Ainv = np.linlag.pinv(A)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVDinv(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
@@ -315,8 +328,22 @@ A = np.t
 B = np.linalg.inv(A)  # here we could use np.linalg.pinv(A)
 C = SVDinv(A)
 print(np.abs(B-C))
-
    
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    - - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs008.html b/doc/pub/week36/html/._week36-bs008.html index e7d25780e..4e33e090a 100644 --- a/doc/pub/week36/html/._week36-bs008.html +++ b/doc/pub/week36/html/._week36-bs008.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,39 +258,37 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Inverse of Rectangular Matrix

    -

    -Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular. +

    Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.

    -

    -The pseudoinverse is the generalization of the matrix inverse for square matrices to +

    The pseudoinverse is the generalization of the matrix inverse for square matrices to rectangular matrices where the number of rows and columns are not equal. +

    -

    -It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. +

    It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices. +

    -

    -Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \)) +

    Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \))

    $$ \boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T, $$ -where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD. +

    where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVDinv(A):
     U, s, VT = np.linalg.svd(A)
@@ -317,11 +311,23 @@ B = np.l
 # Compare our own algorithm with pinv
 C = SVDinv(A)
 print(np.abs(C-B))
-
    
-
    
-As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.

    -

      @@ -345,29 +351,22 @@ As you can see from this example, our own decomposition based on the SVD agrees
    • 17
    • 18
    • ...
      • -
    • 51
      • +
    • 53
    • »
    - - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs009.html b/doc/pub/week36/html/._week36-bs009.html index 3a51c650e..c37e2edcb 100644 --- a/doc/pub/week36/html/._week36-bs009.html +++ b/doc/pub/week36/html/._week36-bs009.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,35 +258,31 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Ridge and LASSO Regression

    -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -or we can state it as +

    or we can state it as

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, $$ -where we have used the definition of a norm-2 vector, that is +

    where we have used the definition of a norm-2 vector, that is

    $$ \vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs010.html b/doc/pub/week36/html/._week36-bs010.html index 29afe5478..88ab182e5 100644 --- a/doc/pub/week36/html/._week36-bs010.html +++ b/doc/pub/week36/html/._week36-bs010.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,52 +258,46 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    From OLS to Ridge and Lasso

    -

    -By minimizing the above equation with respect to the parameters +

    By minimizing the above equation with respect to the parameters \( \boldsymbol{\beta} \) we could then obtain an analytical expression for the parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by defining a new cost function to be optimized, that is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

    -which leads to the Ridge regression minimization problem where we +

    which leads to the Ridge regression minimization problem where we require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is a finite number larger than zero. By defining +

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, $$ -

    -we have a new optimization equation +

    we have a new optimization equation

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 $$ -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    -

    -Here we have defined the norm-1 as +

    Here we have defined the norm-1 as

    $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ -

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs011.html b/doc/pub/week36/html/._week36-bs011.html index 4a8bf211a..59051de07 100644 --- a/doc/pub/week36/html/._week36-bs011.html +++ b/doc/pub/week36/html/._week36-bs011.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,58 +258,36 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - -

    Deriving the Ridge Regression Equations

    -

    -Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -and +

    and taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters +

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that +

    with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

    $$ \sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ -

    -with \( t \) a finite positive number. +

    with \( t \) a finite positive number.

    -

    -When we compare this with the ordinary least squares result we have -$$ -\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, -$$ - -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). - -

    -We see that Ridge regression is nothing but the standard OLS with a -modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in -particular for our discussion of the bias-variance tradeoff are rather -interesting. We will see that for specific values of \( \lambda \), we may -even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. - -

      @@ -339,29 +313,22 @@ even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). The
    • 20
    • 21
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs012.html b/doc/pub/week36/html/._week36-bs012.html index 386eaf6f5..3b85471f8 100644 --- a/doc/pub/week36/html/._week36-bs012.html +++ b/doc/pub/week36/html/._week36-bs012.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,34 +258,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Note on Scikit-Learn

    -

    SVD analysis

    - -

    -Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) -We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as -$$ -\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. -$$ - -

    -For Ridge regression this becomes +

    Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \beta \) becomes

    $$ -\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). +

    In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function.

    -

      @@ -315,29 +297,22 @@ with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \
    • 21
    • 22
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs013.html b/doc/pub/week36/html/._week36-bs013.html index 8f5e2a6aa..8c848f759 100644 --- a/doc/pub/week36/html/._week36-bs013.html +++ b/doc/pub/week36/html/._week36-bs013.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,34 +258,25 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - - -

    Interpreting the Ridge results

    - -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have - +

    Comparison with OLS

    +

    When we compare this with the ordinary least squares result we have

    $$ -\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. +\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the -orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by -\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has -eigenvalues ordered in a descending way, that is \( \sigma_i \geq -\sigma_{i+1} \). +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods. +

    We see that Ridge regression is nothing but the standard OLS with a +modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in +particular for our discussion of the bias-variance tradeoff are rather +interesting. We will see that for specific values of \( \lambda \), we may +even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    -

      @@ -315,29 +302,22 @@ For small eigenvalues \( \sigma_i \) it means that their contributions become le
    • 22
    • 23
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs014.html b/doc/pub/week36/html/._week36-bs014.html index b8644c7f8..aa210769c 100644 --- a/doc/pub/week36/html/._week36-bs014.html +++ b/doc/pub/week36/html/._week36-bs014.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,49 +258,27 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    SVD analysis

    -

    More interpretations

    - -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is - +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

    $$ -\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. +\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$ -

    -In this case the standard OLS results in -$$ -\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, -$$ - -

    -and +

    For Ridge regression this becomes

    $$ -\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$ -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and -the Ridge estimator converges to zero when the hyperparameter goes to -infinity. +

    with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

    -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. - -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. -Similarly, Mehta et al's article is also recommended. - -

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs015.html b/doc/pub/week36/html/._week36-bs015.html index 64ae4999a..6550fc426 100644 --- a/doc/pub/week36/html/._week36-bs015.html +++ b/doc/pub/week36/html/._week36-bs015.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,43 +258,27 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Interpreting the Ridge results

    -

    Deriving the Lasso Regression Equations

    - -

    -Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function +

    Since \( \lambda \geq 0 \), it means that compared to OLS, we have

    $$ -C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +\frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. $$ -

    -Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) -$$ -\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. -$$ +

    Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by +\( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has +eigenvalues ordered in a descending way, that is \( \sigma_i \geq +\sigma_{i+1} \). +

    -we have that the derivative of the cost function is +

    For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

    -$$ -\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, -$$ - -and reordering we have -$$ -\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. -$$ - -This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. - -

      @@ -324,29 +304,22 @@ This equation does not lead to a nice analytical equation as in Ridge regression
    • 24
    • 25
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs016.html b/doc/pub/week36/html/._week36-bs016.html index 6d3c4cb86..549f05f45 100644 --- a/doc/pub/week36/html/._week36-bs016.html +++ b/doc/pub/week36/html/._week36-bs016.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,32 +258,40 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    More interpretations

    -

    Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression

    +

    For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

    -

    -Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the -diagonal. In this case we have an equal number of rows and columns \( n=p \). - -

    -Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, +\boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -and minimizing we have that +

    In this case the standard OLS results in

    $$ -\hat{\beta}_i^{\mathrm{OLS}} = y_i. +\boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -

    +

    and

    + +$$ +\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, +$$ + +

    that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +the Ridge estimator converges to zero when the hyperparameter goes to +infinity. +

    + +

    We will come back to more interpreations after we have gone through some of the statistical analysis part.

    + +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +Similarly, Mehta et al's article is also recommended. +

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs017.html b/doc/pub/week36/html/._week36-bs017.html index 80aadb339..c5808165b 100644 --- a/doc/pub/week36/html/._week36-bs017.html +++ b/doc/pub/week36/html/._week36-bs017.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,28 +258,36 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Deriving the Lasso Regression Equations

    -

    Ridge Regression

    +

    Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function

    -

    -For Ridge regression our cost function is $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, +C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -and minimizing we have that +

    Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

    $$ -\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. +\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. $$ -

    +

    we have that the derivative of the cost function is

    + +$$ +\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, +$$ + +

    and reordering we have

    +$$ +\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

    This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs018.html b/doc/pub/week36/html/._week36-bs018.html index fe1aa55a6..27b43178c 100644 --- a/doc/pub/week36/html/._week36-bs018.html +++ b/doc/pub/week36/html/._week36-bs018.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,38 +258,27 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression

    -

    Lasso Regression

    +

    Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the +diagonal. In this case we have an equal number of rows and columns \( n=p \). +

    -

    -For Lasso regression our cost function is +

    Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))

    $$ -C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, $$ -and minimizing we have that +

    and minimizing we have that

    $$ --2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, +\hat{\beta}_i^{\mathrm{OLS}} = y_i. $$ -which leads to -$$ -\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ - y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ - 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. -$$ -

    -Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \). - -

      @@ -319,29 +304,22 @@ Plotting these results (27
    • 28
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs019.html b/doc/pub/week36/html/._week36-bs019.html index 12d0bb9c7..4cd6504a8 100644 --- a/doc/pub/week36/html/._week36-bs019.html +++ b/doc/pub/week36/html/._week36-bs019.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,31 +258,23 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Ridge Regression

    -

    Yet another Example

    - -

    -Let us assume we have a data set with outputs/targets given by the vector - +

    For Ridge regression our cost function is

    $$ -\boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix}, +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ -and our inputs as a \( 3\times 2 \) design matrix +

    and minimizing we have that

    $$ -\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, +\hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. $$ -meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. -

      @@ -312,29 +300,22 @@ meaning that we have two features and two unknown parameters \( \beta_0 \) and \
    • 28
    • 29
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs020.html b/doc/pub/week36/html/._week36-bs020.html index 2cbf3a2ac..254e3b2da 100644 --- a/doc/pub/week36/html/._week36-bs020.html +++ b/doc/pub/week36/html/._week36-bs020.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,33 +258,31 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Lasso Regression

    -

    The OLS case

    - -

    -For ordinary least squares (OLS) we know that the optimal solution is - +

    For Lasso regression our cost function is

    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ -Inserting the above values we obtain that - +

    and minimizing we have that

    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, +-2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, $$ -

    -The code which implements this simpler case is presented after the discussion of Ridge and Lasso. +

    which leads to

    +$$ +\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ + y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ + 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. +$$ + +

    Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

    -

      @@ -314,29 +308,22 @@ The code which implements this simpler case is presented after the discussion of
    • 29
    • 30
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs021.html b/doc/pub/week36/html/._week36-bs021.html index 3d4cd8ea6..b6df10554 100644 --- a/doc/pub/week36/html/._week36-bs021.html +++ b/doc/pub/week36/html/._week36-bs021.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,37 +258,25 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Yet another Example

    -

    The Ridge case

    - -

    -For Ridge regression we have +

    Let us assume we have a data set with outputs/targets given by the vector

    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +\boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix}, $$ -Inserting the above values we obtain that - +

    and our inputs as a \( 3\times 2 \) design matrix

    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, +\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$ -

    -There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). -Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \). +

    meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.

    -

    -To see this, let us write the cost function for Ridge regression. - -

      @@ -318,29 +302,22 @@ To see this, let us write the cost function for Ridge regression.
    • 30
    • 31
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs022.html b/doc/pub/week36/html/._week36-bs022.html index 6025dfac6..28519a0fc 100644 --- a/doc/pub/week36/html/._week36-bs022.html +++ b/doc/pub/week36/html/._week36-bs022.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,46 +258,26 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    The OLS case

    -

    Writing the Cost Function

    - -

    -We define the MSE without the \( 1/n \) factor and have then, using that -$$ -\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, -$$ - +

    For ordinary least squares (OLS) we know that the optimal solution is

    $$ -C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -and taking the derivative with respect to \( \beta_0 \) we get +

    Inserting the above values we obtain that

    + $$ -\beta_0=\frac{8}{4+\lambda}, +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$ -and for \( \beta_1 \) we obtain -$$ -\beta_1=\frac{2}{1+\lambda}, -$$ +

    The code which implements this simpler case is presented after the discussion of Ridge and Lasso.

    -

    -Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving -$$ -\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, -$$ - -which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \). - -

      @@ -327,29 +303,22 @@ which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
    • 31
    • 32
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs023.html b/doc/pub/week36/html/._week36-bs023.html index 4f200a2da..a1cd5ec8d 100644 --- a/doc/pub/week36/html/._week36-bs023.html +++ b/doc/pub/week36/html/._week36-bs023.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,41 +258,29 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    The Ridge case

    -

    Lasso case

    - -

    -For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) -and \( \beta_1 \). This gives us the following derivatives of the cost function -$$ -C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), -$$ - +

    For Ridge regression we have

    $$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -and +

    Inserting the above values we obtain that

    + $$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. +\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$ -We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely +

    There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \). +

    -
      -
    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
      2. -
    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
      3. -
    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
      4. -
    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).
      5. -
    +

    To see this, let us write the cost function for Ridge regression.

    @@ -323,29 +307,22 @@ We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \

  • 32
  • 33
  • ...
    • -
  • 51
    • +
  • 53
  • »
    • -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs024.html b/doc/pub/week36/html/._week36-bs024.html index 99b69fcd4..9fb4281bb 100644 --- a/doc/pub/week36/html/._week36-bs024.html +++ b/doc/pub/week36/html/._week36-bs024.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,42 +258,39 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Writing the Cost Function

    -

    The first Case

    - -

    -If we consider the first case, we have then +

    We define the MSE without the \( 1/n \) factor and have then, using that

    $$ --4(4-2\beta_0)+\lambda=0, +\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, $$ -and -$$ --2(2-\beta_1)+\lambda=0. -$$ - -which yields $$ -\beta_0=\frac{16+\lambda}{8}, +C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), $$ -and +

    and taking the derivative with respect to \( \beta_0 \) we get

    $$ -\beta_1=\frac{4+\lambda}{2}. +\beta_0=\frac{8}{4+\lambda}, $$ -

    -Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. +

    and for \( \beta_1 \) we obtain

    +$$ +\beta_1=\frac{2}{1+\lambda}, +$$ + +

    Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving

    +$$ +\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, +$$ + +

    which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).

    -

      @@ -323,29 +316,22 @@ Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the opt
    • 33
    • 34
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs025.html b/doc/pub/week36/html/._week36-bs025.html index 2b4498097..cb1933e1c 100644 --- a/doc/pub/week36/html/._week36-bs025.html +++ b/doc/pub/week36/html/._week36-bs025.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,79 +258,36 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Lasso case

    -

    Simple code for solving the above problem

    - -

    -Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. - -

    -First we study and compare the OLS and Ridge results. The next code compares all three methods. - -

    - - -

    import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-
-def R2(y_data, y_model):
-    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
-    n = np.size(y_model)
-    return np.sum((y_data-y_model)**2)/n
+
    For Lasso we need now, keeping a  constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \),  to take the derivative of the absolute values of \( \beta_0 \)
+and \( \beta_1 \). This gives us the following derivatives of the cost function
+
    
+$$
+C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
+$$
 
 
-# A seed just to ensure that the random numbers are the same for every run.
-# Useful for eventual debugging.
+$$
+\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0,
+$$
 
-X = np.array( [ [ 2, 0], [0, 1], [0,0]])
-y = np.array( [4, 2, 3])
+
    and
    
+$$
+\frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0.
+$$
 
-
-# matrix inversion to find beta
-OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
-print(OLSbeta)
-# and then make the prediction
-ytildeOLS = X @ OLSbeta
-print("Training MSE for OLS")
-print(MSE(y,ytildeOLS))
-ypredictOLS = X @ OLSbeta
-
-# Repeat now for Ridge regression and various values of the regularization parameter
-I = np.eye(2,2)
-# Decide which values of lambda to use
-nlambdas = 100
-MSEPredict = np.zeros(nlambdas)
-lambdas = np.logspace(-4, 4, nlambdas)
-for i in range(nlambdas):
-    lmb = lambdas[i]
-    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
-#    print(Ridgebeta)
-    # and then make the prediction
-    ypredictRidge = X @ Ridgebeta
-    MSEPredict[i] = MSE(y,ypredictRidge)
-#    print(MSEPredict[i])
-    # Now plot the results
-plt.figure()
-plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Train')
-plt.xlabel('log10(lambda)')
-plt.ylabel('MSE')
-plt.legend()
-plt.show()
-
    -

    -We see here that we reach a plateau. What is actually happening? - -

    +

    We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely

    +
      +
    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
      2. +
    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
      3. +
    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
      4. +
    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).
      5. +

      @@ -360,29 +313,22 @@ We see here that we reach a plateau. What is actually happening?
    • 34
    • 35
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs026.html b/doc/pub/week36/html/._week36-bs026.html index 5541ab5e9..b422666e4 100644 --- a/doc/pub/week36/html/._week36-bs026.html +++ b/doc/pub/week36/html/._week36-bs026.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,77 +258,35 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    The first Case

    -

    With Lasso Regression

    +

    If we consider the first case, we have then

    +$$ +-4(4-2\beta_0)+\lambda=0, +$$ -

    +

    and

    +$$ +-2(2-\beta_1)+\lambda=0. +$$ - -
    import os
-import numpy as np
-import pandas as pd
-import matplotlib.pyplot as plt
-from sklearn import linear_model
+
    which yields
    
 
-def R2(y_data, y_model):
-    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
-    n = np.size(y_model)
-    return np.sum((y_data-y_model)**2)/n
+$$
+\beta_0=\frac{16+\lambda}{8},
+$$
 
+
    and
    
+$$
+\beta_1=\frac{4+\lambda}{2}.
+$$
 
-# A seed just to ensure that the random numbers are the same for every run.
-# Useful for eventual debugging.
+
    Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.
    
 
-X = np.array( [ [ 2, 0], [0, 1], [0,0]])
-y = np.array( [4, 2, 3])
-
-
-# matrix inversion to find beta
-OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
-print(OLSbeta)
-# and then make the prediction
-ytildeOLS = X @ OLSbeta
-print("Training MSE for OLS")
-print(MSE(y,ytildeOLS))
-ypredictOLS = X @ OLSbeta
-
-# Repeat now for Ridge regression and various values of the regularization parameter
-I = np.eye(2,2)
-# Decide which values of lambda to use
-nlambdas = 100
-MSERidgePredict = np.zeros(nlambdas)
-MSELassoPredict = np.zeros(nlambdas)
-lambdas = np.logspace(-4, 4, nlambdas)
-for i in range(nlambdas):
-    lmb = lambdas[i]
-    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
-    print(Ridgebeta)
-    # and then make the prediction
-    ypredictRidge = X @ Ridgebeta
-    MSERidgePredict[i] = MSE(y,ypredictRidge)
-    RegLasso = linear_model.Lasso(lmb)
-    RegLasso.fit(X,y)
-    ypredictLasso = RegLasso.predict(X)
-    print(RegLasso.coef_)
-    MSELassoPredict[i] = MSE(y,ypredictLasso)
-# Now plot the results
-plt.figure()
-plt.plot(np.log10(lambdas), MSERidgePredict, 'r--', label = 'MSE Ridge Train')
-plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Train')
-plt.xlabel('log10(lambda)')
-plt.ylabel('MSE')
-plt.legend()
-plt.show()
-
    -

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs027.html b/doc/pub/week36/html/._week36-bs027.html index 7e9f17121..005aa3ed3 100644 --- a/doc/pub/week36/html/._week36-bs027.html +++ b/doc/pub/week36/html/._week36-bs027.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,25 +258,27 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Simple code for solving the above problem

    -

    Another Example, now with a polynomial fit

    +

    Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.

    + +

    First we study and compare the OLS and Ridge results. The next code compares all three methods.

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn import linear_model
 
 def R2(y_data, y_model):
     return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
@@ -291,70 +289,58 @@ MathJax.Hub.Config({
 
 # A seed just to ensure that the random numbers are the same for every run.
 # Useful for eventual debugging.
-np.random.seed(3155)
 
-x = np.random.rand(100)
-y = 2.0+5*x*x+0.1*np.random.randn(100)
+X = np.array( [ [ 2, 0], [0, 1], [0,0]])
+y = np.array( [4, 2, 3])
 
-# number of features p (here degree of polynomial
-p = 3
-#  The design matrix now as function of a given polynomial
-X = np.zeros((len(x),p))
-X[:,0] = 1.0
-X[:,1] = x
-X[:,2] = x*x
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
 
 # matrix inversion to find beta
-OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
+OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
 print(OLSbeta)
 # and then make the prediction
-ytildeOLS = X_train @ OLSbeta
+ytildeOLS = X @ OLSbeta
 print("Training MSE for OLS")
-print(MSE(y_train,ytildeOLS))
-ypredictOLS = X_test @ OLSbeta
-print("Test MSE OLS")
-print(MSE(y_test,ypredictOLS))
+print(MSE(y,ytildeOLS))
+ypredictOLS = X @ OLSbeta
 
-# Repeat now for Lasso and Ridge regression and various values of the regularization parameter
-I = np.eye(p,p)
+# Repeat now for Ridge regression and various values of the regularization parameter
+I = np.eye(2,2)
 # Decide which values of lambda to use
 nlambdas = 100
 MSEPredict = np.zeros(nlambdas)
-MSETrain = np.zeros(nlambdas)
-MSELassoPredict = np.zeros(nlambdas)
-MSELassoTrain = np.zeros(nlambdas)
 lambdas = np.logspace(-4, 4, nlambdas)
 for i in range(nlambdas):
     lmb = lambdas[i]
-    Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
-    # include lasso using Scikit-Learn
-    RegLasso = linear_model.Lasso(lmb)
-    RegLasso.fit(X_train,y_train)
+    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
+#    print(Ridgebeta)
     # and then make the prediction
-    ytildeRidge = X_train @ Ridgebeta
-    ypredictRidge = X_test @ Ridgebeta
-    ytildeLasso = RegLasso.predict(X_train)
-    ypredictLasso = RegLasso.predict(X_test)
-    MSEPredict[i] = MSE(y_test,ypredictRidge)
-    MSETrain[i] = MSE(y_train,ytildeRidge)
-    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
-    MSELassoTrain[i] = MSE(y_train,ytildeLasso)
-
-# Now plot the results
+    ypredictRidge = X @ Ridgebeta
+    MSEPredict[i] = MSE(y,ypredictRidge)
+#    print(MSEPredict[i])
+    # Now plot the results
 plt.figure()
-plt.plot(np.log10(lambdas), MSETrain, label = 'MSE Ridge train')
-plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Test')
-plt.plot(np.log10(lambdas), MSELassoTrain, label = 'MSE Lasso train')
-plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Test')
-
+plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Train')
 plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We see here that we reach a plateau. What is actually happening?

    +

    - - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs028.html b/doc/pub/week36/html/._week36-bs028.html index bc624f22d..3c73490a9 100644 --- a/doc/pub/week36/html/._week36-bs028.html +++ b/doc/pub/week36/html/._week36-bs028.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,290 +258,91 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    With Lasso Regression

    -

    Using CVXOPT

    - -

    -As a small addendum, we note that you can also solve this problem -using the convex optimization package -CVXOPT. This -requires, in addition to having installed CVXOPT, you need to -download the file l1regl.py. The following code example solves the -simpler problem we discussed above, where we have added the latter -python file. - -

    -We list first the needed file. Thereafter we set up the problem to solve. - -

    -

    from cvxopt import matrix, spdiag, mul, div, sqrt, normal, setseed
-from cvxopt import blas, lapack, solvers, sparse, spmatrix
-import math
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn import linear_model
 
-try:
-    import mosek
-    import sys
-    __MOSEK = True
-except: __MOSEK = False
-
-if __MOSEK:
-
-    def l1regls_mosek(A, b):
-        """
-
-        Returns the solution of l1-norm regularized least-squares problem
-
-            minimize    || A*x - b ||_2^2  + e'*u
-
-            subject to  -u <= x <= u
-
-        """
-
-        m, n = A.size
-
-        env  = mosek.Env()
-        task = env.Task(0,0)
-        task.set_Stream(mosek.streamtype.log, lambda x: sys.stdout.write(x))
-
-        task.appendvars( 2*n)            # number of variables
-        task.appendcons( 2*n)            # number of constraints
-
-        # input quadratic objective
-        Q = matrix(0.0, (n,n)) 
-        blas.syrk(A, Q, alpha = 2.0, trans='T')
-
-        I = []
-        for i in range(n):
-            I.extend(range(i,n))
-
-        J = []
-        for i in range(n):
-            J.extend((n-i)*[i])
-
-        task.putqobj(I, J, list(Q[matrix(I) + matrix(J)*n]))
-        task.putclist(range(2*n), list(-2*A.T*b) + n*[1.0])  # setup linear objective
-
-        # input constraint matrix row by row
-        for i in range(n):
-            task.putarow(   i, [i, n+i], [1.0, -1.0])
-            task.putarow( n+i, [i, n+i], [1.0,  1.0])
-
-        # setup bounds on constraints
-        task.putboundslice(mosek.accmode.con,
-                           0, n, n*[mosek.boundkey.up], n*[0.0], n*[0.0])
-        task.putboundslice(mosek.accmode.con,
-                           n, 2*n, n*[mosek.boundkey.lo], n*[0.0], n*[0.0])
-
-        # setup variable bounds
-        task.putboundslice(mosek.accmode.var,
-                           0, 2*n, 2*n*[mosek.boundkey.fr], 2*n*[0.0], 2*n*[0.0])
-
-        # optimize the task
-        task.putobjsense(mosek.objsense.minimize)
-        task.optimize()
-        task.solutionsummary(mosek.streamtype.log)
-        x = n*[0.0]
-        task.getsolutionslice(mosek.soltype.itr, mosek.solitem.xx, 0, n, x)
-
-        return matrix(x)
-
-    def l1regls_mosek2(A, b):
-        """
-
-        Returns the solution of l1-norm regularized least-squares problem
-
-            minimize     w'*w + e'*u
-
-            subject to  -u <= x <= u
-
-                         A*x - w = b
-
-        """
-
-        m, n = A.size
-
-        env  = mosek.Env()
-        task = env.Task(0,0)
-        task.set_Stream(mosek.streamtype.log, lambda x: sys.stdout.write(x))
-
-        task.appendvars(2*n + m)     # number of variables
-        task.appendcons(2*n + m)     # number of constraints
-
-        # input quadratic objective
-        task.putqobj(range(2*n,2*n+m), range(2*n,2*n+m), m*[2.0])
-
-        task.putclist(range(2*n+m), n*[0.0] + n*[1.0] + m*[0.0])  # setup linear objective
-
-        # input constraint matrix row by row
-        for i in range(n):
-            task.putarow(   i, [i, n+i], [1.0, -1.0])
-            task.putarow( n+i, [i, n+i], [1.0,  1.0])
-
-        for i in range(m):
-            task.putarow( 2*n+i, range(n) + [2*n+i], list(A[i,:]) + [-1.0])
-
-        # setup bounds on constraints
-        task.putboundslice(mosek.accmode.con,
-                           0, n, n*[mosek.boundkey.up], n*[0.0], n*[0.0])
-        task.putboundslice(mosek.accmode.con,
-                           n, 2*n, n*[mosek.boundkey.lo], n*[0.0], n*[0.0])
-        task.putboundslice(mosek.accmode.con,
-                           2*n, 2*n+m, m*[mosek.boundkey.fx], list(b), list(b))
-
-        # setup variable bounds
-        task.putboundslice(mosek.accmode.var, 0, 2*n+m, (2*n+m)*[mosek.boundkey.fr], 
-                           (2*n+m)*[0.0], (2*n+m)*[0.0])
-
-        # optimize the task
-        task.putobjsense(mosek.objsense.minimize)
-        task.optimize()
-        task.solutionsummary(mosek.streamtype.log)
-        x = n*[0.0]
-        task.getsolutionslice(mosek.soltype.itr, mosek.solitem.xx, 0, n, x)
-
-        return matrix(x)
-
-def l1regls(A, b):
-    """
-    
-    Returns the solution of l1-norm regularized least-squares problem
-  
-        minimize || A*x - b ||_2^2  + || x ||_1.
-
-    """
-
-    m, n = A.size
-    q = matrix(1.0, (2*n,1))
-    q[:n] = -2.0 * A.T * b
-
-    def P(u, v, alpha = 1.0, beta = 0.0 ):
-        """
-            v := alpha * 2.0 * [ A'*A, 0; 0, 0 ] * u + beta * v 
-        """
-        v *= beta
-        v[:n] += alpha * 2.0 * A.T * (A * u[:n])
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
 
 
-    def G(u, v, alpha=1.0, beta=0.0, trans='N'):
-        """
-            v := alpha*[I, -I; -I, -I] * u + beta * v  (trans = 'N' or 'T')
-        """
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
 
-        v *= beta
-        v[:n] += alpha*(u[:n] - u[n:])
-        v[n:] += alpha*(-u[:n] - u[n:])
-
-    h = matrix(0.0, (2*n,1))
+X = np.array( [ [ 2, 0], [0, 1], [0,0]])
+y = np.array( [4, 2, 3])
 
 
-    # Customized solver for the KKT system 
-    #
-    #     [  2.0*A'*A  0    I      -I     ] [x[:n] ]     [bx[:n] ]
-    #     [  0         0   -I      -I     ] [x[n:] ]  =  [bx[n:] ].
-    #     [  I        -I   -D1^-1   0     ] [zl[:n]]     [bzl[:n]]
-    #     [ -I        -I    0      -D2^-1 ] [zl[n:]]     [bzl[n:]]
-    #
-    # where D1 = W['di'][:n]**2, D2 = W['di'][:n]**2.
-    #    
-    # We first eliminate zl and x[n:]:
-    #
-    #     ( 2*A'*A + 4*D1*D2*(D1+D2)^-1 ) * x[:n] = 
-    #         bx[:n] - (D2-D1)*(D1+D2)^-1 * bx[n:] + 
-    #         D1 * ( I + (D2-D1)*(D1+D2)^-1 ) * bzl[:n] - 
-    #         D2 * ( I - (D2-D1)*(D1+D2)^-1 ) * bzl[n:]           
-    #
-    #     x[n:] = (D1+D2)^-1 * ( bx[n:] - D1*bzl[:n]  - D2*bzl[n:] ) 
-    #         - (D2-D1)*(D1+D2)^-1 * x[:n]         
-    #
-    #     zl[:n] = D1 * ( x[:n] - x[n:] - bzl[:n] )
-    #     zl[n:] = D2 * (-x[:n] - x[n:] - bzl[n:] ).
-    #
-    # The first equation has the form
-    #
-    #     (A'*A + D)*x[:n]  =  rhs
-    #
-    # and is equivalent to
-    #
-    #     [ D    A' ] [ x:n] ]  = [ rhs ]
-    #     [ A   -I  ] [ v    ]    [ 0   ].
-    #
-    # It can be solved as 
-    #
-    #     ( A*D^-1*A' + I ) * v = A * D^-1 * rhs
-    #     x[:n] = D^-1 * ( rhs - A'*v ).
+# matrix inversion to find beta
+OLSbeta = np.linalg.inv(X.T @ X) @ X.T @ y
+print(OLSbeta)
+# and then make the prediction
+ytildeOLS = X @ OLSbeta
+print("Training MSE for OLS")
+print(MSE(y,ytildeOLS))
+ypredictOLS = X @ OLSbeta
 
-    S = matrix(0.0, (m,m))
-    Asc = matrix(0.0, (m,n))
-    v = matrix(0.0, (m,1))
+# Repeat now for Ridge regression and various values of the regularization parameter
+I = np.eye(2,2)
+# Decide which values of lambda to use
+nlambdas = 100
+MSERidgePredict = np.zeros(nlambdas)
+MSELassoPredict = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 4, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y
+    print(Ridgebeta)
+    # and then make the prediction
+    ypredictRidge = X @ Ridgebeta
+    MSERidgePredict[i] = MSE(y,ypredictRidge)
+    RegLasso = linear_model.Lasso(lmb)
+    RegLasso.fit(X,y)
+    ypredictLasso = RegLasso.predict(X)
+    print(RegLasso.coef_)
+    MSELassoPredict[i] = MSE(y,ypredictLasso)
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSERidgePredict, 'r--', label = 'MSE Ridge Train')
+plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Train')
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
    
+
    
+      
    
+    
    
+  
    
+  
    
+    
    
+      
    
+        
              
+        
    
+      
    
+    
    
+  
    
+
    
 
-    def Fkkt(W):
 
-        # Factor 
-        #
-        #     S = A*D^-1*A' + I 
-        #
-        # where D = 2*D1*D2*(D1+D2)^-1, D1 = d[:n]**-2, D2 = d[n:]**-2.
-
-        d1, d2 = W['di'][:n]**2, W['di'][n:]**2
-
-        # ds is square root of diagonal of D
-        ds = math.sqrt(2.0) * div( mul( W['di'][:n], W['di'][n:]), 
-            sqrt(d1+d2) )
-        d3 =  div(d2 - d1, d1 + d2)
-     
-        # Asc = A*diag(d)^-1/2
-        Asc = A * spdiag(ds**-1)
-
-        # S = I + A * D^-1 * A'
-        blas.syrk(Asc, S)
-        S[::m+1] += 1.0 
-        lapack.potrf(S)
-
-        def g(x, y, z):
-
-            x[:n] = 0.5 * ( x[:n] - mul(d3, x[n:]) + 
-                mul(d1, z[:n] + mul(d3, z[:n])) - mul(d2, z[n:] - 
-                mul(d3, z[n:])) )
-            x[:n] = div( x[:n], ds) 
-
-            # Solve
-            #
-            #     S * v = 0.5 * A * D^-1 * ( bx[:n] - 
-            #         (D2-D1)*(D1+D2)^-1 * bx[n:] + 
-            #         D1 * ( I + (D2-D1)*(D1+D2)^-1 ) * bzl[:n] - 
-            #         D2 * ( I - (D2-D1)*(D1+D2)^-1 ) * bzl[n:] )
-                
-            blas.gemv(Asc, x, v)
-            lapack.potrs(S, v)
-            
-            # x[:n] = D^-1 * ( rhs - A'*v ).
-            blas.gemv(Asc, v, x, alpha=-1.0, beta=1.0, trans='T')
-            x[:n] = div(x[:n], ds)
-
-            # x[n:] = (D1+D2)^-1 * ( bx[n:] - D1*bzl[:n]  - D2*bzl[n:] ) 
-            #         - (D2-D1)*(D1+D2)^-1 * x[:n]         
-            x[n:] = div( x[n:] - mul(d1, z[:n]) - mul(d2, z[n:]), d1+d2 )\
-                - mul( d3, x[:n] )
-                
-            # zl[:n] = D1^1/2 * (  x[:n] - x[n:] - bzl[:n] )
-            # zl[n:] = D2^1/2 * ( -x[:n] - x[n:] - bzl[n:] ).
-            z[:n] = mul( W['di'][:n],  x[:n] - x[n:] - z[:n] ) 
-            z[n:] = mul( W['di'][n:], -x[:n] - x[n:] - z[n:] ) 
-
-        return g
-
-    return solvers.coneqp(P, q, G, h, kktsolver = Fkkt)['x'][:n]
-
    -

      @@ -571,29 +368,22 @@ We list first the needed file. Thereafter we set up the problem to solve.
    • 37
    • 38
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs029.html b/doc/pub/week36/html/._week36-bs029.html index b6dfb66b2..97f14952a 100644 --- a/doc/pub/week36/html/._week36-bs029.html +++ b/doc/pub/week36/html/._week36-bs029.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,26 +258,113 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Another Example, now with a polynomial fit

    -

    The simpler Example

    - -

    -

    from cvxopt import matrix, normal
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
+
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
+
+
+# A seed just to ensure that the random numbers are the same for every run.
+# Useful for eventual debugging.
+np.random.seed(3155)
+
+x = np.random.rand(100)
+y = 2.0+5*x*x+0.1*np.random.randn(100)
+
+# number of features p (here degree of polynomial
+p = 3
+#  The design matrix now as function of a given polynomial
+X = np.zeros((len(x),p))
+X[:,0] = 1.0
+X[:,1] = x
+X[:,2] = x*x
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+# matrix inversion to find beta
+OLSbeta = np.linalg.inv(X_train.T @ X_train) @ X_train.T @ y_train
+print(OLSbeta)
+# and then make the prediction
+ytildeOLS = X_train @ OLSbeta
+print("Training MSE for OLS")
+print(MSE(y_train,ytildeOLS))
+ypredictOLS = X_test @ OLSbeta
+print("Test MSE OLS")
+print(MSE(y_test,ypredictOLS))
+
+# Repeat now for Lasso and Ridge regression and various values of the regularization parameter
+I = np.eye(p,p)
+# Decide which values of lambda to use
+nlambdas = 100
+MSEPredict = np.zeros(nlambdas)
+MSETrain = np.zeros(nlambdas)
+MSELassoPredict = np.zeros(nlambdas)
+MSELassoTrain = np.zeros(nlambdas)
+lambdas = np.logspace(-4, 4, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train
+    # include lasso using Scikit-Learn
+    RegLasso = linear_model.Lasso(lmb)
+    RegLasso.fit(X_train,y_train)
+    # and then make the prediction
+    ytildeRidge = X_train @ Ridgebeta
+    ypredictRidge = X_test @ Ridgebeta
+    ytildeLasso = RegLasso.predict(X_train)
+    ypredictLasso = RegLasso.predict(X_test)
+    MSEPredict[i] = MSE(y_test,ypredictRidge)
+    MSETrain[i] = MSE(y_train,ytildeRidge)
+    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
+    MSELassoTrain[i] = MSE(y_train,ytildeLasso)
+
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSETrain, label = 'MSE Ridge train')
+plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Test')
+plt.plot(np.log10(lambdas), MSELassoTrain, label = 'MSE Lasso train')
+plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Test')
+
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
    
+
    
+      
    
+    
    
+  
    
+  
    
+    
    
+      
    
+        
              
+        
    
+      
    
+    
    
+  
    
+
    
+
 
-X = matrix( [ [ 2, 0, 1], [0, 1, 3]])
-y = matrix( [4, 2, 3])
-x = l1regls(X,y)
-
    -

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs030.html b/doc/pub/week36/html/._week36-bs030.html index 8ec9fd839..f01c67811 100644 --- a/doc/pub/week36/html/._week36-bs030.html +++ b/doc/pub/week36/html/._week36-bs030.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,17 +258,303 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Using CVXOPT

    + +

    As a small addendum, we note that you can also solve this problem +using the convex optimization package +CVXOPT. This +requires, in addition to having installed CVXOPT, you need to +download the file l1regl.py. The following code example solves the +simpler problem we discussed above, where we have added the latter +python file. +

    + +

    We list first the needed file. Thereafter we set up the problem to solve.

    + + + +
    +
    +
    +
    +
    + 
    from cvxopt import matrix, spdiag, mul, div, sqrt, normal, setseed
+from cvxopt import blas, lapack, solvers, sparse, spmatrix
+import math
+
+try:
+    import mosek
+    import sys
+    __MOSEK = True
+except: __MOSEK = False
+
+if __MOSEK:
+
+    def l1regls_mosek(A, b):
+        """
+
+        Returns the solution of l1-norm regularized least-squares problem
+
+            minimize    || A*x - b ||_2^2  + e'*u
+
+            subject to  -u <= x <= u
+
+        """
+
+        m, n = A.size
+
+        env  = mosek.Env()
+        task = env.Task(0,0)
+        task.set_Stream(mosek.streamtype.log, lambda x: sys.stdout.write(x))
+
+        task.appendvars( 2*n)            # number of variables
+        task.appendcons( 2*n)            # number of constraints
+
+        # input quadratic objective
+        Q = matrix(0.0, (n,n)) 
+        blas.syrk(A, Q, alpha = 2.0, trans='T')
+
+        I = []
+        for i in range(n):
+            I.extend(range(i,n))
+
+        J = []
+        for i in range(n):
+            J.extend((n-i)*[i])
+
+        task.putqobj(I, J, list(Q[matrix(I) + matrix(J)*n]))
+        task.putclist(range(2*n), list(-2*A.T*b) + n*[1.0])  # setup linear objective
+
+        # input constraint matrix row by row
+        for i in range(n):
+            task.putarow(   i, [i, n+i], [1.0, -1.0])
+            task.putarow( n+i, [i, n+i], [1.0,  1.0])
+
+        # setup bounds on constraints
+        task.putboundslice(mosek.accmode.con,
+                           0, n, n*[mosek.boundkey.up], n*[0.0], n*[0.0])
+        task.putboundslice(mosek.accmode.con,
+                           n, 2*n, n*[mosek.boundkey.lo], n*[0.0], n*[0.0])
+
+        # setup variable bounds
+        task.putboundslice(mosek.accmode.var,
+                           0, 2*n, 2*n*[mosek.boundkey.fr], 2*n*[0.0], 2*n*[0.0])
+
+        # optimize the task
+        task.putobjsense(mosek.objsense.minimize)
+        task.optimize()
+        task.solutionsummary(mosek.streamtype.log)
+        x = n*[0.0]
+        task.getsolutionslice(mosek.soltype.itr, mosek.solitem.xx, 0, n, x)
+
+        return matrix(x)
+
+    def l1regls_mosek2(A, b):
+        """
+
+        Returns the solution of l1-norm regularized least-squares problem
+
+            minimize     w'*w + e'*u
+
+            subject to  -u <= x <= u
+
+                         A*x - w = b
+
+        """
+
+        m, n = A.size
+
+        env  = mosek.Env()
+        task = env.Task(0,0)
+        task.set_Stream(mosek.streamtype.log, lambda x: sys.stdout.write(x))
+
+        task.appendvars(2*n + m)     # number of variables
+        task.appendcons(2*n + m)     # number of constraints
+
+        # input quadratic objective
+        task.putqobj(range(2*n,2*n+m), range(2*n,2*n+m), m*[2.0])
+
+        task.putclist(range(2*n+m), n*[0.0] + n*[1.0] + m*[0.0])  # setup linear objective
+
+        # input constraint matrix row by row
+        for i in range(n):
+            task.putarow(   i, [i, n+i], [1.0, -1.0])
+            task.putarow( n+i, [i, n+i], [1.0,  1.0])
+
+        for i in range(m):
+            task.putarow( 2*n+i, range(n) + [2*n+i], list(A[i,:]) + [-1.0])
+
+        # setup bounds on constraints
+        task.putboundslice(mosek.accmode.con,
+                           0, n, n*[mosek.boundkey.up], n*[0.0], n*[0.0])
+        task.putboundslice(mosek.accmode.con,
+                           n, 2*n, n*[mosek.boundkey.lo], n*[0.0], n*[0.0])
+        task.putboundslice(mosek.accmode.con,
+                           2*n, 2*n+m, m*[mosek.boundkey.fx], list(b), list(b))
+
+        # setup variable bounds
+        task.putboundslice(mosek.accmode.var, 0, 2*n+m, (2*n+m)*[mosek.boundkey.fr], 
+                           (2*n+m)*[0.0], (2*n+m)*[0.0])
+
+        # optimize the task
+        task.putobjsense(mosek.objsense.minimize)
+        task.optimize()
+        task.solutionsummary(mosek.streamtype.log)
+        x = n*[0.0]
+        task.getsolutionslice(mosek.soltype.itr, mosek.solitem.xx, 0, n, x)
+
+        return matrix(x)
+
+def l1regls(A, b):
+    """
+    
+    Returns the solution of l1-norm regularized least-squares problem
+  
+        minimize || A*x - b ||_2^2  + || x ||_1.
+
+    """
+
+    m, n = A.size
+    q = matrix(1.0, (2*n,1))
+    q[:n] = -2.0 * A.T * b
+
+    def P(u, v, alpha = 1.0, beta = 0.0 ):
+        """
+            v := alpha * 2.0 * [ A'*A, 0; 0, 0 ] * u + beta * v 
+        """
+        v *= beta
+        v[:n] += alpha * 2.0 * A.T * (A * u[:n])
+
+
+    def G(u, v, alpha=1.0, beta=0.0, trans='N'):
+        """
+            v := alpha*[I, -I; -I, -I] * u + beta * v  (trans = 'N' or 'T')
+        """
+
+        v *= beta
+        v[:n] += alpha*(u[:n] - u[n:])
+        v[n:] += alpha*(-u[:n] - u[n:])
+
+    h = matrix(0.0, (2*n,1))
+
+
+    # Customized solver for the KKT system 
+    #
+    #     [  2.0*A'*A  0    I      -I     ] [x[:n] ]     [bx[:n] ]
+    #     [  0         0   -I      -I     ] [x[n:] ]  =  [bx[n:] ].
+    #     [  I        -I   -D1^-1   0     ] [zl[:n]]     [bzl[:n]]
+    #     [ -I        -I    0      -D2^-1 ] [zl[n:]]     [bzl[n:]]
+    #
+    # where D1 = W['di'][:n]**2, D2 = W['di'][:n]**2.
+    #    
+    # We first eliminate zl and x[n:]:
+    #
+    #     ( 2*A'*A + 4*D1*D2*(D1+D2)^-1 ) * x[:n] = 
+    #         bx[:n] - (D2-D1)*(D1+D2)^-1 * bx[n:] + 
+    #         D1 * ( I + (D2-D1)*(D1+D2)^-1 ) * bzl[:n] - 
+    #         D2 * ( I - (D2-D1)*(D1+D2)^-1 ) * bzl[n:]           
+    #
+    #     x[n:] = (D1+D2)^-1 * ( bx[n:] - D1*bzl[:n]  - D2*bzl[n:] ) 
+    #         - (D2-D1)*(D1+D2)^-1 * x[:n]         
+    #
+    #     zl[:n] = D1 * ( x[:n] - x[n:] - bzl[:n] )
+    #     zl[n:] = D2 * (-x[:n] - x[n:] - bzl[n:] ).
+    #
+    # The first equation has the form
+    #
+    #     (A'*A + D)*x[:n]  =  rhs
+    #
+    # and is equivalent to
+    #
+    #     [ D    A' ] [ x:n] ]  = [ rhs ]
+    #     [ A   -I  ] [ v    ]    [ 0   ].
+    #
+    # It can be solved as 
+    #
+    #     ( A*D^-1*A' + I ) * v = A * D^-1 * rhs
+    #     x[:n] = D^-1 * ( rhs - A'*v ).
+
+    S = matrix(0.0, (m,m))
+    Asc = matrix(0.0, (m,n))
+    v = matrix(0.0, (m,1))
+
+    def Fkkt(W):
+
+        # Factor 
+        #
+        #     S = A*D^-1*A' + I 
+        #
+        # where D = 2*D1*D2*(D1+D2)^-1, D1 = d[:n]**-2, D2 = d[n:]**-2.
+
+        d1, d2 = W['di'][:n]**2, W['di'][n:]**2
+
+        # ds is square root of diagonal of D
+        ds = math.sqrt(2.0) * div( mul( W['di'][:n], W['di'][n:]), 
+            sqrt(d1+d2) )
+        d3 =  div(d2 - d1, d1 + d2)
+     
+        # Asc = A*diag(d)^-1/2
+        Asc = A * spdiag(ds**-1)
+
+        # S = I + A * D^-1 * A'
+        blas.syrk(Asc, S)
+        S[::m+1] += 1.0 
+        lapack.potrf(S)
+
+        def g(x, y, z):
+
+            x[:n] = 0.5 * ( x[:n] - mul(d3, x[n:]) + 
+                mul(d1, z[:n] + mul(d3, z[:n])) - mul(d2, z[n:] - 
+                mul(d3, z[n:])) )
+            x[:n] = div( x[:n], ds) 
+
+            # Solve
+            #
+            #     S * v = 0.5 * A * D^-1 * ( bx[:n] - 
+            #         (D2-D1)*(D1+D2)^-1 * bx[n:] + 
+            #         D1 * ( I + (D2-D1)*(D1+D2)^-1 ) * bzl[:n] - 
+            #         D2 * ( I - (D2-D1)*(D1+D2)^-1 ) * bzl[n:] )
+                
+            blas.gemv(Asc, x, v)
+            lapack.potrs(S, v)
+            
+            # x[:n] = D^-1 * ( rhs - A'*v ).
+            blas.gemv(Asc, v, x, alpha=-1.0, beta=1.0, trans='T')
+            x[:n] = div(x[:n], ds)
+
+            # x[n:] = (D1+D2)^-1 * ( bx[n:] - D1*bzl[:n]  - D2*bzl[n:] ) 
+            #         - (D2-D1)*(D1+D2)^-1 * x[:n]         
+            x[n:] = div( x[n:] - mul(d1, z[:n]) - mul(d2, z[n:]), d1+d2 )\
+                - mul( d3, x[:n] )
+                
+            # zl[:n] = D1^1/2 * (  x[:n] - x[n:] - bzl[:n] )
+            # zl[n:] = D2^1/2 * ( -x[:n] - x[n:] - bzl[n:] ).
+            z[:n] = mul( W['di'][:n],  x[:n] - x[n:] - z[:n] ) 
+            z[n:] = mul( W['di'][n:], -x[:n] - x[n:] - z[n:] ) 
+
+        return g
+
+    return solvers.coneqp(P, q, G, h, kktsolver = Fkkt)['x'][:n]
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    Friday September 10

    -

      @@ -298,29 +580,22 @@ MathJax.Hub.Config({
    • 39
    • 40
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs031.html b/doc/pub/week36/html/._week36-bs031.html index 2d74d284b..4161cb102 100644 --- a/doc/pub/week36/html/._week36-bs031.html +++ b/doc/pub/week36/html/._week36-bs031.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,58 +258,40 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - - + +

    The simpler Example

    -

    Linking the regression analysis with a statistical interpretation

    -

    -We will now couple the discussions of ordinary least squares, Ridge -and Lasso regression with a statistical interpretation, that is we -move from a linear algebra analysis to a statistical analysis. In -particular, we will focus on what the regularization terms can result -in. We will amongst other things show that the regularization -parameter can reduce considerably the variance of the parameters -\( \beta \). + +

    +
    +
    +
    +
    + 
    from cvxopt import matrix, normal
 
-
    
-The
-advantage of doing linear regression is that we actually end up with
-analytical expressions for several statistical quantities.  
-Standard least squares and Ridge regression  allow us to
-derive quantities like the variance and other expectation values in a
-rather straightforward way.
+X = matrix( [ [ 2, 0, 1], [0, 1, 3]])
+y = matrix( [4, 2, 3])
+x = l1regls(X,y)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    -

    -It is assumed that \( \varepsilon_i -\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are -independent, i.e.: -$$ -\begin{align*} -\mbox{Cov}(\varepsilon_{i_1}, -\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} -& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. -\end{align*} -$$ -The randomness of \( \varepsilon_i \) implies that -\( \mathbf{y}_i \) is also a random variable. In particular, -\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim -\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a -non-random scalar. To specify the parameters of the distribution of -\( \mathbf{y}_i \) we need to calculate its first two moments. - -

    -Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The -notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the -row number \( i \) and perform a sum over all values \( p \). - -

      @@ -339,29 +317,22 @@ row number \( i \) and perform a sum over all values \( p \).
    • 40
    • 41
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs032.html b/doc/pub/week36/html/._week36-bs032.html index cad7fff92..7cae91dc9 100644 --- a/doc/pub/week36/html/._week36-bs032.html +++ b/doc/pub/week36/html/._week36-bs032.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,32 +258,12 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Friday September 10

    -

    Assumptions made

    - -

    -The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) -that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) -which describe our data -$$ -\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} -$$ - -

    -We approximate this function with our model from the solution of the linear regression equations, that is our -function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with -$$ -\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. -$$ - -

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs033.html b/doc/pub/week36/html/._week36-bs033.html index fedc3e064..7154e1ef9 100644 --- a/doc/pub/week36/html/._week36-bs033.html +++ b/doc/pub/week36/html/._week36-bs033.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,47 +258,54 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - - + +

    Linking the regression analysis with a statistical interpretation

    -

    Expectation value and variance

    +

    We will now couple the discussions of ordinary least squares, Ridge +and Lasso regression with a statistical interpretation, that is we +move from a linear algebra analysis to a statistical analysis. In +particular, we will focus on what the regularization terms can result +in. We will amongst other things show that the regularization +parameter can reduce considerably the variance of the parameters +\( \beta \). +

    -

    -We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +

    The +advantage of doing linear regression is that we actually end up with +analytical expressions for several statistical quantities. +Standard least squares and Ridge regression allow us to +derive quantities like the variance and other expectation values in a +rather straightforward way. +

    + +

    It is assumed that \( \varepsilon_i +\sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are +independent, i.e.: +

    $$ \begin{align*} -\mathbb{E}(y_i) & = -\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) -\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\mbox{Cov}(\varepsilon_{i_1}, +\varepsilon_{i_2}) & = \left\{ \begin{array}{lcc} \sigma^2 & \mbox{if} +& i_1 = i_2, \\ 0 & \mbox{if} & i_1 \not= i_2. \end{array} \right. \end{align*} $$ -while -its variance is -$$ -\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i -- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - -[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, -\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & -= \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i -\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, -\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 -\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + -\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 -\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, -\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. -\end{align*} -$$ +

    The randomness of \( \varepsilon_i \) implies that +\( \mathbf{y}_i \) is also a random variable. In particular, +\( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim +\mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a +non-random scalar. To specify the parameters of the distribution of +\( \mathbf{y}_i \) we need to calculate its first two moments. +

    -Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with -mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

    Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the +row number \( i \) and perform a sum over all values \( p \). +

    -

      @@ -328,29 +331,22 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
    • 42
    • 43
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs034.html b/doc/pub/week36/html/._week36-bs034.html index d98b12724..5cd514827 100644 --- a/doc/pub/week36/html/._week36-bs034.html +++ b/doc/pub/week36/html/._week36-bs034.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,96 +258,28 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Assumptions made

    -

    Expectation value and variance for \( \boldsymbol{\beta} \)

    - -

    -With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +

    The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) +that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) +which describe our data +

    $$ -\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. +\boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} $$ -This means that the estimator of the regression parameters is unbiased. - -

    -We can also calculate the variance - -

    -The variance of \( \boldsymbol{\beta} \) is +

    We approximate this function with our model from the solution of the linear regression equations, that is our +function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +

    $$ -\begin{eqnarray*} -\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} -\\ -& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} -\\ -% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -% \\ -% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -% \\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -\\ -& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -% \\ -% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} -% \\ -% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T -\\ -& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} -\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, -\end{eqnarray*} +\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. $$ -

    -where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = -\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + -\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 -\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the -variance of the estimate of the \( j \)-th regression coefficient: -\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to -construct a confidence interval for the estimates. -

    -In a similar way, we can obtain analytical expressions for say the -expectation values of the parameters \( \boldsymbol{\beta} \) and their variance -when we employ Ridge regression, allowing us again to define a confidence interval. - -

    -It is rather straightforward to show that -$$ -\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. -$$ - -We see clearly that -\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. - -

    -We can also compute the variance as - -$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, -$$ - -and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. - -

    -With this, we can compute the difference - -$$ -\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. -$$ - -The difference is non-negative definite since each component of the -matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. - -

      @@ -377,29 +305,22 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
    • 43
    • 44
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs035.html b/doc/pub/week36/html/._week36-bs035.html index 5a6d754fb..1f92f961b 100644 --- a/doc/pub/week36/html/._week36-bs035.html +++ b/doc/pub/week36/html/._week36-bs035.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,36 +258,43 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Expectation value and variance

    -

    Deriving OLS from a probability distribution

    - -

    -Our basic assumption when we derived the OLS equations was to assume -that our output is determined by a given continuous function -\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal -distribution with zero mean value and an undetermined variance -\( \sigma^2 \). - -

    -We found above that the outputs \( \boldsymbol{y} \) have a mean value given by -\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to -the design matrix are not stochastic variables, we can assume that the -probability distribution of our targets is also a normal distribution -but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a -single output \( y_i \) is given by the Gaussian distribution - +

    We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)

    $$ -y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +\begin{align*} +\mathbb{E}(y_i) & = +\mathbb{E}(\mathbf{X}_{i, \ast} \, \boldsymbol{\beta}) + \mathbb{E}(\varepsilon_i) +\, \, \, = \, \, \, \mathbf{X}_{i, \ast} \, \beta, +\end{align*} $$ -

    +

    while +its variance is +

    +$$ +\begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i +- \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - +[\mathbb{E}(y_i)]^2 \\ & = \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, +\beta + \varepsilon_i )^2] - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 \\ & += \mathbb{E} [ ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 \varepsilon_i +\mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + \varepsilon_i^2 ] - ( \mathbf{X}_{i, +\ast} \, \beta)^2 \\ & = ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 + 2 +\mathbb{E}(\varepsilon_i) \mathbf{X}_{i, \ast} \, \boldsymbol{\beta} + +\mathbb{E}(\varepsilon_i^2 ) - ( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta})^2 +\\ & = \mathbb{E}(\varepsilon_i^2 ) \, \, \, = \, \, \, +\mbox{Var}(\varepsilon_i) \, \, \, = \, \, \, \sigma^2. +\end{align*} +$$ + +

    Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs036.html b/doc/pub/week36/html/._week36-bs036.html index 614c2dde9..d6ab301e5 100644 --- a/doc/pub/week36/html/._week36-bs036.html +++ b/doc/pub/week36/html/._week36-bs036.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,49 +258,87 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Expectation value and variance for \( \boldsymbol{\beta} \)

    -

    Independent and Identically Distrubuted (iid)

    - -

    -We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. -We define this distribution as +

    With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value

    $$ -p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, +\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. $$ -which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \). +

    This means that the estimator of the regression parameters is unbiased.

    -

    -Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have +

    We can also calculate the variance

    +

    The variance of \( \boldsymbol{\beta} \) is

    $$ -p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). +\begin{eqnarray*} +\mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} +\\ +& = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} - \boldsymbol{\beta}]^{T} \} +\\ +% & = & \mathbb{E} \{ [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}] \, [(\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y}]^{T} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & \mathbb{E} \{ (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \mathbf{Y} \, \mathbf{Y}^{T} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} \} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \mathbb{E} \{ \mathbf{Y} \, \mathbf{Y}^{T} \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\\ +& = & (\mathbf{X}^{T} \mathbf{X})^{-1} \, \mathbf{X}^{T} \, \{ \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \} \, \mathbf{X} \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +% \\ +% & = & (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^T \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T % \mathbf{X})^{-1} +% \\ +% & & + \, \, \sigma^2 \, (\mathbf{X}^T \mathbf{X})^{-1} \, \mathbf{X}^T \, \mathbf{X} \, (\mathbf{X}^T \mathbf{X})^{-1} - \boldsymbol{\beta} \boldsymbol{\beta}^T +\\ +& = & \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} + \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} - \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} +\, \, \, = \, \, \, \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1}, +\end{eqnarray*} $$ -

    -We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is -in case we have a simple one-dimensional input and output case +

    where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +\mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + +\sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 +\, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the +variance of the estimate of the \( j \)-th regression coefficient: +\( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to +construct a confidence interval for the estimates. +

    + +

    In a similar way, we can obtain analytical expressions for say the +expectation values of the parameters \( \boldsymbol{\beta} \) and their variance +when we employ Ridge regression, allowing us again to define a confidence interval. +

    + +

    It is rather straightforward to show that

    $$ -\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})]. +\mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. $$ -In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). -We can now rewrite the above probability as +

    We see clearly that +\( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. +

    + +

    We can also compute the variance as

    + $$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, $$ -

    -It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \). +

    and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.

    + +

    With this, we can compute the difference

    + +$$ +\mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. +$$ + +

    The difference is non-negative definite since each component of the +matrix product is non-negative definite. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +

    -

      @@ -330,29 +364,22 @@ It is a conditional probability (see below) and reads as the likelihood of a dom
    • 45
    • 46
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs037.html b/doc/pub/week36/html/._week36-bs037.html index 0acc4016c..7313b6bf4 100644 --- a/doc/pub/week36/html/._week36-bs037.html +++ b/doc/pub/week36/html/._week36-bs037.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,42 +258,32 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Deriving OLS from a probability distribution

    -

    Maximum Likelihood Estimation (MLE)

    +

    Our basic assumption when we derived the OLS equations was to assume +that our output is determined by a given continuous function +\( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal +distribution with zero mean value and an undetermined variance +\( \sigma^2 \). +

    -

    -In statistics, maximum likelihood estimation (MLE) is a method of -estimating the parameters of an assumed probability distribution, -given some observed data. This is achieved by maximizing a likelihood -function so that, under the assumed statistical model, the observed -data is the most probable. +

    We found above that the outputs \( \boldsymbol{y} \) have a mean value given by +\( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to +the design matrix are not stochastic variables, we can assume that the +probability distribution of our targets is also a normal distribution +but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a +single output \( y_i \) is given by the Gaussian distribution +

    -

    -We will assume here that our events are given by the above Gaussian -distribution and we will determine the optimal parameters \( \beta \) by -maximizing the above PDF. However, computing the derivatives of a -product function is cumbersome and can easily lead to overflow and/or -underflowproblems, with potentials for loss of numerical precision. +$$ +y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +$$ -

    -In practice, it is more convenient to maximize the logarithm of the -PDF because it is a monotonically increasing function of the argument. -Alternatively, and this will be our option, we will minimize the -negative of the logarithm since this is a monotonically decreasing -function. -

    -Note also that maximization/minimization of the logarithm of the PDF -is equivalent to the maximization/minimization of the function itself. - -

      @@ -323,29 +309,22 @@ is equivalent to the maximization/minimization of the function itself.
    • 46
    • 47
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs038.html b/doc/pub/week36/html/._week36-bs038.html index 5ff7c9705..0b29d55d6 100644 --- a/doc/pub/week36/html/._week36-bs038.html +++ b/doc/pub/week36/html/._week36-bs038.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,44 +258,43 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Independent and Identically Distrubuted (iid)

    -

    A new Cost Function

    - -

    -We could now define a new cost function to minimize, namely the negative logarithm of the above PDF - +

    We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +We define this distribution as +

    $$ -C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, +p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, $$ -which becomes -$$ -C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. -$$ +

    which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).

    -

    -Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely +

    Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have

    $$ -\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, +p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). $$ -which leads to the well-known OLS equation for the optimal paramters \( \beta \) +

    We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is +in case we have a simple one-dimensional input and output case +

    $$ -\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})]. $$ -

    -Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics. +

    In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). +We can now rewrite the above probability as +

    +$$ +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. +$$ + +

    It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).

    -

      @@ -325,29 +320,22 @@ Before we make a similar analysis for Ridge and Lasso regression, we need a shor
    • 47
    • 48
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs039.html b/doc/pub/week36/html/._week36-bs039.html index dd46a25c2..3b525c29a 100644 --- a/doc/pub/week36/html/._week36-bs039.html +++ b/doc/pub/week36/html/._week36-bs039.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,55 +258,37 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Maximum Likelihood Estimation (MLE)

    -

    More basic Statistics and Bayes' theorem

    +

    In statistics, maximum likelihood estimation (MLE) is a method of +estimating the parameters of an assumed probability distribution, +given some observed data. This is achieved by maximizing a likelihood +function so that, under the assumed statistical model, the observed +data is the most probable. +

    -

    -A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. -Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. +

    We will assume here that our events are given by the above Gaussian +distribution and we will determine the optimal parameters \( \beta \) by +maximizing the above PDF. However, computing the derivatives of a +product function is cumbersome and can easily lead to overflow and/or +underflowproblems, with potentials for loss of numerical precision. +

    -

    -Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). +

    In practice, it is more convenient to maximize the logarithm of the +PDF because it is a monotonically increasing function of the argument. +Alternatively, and this will be our option, we will minimize the +negative of the logarithm since this is a monotonically decreasing +function. +

    -

    -We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. -The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

    Note also that maximization/minimization of the logarithm of the PDF +is equivalent to the maximization/minimization of the function itself. +

    -

    -

    -
    -

    -$$ -p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). -$$ -

    -
    - - -

    -

    -
    -

    -$$ -p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X), -$$ - -where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). -

    -
    - - -

    -If we have independent events then \( p(X,Y)=p(X)p(Y) \). - -

      @@ -336,29 +314,22 @@ If we have independent events then \( p(X,Y)=p(X)p(Y) \).
    • 48
    • 49
    • ...
      • -
    • 51
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs040.html b/doc/pub/week36/html/._week36-bs040.html index 7f38a5df1..86f6bf7fb 100644 --- a/doc/pub/week36/html/._week36-bs040.html +++ b/doc/pub/week36/html/._week36-bs040.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,29 +258,36 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    A new Cost Function

    -

    Marginal Probability

    +

    We could now define a new cost function to minimize, namely the negative logarithm of the above PDF

    -

    -The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have -

    -
    -

    $$ -p(X)=\sum_{i=0}^{n-1}p(X,Y=y_i)=\sum_{i=0}^{n-1}p(X\vert Y=y_i)p(Y=y_i)=\sum_{i=0}^{n-1}p(X\vert y_i)p(y_i). +C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, $$ -

    -
    +

    which becomes

    +$$ +C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. +$$ + +

    Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely

    + +$$ +\boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, +$$ + +

    which leads to the well-known OLS equation for the optimal paramters \( \beta \)

    +$$ +\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! +$$ + +

    Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.

    -

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs041.html b/doc/pub/week36/html/._week36-bs041.html index e96ca3f40..16725783f 100644 --- a/doc/pub/week36/html/._week36-bs041.html +++ b/doc/pub/week36/html/._week36-bs041.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,29 +258,46 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    More basic Statistics and Bayes' theorem

    -

    Conditional Probability

    +

    A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. +

    + +

    Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).

    + +

    We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

    -

    -The conditional probability, if \( p(Y) > 0 \), is

    -

    + $$ -p(X\vert Y)= \frac{p(X,Y)}{p(Y)}=\frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}. +p(X \cup Y)= p(X)+p(Y)-p(X \cap Y). $$

    -

    +

    +
    + +$$ +p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X), +$$ + +

    where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).

    +
    +
    + + +

    If we have independent events then \( p(X,Y)=p(X)p(Y) \).

    +

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs042.html b/doc/pub/week36/html/._week36-bs042.html index 5e87f317a..f834db0e5 100644 --- a/doc/pub/week36/html/._week36-bs042.html +++ b/doc/pub/week36/html/._week36-bs042.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,31 +258,23 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Marginal Probability

    -

    Bayes' Theorem

    - -

    -If we combine the conditional probability with the marginal probability and the standard product rule, we have +

    The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have

    +
    +
    + $$ -p(X\vert Y)= \frac{p(X,Y)}{p(Y)}, +p(X)=\sum_{i=0}^{n-1}p(X,Y=y_i)=\sum_{i=0}^{n-1}p(X\vert Y=y_i)p(Y=y_i)=\sum_{i=0}^{n-1}p(X\vert y_i)p(y_i). $$ +
    +
    -which we can rewrite as -$$ -p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}, -$$ - -which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \). - -

      @@ -310,28 +298,24 @@ which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \)
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • ...
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs043.html b/doc/pub/week36/html/._week36-bs043.html index c907e1798..385cbccd0 100644 --- a/doc/pub/week36/html/._week36-bs043.html +++ b/doc/pub/week36/html/._week36-bs043.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,29 +258,23 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Conditional Probability

    -

    Interpretations of Bayes' Theorem

    +

    The conditional probability, if \( p(Y) > 0 \), is

    +
    +
    + +$$ +p(X\vert Y)= \frac{p(X,Y)}{p(Y)}=\frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}. +$$ +
    +
    -

    -The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is -evaluated for the observed data \( Y \) and can be viewed as a function of -the parameter space represented by \( X \). This function is not -necesseraly normalized and is normally called the likelihood function. -

    -The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution. - -

    -Let us try to illustrate Bayes' theorem through an example. - -

      @@ -307,28 +297,23 @@ Let us try to illustrate Bayes' theorem through an example.
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs044.html b/doc/pub/week36/html/._week36-bs044.html index 75e5def13..01b8b4d66 100644 --- a/doc/pub/week36/html/._week36-bs044.html +++ b/doc/pub/week36/html/._week36-bs044.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,39 +258,25 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Bayes' Theorem

    -

    Example of Usage of Bayes' theorem

    - -

    -Let us suppose that you are undergoing a series of mammography scans in -order to rule out possible breast cancer cases. We define the -sensitivity for a positive event by the variable \( X \). It takes binary -values with \( X=1 \) representing a positive event and \( X=0 \) being a -negative event. We reserve \( Y \) as a classification parameter for -either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing). - -

    -We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not. - -

    -Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have - +

    If we combine the conditional probability with the marginal probability and the standard product rule, we have

    $$ -p(X=1\vert Y=1) =0.8. +p(X\vert Y)= \frac{p(X,Y)}{p(Y)}, $$ -

    -This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. -It is however not correct, as the following Bayesian analysis shows. +

    which we can rewrite as

    + +$$ +p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}, +$$ + +

    which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \).

    -

      @@ -316,28 +298,23 @@ It is however not correct, as the following Bayesian analysis shows.
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs045.html b/doc/pub/week36/html/._week36-bs045.html index 4ae0e583f..9b9d7b602 100644 --- a/doc/pub/week36/html/._week36-bs045.html +++ b/doc/pub/week36/html/._week36-bs045.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,40 +258,22 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Interpretations of Bayes' Theorem

    -

    Doing it correctly

    +

    The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is +evaluated for the observed data \( Y \) and can be viewed as a function of +the parameter space represented by \( X \). This function is not +necesseraly normalized and is normally called the likelihood function. +

    -

    -If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. -Let us assume that the prior probability in the population as a whole is +

    The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.

    -$$ -p(Y=1) =0.004. -$$ +

    Let us try to illustrate Bayes' theorem through an example.

    -

    -We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have -$$ -p(X=1\vert Y=0) =0.1. -$$ - -

    -Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute - -$$ -p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031. -$$ - -That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer! - -

      @@ -316,28 +294,23 @@ That is, in case of a positive test, there is only a \( 3\% \) chance of having
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs046.html b/doc/pub/week36/html/._week36-bs046.html index c0a166068..3246ce57b 100644 --- a/doc/pub/week36/html/._week36-bs046.html +++ b/doc/pub/week36/html/._week36-bs046.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,26 +258,32 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Example of Usage of Bayes' theorem

    -

    Bayes' Theorem and Ridge and Lasso Regression

    +

    Let us suppose that you are undergoing a series of mammography scans in +order to rule out possible breast cancer cases. We define the +sensitivity for a positive event by the variable \( X \). It takes binary +values with \( X=1 \) representing a positive event and \( X=0 \) being a +negative event. We reserve \( Y \) as a classification parameter for +either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing). +

    -

    -Hitherto we have discussed Ridge and Lasso regression in terms of a -linear analysis. This may to many of you feel rather technical and -perhaps not that intuitive. The question is whether we can develop a -more intuitive way of understanding what Ridge and Lasso express. +

    We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not.

    -

    -Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit. +

    Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have

    + +$$ +p(X=1\vert Y=1) =0.8. +$$ + +

    This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. +It is however not correct, as the following Bayesian analysis shows. +

    -

      @@ -301,28 +303,23 @@ Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomia
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs047.html b/doc/pub/week36/html/._week36-bs047.html index 64d5af23a..1ff897c7f 100644 --- a/doc/pub/week36/html/._week36-bs047.html +++ b/doc/pub/week36/html/._week36-bs047.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,97 +258,33 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Doing it correctly

    -

    Test Function for what happens with OLS, Ridge and Lasso

    +

    If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. +Let us assume that the prior probability in the population as a whole is +

    -

    -We will play around with a study of the values for the optimal -parameters \( \boldsymbol{\beta} \) using OLS, Ridge and Lasso regression. For -OLS, you will notice as function of the noise and polynomial degree, -that the parameters \( \beta \) will fluctuate from order to order in the -polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS. +$$ +p(Y=1) =0.004. +$$ -

    -For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one. +

    We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have

    +$$ +p(X=1\vert Y=0) =0.1. +$$ -

    +

    Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute

    - -
    import numpy as np
-import matplotlib.pyplot as plt
-from sklearn.model_selection import train_test_split
-from sklearn import linear_model
+$$
+p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031.
+$$
 
-def R2(y_data, y_model):
-    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
-def MSE(y_data,y_model):
-    n = np.size(y_model)
-    return np.sum((y_data-y_model)**2)/n
+
    That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer!
    
 
-# Make data set.
-n = 10000
-x = np.random.rand(n)
-y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)
-
-Maxpolydegree = 5
-X = np.zeros((len(x),Maxpolydegree))
-X[:,0] = 1.0
-
-for polydegree in range(1, Maxpolydegree):
-    for degree in range(polydegree):
-        X[:,degree] = x**(degree)
-
-
-# We split the data in test and training data
-X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
-
-# matrix inversion to find beta
-OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
-print(OLSbeta)
-ypredictOLS = X_test @ OLSbeta
-print("Test MSE OLS")
-print(MSE(y_test,ypredictOLS))
-# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
-# Decide which values of lambda to use
-nlambdas = 4
-MSERidgePredict = np.zeros(nlambdas)
-MSELassoPredict = np.zeros(nlambdas)
-lambdas = np.logspace(-3, 1, nlambdas)
-for i in range(nlambdas):
-    lmb = lambdas[i]
-    # Make the fit using Ridge and Lasso
-    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
-    RegRidge.fit(X_train,y_train)
-    RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
-    RegLasso.fit(X_train,y_train)
-    # and then make the prediction
-    ypredictRidge = RegRidge.predict(X_test)
-    ypredictLasso = RegLasso.predict(X_test)
-    # Compute the MSE and print it
-    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
-    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
-    print(lmb,RegRidge.coef_)
-    print(lmb,RegLasso.coef_)
-# Now plot the results
-plt.figure()
-plt.plot(np.log10(lambdas), MSERidgePredict, 'b', label = 'MSE Ridge Test')
-plt.plot(np.log10(lambdas), MSELassoPredict, 'r', label = 'MSE Lasso Test')
-plt.xlabel('log10(lambda)')
-plt.ylabel('MSE')
-plt.legend()
-plt.show()
-
    -

    -How can we understand this? - -

      @@ -371,28 +303,23 @@ How can we understand this?
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs048.html b/doc/pub/week36/html/._week36-bs048.html index 5a9c8b66e..ec265ff05 100644 --- a/doc/pub/week36/html/._week36-bs048.html +++ b/doc/pub/week36/html/._week36-bs048.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,47 +258,20 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Bayes' Theorem and Ridge and Lasso Regression

    -

    Invoking Bayes' theorem

    +

    Hitherto we have discussed Ridge and Lasso regression in terms of a +linear analysis. This may to many of you feel rather technical and +perhaps not that intuitive. The question is whether we can develop a +more intuitive way of understanding what Ridge and Lasso express. +

    -

    -Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression. +

    Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.

    -

    -For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case) -$$ -\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})], -$$ - -is given by -$$ -p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. -$$ - -

    -In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability - -$$ -p(\boldsymbol{\beta}\vert\boldsymbol{D}). -$$ - -

    -Bayes' theorem comes to our rescue here since (omitting the normalization constant) -$$ -p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}). -$$ - -

    -We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)! - -

      @@ -320,28 +289,23 @@ We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs049.html b/doc/pub/week36/html/._week36-bs049.html index 9edc8781f..8310c8f65 100644 --- a/doc/pub/week36/html/._week36-bs049.html +++ b/doc/pub/week36/html/._week36-bs049.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,53 +258,108 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Test Function for what happens with OLS, Ridge and Lasso

    -

    Ridge and Bayes

    +

    We will play around with a study of the values for the optimal +parameters \( \boldsymbol{\beta} \) using OLS, Ridge and Lasso regression. For +OLS, you will notice as function of the noise and polynomial degree, +that the parameters \( \beta \) will fluctuate from order to order in the +polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS. +

    -

    -With the posterior probability defined by a likelihood which we have -already modeled and an unknown prior, we are now ready to make -additional models for the prior. +

    For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.

    -

    -We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is -$$ -p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. -$$ + +

    +
    +
    +
    +
    + 
    import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.model_selection import train_test_split
+from sklearn import linear_model
 
-
    
-Our posterior probability becomes then (omitting the normalization factor which is just a constant)
-$$
-p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}.
-$$
+def R2(y_data, y_model):
+    return 1 - np.sum((y_data - y_model) ** 2) / np.sum((y_data - np.mean(y_data)) ** 2)
+def MSE(y_data,y_model):
+    n = np.size(y_model)
+    return np.sum((y_data-y_model)**2)/n
 
-
    
-We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we
-did for OLS, this is most conveniently done by taking the negative
-logarithm of the posterior probability. Doing so and leaving out the
-constants terms that do not depend on \( \beta \), we have
+# Make data set.
+n = 10000
+x = np.random.rand(n)
+y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)
 
-$$
-C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
-$$
+Maxpolydegree = 5
+X = np.zeros((len(x),Maxpolydegree))
+X[:,0] = 1.0
 
-and replacing \( 1/2\tau^2 \) with \( \lambda \) we have
+for polydegree in range(1, Maxpolydegree):
+    for degree in range(polydegree):
+        X[:,degree] = x**(degree)
 
-$$
-C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2,
-$$
 
-which is our Ridge cost function!  Nice, isn't it?
+# We split the data in test and training data
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
+
+# matrix inversion to find beta
+OLSbeta = np.linalg.pinv(X_train.T @ X_train) @ X_train.T @ y_train
+print(OLSbeta)
+ypredictOLS = X_test @ OLSbeta
+print("Test MSE OLS")
+print(MSE(y_test,ypredictOLS))
+# Repeat now for Lasso and Ridge regression and various values of the regularization parameter using Scikit-Learn
+# Decide which values of lambda to use
+nlambdas = 4
+MSERidgePredict = np.zeros(nlambdas)
+MSELassoPredict = np.zeros(nlambdas)
+lambdas = np.logspace(-3, 1, nlambdas)
+for i in range(nlambdas):
+    lmb = lambdas[i]
+    # Make the fit using Ridge and Lasso
+    RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
+    RegRidge.fit(X_train,y_train)
+    RegLasso = linear_model.Lasso(lmb,fit_intercept=False)
+    RegLasso.fit(X_train,y_train)
+    # and then make the prediction
+    ypredictRidge = RegRidge.predict(X_test)
+    ypredictLasso = RegLasso.predict(X_test)
+    # Compute the MSE and print it
+    MSERidgePredict[i] = MSE(y_test,ypredictRidge)
+    MSELassoPredict[i] = MSE(y_test,ypredictLasso)
+    print(lmb,RegRidge.coef_)
+    print(lmb,RegLasso.coef_)
+# Now plot the results
+plt.figure()
+plt.plot(np.log10(lambdas), MSERidgePredict, 'b', label = 'MSE Ridge Test')
+plt.plot(np.log10(lambdas), MSELassoPredict, 'r', label = 'MSE Lasso Test')
+plt.xlabel('log10(lambda)')
+plt.ylabel('MSE')
+plt.legend()
+plt.show()
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    How can we understand this?

    -

      @@ -325,28 +376,23 @@ which is our Ridge cost function! Nice, isn't it?
    • 49
    • 50
    • 51
      • +
    • 52
      • +
    • 53
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs050.html b/doc/pub/week36/html/._week36-bs050.html index b2669039c..399ac3c79 100644 --- a/doc/pub/week36/html/._week36-bs050.html +++ b/doc/pub/week36/html/._week36-bs050.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,45 +258,36 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Invoking Bayes' theorem

    -

    Lasso and Bayes

    - -

    -To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is +

    Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression.

    +

    For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case)

    $$ -p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})], $$ -

    -Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

    is given by

    $$ -p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -Taking the negative -logarithm of the posterior probability and leaving out the -constants terms that do not depend on \( \beta \), we have +

    In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability

    $$ -C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, +p(\boldsymbol{\beta}\vert\boldsymbol{D}). $$ -and replacing \( 1/\tau \) with \( \lambda \) we have - +

    Bayes' theorem comes to our rescue here since (omitting the normalization constant)

    $$ -C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}). $$ -which is our Lasso cost function! +

    We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)!

    @@ -317,27 +304,23 @@ which is our Lasso cost function!

  • 49
  • 50
  • 51
    • +
  • 52
    • +
  • 53
    • +
  • »
    • -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs051.html b/doc/pub/week36/html/._week36-bs051.html index a8df839bd..6a378c55d 100644 --- a/doc/pub/week36/html/._week36-bs051.html +++ b/doc/pub/week36/html/._week36-bs051.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - - @@ -275,8 +184,6 @@ MathJax.Hub.Config({ - -
    @@ -287,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -377,29 +258,46 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Ridge and Bayes

    -

    Why resampling methods

    +

    With the posterior probability defined by a likelihood which we have +already modeled and an unknown prior, we are now ready to make +additional models for the prior. +

    -

    -Before we proceed, we need to rethink what we have been doing. In our -eager to fit the data, we have omitted several important elements in -our regression analysis. In what follows we will +

    We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is

    -
      -
    1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
      2. -
    2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
      3. -
    +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +$$ -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +

    Our posterior probability becomes then (omitting the normalization factor which is just a constant)

    +$$ +p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. +$$ + +

    We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +did for OLS, this is most conveniently done by taking the negative +logarithm of the posterior probability. Doing so and leaving out the +constants terms that do not depend on \( \beta \), we have +

    + +$$ +C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +

    and replacing \( 1/2\tau^2 \) with \( \lambda \) we have

    + +$$ +C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, +$$ + +

    which is our Ridge cost function! Nice, isn't it?

    -

      @@ -416,38 +314,21 @@ This will allow us to link the standard linear algebra methods we have discussed
    • 51
    • 52
    • 53
      • -
    • 54
      • -
    • 55
      • -
    • 56
      • -
    • 57
      • -
    • 58
      • -
    • 59
      • -
    • 60
      • -
    • 61
      • -
    • ...
      • -
    • 78
    • »
    -
    - - -
    - - - diff --git a/doc/pub/week36/html/._week36-bs052.html b/doc/pub/week36/html/._week36-bs052.html index 6a9ebb25d..2c1b2b471 100644 --- a/doc/pub/week36/html/._week36-bs052.html +++ b/doc/pub/week36/html/._week36-bs052.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - - @@ -275,8 +184,6 @@ MathJax.Hub.Config({ - -
    @@ -287,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -377,46 +258,39 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - +

    Lasso and Bayes

    -

    Resampling methods

    -
    -
    -

    -Resampling methods are an indispensable tool in modern -statistics. They involve repeatedly drawing samples from a training -set and refitting a model of interest on each sample in order to -obtain additional information about the fitted model. For example, in -order to estimate the variability of a linear regression fit, we can -repeatedly draw different samples from the training data, fit a linear -regression to each new sample, and then examine the extent to which -the resulting fits differ. Such an approach may allow us to obtain -information that would not be available from fitting the model only -once using the original training sample. +

    To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is

    -

    -Two resampling methods are often used in Machine Learning analyses, +$$ +p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +$$ -

      -
    1. The bootstrap method
      2. -
    2. and Cross-Validation
      3. -
    +

    Our posterior probability becomes then (omitting the normalization factor which is just a constant)

    +$$ +p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. +$$ -In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular -cross-validation and the bootstrap method. +

    Taking the negative +logarithm of the posterior probability and leaving out the +constants terms that do not depend on \( \beta \), we have +

    -

    -

    -
    +$$ +C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ +

    and replacing \( 1/\tau \) with \( \lambda \) we have

    -

    +$$ +C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, +$$ + +

    which is our Lasso cost function!

    -
    - - -
    - - - diff --git a/doc/pub/week36/html/week36-bs.html b/doc/pub/week36/html/week36-bs.html index 869b857dc..7cbe28630 100644 --- a/doc/pub/week36/html/week36-bs.html +++ b/doc/pub/week36/html/week36-bs.html @@ -1,6 +1,7 @@ @@ -8,24 +9,20 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - + - - -
    @@ -199,7 +194,6 @@ MathJax.Hub.Config({ Week 36: Statistical interpretation of Linear Regression and Resampling techniques
    -
    @@ -262,36 +258,32 @@ MathJax.Hub.Config({
    -
    -

     

     

     

    - - - -
    -

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    +
    +

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    +
    -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 16, 2021

    -
    -

    Read »

    @@ -313,29 +305,22 @@ MathJax.Hub.Config({
  • 9
  • 10
  • ...
    • -
  • 51
    • +
  • 53
  • »
    • -
    - - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html index e8ff537a8..524ffd6e6 100644 --- a/doc/pub/week36/html/week36-reveal.html +++ b/doc/pub/week36/html/week36-reveal.html @@ -1,18 +1,17 @@ + - + + - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - - - - - - @@ -55,36 +54,81 @@ document.getElementsByTagName( 'head' )[0].appendChild( link ); - - - +
    +

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    +
    - - -

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 16, 2021

    -
    -











    -

    Plans for week 36

    • Thursday: Summary from last week on SVD, Statistics, probability theory and linear regression. Video of Lecture.
    • Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project. Video of Lecture
    - -Recommended Reading: - +

    Recommended Reading:

    1. Lectures on Regression
    2. Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1
    3. Hastie et al chapter 3
    -









    -

    Thursday September 9

    -











    -

    Summary from last Week and Examples

    -











    -

    Linear Regression and the SVD

    -

    -We used the SVD to analyse the matrix to invert in ordinary lineat regression +

    We used the SVD to analyse the matrix to invert in ordinary lineat regression

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix +

    Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix}, $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is +

    where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

    $$ \tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ @@ -285,118 +276,132 @@ $$ \end{bmatrix}, $$ -meaning we can write +

    meaning we can write

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -









    +









    What does it mean?

    -

    -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +

    In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that the columns of \( \boldsymbol{V} \) are the eigenvectors of \( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of \( \boldsymbol{X} \) in a descending order, it means that the column vectors \( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). +encode from the columns of \( \boldsymbol{X} \). +

    -

    -Note that these are also the eigenvectors and eigenvalues of the +

    Note that these are also the eigenvectors and eigenvalues of the Hessian matrix. +

    -

    -If we now recall the definition of the covariance matrix (not using +

    If we now recall the definition of the covariance matrix (not using Bessel's correction) we have +

    $$ \boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ -

    -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +

    meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    -











    -

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    -

    -For \( \boldsymbol{X}\boldsymbol{X}^T \) we found +

    For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -Since the matrices here have dimension \( n\times n \), we have +

    Since the matrices here have dimension \( n\times n \), we have

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -leading to +

    leading to

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -

    -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +

    Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

    $$ (\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$ -

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +

    It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

    -

    -Since we will mainly be interested in the correlations among the features +

    Since we will mainly be interested in the correlations among the features of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +

    -











    -

    Code for SVD and Inversion of Matrices

    -

    -How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? +

    How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? The simple answer is to use the linear algebra function for pseudoinvers, that is -

    +

    -
    Ainv = np.linlag.pinv(A)
-
    -

    -Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD. +

    +
    +
    +
    +
    + 
    Ainv = np.linlag.pinv(A)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVDinv(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
@@ -425,35 +430,50 @@ A = np.transpose(X) @ X
 B = np.linalg.inv(A)  # here we could use np.linalg.pinv(A)
 C = SVDinv(A)
 print(np.abs(B-C))
-
    
-
    
-









    
+
    +
    +
    + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Inverse of Rectangular Matrix

    -

    -Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular. +

    Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.

    -

    -The pseudoinverse is the generalization of the matrix inverse for square matrices to +

    The pseudoinverse is the generalization of the matrix inverse for square matrices to rectangular matrices where the number of rows and columns are not equal. +

    -

    -It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. +

    It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices. +

    -

    -Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \)) +

    Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \))

    $$ \boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T, $$ -where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD. +

    where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVDinv(A):
     U, s, VT = np.linalg.svd(A)
@@ -476,365 +496,342 @@ B = np.linalg.pinv(A)
 # Compare our own algorithm with pinv
 C = SVDinv(A)
 print(np.abs(C-B))
-
    
-
    
-As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.

    -











    -

    Ridge and LASSO Regression

    -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -or we can state it as +

    or we can state it as

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, $$ -where we have used the definition of a norm-2 vector, that is +

    where we have used the definition of a norm-2 vector, that is

    $$ \vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -

    -









    +









    From OLS to Ridge and Lasso

    -

    -By minimizing the above equation with respect to the parameters +

    By minimizing the above equation with respect to the parameters \( \boldsymbol{\beta} \) we could then obtain an analytical expression for the parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by defining a new cost function to be optimized, that is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

    -which leads to the Ridge regression minimization problem where we +

    which leads to the Ridge regression minimization problem where we require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is a finite number larger than zero. By defining +

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, $$ -

    -we have a new optimization equation +

    we have a new optimization equation

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 $$ -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    -

    -Here we have defined the norm-1 as +

    Here we have defined the norm-1 as

    $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ -

    -









    +









    Deriving the Ridge Regression Equations

    -

    -Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -and +

    and taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters +

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that +

    with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

    $$ \sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ -

    -with \( t \) a finite positive number. +

    with \( t \) a finite positive number.

    -

    -When we compare this with the ordinary least squares result we have +









    +

    Note on Scikit-Learn

    + +

    Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \beta \) becomes

    + +$$ +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

    In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function.

    + +









    +

    Comparison with OLS

    +

    When we compare this with the ordinary least squares result we have

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

    -

    -We see that Ridge regression is nothing but the standard OLS with a +

    We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    -











    -

    SVD analysis

    -

    -Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

    $$ \tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$ -

    -For Ridge regression this becomes +

    For Ridge regression this becomes

    $$ \tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$ -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). +

    with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

    -











    -

    Interpreting the Ridge results

    -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have +

    Since \( \lambda \geq 0 \), it means that compared to OLS, we have

    $$ \frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. $$ -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +

    Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by \( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has eigenvalues ordered in a descending way, that is \( \sigma_i \geq \sigma_{i+1} \). +

    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods. +

    For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

    -











    -

    More interpretations

    -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is +

    For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

    $$ \boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -

    -In this case the standard OLS results in +

    In this case the standard OLS results in

    $$ \boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -

    -and +

    and

    $$ \boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, $$ -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +

    that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and the Ridge estimator converges to zero when the hyperparameter goes to infinity. +

    -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. +

    We will come back to more interpreations after we have gone through some of the statistical analysis part.

    -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -











    -

    Deriving the Lasso Regression Equations

    -

    -Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function +

    Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

    -Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +

    Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

    $$ \frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. $$ -we have that the derivative of the cost function is +

    we have that the derivative of the cost function is

    $$ \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, $$ -and reordering we have +

    and reordering we have

    $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. $$ -This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. +

    This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    -











    -

    Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression

    -

    -Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the +

    Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the diagonal. In this case we have an equal number of rows and columns \( n=p \). +

    -

    -Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) +

    Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))

    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, $$ -and minimizing we have that +

    and minimizing we have that

    $$ \hat{\beta}_i^{\mathrm{OLS}} = y_i. $$ -

    -









    +









    Ridge Regression

    -

    -For Ridge regression our cost function is +

    For Ridge regression our cost function is

    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ -and minimizing we have that +

    and minimizing we have that

    $$ \hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. $$ -

    -









    +









    Lasso Regression

    -

    -For Lasso regression our cost function is +

    For Lasso regression our cost function is

    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ -and minimizing we have that +

    and minimizing we have that

    $$ -2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, $$ -which leads to +

    which leads to

    $$ \hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. $$ -

    -Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \). +

    Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

    -











    -

    Yet another Example

    -

    -Let us assume we have a data set with outputs/targets given by the vector +

    Let us assume we have a data set with outputs/targets given by the vector

    $$ \boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix}, $$ -and our inputs as a \( 3\times 2 \) design matrix +

    and our inputs as a \( 3\times 2 \) design matrix

    $$ \boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$ -meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. +

    meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.

    -











    -

    The OLS case

    -

    -For ordinary least squares (OLS) we know that the optimal solution is +

    For ordinary least squares (OLS) we know that the optimal solution is

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -Inserting the above values we obtain that +

    Inserting the above values we obtain that

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$ -

    -The code which implements this simpler case is presented after the discussion of Ridge and Lasso. +

    The code which implements this simpler case is presented after the discussion of Ridge and Lasso.

    -











    -

    The Ridge case

    -

    -For Ridge regression we have +

    For Ridge regression we have

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -Inserting the above values we obtain that +

    Inserting the above values we obtain that

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$ -

    -There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +

    There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \). +

    -

    -To see this, let us write the cost function for Ridge regression. +

    To see this, let us write the cost function for Ridge regression.

    -











    -

    Writing the Cost Function

    -

    -We define the MSE without the \( 1/n \) factor and have then, using that +

    We define the MSE without the \( 1/n \) factor and have then, using that

    $$ \boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, $$ @@ -844,32 +841,29 @@ $$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), $$ -and taking the derivative with respect to \( \beta_0 \) we get +

    and taking the derivative with respect to \( \beta_0 \) we get

    $$ \beta_0=\frac{8}{4+\lambda}, $$ -and for \( \beta_1 \) we obtain +

    and for \( \beta_1 \) we obtain

    $$ \beta_1=\frac{2}{1+\lambda}, $$ -

    -Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving +

    Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving

    $$ \left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, $$ -which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \). +

    which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).

    -











    -

    Lasso case

    -

    -For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) +

    For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) and \( \beta_1 \). This gives us the following derivatives of the cost function +

    $$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), $$ @@ -879,64 +873,59 @@ $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, $$ -and +

    and

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. $$ -We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely - +

    We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely

    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).
    -









    -

    The first Case

    -

    -If we consider the first case, we have then +

    If we consider the first case, we have then

    $$ -4(4-2\beta_0)+\lambda=0, $$ -and +

    and

    $$ -2(2-\beta_1)+\lambda=0. $$ -which yields +

    which yields

    $$ \beta_0=\frac{16+\lambda}{8}, $$ -and +

    and

    $$ \beta_1=\frac{4+\lambda}{2}. $$ -

    -Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. +

    Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.

    -











    -

    Simple code for solving the above problem

    -

    -Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. +

    Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.

    -

    -First we study and compare the OLS and Ridge results. The next code compares all three methods. +

    First we study and compare the OLS and Ridge results. The next code compares all three methods.

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -985,19 +974,34 @@ plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-We see here that we reach a plateau. What is actually happening?
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We see here that we reach a plateau. What is actually happening?

    -











    -

    With Lasso Regression

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -1053,16 +1057,33 @@ plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Another Example, now with a polynomial fit

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -1140,28 +1161,44 @@ plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Using CVXOPT

    -

    -As a small addendum, we note that you can also solve this problem +

    As a small addendum, we note that you can also solve this problem using the convex optimization package CVXOPT. This requires, in addition to having installed CVXOPT, you need to download the file l1regl.py. The following code example solves the simpler problem we discussed above, where we have added the latter python file. +

    -

    -We list first the needed file. Thereafter we set up the problem to solve. +

    We list first the needed file. Thereafter we set up the problem to solve.

    -

    -

    from cvxopt import matrix, spdiag, mul, div, sqrt, normal, setseed
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from cvxopt import matrix, spdiag, mul, div, sqrt, normal, setseed
 from cvxopt import blas, lapack, solvers, sparse, spmatrix
 import math
 
@@ -1418,52 +1455,80 @@ We list first the needed file. Thereafter we set up the problem to solve.
         return g
 
     return solvers.coneqp(P, q, G, h, kktsolver = Fkkt)['x'][:n]
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    The simpler Example

    -

    -

    from cvxopt import matrix, normal
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from cvxopt import matrix, normal
 
 X = matrix( [ [ 2, 0, 1], [0, 1, 3]])
 y = matrix( [4, 2, 3])
 x = l1regls(X,y)
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Friday September 10

    -

    -

    Linking the regression analysis with a statistical interpretation

    -

    -We will now couple the discussions of ordinary least squares, Ridge +

    We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \). +

    -

    -The +

    The advantage of doing linear regression is that we actually end up with analytical expressions for several statistical quantities. Standard least squares and Ridge regression allow us to derive quantities like the variance and other expectation values in a rather straightforward way. +

    -

    -It is assumed that \( \varepsilon_i +

    It is assumed that \( \varepsilon_i \sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are independent, i.e.: +

    $$ \begin{align*} \mbox{Cov}(\varepsilon_{i_1}, @@ -1472,45 +1537,42 @@ $$ \end{align*} $$ -The randomness of \( \varepsilon_i \) implies that +

    The randomness of \( \varepsilon_i \) implies that \( \mathbf{y}_i \) is also a random variable. In particular, \( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim \mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a non-random scalar. To specify the parameters of the distribution of -\( \mathbf{y}_i \) we need to calculate its first two moments. +\( \mathbf{y}_i \) we need to calculate its first two moments. +

    -

    -Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +

    Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the row number \( i \) and perform a sum over all values \( p \). +

    -











    -

    Assumptions made

    -

    -The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) +

    The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) which describe our data +

    $$ \boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} $$ -

    -We approximate this function with our model from the solution of the linear regression equations, that is our +

    We approximate this function with our model from the solution of the linear regression equations, that is our function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +

    $$ \boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. $$ -

    -









    +









    Expectation value and variance

    -

    -We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +

    We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)

    $$ \begin{align*} \mathbb{E}(y_i) & = @@ -1519,8 +1581,9 @@ $$ \end{align*} $$ -while +

    while its variance is +

    $$ \begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - @@ -1536,27 +1599,23 @@ $$ \end{align*} $$ -Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with -mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

    Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

    -











    -

    Expectation value and variance for \( \boldsymbol{\beta} \)

    -

    -With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +

    With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value

    $$ \mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. $$ -This means that the estimator of the regression parameters is unbiased. +

    This means that the estimator of the regression parameters is unbiased.

    -

    -We can also calculate the variance +

    We can also calculate the variance

    -

    -The variance of \( \boldsymbol{\beta} \) is +

    The variance of \( \boldsymbol{\beta} \) is

    $$ \begin{eqnarray*} \mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} @@ -1580,189 +1639,173 @@ $$ \end{eqnarray*} $$ -

    -where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +

    where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the variance of the estimate of the \( j \)-th regression coefficient: \( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to construct a confidence interval for the estimates. +

    -

    -In a similar way, we can obtain analytical expressions for say the +

    In a similar way, we can obtain analytical expressions for say the expectation values of the parameters \( \boldsymbol{\beta} \) and their variance -when we employ Ridge regression, allowing us again to define a confidence interval. +when we employ Ridge regression, allowing us again to define a confidence interval. +

    -

    -It is rather straightforward to show that +

    It is rather straightforward to show that

    $$ \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. $$ -We see clearly that +

    We see clearly that \( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. +

    -

    -We can also compute the variance as +

    We can also compute the variance as

    $$ \mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, $$ -and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. +

    and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.

    -

    -With this, we can compute the difference +

    With this, we can compute the difference

    $$ \mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. $$ -The difference is non-negative definite since each component of the +

    The difference is non-negative definite since each component of the matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +

    -











    -

    Deriving OLS from a probability distribution

    -

    -Our basic assumption when we derived the OLS equations was to assume +

    Our basic assumption when we derived the OLS equations was to assume that our output is determined by a given continuous function \( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal distribution with zero mean value and an undetermined variance \( \sigma^2 \). +

    -

    -We found above that the outputs \( \boldsymbol{y} \) have a mean value given by +

    We found above that the outputs \( \boldsymbol{y} \) have a mean value given by \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a single output \( y_i \) is given by the Gaussian distribution +

    $$ y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -









    +









    Independent and Identically Distrubuted (iid)

    -

    -We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +

    We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. We define this distribution as +

    $$ p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, $$ -which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \). +

    which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).

    -

    -Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have +

    Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have

    $$ p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). $$ -

    -We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is +

    We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is in case we have a simple one-dimensional input and output case +

    $$ \boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})]. $$ -In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). +

    In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). We can now rewrite the above probability as +

    $$ p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \). +

    It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).

    -











    -

    Maximum Likelihood Estimation (MLE)

    -

    -In statistics, maximum likelihood estimation (MLE) is a method of +

    In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed -data is the most probable. +data is the most probable. +

    -

    -We will assume here that our events are given by the above Gaussian +

    We will assume here that our events are given by the above Gaussian distribution and we will determine the optimal parameters \( \beta \) by maximizing the above PDF. However, computing the derivatives of a product function is cumbersome and can easily lead to overflow and/or underflowproblems, with potentials for loss of numerical precision. +

    -

    -In practice, it is more convenient to maximize the logarithm of the +

    In practice, it is more convenient to maximize the logarithm of the PDF because it is a monotonically increasing function of the argument. Alternatively, and this will be our option, we will minimize the negative of the logarithm since this is a monotonically decreasing function. +

    -

    -Note also that maximization/minimization of the logarithm of the PDF +

    Note also that maximization/minimization of the logarithm of the PDF is equivalent to the maximization/minimization of the function itself. +

    -











    -

    A new Cost Function

    -

    -We could now define a new cost function to minimize, namely the negative logarithm of the above PDF +

    We could now define a new cost function to minimize, namely the negative logarithm of the above PDF

    $$ C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, $$ -which becomes +

    which becomes

    $$ C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. $$ -

    -Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely +

    Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely

    $$ \boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, $$ -which leads to the well-known OLS equation for the optimal paramters \( \beta \) +

    which leads to the well-known OLS equation for the optimal paramters \( \beta \)

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$ -

    -Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics. +

    Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.

    -











    -

    More basic Statistics and Bayes' theorem

    -

    -A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +

    A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. +

    -

    -Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). +

    Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).

    -

    -We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. -The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

    We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

    -

    Union of events is given by

    @@ -1772,7 +1815,6 @@ $$

    -

    The product rule (aka joint probability) is given by

    @@ -1780,20 +1822,16 @@ $$ p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X), $$ -where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +

    where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).

    -

    -If we have independent events then \( p(X,Y)=p(X)p(Y) \). +

    If we have independent events then \( p(X,Y)=p(X)p(Y) \).

    -











    -

    Marginal Probability

    -

    -The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have +

    The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have

    @@ -1803,13 +1841,10 @@ $$

    -











    -

    Conditional Probability

    -

    -The conditional probability, if \( p(Y) > 0 \), is +

    The conditional probability, if \( p(Y) > 0 \), is

    @@ -1819,130 +1854,113 @@ $$

    -











    -

    Bayes' Theorem

    -

    -If we combine the conditional probability with the marginal probability and the standard product rule, we have +

    If we combine the conditional probability with the marginal probability and the standard product rule, we have

    $$ p(X\vert Y)= \frac{p(X,Y)}{p(Y)}, $$ -which we can rewrite as +

    which we can rewrite as

    $$ p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}, $$ -which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \). +

    which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \).

    -











    -

    Interpretations of Bayes' Theorem

    -

    -The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is +

    The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is evaluated for the observed data \( Y \) and can be viewed as a function of the parameter space represented by \( X \). This function is not necesseraly normalized and is normally called the likelihood function. +

    -

    -The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution. +

    The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.

    -

    -Let us try to illustrate Bayes' theorem through an example. +

    Let us try to illustrate Bayes' theorem through an example.

    -











    -

    Example of Usage of Bayes' theorem

    -

    -Let us suppose that you are undergoing a series of mammography scans in +

    Let us suppose that you are undergoing a series of mammography scans in order to rule out possible breast cancer cases. We define the sensitivity for a positive event by the variable \( X \). It takes binary values with \( X=1 \) representing a positive event and \( X=0 \) being a negative event. We reserve \( Y \) as a classification parameter for either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing). +

    -

    -We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not. +

    We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not.

    -

    -Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have +

    Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have

    $$ p(X=1\vert Y=1) =0.8. $$ -

    -This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. +

    This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. It is however not correct, as the following Bayesian analysis shows. +

    -











    -

    Doing it correctly

    -

    -If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. +

    If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. Let us assume that the prior probability in the population as a whole is +

    $$ p(Y=1) =0.004. $$ -

    -We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have +

    We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have

    $$ p(X=1\vert Y=0) =0.1. $$ -

    -Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute +

    Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute

    $$ p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031. $$ -That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer! +

    That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer!

    -











    -

    Bayes' Theorem and Ridge and Lasso Regression

    -

    -Hitherto we have discussed Ridge and Lasso regression in terms of a +

    Hitherto we have discussed Ridge and Lasso regression in terms of a linear analysis. This may to many of you feel rather technical and perhaps not that intuitive. The question is whether we can develop a more intuitive way of understanding what Ridge and Lasso express. +

    -

    -Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit. +

    Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.

    -











    -

    Test Function for what happens with OLS, Ridge and Lasso

    -

    -We will play around with a study of the values for the optimal +

    We will play around with a study of the values for the optimal parameters \( \boldsymbol{\beta} \) using OLS, Ridge and Lasso regression. For OLS, you will notice as function of the noise and polynomial degree, that the parameters \( \beta \) will fluctuate from order to order in the polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS. +

    -

    -For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one. +

    For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import matplotlib.pyplot as plt
 from sklearn.model_selection import train_test_split
 from sklearn import linear_model
@@ -2005,130 +2023,122 @@ plt.xlabel('log10(lambda)')
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-How can we understand this?
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    How can we understand this?

    -











    -

    Invoking Bayes' theorem

    -

    -Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression. +

    Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression.

    -

    -For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case) +

    For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case)

    $$ \boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})], $$ -is given by +

    is given by

    $$ p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability +

    In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability

    $$ p(\boldsymbol{\beta}\vert\boldsymbol{D}). $$ -

    -Bayes' theorem comes to our rescue here since (omitting the normalization constant) +

    Bayes' theorem comes to our rescue here since (omitting the normalization constant)

    $$ p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}). $$ -

    -We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)! +

    We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)!

    -











    -

    Ridge and Bayes

    -

    -With the posterior probability defined by a likelihood which we have +

    With the posterior probability defined by a likelihood which we have already modeled and an unknown prior, we are now ready to make additional models for the prior. +

    -

    -We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is +

    We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is

    $$ p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. $$ -

    -Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

    Our posterior probability becomes then (omitting the normalization factor which is just a constant)

    $$ p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. $$ -

    -We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +

    We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we did for OLS, this is most conveniently done by taking the negative logarithm of the posterior probability. Doing so and leaving out the constants terms that do not depend on \( \beta \), we have +

    $$ C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, $$ -and replacing \( 1/2\tau^2 \) with \( \lambda \) we have +

    and replacing \( 1/2\tau^2 \) with \( \lambda \) we have

    $$ C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, $$ -which is our Ridge cost function! Nice, isn't it? +

    which is our Ridge cost function! Nice, isn't it?

    -











    -

    Lasso and Bayes

    -

    -To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is +

    To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is

    $$ p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. $$ -

    -Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

    Our posterior probability becomes then (omitting the normalization factor which is just a constant)

    $$ p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. $$ -

    -Taking the negative +

    Taking the negative logarithm of the posterior probability and leaving out the constants terms that do not depend on \( \beta \), we have +

    $$ C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -and replacing \( 1/\tau \) with \( \lambda \) we have +

    and replacing \( 1/\tau \) with \( \lambda \) we have

    $$ C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -which is our Lasso cost function! - +

    which is our Lasso cost function!

    - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html index 933870fce..8628be22f 100644 --- a/doc/pub/week36/html/week36.html +++ b/doc/pub/week36/html/week36.html @@ -1,6 +1,7 @@ @@ -8,29 +9,97 @@ Automatically generated HTML file from DocOnce source - Week 36: Statistical interpretation of Linear Regression and Resampling techniques - - - - +
    +

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    +
    - - -

    Week 36: Statistical interpretation of Linear Regression and Resampling techniques

    - -

    -

    Morten Hjorth-Jensen [1, 2]
    - -

    +

    +[1] Department of Physics, University of Oslo +
    +
    +[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University +
    +
    +
    +

    Nov 3, 2021

    +
    +
    -
    [1] Department of Physics, University of Oslo
    -
    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
    -
    -

    -

    Sep 16, 2021

    -
    -











    -

    Plans for week 36

    • Thursday: Summary from last week on SVD, Statistics, probability theory and linear regression. Video of Lecture.
    • Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project. Video of Lecture
    - -Recommended Reading: - +

    Recommended Reading:

    1. Lectures on Regression
    2. Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1
    3. Hastie et al chapter 3
    -









    -

    Thursday September 9

    -











    -

    Summary from last Week and Examples

    -











    -

    Linear Regression and the SVD

    -

    -We used the SVD to analyse the matrix to invert in ordinary lineat regression +

    We used the SVD to analyse the matrix to invert in ordinary lineat regression

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. $$ -Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix +

    Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix

    $$ \boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\end{bmatrix}, $$ -where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is +

    where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is

    $$ \tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\ @@ -290,118 +353,132 @@ $$ \end{bmatrix}, $$ -meaning we can write +

    meaning we can write

    $$ \boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T. $$ -Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get +

    Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2. $$ -

    -









    +









    What does it mean?

    -

    -This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) +

    This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues given by the singular values squared, that is +

    $$ \left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2. $$ -

    -In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive +

    In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that the columns of \( \boldsymbol{V} \) are the eigenvectors of \( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of \( \boldsymbol{X} \) in a descending order, it means that the column vectors \( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they -encode from the columns of \( \boldsymbol{X} \). +encode from the columns of \( \boldsymbol{X} \). +

    -

    -Note that these are also the eigenvectors and eigenvalues of the +

    Note that these are also the eigenvectors and eigenvalues of the Hessian matrix. +

    -

    -If we now recall the definition of the covariance matrix (not using +

    If we now recall the definition of the covariance matrix (not using Bessel's correction) we have +

    $$ \boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}, $$ -

    -meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +

    meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). +

    -











    -

    And finally \( \boldsymbol{X}\boldsymbol{X}^T \)

    -

    -For \( \boldsymbol{X}\boldsymbol{X}^T \) we found +

    For \( \boldsymbol{X}\boldsymbol{X}^T \) we found

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ -Since the matrices here have dimension \( n\times n \), we have +

    Since the matrices here have dimension \( n\times n \), we have

    $$ \boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ -leading to +

    leading to

    $$ \boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$ -

    -Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +

    Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem

    $$ (\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$ -

    -It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by +

    It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \boldsymbol{X} \). +

    -

    -Since we will mainly be interested in the correlations among the features +

    Since we will mainly be interested in the correlations among the features of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \). +

    -











    -

    Code for SVD and Inversion of Matrices

    -

    -How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? +

    How do we use the SVD to invert a matrix \( \boldsymbol{X}^\boldsymbol{X} \) which is singular or near singular? The simple answer is to use the linear algebra function for pseudoinvers, that is -

    +

    -
    Ainv = np.linlag.pinv(A)
-
    -

    -Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD. +

    +
    +
    +
    +
    + 
    Ainv = np.linlag.pinv(A)
+
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    +
    + +

    Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVDinv(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
@@ -430,35 +507,50 @@ A = np.t
 B = np.linalg.inv(A)  # here we could use np.linalg.pinv(A)
 C = SVDinv(A)
 print(np.abs(B-C))
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Inverse of Rectangular Matrix

    -

    -Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular. +

    Although our matrix to invert \( \boldsymbol{X}^T\boldsymbol{X} \) is a square matrix, our matrix may be singular.

    -

    -The pseudoinverse is the generalization of the matrix inverse for square matrices to +

    The pseudoinverse is the generalization of the matrix inverse for square matrices to rectangular matrices where the number of rows and columns are not equal. +

    -

    -It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. +

    It is also called the the Moore-Penrose Inverse after two independent discoverers of the method or the Generalized Inverse. It is used for the calculation of the inverse for singular or near singular matrices and for rectangular matrices. +

    -

    -Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \)) +

    Using the SVD we can obtain the pseudoinverse of a matrix \( \boldsymbol{A} \) (labeled here as \( \boldsymbol{A}_{\mathrm{PI}} \))

    $$ \boldsymbol{A}_{\mathrm{PI}}= \boldsymbol{V}\boldsymbol{D}_{\mathrm{PI}}\boldsymbol{U}^T, $$ -where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD. +

    where \( \boldsymbol{D}_{\mathrm{PI}} \) can be calculated by creating a diagonal matrix from \( \boldsymbol{\Sigma} \) where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 # SVD inversion
 def SVDinv(A):
     U, s, VT = np.linalg.svd(A)
@@ -481,365 +573,342 @@ B = np.l
 # Compare our own algorithm with pinv
 C = SVDinv(A)
 print(np.abs(C-B))
-
    
-
    
-As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by Numpy.

    -











    -

    Ridge and LASSO Regression

    -

    -Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is +

    Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is our optimization problem is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\left\{\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right\}. $$ -or we can state it as +

    or we can state it as

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\sum_{i=0}^{n-1}\left(y_i-\tilde{y}_i\right)^2=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2, $$ -where we have used the definition of a norm-2 vector, that is +

    where we have used the definition of a norm-2 vector, that is

    $$ \vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}. $$ -

    -









    +









    From OLS to Ridge and Lasso

    -

    -By minimizing the above equation with respect to the parameters +

    By minimizing the above equation with respect to the parameters \( \boldsymbol{\beta} \) we could then obtain an analytical expression for the parameters \( \boldsymbol{\beta} \). We can add a regularization parameter \( \lambda \) by defining a new cost function to be optimized, that is +

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_2^2 $$ -

    -which leads to the Ridge regression minimization problem where we +

    which leads to the Ridge regression minimization problem where we require that \( \vert\vert \boldsymbol{\beta}\vert\vert_2^2\le t \), where \( t \) is a finite number larger than zero. By defining +

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1, $$ -

    -we have a new optimization equation +

    we have a new optimization equation

    $$ {\displaystyle \min_{\boldsymbol{\beta}\in {\mathbb{R}}^{p}}}\frac{1}{n}\vert\vert \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\vert\vert_2^2+\lambda\vert\vert \boldsymbol{\beta}\vert\vert_1 $$ -which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. +

    which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator.

    -

    -Here we have defined the norm-1 as +

    Here we have defined the norm-1 as

    $$ \vert\vert \boldsymbol{x}\vert\vert_1 = \sum_i \vert x_i\vert. $$ -

    -









    +









    Deriving the Ridge Regression Equations

    -

    -Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have +

    Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta}, $$ -and +

    and taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then a slightly modified matrix inversion problem which for finite values of \( \lambda \) does not suffer from singularity problems. We obtain the optimal parameters +

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -

    -with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that +

    with \( \boldsymbol{I} \) being a \( p\times p \) identity matrix with the constraint that

    $$ \sum_{i=0}^{p-1} \beta_i^2 \leq t, $$ -

    -with \( t \) a finite positive number. +

    with \( t \) a finite positive number.

    -

    -When we compare this with the ordinary least squares result we have +









    +

    Note on Scikit-Learn

    + +

    Note well that a library like Scikit-Learn does not include the \( 1/n \) factor in the expression for the mean-squared error. If you include it, the optimal parameter \( \beta \) becomes

    + +$$ +\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+n\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +

    In our codes where we compare our own codes with Scikit-Learn, we do thus not include the \( 1/n \) factor in the cost function.

    + +









    +

    Comparison with OLS

    +

    When we compare this with the ordinary least squares result we have

    $$ \hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, $$ -which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). +

    which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).

    -

    -We see that Ridge regression is nothing but the standard OLS with a +

    We see that Ridge regression is nothing but the standard OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in particular for our discussion of the bias-variance tradeoff are rather interesting. We will see that for specific values of \( \lambda \), we may even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here. +

    -











    -

    SVD analysis

    -

    -Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) +

    Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as +

    $$ \tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$ -

    -For Ridge regression this becomes +

    For Ridge regression this becomes

    $$ \tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$ -

    -with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). +

    with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).

    -











    -

    Interpreting the Ridge results

    -

    -Since \( \lambda \geq 0 \), it means that compared to OLS, we have +

    Since \( \lambda \geq 0 \), it means that compared to OLS, we have

    $$ \frac{\sigma_j^2}{\sigma_j^2+\lambda} \leq 1. $$ -

    -Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the +

    Ridge regression finds the coordinates of \( \boldsymbol{y} \) with respect to the orthonormal basis \( \boldsymbol{U} \), it then shrinks the coordinates by \( \frac{\sigma_j^2}{\sigma_j^2+\lambda} \). Recall that the SVD has eigenvalues ordered in a descending way, that is \( \sigma_i \geq \sigma_{i+1} \). +

    -

    -For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods. +

    For small eigenvalues \( \sigma_i \) it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.

    -











    -

    More interpretations

    -

    -For the sake of simplicity, let us assume that the design matrix is orthonormal, that is +

    For the sake of simplicity, let us assume that the design matrix is orthonormal, that is

    $$ \boldsymbol{X}^T\boldsymbol{X}=(\boldsymbol{X}^T\boldsymbol{X})^{-1} =\boldsymbol{I}. $$ -

    -In this case the standard OLS results in +

    In this case the standard OLS results in

    $$ \boldsymbol{\beta}^{\mathrm{OLS}} = \boldsymbol{X}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\boldsymbol{y}, $$ -

    -and +

    and

    $$ \boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{I}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\left(1+\lambda\right)^{-1}\boldsymbol{\beta}^{\mathrm{OLS}}, $$ -

    -that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and +

    that is the Ridge estimator scales the OLS estimator by the inverse of a factor \( 1+\lambda \), and the Ridge estimator converges to zero when the hyperparameter goes to infinity. +

    -

    -We will come back to more interpreations after we have gone through some of the statistical analysis part. +

    We will come back to more interpreations after we have gone through some of the statistical analysis part.

    -

    -For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. +

    For more discussions of Ridge and Lasso regression, Wessel van Wieringen's article is highly recommended. Similarly, Mehta et al's article is also recommended. +

    -











    -

    Deriving the Lasso Regression Equations

    -

    -Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function +

    Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function

    $$ C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -

    -Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty) +

    Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)

    $$ \frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right. $$ -we have that the derivative of the cost function is +

    we have that the derivative of the cost function is

    $$ \frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0, $$ -and reordering we have +

    and reordering we have

    $$ \boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}. $$ -This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later. +

    This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.

    -











    -

    Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression

    -

    -Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the +

    Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the diagonal. In this case we have an equal number of rows and columns \( n=p \). +

    -

    -Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \)) +

    Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))

    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2, $$ -and minimizing we have that +

    and minimizing we have that

    $$ \hat{\beta}_i^{\mathrm{OLS}} = y_i. $$ -

    -









    +









    Ridge Regression

    -

    -For Ridge regression our cost function is +

    For Ridge regression our cost function is

    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2, $$ -and minimizing we have that +

    and minimizing we have that

    $$ \hat{\beta}_i^{\mathrm{Ridge}} = \frac{y_i}{1+\lambda}. $$ -

    -









    +









    Lasso Regression

    -

    -For Lasso regression our cost function is +

    For Lasso regression our cost function is

    $$ C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\vert\beta_i\vert=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\sqrt{\beta_i^2}, $$ -and minimizing we have that +

    and minimizing we have that

    $$ -2\sum_{i=0}^{p-1}(y_i-\beta_i)+\lambda \sum_{i=0}^{p-1}\frac{(\beta_i)}{\vert\beta_i\vert}=0, $$ -which leads to +

    which leads to

    $$ \hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\ y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\ 0 &\mathrm{if} & \vert y_i\vert\le \frac{\lambda}{2}\end{array}\right.\\. $$ -

    -Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \). +

    Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).

    -











    -

    Yet another Example

    -

    -Let us assume we have a data set with outputs/targets given by the vector +

    Let us assume we have a data set with outputs/targets given by the vector

    $$ \boldsymbol{y}=\begin{bmatrix}4 \\ 2 \\3\end{bmatrix}, $$ -and our inputs as a \( 3\times 2 \) design matrix +

    and our inputs as a \( 3\times 2 \) design matrix

    $$ \boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix}, $$ -meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression. +

    meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.

    -











    -

    The OLS case

    -

    -For ordinary least squares (OLS) we know that the optimal solution is +

    For ordinary least squares (OLS) we know that the optimal solution is

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left( \boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -Inserting the above values we obtain that +

    Inserting the above values we obtain that

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix}, $$ -

    -The code which implements this simpler case is presented after the discussion of Ridge and Lasso. +

    The code which implements this simpler case is presented after the discussion of Ridge and Lasso.

    -











    -

    The Ridge case

    -

    -For Ridge regression we have +

    For Ridge regression we have

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\left( \boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}. $$ -Inserting the above values we obtain that +

    Inserting the above values we obtain that

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix}, $$ -

    -There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). +

    There is normally a constraint on the value of \( \vert\vert \boldsymbol{\beta}\vert\vert_2 \) via the parameter \( \lambda \). Let us for simplicity assume that \( \beta_0^2+\beta_1^2=1 \) as constraint. This will allow us to find an expression for the optimal values of \( \beta \) and \( \lambda \). +

    -

    -To see this, let us write the cost function for Ridge regression. +

    To see this, let us write the cost function for Ridge regression.

    -











    -

    Writing the Cost Function

    -

    -We define the MSE without the \( 1/n \) factor and have then, using that +

    We define the MSE without the \( 1/n \) factor and have then, using that

    $$ \boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix}, $$ @@ -849,32 +918,29 @@ $$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2), $$ -and taking the derivative with respect to \( \beta_0 \) we get +

    and taking the derivative with respect to \( \beta_0 \) we get

    $$ \beta_0=\frac{8}{4+\lambda}, $$ -and for \( \beta_1 \) we obtain +

    and for \( \beta_1 \) we obtain

    $$ \beta_1=\frac{2}{1+\lambda}, $$ -

    -Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving +

    Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving

    $$ \left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1, $$ -which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \). +

    which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).

    -











    -

    Lasso case

    -

    -For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) +

    For Lasso we need now, keeping a constraint on \( \vert\beta_0\vert+\vert\beta_1\vert=1 \), to take the derivative of the absolute values of \( \beta_0 \) and \( \beta_1 \). This gives us the following derivatives of the cost function +

    $$ C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert), $$ @@ -884,64 +950,59 @@ $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_0}=-4(4-2\beta_0)+\lambda\mathrm{sgn}(\beta_0)=0, $$ -and +

    and

    $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}(\beta_1)=0. $$ -We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely - +

    We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely

    1. \( \beta_0 > 0 \) and \( \beta_1 > 0 \),
    2. \( \beta_0 > 0 \) and \( \beta_1 < 0 \),
    3. \( \beta_0 < 0 \) and \( \beta_1 > 0 \),
    4. \( \beta_0 < 0 \) and \( \beta_1 < 0 \).
    -









    -

    The first Case

    -

    -If we consider the first case, we have then +

    If we consider the first case, we have then

    $$ -4(4-2\beta_0)+\lambda=0, $$ -and +

    and

    $$ -2(2-\beta_1)+\lambda=0. $$ -which yields +

    which yields

    $$ \beta_0=\frac{16+\lambda}{8}, $$ -and +

    and

    $$ \beta_1=\frac{4+\lambda}{2}. $$ -

    -Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you. +

    Using the constraint on \( \beta_0 \) and \( \beta_1 \) we can then find the optimal value of \( \lambda \) for the different cases. We leave this as an exercise to you.

    -











    -

    Simple code for solving the above problem

    -

    -Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values. +

    Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of \( \lambda \), meaning that we need to perform a search in order to find the optimal values.

    -

    -First we study and compare the OLS and Ridge results. The next code compares all three methods. +

    First we study and compare the OLS and Ridge results. The next code compares all three methods.

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -990,19 +1051,34 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-We see here that we reach a plateau. What is actually happening?
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    We see here that we reach a plateau. What is actually happening?

    -











    -

    With Lasso Regression

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -1058,16 +1134,33 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Another Example, now with a polynomial fit

    -

    -

    import os
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import os
 import numpy as np
 import pandas as pd
 import matplotlib.pyplot as plt
@@ -1145,28 +1238,44 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Using CVXOPT

    -

    -As a small addendum, we note that you can also solve this problem +

    As a small addendum, we note that you can also solve this problem using the convex optimization package CVXOPT. This requires, in addition to having installed CVXOPT, you need to download the file l1regl.py. The following code example solves the simpler problem we discussed above, where we have added the latter python file. +

    -

    -We list first the needed file. Thereafter we set up the problem to solve. +

    We list first the needed file. Thereafter we set up the problem to solve.

    -

    -

    from cvxopt import matrix, spdiag, mul, div, sqrt, normal, setseed
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from cvxopt import matrix, spdiag, mul, div, sqrt, normal, setseed
 from cvxopt import blas, lapack, solvers, sparse, spmatrix
 import math
 
@@ -1423,52 +1532,80 @@ We list first the needed file. Thereafter we set up the problem to solve.
         return g
 
     return solvers.coneqp(P, q, G, h, kktsolver = Fkkt)['x'][:n]
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    The simpler Example

    -

    -

    from cvxopt import matrix, normal
+
    
+  
    
+    
    
+      
    
+        
    
+  
    from cvxopt import matrix, normal
 
 X = matrix( [ [ 2, 0, 1], [0, 1, 3]])
 y = matrix( [4, 2, 3])
 x = l1regls(X,y)
-
    
-
    
-









    
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +









    Friday September 10

    -

    -

    Linking the regression analysis with a statistical interpretation

    -

    -We will now couple the discussions of ordinary least squares, Ridge +

    We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in. We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters \( \beta \). +

    -

    -The +

    The advantage of doing linear regression is that we actually end up with analytical expressions for several statistical quantities. Standard least squares and Ridge regression allow us to derive quantities like the variance and other expectation values in a rather straightforward way. +

    -

    -It is assumed that \( \varepsilon_i +

    It is assumed that \( \varepsilon_i \sim \mathcal{N}(0, \sigma^2) \) and the \( \varepsilon_{i} \) are independent, i.e.: +

    $$ \begin{align*} \mbox{Cov}(\varepsilon_{i_1}, @@ -1477,45 +1614,42 @@ $$ \end{align*} $$ -The randomness of \( \varepsilon_i \) implies that +

    The randomness of \( \varepsilon_i \) implies that \( \mathbf{y}_i \) is also a random variable. In particular, \( \mathbf{y}_i \) is normally distributed, because \( \varepsilon_i \sim \mathcal{N}(0, \sigma^2) \) and \( \mathbf{X}_{i,\ast} \, \boldsymbol{\beta} \) is a non-random scalar. To specify the parameters of the distribution of -\( \mathbf{y}_i \) we need to calculate its first two moments. +\( \mathbf{y}_i \) we need to calculate its first two moments. +

    -

    -Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The +

    Recall that \( \boldsymbol{X} \) is a matrix of dimensionality \( n\times p \). The notation above \( \mathbf{X}_{i,\ast} \) means that we are looking at the row number \( i \) and perform a sum over all values \( p \). +

    -











    -

    Assumptions made

    -

    -The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) +

    The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off) that there exists a function \( f(\boldsymbol{x}) \) and a normal distributed error \( \boldsymbol{\varepsilon}\sim \mathcal{N}(0, \sigma^2) \) which describe our data +

    $$ \boldsymbol{y} = f(\boldsymbol{x})+\boldsymbol{\varepsilon} $$ -

    -We approximate this function with our model from the solution of the linear regression equations, that is our +

    We approximate this function with our model from the solution of the linear regression equations, that is our function \( f \) is approximated by \( \boldsymbol{\tilde{y}} \) where we want to minimize \( (\boldsymbol{y}-\boldsymbol{\tilde{y}})^2 \), our MSE, with +

    $$ \boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta}. $$ -

    -









    +









    Expectation value and variance

    -

    -We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \) +

    We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)

    $$ \begin{align*} \mathbb{E}(y_i) & = @@ -1524,8 +1658,9 @@ $$ \end{align*} $$ -while +

    while its variance is +

    $$ \begin{align*} \mbox{Var}(y_i) & = \mathbb{E} \{ [y_i - \mathbb{E}(y_i)]^2 \} \, \, \, = \, \, \, \mathbb{E} ( y_i^2 ) - @@ -1541,27 +1676,23 @@ $$ \end{align*} $$ -Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with -mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

    Hence, \( y_i \sim \mathcal{N}( \mathbf{X}_{i, \ast} \, \boldsymbol{\beta}, \sigma^2) \), that is \( \boldsymbol{y} \) follows a normal distribution with +mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (not be confused with the singular values of the SVD). +

    -











    -

    Expectation value and variance for \( \boldsymbol{\beta} \)

    -

    -With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value +

    With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value

    $$ \mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}. $$ -This means that the estimator of the regression parameters is unbiased. +

    This means that the estimator of the regression parameters is unbiased.

    -

    -We can also calculate the variance +

    We can also calculate the variance

    -

    -The variance of \( \boldsymbol{\beta} \) is +

    The variance of \( \boldsymbol{\beta} \) is

    $$ \begin{eqnarray*} \mbox{Var}(\boldsymbol{\beta}) & = & \mathbb{E} \{ [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})] [\boldsymbol{\beta} - \mathbb{E}(\boldsymbol{\beta})]^{T} \} @@ -1585,189 +1716,173 @@ $$ \end{eqnarray*} $$ -

    -where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = +

    where we have used that \( \mathbb{E} (\mathbf{Y} \mathbf{Y}^{T}) = \mathbf{X} \, \boldsymbol{\beta} \, \boldsymbol{\beta}^{T} \, \mathbf{X}^{T} + \sigma^2 \, \mathbf{I}_{nn} \). From \( \mbox{Var}(\boldsymbol{\beta}) = \sigma^2 \, (\mathbf{X}^{T} \mathbf{X})^{-1} \), one obtains an estimate of the variance of the estimate of the \( j \)-th regression coefficient: \( \boldsymbol{\sigma}^2 (\boldsymbol{\beta}_j ) = \boldsymbol{\sigma}^2 [(\mathbf{X}^{T} \mathbf{X})^{-1}]_{jj} \). This may be used to construct a confidence interval for the estimates. +

    -

    -In a similar way, we can obtain analytical expressions for say the +

    In a similar way, we can obtain analytical expressions for say the expectation values of the parameters \( \boldsymbol{\beta} \) and their variance -when we employ Ridge regression, allowing us again to define a confidence interval. +when we employ Ridge regression, allowing us again to define a confidence interval. +

    -

    -It is rather straightforward to show that +

    It is rather straightforward to show that

    $$ \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big]=(\mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I}_{pp})^{-1} (\mathbf{X}^{\top} \mathbf{X})\boldsymbol{\beta}^{\mathrm{OLS}}. $$ -We see clearly that +

    We see clearly that \( \mathbb{E} \big[ \boldsymbol{\beta}^{\mathrm{Ridge}} \big] \not= \boldsymbol{\beta}^{\mathrm{OLS}} \) for any \( \lambda > 0 \). We say then that the ridge estimator is biased. +

    -

    -We can also compute the variance as +

    We can also compute the variance as

    $$ \mbox{Var}[\boldsymbol{\beta}^{\mathrm{Ridge}}]=\sigma^2[ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1} \mathbf{X}^{T} \mathbf{X} \{ [ \mathbf{X}^{\top} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}, $$ -and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero. +

    and it is easy to see that if the parameter \( \lambda \) goes to infinity then the variance of Ridge parameters \( \boldsymbol{\beta} \) goes to zero.

    -

    -With this, we can compute the difference +

    With this, we can compute the difference

    $$ \mbox{Var}[\boldsymbol{\beta}^{\mathrm{OLS}}]-\mbox{Var}(\boldsymbol{\beta}^{\mathrm{Ridge}})=\sigma^2 [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}[ 2\lambda\mathbf{I} + \lambda^2 (\mathbf{X}^{T} \mathbf{X})^{-1} ] \{ [ \mathbf{X}^{T} \mathbf{X} + \lambda \mathbf{I} ]^{-1}\}^{T}. $$ -The difference is non-negative definite since each component of the +

    The difference is non-negative definite since each component of the matrix product is non-negative definite. -This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +This means the variance we obtain with the standard OLS will always for \( \lambda > 0 \) be larger than the variance of \( \boldsymbol{\beta} \) obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. +

    -











    -

    Deriving OLS from a probability distribution

    -

    -Our basic assumption when we derived the OLS equations was to assume +

    Our basic assumption when we derived the OLS equations was to assume that our output is determined by a given continuous function \( f(\boldsymbol{x}) \) and a random noise \( \boldsymbol{\epsilon} \) given by the normal distribution with zero mean value and an undetermined variance \( \sigma^2 \). +

    -

    -We found above that the outputs \( \boldsymbol{y} \) have a mean value given by +

    We found above that the outputs \( \boldsymbol{y} \) have a mean value given by \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \) and variance \( \sigma^2 \). Since the entries to the design matrix are not stochastic variables, we can assume that the probability distribution of our targets is also a normal distribution but now with mean value \( \boldsymbol{X}\hat{\boldsymbol{\beta}} \). This means that a single output \( y_i \) is given by the Gaussian distribution +

    $$ y_i\sim \mathcal{N}(\boldsymbol{X}_{i,*}\boldsymbol{\beta}, \sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -









    +









    Independent and Identically Distrubuted (iid)

    -

    -We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. +

    We assume now that the various \( y_i \) values are stochastically distributed according to the above Gaussian distribution. We define this distribution as +

    $$ p(y_i, \boldsymbol{X}\vert\boldsymbol{\beta})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}, $$ -which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \). +

    which reads as finding the likelihood of an event \( y_i \) with the input variables \( \boldsymbol{X} \) given the parameters (to be determined) \( \boldsymbol{\beta} \).

    -

    -Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have +

    Since these events are assumed to be independent and identicall distributed we can build the probability distribution function (PDF) for all possible event \( \boldsymbol{y} \) as the product of the single events, that is we have

    $$ p(\boldsymbol{y},\boldsymbol{X}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}=\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta}). $$ -

    -We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is +

    We will write this in a more compact form reserving \( \boldsymbol{D} \) for the domain of events, including the ouputs (targets) and the inputs. That is in case we have a simple one-dimensional input and output case +

    $$ \boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})]. $$ -In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). +

    In the more general case the various inputs should be replaced by the possible features represented by the input data set \( \boldsymbol{X} \). We can now rewrite the above probability as +

    $$ p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \). +

    It is a conditional probability (see below) and reads as the likelihood of a domain of events \( \boldsymbol{D} \) given a set of parameters \( \boldsymbol{\beta} \).

    -











    -

    Maximum Likelihood Estimation (MLE)

    -

    -In statistics, maximum likelihood estimation (MLE) is a method of +

    In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statistical model, the observed -data is the most probable. +data is the most probable. +

    -

    -We will assume here that our events are given by the above Gaussian +

    We will assume here that our events are given by the above Gaussian distribution and we will determine the optimal parameters \( \beta \) by maximizing the above PDF. However, computing the derivatives of a product function is cumbersome and can easily lead to overflow and/or underflowproblems, with potentials for loss of numerical precision. +

    -

    -In practice, it is more convenient to maximize the logarithm of the +

    In practice, it is more convenient to maximize the logarithm of the PDF because it is a monotonically increasing function of the argument. Alternatively, and this will be our option, we will minimize the negative of the logarithm since this is a monotonically decreasing function. +

    -

    -Note also that maximization/minimization of the logarithm of the PDF +

    Note also that maximization/minimization of the logarithm of the PDF is equivalent to the maximization/minimization of the function itself. +

    -











    -

    A new Cost Function

    -

    -We could now define a new cost function to minimize, namely the negative logarithm of the above PDF +

    We could now define a new cost function to minimize, namely the negative logarithm of the above PDF

    $$ C(\boldsymbol{\beta}=-\log{\prod_{i=0}^{n-1}p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}=-\sum_{i=0}^{n-1}\log{p(y_i,\boldsymbol{X}\vert\boldsymbol{\beta})}, $$ -which becomes +

    which becomes

    $$ C(\boldsymbol{\beta}=\frac{n}{2}\log{2\pi\sigma^2}+\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}. $$ -

    -Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely +

    Taking the derivative of the new cost function with respect to the parameters \( \beta \) we recognize our familiar OLS equation, namely

    $$ \boldsymbol{X}^T\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right) =0, $$ -which leads to the well-known OLS equation for the optimal paramters \( \beta \) +

    which leads to the well-known OLS equation for the optimal paramters \( \beta \)

    $$ \hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}! $$ -

    -Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics. +

    Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics.

    -











    -

    More basic Statistics and Bayes' theorem

    -

    -A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. +

    A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry. Bayes' theorem is extremely simple to derive. But to do so we need some basic axioms from statistics. +

    -

    -Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \). +

    Assume we have two domains of events \( X=[x_0,x_1,\dots,x_{n-1}] \) and \( Y=[y_0,y_1,\dots,y_{n-1}] \).

    -

    -We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. -The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

    We define also the likelihood for \( X \) and \( Y \) as \( p(X) \) and \( p(Y) \) respectively. +The likelihood of a specific event \( x_i \) (or \( y_i \)) is then written as \( p(X=x_i) \) or just \( p(x_i)=p_i \). +

    -

    Union of events is given by

    @@ -1777,7 +1892,6 @@ $$

    -

    The product rule (aka joint probability) is given by

    @@ -1785,20 +1899,16 @@ $$ p(X \cup Y)= p(X,Y)= p(X\vert Y)p(Y)=p(Y\vert X)p(X), $$ -where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \). +

    where we read \( p(X\vert Y) \) as the likelihood of obtaining \( X \) given \( Y \).

    -

    -If we have independent events then \( p(X,Y)=p(X)p(Y) \). +

    If we have independent events then \( p(X,Y)=p(X)p(Y) \).

    -











    -

    Marginal Probability

    -

    -The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have +

    The marginal probability is defined in terms of only one of the set of variables \( X,Y \). For a discrete probability we have

    @@ -1808,13 +1918,10 @@ $$

    -











    -

    Conditional Probability

    -

    -The conditional probability, if \( p(Y) > 0 \), is +

    The conditional probability, if \( p(Y) > 0 \), is

    @@ -1824,130 +1931,113 @@ $$

    -











    -

    Bayes' Theorem

    -

    -If we combine the conditional probability with the marginal probability and the standard product rule, we have +

    If we combine the conditional probability with the marginal probability and the standard product rule, we have

    $$ p(X\vert Y)= \frac{p(X,Y)}{p(Y)}, $$ -which we can rewrite as +

    which we can rewrite as

    $$ p(X\vert Y)= \frac{p(X,Y)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}=\frac{p(Y\vert X)p(X)}{\sum_{i=0}^{n-1}p(Y\vert X=x_i)p(x_i)}, $$ -which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \). +

    which is Bayes' theorem. It allows us to evaluate the uncertainty in in \( X \) after we have observed \( Y \). We can easily interchange \( X \) with \( Y \).

    -











    -

    Interpretations of Bayes' Theorem

    -

    -The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is +

    The quantity \( p(Y\vert X) \) on the right-hand side of the theorem is evaluated for the observed data \( Y \) and can be viewed as a function of the parameter space represented by \( X \). This function is not necesseraly normalized and is normally called the likelihood function. +

    -

    -The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution. +

    The function \( p(X) \) on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.

    -

    -Let us try to illustrate Bayes' theorem through an example. +

    Let us try to illustrate Bayes' theorem through an example.

    -











    -

    Example of Usage of Bayes' theorem

    -

    -Let us suppose that you are undergoing a series of mammography scans in +

    Let us suppose that you are undergoing a series of mammography scans in order to rule out possible breast cancer cases. We define the sensitivity for a positive event by the variable \( X \). It takes binary values with \( X=1 \) representing a positive event and \( X=0 \) being a negative event. We reserve \( Y \) as a classification parameter for either a negative or a positive breast cancer confirmation. (Short note on wordings: positive here means having breast cancer, although none of us would consider this being a positive thing). +

    -

    -We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not. +

    We let \( Y=1 \) represent the the case of having breast cancer and \( Y=0 \) as not.

    -

    -Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have +

    Let us assume that if you have breast cancer, the test will be positive with a probability of \( 0.8 \), that is we have

    $$ p(X=1\vert Y=1) =0.8. $$ -

    -This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. +

    This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of \( 80\% \) for having cancer. It is however not correct, as the following Bayesian analysis shows. +

    -











    -

    Doing it correctly

    -

    -If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. +

    If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number. Let us assume that the prior probability in the population as a whole is +

    $$ p(Y=1) =0.004. $$ -

    -We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have +

    We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have

    $$ p(X=1\vert Y=0) =0.1. $$ -

    -Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute +

    Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute

    $$ p(Y=1\vert X=1)=\frac{p(X=1\vert Y=1)p(Y=1)}{p(X=1\vert Y=1)p(Y=1)+p(X=1\vert Y=0)p(Y=0)}=\frac{0.8\times 0.004}{0.8\times 0.004+0.1\times 0.996}=0.031. $$ -That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer! +

    That is, in case of a positive test, there is only a \( 3\% \) chance of having breast cancer!

    -











    -

    Bayes' Theorem and Ridge and Lasso Regression

    -

    -Hitherto we have discussed Ridge and Lasso regression in terms of a +

    Hitherto we have discussed Ridge and Lasso regression in terms of a linear analysis. This may to many of you feel rather technical and perhaps not that intuitive. The question is whether we can develop a more intuitive way of understanding what Ridge and Lasso express. +

    -

    -Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit. +

    Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit.

    -











    -

    Test Function for what happens with OLS, Ridge and Lasso

    -

    -We will play around with a study of the values for the optimal +

    We will play around with a study of the values for the optimal parameters \( \boldsymbol{\beta} \) using OLS, Ridge and Lasso regression. For OLS, you will notice as function of the noise and polynomial degree, that the parameters \( \beta \) will fluctuate from order to order in the polynomial fit and that for larger and larger polynomial degrees of freedom, the parameters will tend to increase in value for OLS. +

    -

    -For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one. +

    For Ridge and Lasso regression, the higher order parameters will typically be reduced, providing thereby less fluctuations from one order to another one.

    -

    -

    import numpy as np
+
    
+  
    
+    
    
+      
    
+        
    
+  
    import numpy as np
 import matplotlib.pyplot as plt
 from sklearn.model_selection import train_test_split
 from sklearn import linear_model
@@ -2010,130 +2100,122 @@ plt.xlabel('
 plt.ylabel('MSE')
 plt.legend()
 plt.show()
-
    
-
    
-How can we understand this?
+
    +
    + + + +
    +
    +
    +
    +
    +
    +
    +
    + + +

    How can we understand this?

    -











    -

    Invoking Bayes' theorem

    -

    -Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression. +

    Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression.

    -

    -For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case) +

    For ordinary least squares we postulated that the maximum likelihood for the doamin of events \( \boldsymbol{D} \) (one-dimensional case)

    $$ \boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\dots, (x_{n-1},y_{n-1})], $$ -is given by +

    is given by

    $$ p(\boldsymbol{D}\vert\boldsymbol{\beta})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}. $$ -

    -In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability +

    In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set \( \boldsymbol{\beta} \) given a domain of events \( \boldsymbol{D} \)? That is, how can we define the posterior probability

    $$ p(\boldsymbol{\beta}\vert\boldsymbol{D}). $$ -

    -Bayes' theorem comes to our rescue here since (omitting the normalization constant) +

    Bayes' theorem comes to our rescue here since (omitting the normalization constant)

    $$ p(\boldsymbol{\beta}\vert\boldsymbol{D})\propto p(\boldsymbol{D}\vert\boldsymbol{\beta})p(\boldsymbol{\beta}). $$ -

    -We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)! +

    We have a model for \( p(\boldsymbol{D}\vert\boldsymbol{\beta}) \) but need one for the prior \( p(\boldsymbol{\beta} \)!

    -











    -

    Ridge and Bayes

    -

    -With the posterior probability defined by a likelihood which we have +

    With the posterior probability defined by a likelihood which we have already modeled and an unknown prior, we are now ready to make additional models for the prior. +

    -

    -We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is +

    We can, based on our discussions of the variance of \( \boldsymbol{\beta} \) and the mean value, assume that the prior for the values \( \boldsymbol{\beta} \) is given by a Gaussian with mean value zero and variance \( \tau^2 \), that is

    $$ p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. $$ -

    -Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

    Our posterior probability becomes then (omitting the normalization factor which is just a constant)

    $$ p(\boldsymbol{\beta\vert\boldsymbol{D})}=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\beta_j^2}{2\tau^2}\right)}. $$ -

    -We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we +

    We can now optimize this quantity with respect to \( \boldsymbol{\beta} \). As we did for OLS, this is most conveniently done by taking the negative logarithm of the posterior probability. Doing so and leaving out the constants terms that do not depend on \( \beta \), we have +

    $$ C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{2\tau^2}\vert\vert\boldsymbol{\beta}\vert\vert_2^2, $$ -and replacing \( 1/2\tau^2 \) with \( \lambda \) we have +

    and replacing \( 1/2\tau^2 \) with \( \lambda \) we have

    $$ C(\boldsymbol{\beta})=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_2^2, $$ -which is our Ridge cost function! Nice, isn't it? +

    which is our Ridge cost function! Nice, isn't it?

    -











    -

    Lasso and Bayes

    -

    -To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is +

    To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution (Laplace in this case) with zero mean value, that is

    $$ p(\boldsymbol{\beta})=\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. $$ -

    -Our posterior probability becomes then (omitting the normalization factor which is just a constant) +

    Our posterior probability becomes then (omitting the normalization factor which is just a constant)

    $$ p(\boldsymbol{\beta}\vert\boldsymbol{D})=\prod_{i=0}^{n-1}\frac{1}{\sqrt{2\pi\sigma^2}}\exp{\left[-\frac{(y_i-\boldsymbol{X}_{i,*}\boldsymbol{\beta})^2}{2\sigma^2}\right]}\prod_{j=0}^{p-1}\exp{\left(-\frac{\vert\beta_j\vert}{\tau}\right)}. $$ -

    -Taking the negative +

    Taking the negative logarithm of the posterior probability and leaving out the constants terms that do not depend on \( \beta \), we have +

    $$ C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\frac{1}{\tau}\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -and replacing \( 1/\tau \) with \( \lambda \) we have +

    and replacing \( 1/\tau \) with \( \lambda \) we have

    $$ C(\boldsymbol{\beta}=\frac{\vert\vert (\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\vert\vert_2^2}{2\sigma^2}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1, $$ -which is our Lasso cost function! - +

    which is our Lasso cost function!

    - -
    © 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
    - - - diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz index c22e35956..f4b2d4084 100644 Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb index 7812628d6..5f5aee18c 100644 --- a/doc/pub/week36/ipynb/week36.ipynb +++ b/doc/pub/week36/ipynb/week36.ipynb @@ -2,21 +2,38 @@ "cells": [ { "cell_type": "markdown", - "metadata": {}, + "id": "2c14ba1f", + "metadata": { + "editable": true + }, + "source": [ + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "c1a05e8b", + "metadata": { + "editable": true + }, "source": [ - "\n", "# Week 36: Statistical interpretation of Linear Regression and Resampling techniques\n", - "\n", - " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 16, 2021**\n", - "\n", - "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", - "\n", - "\n", - "\n", + "Date: **Nov 3, 2021**\n", "\n", + "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" + ] + }, + { + "cell_type": "markdown", + "id": "d4122722", + "metadata": { + "editable": true + }, + "source": [ "## Plans for week 36\n", "\n", "* Thursday: Summary from last week on SVD, Statistics, probability theory and linear regression. [Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember9.mp4?vrtx=view-as-webpage).\n", @@ -28,13 +45,36 @@ "\n", "2. Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1\n", "\n", - "3. Hastie et al chapter 3\n", - "\n", - "## Thursday September 9\n", - "\n", - "\n", - "## Summary from last Week and Examples\n", - "\n", + "3. Hastie et al chapter 3" + ] + }, + { + "cell_type": "markdown", + "id": "a62e4521", + "metadata": { + "editable": true + }, + "source": [ + "## Thursday September 9" + ] + }, + { + "cell_type": "markdown", + "id": "7136a1eb", + "metadata": { + "editable": true + }, + "source": [ + "## Summary from last Week and Examples" + ] + }, + { + "cell_type": "markdown", + "id": "cf306c42", + "metadata": { + "editable": true + }, + "source": [ "## Linear Regression and the SVD\n", "\n", "We used the SVD to analyse the matrix to invert in ordinary lineat regression" @@ -42,7 +82,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "22d97953", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -51,14 +94,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "45d6f030", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "e6f24acd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n", @@ -67,14 +116,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0851aef0", + "metadata": { + "editable": true + }, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "186d0a05", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -88,14 +143,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "77441584", + "metadata": { + "editable": true + }, "source": [ "meaning we can write" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a77bf8bd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -104,14 +165,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2df69db6", + "metadata": { + "editable": true + }, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b076a29d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -120,7 +187,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b72d31e4", + "metadata": { + "editable": true + }, "source": [ "## What does it mean?\n", "\n", @@ -131,7 +201,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "39d0f309", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -140,7 +213,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ebecab45", + "metadata": { + "editable": true + }, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -150,7 +226,6 @@ "$\\boldsymbol{v}_i$ are hierarchically ordered by how much correlation they\n", "encode from the columns of $\\boldsymbol{X}$. \n", "\n", - "\n", "Note that these are also the eigenvectors and eigenvalues of the\n", "Hessian matrix.\n", "\n", @@ -160,7 +235,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "84ecb76e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -169,15 +247,26 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9c80f296", + "metadata": { + "editable": true + }, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", "matrix. Every singular value of $\\boldsymbol{X}$ is thus a positive square\n", "root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. If the matrix $\\boldsymbol{X}$ is\n", "self-adjoint, the singular values of $\\boldsymbol{X}$ are equal to the\n", - "absolute value of the eigenvalues of $\\boldsymbol{X}$.\n", - "\n", + "absolute value of the eigenvalues of $\\boldsymbol{X}$." + ] + }, + { + "cell_type": "markdown", + "id": "fdf15b7a", + "metadata": { + "editable": true + }, + "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", "For $\\boldsymbol{X}\\boldsymbol{X}^T$ we found" @@ -185,7 +274,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "cae5fea5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -194,14 +286,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f67c8a37", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "6854b96f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -210,14 +308,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3e71d9df", + "metadata": { + "editable": true + }, "source": [ "leading to" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1b9e441f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -226,14 +330,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "71a1f65e", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a7070951", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -242,7 +352,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ad965bf9", + "metadata": { + "editable": true + }, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -251,10 +364,16 @@ "\n", "Since we will mainly be interested in the correlations among the features\n", "of our data (the columns of $\\boldsymbol{X}$, the quantity of interest for us are the non-zero singular\n", - "values and the column vectors of $\\boldsymbol{V}$.\n", - "\n", - "\n", - "\n", + "values and the column vectors of $\\boldsymbol{V}$." + ] + }, + { + "cell_type": "markdown", + "id": "7eebb478", + "metadata": { + "editable": true + }, + "source": [ "## Code for SVD and Inversion of Matrices\n", "\n", "How do we use the SVD to invert a matrix $\\boldsymbol{X}^\\boldsymbol{X}$ which is singular or near singular?\n", @@ -263,7 +382,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 1, + "id": "d3599775", "metadata": { "collapsed": false, "editable": true @@ -275,14 +395,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1c1668ea", + "metadata": { + "editable": true + }, "source": [ "Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, + "id": "29014fa6", "metadata": { "collapsed": false, "editable": true @@ -322,7 +446,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e272a9c5", + "metadata": { + "editable": true + }, "source": [ "## Inverse of Rectangular Matrix\n", "\n", @@ -339,7 +466,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2310328a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n", @@ -348,14 +478,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "69d55316", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{\\Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 3, + "id": "ac570461", "metadata": { "collapsed": false, "editable": true @@ -389,12 +523,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9520678e", + "metadata": { + "editable": true + }, + "source": [ + "As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by **Numpy**." + ] + }, + { + "cell_type": "markdown", + "id": "05c8d474", + "metadata": { + "editable": true + }, "source": [ - "As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by **Numpy**.\n", - "\n", - "\n", - "\n", "## Ridge and LASSO Regression\n", "\n", "Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n", @@ -403,7 +546,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2e9c73ab", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -412,14 +558,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c59de7b1", + "metadata": { + "editable": true + }, "source": [ "or we can state it as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "f3449379", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -429,14 +581,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "81954b3e", + "metadata": { + "editable": true + }, "source": [ "where we have used the definition of a norm-2 vector, that is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "7f8177b4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -445,7 +603,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "14fd3461", + "metadata": { + "editable": true + }, "source": [ "## From OLS to Ridge and Lasso\n", "\n", @@ -457,7 +618,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bc470979", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -467,7 +631,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8ed8ff40", + "metadata": { + "editable": true + }, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -476,7 +643,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fc9b353b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -485,14 +655,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "69c5d6be", + "metadata": { + "editable": true + }, "source": [ "we have a new optimization equation" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "698ef684", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -502,7 +678,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0aa04d21", + "metadata": { + "editable": true + }, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -511,7 +690,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b0a573cc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -520,7 +702,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b5341503", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -529,7 +714,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a9ecc1c6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -538,7 +726,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f701b23a", + "metadata": { + "editable": true + }, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -549,7 +740,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "baa0d96e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -558,14 +752,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f619989a", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "334e15a0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -574,16 +774,65 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "524c3df1", + "metadata": { + "editable": true + }, "source": [ - "with $t$ a finite positive number. \n", + "with $t$ a finite positive number." + ] + }, + { + "cell_type": "markdown", + "id": "c502d538", + "metadata": { + "editable": true + }, + "source": [ + "## Note on Scikit-Learn\n", "\n", + "Note well that a library like **Scikit-Learn** does not include the $1/n$ factor in the expression for the mean-squared error. If you include it, the optimal parameter $\\beta$ becomes" + ] + }, + { + "cell_type": "markdown", + "id": "8409b684", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "450166ae", + "metadata": { + "editable": true + }, + "source": [ + "In our codes where we compare our own codes with **Scikit-Learn**, we do thus not include the $1/n$ factor in the cost function." + ] + }, + { + "cell_type": "markdown", + "id": "64882702", + "metadata": { + "editable": true + }, + "source": [ + "## Comparison with OLS\n", "When we compare this with the ordinary least squares result we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "20e8fef3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -592,17 +841,27 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c2f1945b", + "metadata": { + "editable": true + }, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", - "\n", "We see that Ridge regression is nothing but the standard OLS with a\n", "modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n", "particular for our discussion of the bias-variance tradeoff are rather\n", "interesting. We will see that for specific values of $\\lambda$, we may\n", - "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n", - "\n", + "even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here." + ] + }, + { + "cell_type": "markdown", + "id": "5adddee9", + "metadata": { + "editable": true + }, + "source": [ "## SVD analysis\n", "\n", "Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n", @@ -611,7 +870,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e19e150c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -620,14 +882,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "eb2ed2ce", + "metadata": { + "editable": true + }, "source": [ "For Ridge regression this becomes" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "303da425", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -636,10 +904,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e6f5e319", + "metadata": { + "editable": true + }, + "source": [ + "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." + ] + }, + { + "cell_type": "markdown", + "id": "8af91d44", + "metadata": { + "editable": true + }, "source": [ - "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$. \n", - "\n", "## Interpreting the Ridge results\n", "\n", "Since $\\lambda \\geq 0$, it means that compared to OLS, we have" @@ -647,7 +926,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "16e579b0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -656,7 +938,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c932fcca", + "metadata": { + "editable": true + }, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -664,9 +949,16 @@ "eigenvalues ordered in a descending way, that is $\\sigma_i \\geq\n", "\\sigma_{i+1}$.\n", "\n", - "For small eigenvalues $\\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods.\n", - "\n", - "\n", + "For small eigenvalues $\\sigma_i$ it means that their contributions become less important, a fact which can be used to reduce the number of degrees of freedom. More about this when we have covered the material on a statistical interpretation of various linear regression methods." + ] + }, + { + "cell_type": "markdown", + "id": "720d6a8b", + "metadata": { + "editable": true + }, + "source": [ "## More interpretations\n", "\n", "For the sake of simplicity, let us assume that the design matrix is orthonormal, that is" @@ -674,7 +966,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2c85c574", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -683,14 +978,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a398302e", + "metadata": { + "editable": true + }, "source": [ "In this case the standard OLS results in" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8370d84e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\boldsymbol{y},\n", @@ -699,14 +1000,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b64bda86", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "af74a8ca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -715,7 +1022,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "27ba5748", + "metadata": { + "editable": true + }, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -724,8 +1034,16 @@ "We will come back to more interpreations after we have gone through some of the statistical analysis part. \n", "\n", "For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n", - "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n", - "\n", + "Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended." + ] + }, + { + "cell_type": "markdown", + "id": "dc3abdfb", + "metadata": { + "editable": true + }, + "source": [ "## Deriving the Lasso Regression Equations\n", "\n", "Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard mean squared error equation, we have the following **cost** function" @@ -733,7 +1051,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9c81f0ce", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -742,14 +1063,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0e036305", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b270a5c6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\ 0 & \\beta =0\\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -758,14 +1085,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e0db858a", + "metadata": { + "editable": true + }, "source": [ "we have that the derivative of the cost function is" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "90eff3d7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -774,14 +1107,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "66765f18", + "metadata": { + "editable": true + }, "source": [ "and reordering we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "99894702", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -790,13 +1129,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e73e9adb", + "metadata": { + "editable": true + }, + "source": [ + "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." + ] + }, + { + "cell_type": "markdown", + "id": "c4449057", + "metadata": { + "editable": true + }, "source": [ - "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n", - "\n", - "\n", - "\n", - "\n", "## Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression\n", "\n", "Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the\n", @@ -807,7 +1154,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b7936fd4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n", @@ -816,14 +1166,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bd77d3bc", + "metadata": { + "editable": true + }, "source": [ "and minimizing we have that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d70b808e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n", @@ -832,7 +1188,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7192d7f1", + "metadata": { + "editable": true + }, "source": [ "## Ridge Regression\n", "\n", @@ -841,7 +1200,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "506f661c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", @@ -850,14 +1212,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "98e0a4ee", + "metadata": { + "editable": true + }, "source": [ "and minimizing we have that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "da993d06", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", @@ -866,7 +1234,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8b622b33", + "metadata": { + "editable": true + }, "source": [ "## Lasso Regression\n", "\n", @@ -875,7 +1246,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ba7130be", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", @@ -884,14 +1258,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2d8881c1", + "metadata": { + "editable": true + }, "source": [ "and minimizing we have that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "d62dc0c4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", @@ -900,14 +1280,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5c2cae24", + "metadata": { + "editable": true + }, "source": [ "which leads to" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "09ac3e95", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", @@ -918,12 +1304,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fcbca200", + "metadata": { + "editable": true + }, + "source": [ + "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$." + ] + }, + { + "cell_type": "markdown", + "id": "e24a172d", + "metadata": { + "editable": true + }, "source": [ - "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n", - "\n", - "\n", - "\n", "## Yet another Example\n", "\n", "Let us assume we have a data set with outputs/targets given by the vector" @@ -931,7 +1326,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0f123269", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}=\\begin{bmatrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n", @@ -940,14 +1338,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "088005aa", + "metadata": { + "editable": true + }, "source": [ "and our inputs as a $3\\times 2$ design matrix" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3066fb17", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n", @@ -956,10 +1360,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "aa5fe8ee", + "metadata": { + "editable": true + }, + "source": [ + "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression." + ] + }, + { + "cell_type": "markdown", + "id": "22db2181", + "metadata": { + "editable": true + }, "source": [ - "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.\n", - "\n", "## The OLS case\n", "\n", "For ordinary least squares (OLS) we know that the optimal solution is" @@ -967,7 +1382,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9a94f9d5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -976,14 +1394,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3799b7b4", + "metadata": { + "editable": true + }, "source": [ "Inserting the above values we obtain that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "242f2a8f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", @@ -992,10 +1416,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "afe4f30f", + "metadata": { + "editable": true + }, + "source": [ + "The code which implements this simpler case is presented after the discussion of Ridge and Lasso." + ] + }, + { + "cell_type": "markdown", + "id": "36924c32", + "metadata": { + "editable": true + }, "source": [ - "The code which implements this simpler case is presented after the discussion of Ridge and Lasso.\n", - "\n", "## The Ridge case\n", "\n", "For Ridge regression we have" @@ -1003,7 +1438,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "41128d27", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1012,14 +1450,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a5125e47", + "metadata": { + "editable": true + }, "source": [ "Inserting the above values we obtain that" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "96ccaa85", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", @@ -1028,14 +1472,24 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "941a8d10", + "metadata": { + "editable": true + }, "source": [ "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", "\n", - "To see this, let us write the cost function for Ridge regression. \n", - "\n", - "\n", + "To see this, let us write the cost function for Ridge regression." + ] + }, + { + "cell_type": "markdown", + "id": "6a2ec4a2", + "metadata": { + "editable": true + }, + "source": [ "## Writing the Cost Function\n", "\n", "We define the MSE without the $1/n$ factor and have then, using that" @@ -1043,7 +1497,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "016c54e6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", @@ -1052,7 +1509,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e18dc717", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", @@ -1061,14 +1521,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "93f15fe7", + "metadata": { + "editable": true + }, "source": [ "and taking the derivative with respect to $\\beta_0$ we get" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "1255c0f3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0=\\frac{8}{4+\\lambda},\n", @@ -1077,14 +1543,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "521da512", + "metadata": { + "editable": true + }, "source": [ "and for $\\beta_1$ we obtain" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4921d555", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_1=\\frac{2}{1+\\lambda},\n", @@ -1093,14 +1565,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "2c96fd22", + "metadata": { + "editable": true + }, "source": [ "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "270ae9fd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n", @@ -1109,10 +1587,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6499fe87", + "metadata": { + "editable": true + }, + "source": [ + "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$." + ] + }, + { + "cell_type": "markdown", + "id": "ebf882c1", + "metadata": { + "editable": true + }, "source": [ - "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$.\n", - "\n", "## Lasso case\n", "\n", "For Lasso we need now, keeping a constraint on $\\vert\\beta_0\\vert+\\vert\\beta_1\\vert=1$, to take the derivative of the absolute values of $\\beta_0$\n", @@ -1121,7 +1610,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8a9287c9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", @@ -1130,7 +1622,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "79d660ca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n", @@ -1139,14 +1634,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "edbf849f", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "c5d99ca8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n", @@ -1155,7 +1656,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e8f2e204", + "metadata": { + "editable": true + }, "source": [ "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", @@ -1164,8 +1668,16 @@ "\n", "3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n", "\n", - "4. $\\beta_0 < 0$ and $\\beta_1 < 0$.\n", - "\n", + "4. $\\beta_0 < 0$ and $\\beta_1 < 0$." + ] + }, + { + "cell_type": "markdown", + "id": "70a8d049", + "metadata": { + "editable": true + }, + "source": [ "## The first Case\n", "\n", "If we consider the first case, we have then" @@ -1173,7 +1685,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "eda6b2a8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-4(4-2\\beta_0)+\\lambda=0,\n", @@ -1182,14 +1697,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8b525674", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "0640d88e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-2(2-\\beta_1)+\\lambda=0.\n", @@ -1198,14 +1719,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "61d29ee4", + "metadata": { + "editable": true + }, "source": [ "which yields" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "5fcfaafa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0=\\frac{16+\\lambda}{8},\n", @@ -1214,14 +1741,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bcda6dd3", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "da129004", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_1=\\frac{4+\\lambda}{2}.\n", @@ -1230,10 +1763,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6741f6f6", + "metadata": { + "editable": true + }, + "source": [ + "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you." + ] + }, + { + "cell_type": "markdown", + "id": "5d23952b", + "metadata": { + "editable": true + }, "source": [ - "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you.\n", - "\n", "## Simple code for solving the above problem\n", "\n", "Here we set up the OLS, Ridge and Lasso functionality in order to study the above example. Note that here we have opted for a set of values of $\\lambda$, meaning that we need to perform a search in order to find the optimal values.\n", @@ -1243,7 +1787,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 4, + "id": "407caa3b", "metadata": { "collapsed": false, "editable": true @@ -1305,17 +1850,28 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "24e13cf6", + "metadata": { + "editable": true + }, + "source": [ + "We see here that we reach a plateau. What is actually happening?" + ] + }, + { + "cell_type": "markdown", + "id": "881225d9", + "metadata": { + "editable": true + }, "source": [ - "We see here that we reach a plateau. What is actually happening?\n", - "\n", - "\n", "## With Lasso Regression" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, + "id": "fd163937", "metadata": { "collapsed": false, "editable": true @@ -1382,14 +1938,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ce1e0e32", + "metadata": { + "editable": true + }, "source": [ "## Another Example, now with a polynomial fit" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, + "id": "1174a295", "metadata": { "collapsed": false, "editable": true @@ -1478,11 +2038,13 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e658dc18", + "metadata": { + "editable": true + }, "source": [ "## Using CVXOPT\n", "\n", - "\n", "As a small addendum, we note that you can also solve this problem\n", "using the convex optimization package\n", "[CVXOPT](https://cvxopt.org/examples/mlbook/l1regls.html). This\n", @@ -1496,7 +2058,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, + "id": "1d474e90", "metadata": { "collapsed": false, "editable": true @@ -1764,14 +2327,18 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3bb8f2e9", + "metadata": { + "editable": true + }, "source": [ "## The simpler Example" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 8, + "id": "76a8fe1c", "metadata": { "collapsed": false, "editable": true @@ -1787,13 +2354,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "4a979be1", + "metadata": { + "editable": true + }, + "source": [ + "## Friday September 10" + ] + }, + { + "cell_type": "markdown", + "id": "4b6d1de0", + "metadata": { + "editable": true + }, "source": [ - "## Friday September 10\n", - "\n", - "\n", - "\n", - "\n", "## Linking the regression analysis with a statistical interpretation\n", "\n", "We will now couple the discussions of ordinary least squares, Ridge\n", @@ -1804,7 +2379,6 @@ "parameter can reduce considerably the variance of the parameters\n", "$\\beta$.\n", "\n", - "\n", "The\n", "advantage of doing linear regression is that we actually end up with\n", "analytical expressions for several statistical quantities. \n", @@ -1812,7 +2386,6 @@ "derive quantities like the variance and other expectation values in a\n", "rather straightforward way.\n", "\n", - "\n", "It is assumed that $\\varepsilon_i\n", "\\sim \\mathcal{N}(0, \\sigma^2)$ and the $\\varepsilon_{i}$ are\n", "independent, i.e.:" @@ -1820,7 +2393,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "987ba47a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \n", @@ -1833,7 +2409,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "59de689b", + "metadata": { + "editable": true + }, "source": [ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", @@ -1844,9 +2423,16 @@ "\n", "Recall that $\\boldsymbol{X}$ is a matrix of dimensionality $n\\times p$. The\n", "notation above $\\mathbf{X}_{i,\\ast}$ means that we are looking at the\n", - "row number $i$ and perform a sum over all values $p$.\n", - "\n", - "\n", + "row number $i$ and perform a sum over all values $p$." + ] + }, + { + "cell_type": "markdown", + "id": "0afb50c8", + "metadata": { + "editable": true + }, + "source": [ "## Assumptions made\n", "\n", "The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)\n", @@ -1856,7 +2442,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6f4d98b4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -1865,7 +2454,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d7b9c66a", + "metadata": { + "editable": true + }, "source": [ "We approximate this function with our model from the solution of the linear regression equations, that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with" @@ -1873,7 +2465,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1b998fe6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -1882,7 +2477,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "80db870c", + "metadata": { + "editable": true + }, "source": [ "## Expectation value and variance\n", "\n", @@ -1891,7 +2489,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "644b9f4d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \n", @@ -1904,7 +2505,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9aee9651", + "metadata": { + "editable": true + }, "source": [ "while\n", "its variance is" @@ -1912,7 +2516,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b4b31e3f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", @@ -1932,11 +2539,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "48603d92", + "metadata": { + "editable": true + }, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", - "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD). \n", - "\n", + "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)." + ] + }, + { + "cell_type": "markdown", + "id": "81bceae1", + "metadata": { + "editable": true + }, + "source": [ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n", "\n", "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ we can evaluate the expectation value" @@ -1944,7 +2562,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8d4a452e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", @@ -1953,7 +2574,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3b3f638f", + "metadata": { + "editable": true + }, "source": [ "This means that the estimator of the regression parameters is unbiased.\n", "\n", @@ -1964,7 +2588,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1061397e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -1992,7 +2619,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "df637188", + "metadata": { + "editable": true + }, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", @@ -2002,7 +2632,6 @@ "$\\boldsymbol{\\sigma}^2 (\\boldsymbol{\\beta}_j ) = \\boldsymbol{\\sigma}^2 [(\\mathbf{X}^{T} \\mathbf{X})^{-1}]_{jj} $. This may be used to\n", "construct a confidence interval for the estimates.\n", "\n", - "\n", "In a similar way, we can obtain analytical expressions for say the\n", "expectation values of the parameters $\\boldsymbol{\\beta}$ and their variance\n", "when we employ Ridge regression, allowing us again to define a confidence interval. \n", @@ -2012,7 +2641,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ab5d9989", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", @@ -2021,7 +2653,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "bf196170", + "metadata": { + "editable": true + }, "source": [ "We see clearly that \n", "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", @@ -2031,7 +2666,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e740ee91", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", @@ -2040,7 +2678,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6e423029", + "metadata": { + "editable": true + }, "source": [ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", "\n", @@ -2049,7 +2690,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5633fa66", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", @@ -2058,13 +2702,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "75df8680", + "metadata": { + "editable": true + }, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", - "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n", - "\n", - "\n", + "This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below." + ] + }, + { + "cell_type": "markdown", + "id": "9a4ab6d8", + "metadata": { + "editable": true + }, + "source": [ "## Deriving OLS from a probability distribution\n", "\n", "Our basic assumption when we derived the OLS equations was to assume\n", @@ -2083,7 +2737,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "68ed135a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -2092,7 +2749,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c9791339", + "metadata": { + "editable": true + }, "source": [ "## Independent and Identically Distrubuted (iid)\n", "\n", @@ -2102,7 +2762,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "66080e73", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", @@ -2111,7 +2774,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "85b4c34a", + "metadata": { + "editable": true + }, "source": [ "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", "\n", @@ -2120,7 +2786,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f73619b6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", @@ -2129,7 +2798,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "46c218db", + "metadata": { + "editable": true + }, "source": [ "We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n", "in case we have a simple one-dimensional input and output case" @@ -2137,7 +2809,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7e3adbf9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n", @@ -2146,7 +2821,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "40421417", + "metadata": { + "editable": true + }, "source": [ "In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n", "We can now rewrite the above probability as" @@ -2154,7 +2832,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b518f223", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -2163,10 +2844,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "0e18f690", + "metadata": { + "editable": true + }, + "source": [ + "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$." + ] + }, + { + "cell_type": "markdown", + "id": "f2ccec74", + "metadata": { + "editable": true + }, "source": [ - "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$.\n", - "\n", "## Maximum Likelihood Estimation (MLE)\n", "\n", "In statistics, maximum likelihood estimation (MLE) is a method of\n", @@ -2175,14 +2867,12 @@ "function so that, under the assumed statistical model, the observed\n", "data is the most probable. \n", "\n", - "\n", "We will assume here that our events are given by the above Gaussian\n", "distribution and we will determine the optimal parameters $\\beta$ by\n", "maximizing the above PDF. However, computing the derivatives of a\n", "product function is cumbersome and can easily lead to overflow and/or\n", "underflowproblems, with potentials for loss of numerical precision.\n", "\n", - "\n", "In practice, it is more convenient to maximize the logarithm of the\n", "PDF because it is a monotonically increasing function of the argument.\n", "Alternatively, and this will be our option, we will minimize the\n", @@ -2190,10 +2880,16 @@ "function.\n", "\n", "Note also that maximization/minimization of the logarithm of the PDF\n", - "is equivalent to the maximization/minimization of the function itself.\n", - "\n", - "\n", - "\n", + "is equivalent to the maximization/minimization of the function itself." + ] + }, + { + "cell_type": "markdown", + "id": "19d51b04", + "metadata": { + "editable": true + }, + "source": [ "## A new Cost Function\n", "\n", "We could now define a new cost function to minimize, namely the negative logarithm of the above PDF" @@ -2201,7 +2897,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1974af44", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", @@ -2210,14 +2909,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "e4be08a3", + "metadata": { + "editable": true + }, "source": [ "which becomes" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "fbfde0c2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", @@ -2226,14 +2931,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a26db266", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "3d1b896a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", @@ -2242,14 +2953,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "c4936673", + "metadata": { + "editable": true + }, "source": [ "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "dab82d48", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", @@ -2258,10 +2975,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "23164010", + "metadata": { + "editable": true + }, + "source": [ + "Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics." + ] + }, + { + "cell_type": "markdown", + "id": "a14dd3d2", + "metadata": { + "editable": true + }, "source": [ - "Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics. \n", - "\n", "## More basic Statistics and Bayes' theorem\n", "\n", "A central theorem in statistics is Bayes' theorem. This theorem plays a similar role as the good old Pythagoras' theorem in geometry.\n", @@ -2277,7 +3005,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "de6115bf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n", @@ -2286,14 +3017,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b7d42db6", + "metadata": { + "editable": true + }, "source": [ "**The product rule (aka joint probability) is given by.**" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a2e10408", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n", @@ -2302,15 +3039,23 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ea492485", + "metadata": { + "editable": true + }, "source": [ "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n", "\n", - "\n", - "\n", - "If we have independent events then $p(X,Y)=p(X)p(Y)$.\n", - "\n", - "\n", + "If we have independent events then $p(X,Y)=p(X)p(Y)$." + ] + }, + { + "cell_type": "markdown", + "id": "f245dce4", + "metadata": { + "editable": true + }, + "source": [ "## Marginal Probability\n", "\n", "The marginal probability is defined in terms of only one of the set of variables $X,Y$. For a discrete probability we have" @@ -2318,7 +3063,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "62d548ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X)=\\sum_{i=0}^{n-1}p(X,Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert Y=y_i)p(Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert y_i)p(y_i).\n", @@ -2327,7 +3075,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "91c281d0", + "metadata": { + "editable": true + }, "source": [ "## Conditional Probability\n", "\n", @@ -2336,7 +3087,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "86256aec", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)}=\\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}.\n", @@ -2345,7 +3099,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5b302841", + "metadata": { + "editable": true + }, "source": [ "## Bayes' Theorem\n", "\n", @@ -2354,7 +3111,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "516b2342", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n", @@ -2363,14 +3123,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "531e93a3", + "metadata": { + "editable": true + }, "source": [ "which we can rewrite as" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "060b7b4c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n", @@ -2379,10 +3145,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "11d2bdb9", + "metadata": { + "editable": true + }, + "source": [ + "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$." + ] + }, + { + "cell_type": "markdown", + "id": "27ca35db", + "metadata": { + "editable": true + }, "source": [ - "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$. \n", - "\n", "## Interpretations of Bayes' Theorem\n", "\n", "The quantity $p(Y\\vert X)$ on the right-hand side of the theorem is\n", @@ -2392,8 +3169,16 @@ "\n", "The function $p(X)$ on the right hand side is called the prior while the function on the left hand side is the called the posterior probability. The denominator on the right hand side serves as a normalization factor for the posterior distribution.\n", "\n", - "Let us try to illustrate Bayes' theorem through an example.\n", - "\n", + "Let us try to illustrate Bayes' theorem through an example." + ] + }, + { + "cell_type": "markdown", + "id": "cd49cac1", + "metadata": { + "editable": true + }, + "source": [ "## Example of Usage of Bayes' theorem\n", "\n", "Let us suppose that you are undergoing a series of mammography scans in\n", @@ -2410,7 +3195,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f2b6ffc5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X=1\\vert Y=1) =0.8.\n", @@ -2419,11 +3207,22 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "8ed6ee18", + "metadata": { + "editable": true + }, "source": [ "This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n", - "It is however not correct, as the following Bayesian analysis shows.\n", - "\n", + "It is however not correct, as the following Bayesian analysis shows." + ] + }, + { + "cell_type": "markdown", + "id": "954e9167", + "metadata": { + "editable": true + }, + "source": [ "## Doing it correctly\n", "\n", "If we look at various national surveys on breast cancer, the general likelihood of developing breast cancer is a very small number.\n", @@ -2432,7 +3231,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "7c761951", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(Y=1) =0.004.\n", @@ -2441,14 +3243,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "604e1ef2", + "metadata": { + "editable": true + }, "source": [ "We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "8500c910", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(X=1\\vert Y=0) =0.1.\n", @@ -2457,14 +3265,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "51ed965e", + "metadata": { + "editable": true + }, "source": [ "Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "59d4db93", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n", @@ -2473,11 +3287,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "3fa10ac3", + "metadata": { + "editable": true + }, + "source": [ + "That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!" + ] + }, + { + "cell_type": "markdown", + "id": "f0272c14", + "metadata": { + "editable": true + }, "source": [ - "That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!\n", - "\n", - "\n", "## Bayes' Theorem and Ridge and Lasso Regression\n", "\n", "Hitherto we have discussed Ridge and Lasso regression in terms of a\n", @@ -2485,8 +3309,16 @@ "perhaps not that intuitive. The question is whether we can develop a\n", "more intuitive way of understanding what Ridge and Lasso express.\n", "\n", - "Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit. \n", - "\n", + "Before we proceed let us perform a Ridge, Lasso and OLS analysis of a polynomial fit." + ] + }, + { + "cell_type": "markdown", + "id": "e143756b", + "metadata": { + "editable": true + }, + "source": [ "## Test Function for what happens with OLS, Ridge and Lasso\n", "\n", "We will play around with a study of the values for the optimal\n", @@ -2500,7 +3332,8 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 9, + "id": "76cea983", "metadata": { "collapsed": false, "editable": true @@ -2574,11 +3407,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ffe30588", + "metadata": { + "editable": true + }, + "source": [ + "How can we understand this?" + ] + }, + { + "cell_type": "markdown", + "id": "275f37d0", + "metadata": { + "editable": true + }, "source": [ - "How can we understand this? \n", - "\n", - "\n", "## Invoking Bayes' theorem\n", "\n", "Using Bayes' theorem we can gain a better intuition about Ridge and Lasso regression. \n", @@ -2588,7 +3431,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "fc9e564f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n", @@ -2597,14 +3443,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "602792fd", + "metadata": { + "editable": true + }, "source": [ "is given by" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "85c88df7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -2613,14 +3465,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "9a5381c7", + "metadata": { + "editable": true + }, "source": [ "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "a5dc84d2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", @@ -2629,14 +3487,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d93373c1", + "metadata": { + "editable": true + }, "source": [ "Bayes' theorem comes to our rescue here since (omitting the normalization constant)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "6d4241ff", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", @@ -2645,11 +3509,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "22bb3c5d", + "metadata": { + "editable": true + }, + "source": [ + "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!" + ] + }, + { + "cell_type": "markdown", + "id": "d1b6ee2a", + "metadata": { + "editable": true + }, "source": [ - "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$! \n", - "\n", - "\n", "## Ridge and Bayes\n", "\n", "With the posterior probability defined by a likelihood which we have\n", @@ -2661,7 +3535,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "ae7b4c3e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -2670,14 +3547,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5c6fb5fa", + "metadata": { + "editable": true + }, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "b203e66f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -2686,7 +3569,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "b43cbfd1", + "metadata": { + "editable": true + }, "source": [ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", @@ -2696,7 +3582,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "72f2da0c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -2705,14 +3594,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "6f0d440a", + "metadata": { + "editable": true + }, "source": [ "and replacing $1/2\\tau^2$ with $\\lambda$ we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "4f8ead52", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -2721,10 +3616,21 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "f6479b55", + "metadata": { + "editable": true + }, + "source": [ + "which is our Ridge cost function! Nice, isn't it?" + ] + }, + { + "cell_type": "markdown", + "id": "b0aa7607", + "metadata": { + "editable": true + }, "source": [ - "which is our Ridge cost function! Nice, isn't it?\n", - "\n", "## Lasso and Bayes\n", "\n", "To derive the Lasso cost function, we simply replace the Gaussian prior with an exponential distribution ([Laplace in this case](https://en.wikipedia.org/wiki/Laplace_distribution)) with zero mean value, that is" @@ -2732,7 +3638,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5de98401", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -2741,14 +3650,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "a10bfe96", + "metadata": { + "editable": true + }, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "147a60ca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -2757,7 +3672,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "1d97e743", + "metadata": { + "editable": true + }, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", @@ -2766,7 +3684,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "89840052", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -2775,14 +3696,20 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "5f5c045e", + "metadata": { + "editable": true + }, "source": [ "and replacing $1/\\tau$ with $\\lambda$ we have" ] }, { "cell_type": "markdown", - "metadata": {}, + "id": "2bcc2e55", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -2791,7 +3718,10 @@ }, { "cell_type": "markdown", - "metadata": {}, + "id": "d13cae6d", + "metadata": { + "editable": true + }, "source": [ "which is our Lasso cost function!" ] @@ -2799,5 +3729,5 @@ ], "metadata": {}, "nbformat": 4, - "nbformat_minor": 4 + "nbformat_minor": 5 } diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index db87fd4ba..881d1a9b6 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "c4da4e52", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "f29028c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "7f7ee367", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plan for week 39\n", "\n", @@ -56,9 +50,7 @@ { "cell_type": "markdown", "id": "fc2a836f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Thursday September 30\n", "\n", @@ -68,9 +60,7 @@ { "cell_type": "markdown", "id": "9617382d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Searching for Optimal Regularization Parameters $\\lambda$\n", "\n", @@ -87,10 +77,7 @@ "cell_type": "code", "execution_count": 1, "id": "d49166c2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -144,9 +131,7 @@ { "cell_type": "markdown", "id": "19a9a5ab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", @@ -156,9 +141,7 @@ { "cell_type": "markdown", "id": "edeb088d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Grid Search\n", "\n", @@ -171,10 +154,7 @@ "cell_type": "code", "execution_count": 2, "id": "71bdc0a2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -224,9 +204,7 @@ { "cell_type": "markdown", "id": "0ff3515b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "By default the grid search function includes cross validation with\n", "five folds. The [Scikit-Learn\n", @@ -239,9 +217,7 @@ { "cell_type": "markdown", "id": "fb36d7f9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Randomized Grid Search\n", "\n", @@ -259,10 +235,7 @@ "cell_type": "code", "execution_count": 3, "id": "0dc5c1a6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -313,9 +286,7 @@ { "cell_type": "markdown", "id": "c0482352", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -332,9 +303,7 @@ { "cell_type": "markdown", "id": "f13f9cf3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -349,9 +318,7 @@ { "cell_type": "markdown", "id": "563d2108", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -364,9 +331,7 @@ { "cell_type": "markdown", "id": "cfebfdde", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] @@ -374,9 +339,7 @@ { "cell_type": "markdown", "id": "f732663f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations to solve\n", "\n", @@ -390,9 +353,7 @@ { "cell_type": "markdown", "id": "a4f74c15", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -402,9 +363,7 @@ { "cell_type": "markdown", "id": "ce57218f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -413,9 +372,7 @@ { "cell_type": "markdown", "id": "e40a4cac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -425,9 +382,7 @@ { "cell_type": "markdown", "id": "2eafcd65", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This defines what is called the Hessian matrix." ] @@ -435,9 +390,7 @@ { "cell_type": "markdown", "id": "bdb8d75d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -449,9 +402,7 @@ { "cell_type": "markdown", "id": "7304cc0b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -461,9 +412,7 @@ { "cell_type": "markdown", "id": "d04b9c03", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or in matrix form as" ] @@ -471,9 +420,7 @@ { "cell_type": "markdown", "id": "43c885d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -483,9 +430,7 @@ { "cell_type": "markdown", "id": "46bde37c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -495,9 +440,7 @@ { "cell_type": "markdown", "id": "a0b72eb7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -515,9 +458,7 @@ { "cell_type": "markdown", "id": "5b7485f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations\n", "\n", @@ -531,9 +472,7 @@ { "cell_type": "markdown", "id": "4ba6228a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
    \n", @@ -547,9 +486,7 @@ { "cell_type": "markdown", "id": "30380d0c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -558,9 +495,7 @@ { "cell_type": "markdown", "id": "23c99855", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -570,9 +505,7 @@ { "cell_type": "markdown", "id": "08273eaa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "yielding" ] @@ -580,9 +513,7 @@ { "cell_type": "markdown", "id": "e77fa937", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -592,9 +523,7 @@ { "cell_type": "markdown", "id": "52d8ac79", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] @@ -602,9 +531,7 @@ { "cell_type": "markdown", "id": "c749910f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -614,9 +541,7 @@ { "cell_type": "markdown", "id": "ce3de0ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple geometric interpretation\n", "\n", @@ -636,9 +561,7 @@ { "cell_type": "markdown", "id": "7e046c1f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more than one variable\n", "\n", @@ -649,9 +572,7 @@ { "cell_type": "markdown", "id": "6b1748dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -662,9 +583,7 @@ { "cell_type": "markdown", "id": "81266388", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we Taylor expand to obtain" ] @@ -672,9 +591,7 @@ { "cell_type": "markdown", "id": "b8b28a01", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -690,9 +607,7 @@ { "cell_type": "markdown", "id": "a37caf24", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] @@ -700,9 +615,7 @@ { "cell_type": "markdown", "id": "4421201b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -715,9 +628,7 @@ { "cell_type": "markdown", "id": "156aa5ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we can rephrase Newton's method as" ] @@ -725,9 +636,7 @@ { "cell_type": "markdown", "id": "19c663a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -739,9 +648,7 @@ { "cell_type": "markdown", "id": "fab6f264", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -749,9 +656,7 @@ { "cell_type": "markdown", "id": "d3726f16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -763,9 +668,7 @@ { "cell_type": "markdown", "id": "091973c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -778,9 +681,7 @@ { "cell_type": "markdown", "id": "2042ba02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent\n", "\n", @@ -795,9 +696,7 @@ { "cell_type": "markdown", "id": "66e3647e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -807,9 +706,7 @@ { "cell_type": "markdown", "id": "038d29b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -821,9 +718,7 @@ { "cell_type": "markdown", "id": "7b2aec41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on Steepest descent\n", "\n", @@ -836,9 +731,7 @@ { "cell_type": "markdown", "id": "c5344c2b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -848,9 +741,7 @@ { "cell_type": "markdown", "id": "e42471e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -859,9 +750,7 @@ { "cell_type": "markdown", "id": "b38320ec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ideal\n", "\n", @@ -887,9 +776,7 @@ { "cell_type": "markdown", "id": "792e177b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -909,9 +796,7 @@ { "cell_type": "markdown", "id": "760fca67", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex functions\n", "\n", @@ -931,9 +816,7 @@ { "cell_type": "markdown", "id": "52452fa9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex function\n", "\n", @@ -943,9 +826,7 @@ { "cell_type": "markdown", "id": "3dffb542", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditions on convex functions\n", "\n", @@ -980,9 +861,7 @@ { "cell_type": "markdown", "id": "24a9cadc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on convex functions\n", "\n", @@ -1008,9 +887,7 @@ { "cell_type": "markdown", "id": "b9748fe1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some simple problems\n", "\n", @@ -1038,9 +915,7 @@ { "cell_type": "markdown", "id": "4d93454f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Standard steepest descent\n", "\n", @@ -1058,9 +933,7 @@ { "cell_type": "markdown", "id": "3b02576e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -1070,9 +943,7 @@ { "cell_type": "markdown", "id": "c34de3f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the iterative process we end up with a problem like" ] @@ -1080,9 +951,7 @@ { "cell_type": "markdown", "id": "cfc60396", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -1092,9 +961,7 @@ { "cell_type": "markdown", "id": "b6bc2825", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -1104,9 +971,7 @@ { "cell_type": "markdown", "id": "71cce93d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient method\n", "\n", @@ -1116,9 +981,7 @@ { "cell_type": "markdown", "id": "39b1c049", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -1128,9 +991,7 @@ { "cell_type": "markdown", "id": "ab1ecb22", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -1139,9 +1000,7 @@ { "cell_type": "markdown", "id": "a3495238", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent method\n", "\n", @@ -1152,9 +1011,7 @@ { "cell_type": "markdown", "id": "44d95f33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1164,9 +1021,7 @@ { "cell_type": "markdown", "id": "694edeb5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or consider the system" ] @@ -1174,9 +1029,7 @@ { "cell_type": "markdown", "id": "4ec42882", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1186,9 +1039,7 @@ { "cell_type": "markdown", "id": "5e3ada87", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "instead." ] @@ -1196,9 +1047,7 @@ { "cell_type": "markdown", "id": "f996558a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1207,9 +1056,7 @@ { "cell_type": "markdown", "id": "043db570", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1219,9 +1066,7 @@ { "cell_type": "markdown", "id": "b2e31383", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1231,9 +1076,7 @@ { "cell_type": "markdown", "id": "381b2b84", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1243,9 +1086,7 @@ { "cell_type": "markdown", "id": "0025ebc9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -1254,9 +1095,7 @@ { "cell_type": "markdown", "id": "a39f2ec2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -1265,9 +1104,7 @@ { "cell_type": "markdown", "id": "6374696c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1277,9 +1114,7 @@ { "cell_type": "markdown", "id": "acb922cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals" ] @@ -1287,9 +1122,7 @@ { "cell_type": "markdown", "id": "05c95a7a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -1299,9 +1132,7 @@ { "cell_type": "markdown", "id": "9a1924d0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or" ] @@ -1309,9 +1140,7 @@ { "cell_type": "markdown", "id": "86c03b30", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -1321,9 +1150,7 @@ { "cell_type": "markdown", "id": "150d301d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives" ] @@ -1331,9 +1158,7 @@ { "cell_type": "markdown", "id": "e7fab9f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -1343,9 +1168,7 @@ { "cell_type": "markdown", "id": "5447c085", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "leading to the iterative scheme" ] @@ -1353,9 +1176,7 @@ { "cell_type": "markdown", "id": "dee1b865", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k-\\alpha_k\\boldsymbol{r}_{k},\n", @@ -1365,9 +1186,7 @@ { "cell_type": "markdown", "id": "04f189e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent example" ] @@ -1376,10 +1195,7 @@ "cell_type": "code", "execution_count": 4, "id": "fcb1d6ec", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1407,9 +1223,7 @@ { "cell_type": "markdown", "id": "78e14695", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "And then as countor plot" ] @@ -1418,10 +1232,7 @@ "cell_type": "code", "execution_count": 5, "id": "4f10e658", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1432,9 +1243,7 @@ { "cell_type": "markdown", "id": "10fb194e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Find guesses" ] @@ -1443,10 +1252,7 @@ "cell_type": "code", "execution_count": 6, "id": "ee47201f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1456,9 +1262,7 @@ { "cell_type": "markdown", "id": "844b4c66", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Run it!" ] @@ -1467,10 +1271,7 @@ "cell_type": "code", "execution_count": 7, "id": "e68ec7b1", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1485,9 +1286,7 @@ { "cell_type": "markdown", "id": "8a3d5762", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "What happened?" ] @@ -1496,10 +1295,7 @@ "cell_type": "code", "execution_count": 8, "id": "29bd976c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1511,9 +1307,7 @@ { "cell_type": "markdown", "id": "0aee2856", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] @@ -1521,9 +1315,7 @@ { "cell_type": "markdown", "id": "8c7e6157", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1535,9 +1327,7 @@ { "cell_type": "markdown", "id": "cb19fc55", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1547,9 +1337,7 @@ { "cell_type": "markdown", "id": "fb0fa7bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1558,9 +1346,7 @@ { "cell_type": "markdown", "id": "b65328b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1570,9 +1356,7 @@ { "cell_type": "markdown", "id": "efd9856f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1581,9 +1365,7 @@ { "cell_type": "markdown", "id": "d0feaf6a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1592,9 +1374,7 @@ { "cell_type": "markdown", "id": "e2994fb3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1604,9 +1384,7 @@ { "cell_type": "markdown", "id": "a09153f7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is zero unless $i=j$." ] @@ -1614,9 +1392,7 @@ { "cell_type": "markdown", "id": "96825360", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1626,9 +1402,7 @@ { "cell_type": "markdown", "id": "697f4b69", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1638,9 +1412,7 @@ { "cell_type": "markdown", "id": "06baeae1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1650,9 +1422,7 @@ { "cell_type": "markdown", "id": "10331baf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1662,9 +1432,7 @@ { "cell_type": "markdown", "id": "9c07ca94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1673,9 +1441,7 @@ { "cell_type": "markdown", "id": "def0608f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1685,9 +1451,7 @@ { "cell_type": "markdown", "id": "cc38bf53", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] @@ -1695,9 +1459,7 @@ { "cell_type": "markdown", "id": "fb4d2066", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1707,9 +1469,7 @@ { "cell_type": "markdown", "id": "7c605c08", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] @@ -1717,9 +1477,7 @@ { "cell_type": "markdown", "id": "27d71b17", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1729,9 +1487,7 @@ { "cell_type": "markdown", "id": "33a2ec8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1749,9 +1505,7 @@ { "cell_type": "markdown", "id": "874a4bf8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1761,9 +1515,7 @@ { "cell_type": "markdown", "id": "38431600", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or consider the system" ] @@ -1771,9 +1523,7 @@ { "cell_type": "markdown", "id": "a0efad46", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1783,9 +1533,7 @@ { "cell_type": "markdown", "id": "4c79cfc4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "instead." ] @@ -1793,9 +1541,7 @@ { "cell_type": "markdown", "id": "e5be0110", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1804,9 +1550,7 @@ { "cell_type": "markdown", "id": "f9840d01", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1816,9 +1560,7 @@ { "cell_type": "markdown", "id": "2cd5295a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1828,9 +1570,7 @@ { "cell_type": "markdown", "id": "df1c1121", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1840,9 +1580,7 @@ { "cell_type": "markdown", "id": "e0ee3be5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1853,9 +1591,7 @@ { "cell_type": "markdown", "id": "4c6749b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1864,9 +1600,7 @@ { "cell_type": "markdown", "id": "93a6c073", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1876,9 +1610,7 @@ { "cell_type": "markdown", "id": "64e5b77a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1892,9 +1624,7 @@ { "cell_type": "markdown", "id": "7948355f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1904,9 +1634,7 @@ { "cell_type": "markdown", "id": "b203f48c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1915,9 +1643,7 @@ { "cell_type": "markdown", "id": "7e4420a7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1927,9 +1653,7 @@ { "cell_type": "markdown", "id": "8e6faf5a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals" ] @@ -1937,9 +1661,7 @@ { "cell_type": "markdown", "id": "36a01503", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1949,9 +1671,7 @@ { "cell_type": "markdown", "id": "7134e172", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or" ] @@ -1959,9 +1679,7 @@ { "cell_type": "markdown", "id": "b4a913e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1971,9 +1689,7 @@ { "cell_type": "markdown", "id": "5a3c105a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives" ] @@ -1981,9 +1697,7 @@ { "cell_type": "markdown", "id": "341d2ab4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1993,9 +1707,7 @@ { "cell_type": "markdown", "id": "617425bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our first homework\n", "\n", @@ -2017,10 +1729,7 @@ "cell_type": "code", "execution_count": 9, "id": "76c38d50", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -2030,9 +1739,7 @@ { "cell_type": "markdown", "id": "a4f55bbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -2041,9 +1748,7 @@ { "cell_type": "markdown", "id": "c9072fa0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -2053,9 +1758,7 @@ { "cell_type": "markdown", "id": "6e55a2c7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "such that" ] @@ -2063,9 +1766,7 @@ { "cell_type": "markdown", "id": "41bacc77", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -2075,9 +1776,7 @@ { "cell_type": "markdown", "id": "dc8223ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent example\n", "\n", @@ -2089,9 +1788,7 @@ { "cell_type": "markdown", "id": "a8b03557", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -2105,9 +1802,7 @@ { "cell_type": "markdown", "id": "63fbae73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The cost/loss/risk function is given by (" ] @@ -2115,9 +1810,7 @@ { "cell_type": "markdown", "id": "4ad922b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -2127,9 +1820,7 @@ { "cell_type": "markdown", "id": "190ae31f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] @@ -2137,9 +1828,7 @@ { "cell_type": "markdown", "id": "ffff543c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -2149,9 +1838,7 @@ { "cell_type": "markdown", "id": "92aa8f9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2163,9 +1850,7 @@ { "cell_type": "markdown", "id": "7654f8fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $X$ is the design matrix defined above." ] @@ -2173,9 +1858,7 @@ { "cell_type": "markdown", "id": "703447b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -2184,9 +1867,7 @@ { "cell_type": "markdown", "id": "856d4fd1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2199,9 +1880,7 @@ { "cell_type": "markdown", "id": "a05f6979", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] @@ -2209,9 +1888,7 @@ { "cell_type": "markdown", "id": "a6df9f0f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple program\n", "\n", @@ -2221,9 +1898,7 @@ { "cell_type": "markdown", "id": "a643f0c7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -2233,9 +1908,7 @@ { "cell_type": "markdown", "id": "86c2a1fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -2248,9 +1921,7 @@ { "cell_type": "markdown", "id": "f6eb295f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient Descent Example\n", "\n", @@ -2261,10 +1932,7 @@ "cell_type": "code", "execution_count": 10, "id": "14bf489d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2318,9 +1986,7 @@ { "cell_type": "markdown", "id": "fa4674f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -2329,10 +1995,7 @@ "cell_type": "code", "execution_count": 11, "id": "1eef9dcb", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Importing various packages\n", @@ -2356,9 +2019,7 @@ { "cell_type": "markdown", "id": "92ad46a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2368,9 +2029,7 @@ { "cell_type": "markdown", "id": "0bb24dc6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2380,9 +2039,7 @@ { "cell_type": "markdown", "id": "10a9fcb2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] @@ -2390,9 +2047,7 @@ { "cell_type": "markdown", "id": "495dbade", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2404,9 +2059,7 @@ { "cell_type": "markdown", "id": "4b263196", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] @@ -2414,9 +2067,7 @@ { "cell_type": "markdown", "id": "2a0b4fb5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2426,9 +2077,7 @@ { "cell_type": "markdown", "id": "a2b60831", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2437,9 +2086,7 @@ { "cell_type": "markdown", "id": "a82f529c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2452,9 +2099,7 @@ { "cell_type": "markdown", "id": "8fe1c41e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -2466,22 +2111,41 @@ { "cell_type": "markdown", "id": "a03ad61b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program example for gradient descent with Ridge Regression" ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 13, "id": "f0a079df", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Eigenvalues of Hessian Matrix:[0.29294218 4.47046976]\n", + "[[4.03410523]\n", + " [3.03026348]]\n", + "[[4.03408456]\n", + " [3.03028067]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnoklEQVR4nO3de5xdZX3v8c9vJhc6EMVMoiIwE/GO1AtE8NY2nlZFxNrTK3FApJYUPG2xB6sCp4hKqK22ij3H0ogUNSE9vvBS5WjFKqCoIAOCgsEbmUwiAiFckyBJJr/zx1qTrL1nrb2fvffa6zLzfb9e+5XZe91+e+2V9VvreZ71PObuiIiItDNQdgAiIlIPShgiIhJECUNERIIoYYiISBAlDBERCaKEISIiQZQw5jgzmzCz34n/PtfMLi0pjhVmtqWMbc8Vyd+6i2XPNLN7zWy7mQ3nHVvK9i4xs79tMd3N7Jn9jkMaKWFUmJmdZGY3mtkOM7sv/vttZmb92J67X+Tuf9bresxsWfwfel4ecZXNzC43swvLjqMsZjYf+CfgNe5+kLtvy2GdE2b2WJyA7on38UHT0939DHd/f6/bkXwpYVSUmZ0NXAx8EHgq8BTgDOAVwIKMZQYLC1DmkqcABwB3dLqgRbLOM29w94OAFwEvBs7pOkIphBJGBZnZE4H3AW9z9yvd/VGPfN/dx9z98Xi+y83sX8zsy2a2A3iVmb3ezL5vZo+Y2WYzu6Bp3aeY2SYz22Zm5zVNu8DM1ibev9TMvmNmD5nZbWa2IjHtWjN7v5l928weNbOrzWxJPPmb8b8PxVeQL0v5jr8Wx/+gmf0IeEnT9KeZ2WfNbKuZbTSzv0pMO9bMxuPveK+Z/VNi2isTMW82s7fEny80sw+Z2WS8zCVm9mvxtBVmtsXMzo7v5H5pZqfF01YBY8A74+/ypYzf7Llm9jUze8DMfmxmfxx//oz4s6MT3+v+6X1pZqeZ2YZ4H95lZn+eWOd0XO9MxPV7ZnaCmf0kXu+5Tb/flWb2f+P13WJmL8yId8DM3m1mP4+Phc+Y2eKU+Z4N/Djxe34j/vzlZnaTmT0c//vyxDLXmtlqM/s2sBM4Ii2Gae5+D/BVosQxvY6Guzoz+5v4+99tZn/aFOOwmX0pPh5uMrMLzez6dr+NdMHd9arYCzge2APMazPf5cDDRHcdA0RXgSuAX4/fvwC4F/i9eP4jge3AbwILiYoZ9gC/E0+/AFgb/30osA04IV7Xq+P3S+Pp1wI/B54N/Fr8/gPxtGWAt4of+ADwLWAxcDhwO7AlnjYA3AycT3Q3dQRwF/DaePp3gVPivw8CXhr/PQI8CqwE5gPDwIviaR8BvhhvbxHwJeDv4mkr4v3wvni5E4hOdE9K7OcLW3yXA4HNwGnAPOBo4H7g+fH004ENwBDRifFDiWVfDzwDMOC34u0e3RTX+XFcpwNbgSvi7/B84FfAEYnfbzfwh/H87wA2AvPj6ROJ3/rtwA3AYUTHwr8C6zO+X8PvGe/DB4FT4u+7Mn4/nDg2JuP45k1vv2mdyVgOA34IXNx0bF+Y+P9wL3BUvK+viON5Zjz93+PXENExvhm4PuS30avDc1PZAeiV8qPAycA9TZ99B3gIeAz4zfizy4FPtVnXR4APx3+fD/x7YtqBwC7SE8a7gE83reurwKnx39cC/ysx7W3Af8Z/N5xgMuK6Czg+8X4V+xPGccBk0/znAP8W//1N4L3AkpR5Pp+yLQN2AM9IfPYyYGP894p4v85LTL+P/Ylo38kr47v8CfCtps/+FXhP4v0X45PiD4CFLdb1BeCsprgG4/eL4v16XGL+m9l/QXABcENi2gDwS+A34vcTid96A/DbiXkPIUo2M36z5t+TKFF8r2me7wJvSRwb72tzXE4QXbw8Gq/768DBien79jlwGfHFSPz+2fEyzwQG47ifk5h+IfsTRtvfRq/wl4qkqmkbsMQSlcbu/nJ3PzielvzdNicXNLPjzOyauCjnYaJ6j+mioqcl53f3HfH60owCfxQX7TxkZg8BryQ6sUy7J/H3TqKr/VANsQCbmrb9tKZtn0tUlg7wVqKTxp1xEcSJ8eeHE931NFtKdPV5c2J9/xl/Pm2bu+/p8vuMAsc1xTtGVPc07eNEV8j/7HGRIoCZvc7MboiLSx4iurtZklhum7tPxX8/Fv97b2L6Y01xJn/fvcAWon2dFvPnE/FuAKbYv49beRqNvxfx+0PT4mjh99x9EVFifC6N37t5e1nHylKiO4fk9OTfIb+NBFLCqKbvAo8DbwyYt7m74SuIrmYPd/cnApcQXWFDdLV5+PSMZjZEVGyTZjPRHcbBideB7v6BLmJK0xALUXFSctsbm7a9yN1PAHD3n7r7SuDJwN8DV5rZdNHDM1K2dT/RifX5ifU90aMK1xDtvs9m4LqmeA9y9zMBLGr98xHgE8AF03UFZrYQ+CzwIeAp8QXBl9n/e3Uj+fsOEBX33J0R8+uaYj7A3X8RsI27iU7ESSNActngbrDd/TqiO4oPZczS6ljZSlRsd1jis+S8LX8b6YwSRgW5+0NERS4fM7M/NLOD4krKFxEVI7WyCHjA3X9lZscCb0pMuxI4Ma4YXkBUZp91DKwF3mBmrzWzQTM7IK6EPSxj/qStwF5aV3Z+BjjHzJ4Ur/MvE9O+BzxiZu+yqHJ80MyOMrOXAJjZyWa2NL6CfiheZgpYB/yOmf2xmc2LK0NfFM/3ceDDZvbkeB2HmtlrA74LRFf0rb7LVcCzLWpQMD9+vcTMnhdPvxi42aMmy/+PKIlDVD+zkPikZ2avA14TGFOWY8zs9+O707cTXXjckDLfJcBqMxsFMLOlZhZygQJRUnu2mb0p3s9/QlR3cFUPcX8EeHV8jDf7DPAWMzsyvsh5z/SE+O7rc0SJeMjMngu8ObFsu99GOqCEUVHu/g/A/wTeSVSefi9R2eu7iOozsrwNeJ+ZPUpUZ/GZxDrvAP4H0V3IL4kqKlMflnP3zUR3OOcSndA2A39DwDHj7juB1cC342KAl6bM9l6iooWNwNXApxPLTwFvIGo1s5HoDuFS4InxLMcDd5jZdqKT8Unu/it3nyQq0jkbeAC4FZhuJfQu4GfADWb2CPBfwHPafZfYJ4Aj4+/yhZTv+yjRif4koqvve4jufBbGJ+HjiYoGIfpNjzazsXi5vyL6jR4kSu5fDIwpy38QldtPV0r/vrvvTpnv4nhbV8fHyg1EdUdtefQcxolE+3kb0TF6orvf323Q7r4V+BQw42E9d/8KUUL5BtFv+I2mWf6C6Ni4h+g4Wk+UKFv+Nt3GOpeZuwZQEpkNLGpC/Ux3P7nsWMpkZn8PPNXdTy07ltlGdxgiUmvxcxYvsMixRI0iPl92XLPRrOi6QUTmtEVExVBPIyq+/UeiojnJmYqkREQkiIqkREQkSK2KpJYsWeLLli0rOwwRkVq5+eab73f3pe3nbK1WCWPZsmWMj4+XHYaISK2YWfOT+V1RkZSIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSB9TxhmdpmZ3Wdmt6dMe4eZuZllDf4uIiIVUcQdxuVEQ1Q2MLPDgVcDkwXEICIiPep7wnD3bxKNr9zsw0RjAWtADhGRGiilDsPMfhf4hbvfFjDvKjMbN7PxrVu3FhCdiIikKTxhmNkQcB5wfsj87r7G3Ze7+/KlS3vuzl1ERLpUxh3GM4CnA7eZ2QRwGHCLmT21hFhERCRQ4QMoufsPgSdPv4+TxnJ3v7/oWEREJFwRzWrXA98FnmNmW8zsrf3epoiI5K/vdxjuvrLN9GX9jkFERHqnJ71FRCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEqTvCcPMLjOz+8zs9sRnHzSzO83sB2b2eTM7uN9xiIhIb4q4w7gcOL7ps68BR7n7C4CfAOcUEIeIiPSg7wnD3b8JPND02dXuvid+ewNwWL/jEBGR3lShDuNPga9kTTSzVWY2bmbjW7duLTAsERFJKjVhmNl5wB5gXdY87r7G3Ze7+/KlS5cWF5yIiDSYV9aGzexU4ETgt93dy4pDRETClJIwzOx44F3Ab7n7zjJiEBGRzhTRrHY98F3gOWa2xczeCvxvYBHwNTO71cwu6XccIiLSm77fYbj7ypSPP9Hv7YqISL6q0EpKRGS/detg2TIYGIj+XZfZJqY66hhzF0qr9BYRmWHdOli1CnbGVZubNkXvAcbGyourlTrG3CWrUwOl5cuX+/j4eNlhiEi/LFsWnXCbjY7CxETR0YSpQcxmdrO7L+91PSqSEpHqmJzs7PMqqGPMXVLCEJHqGBnp7PMqqGPMXVLCEJHqWL0ahoYaPxsaij6vqjrG3CUlDBGpjrExWLMmKv83i/5ds6balcedxNxpa6rp+c1g3rzo3xJbYanSW0SkCM2tqSC6E2mVXJrnn7ZgASxaBA88EBV9rV7dMqnmVemthCEiUoROW1NlzZ+mVeJBraREROql09ZUnbSy2rkTzjuv85g6pIQhIlKETltTddrKqoBmvEoYIiJF6LQ1Vdr8rSQTTFPl+hJY3HG8KZQwRESKENKaKnmiP+88OPXUaD6AwcHo3+FhmD+/cd3JxDNdWb5pE7jDpk2MwGgeX0GV3iIiVdBJK6p166KEMjk5s5VUSmX5cmDc3XoNUQlDRKQK8uqTamAgurNIyCthqEhKRKQK8uqTqo9dkihhiIhUQV6tpVIqyx32dhlVAyUMEZEqyKtPqpTK9UkIfAKwNSUMEZEqyLMfrbGxqN5j716YmOB+eCCPEJUwRGR2q9PwqU0n+qp1utj3hGFml5nZfWZ2e+KzxWb2NTP7afzvk/odh4jMQSnPJLBqVbWTRoUVcYdxOXB802fvBr7u7s8Cvh6/F5n96nS1m6Zu8Z933szeXovod6mf+6nFunfv3M3H3/xNlg7cjxmc87Jr89sugLv3/QUsA25PvP8xcEj89yHAj0PWc8wxx7hIba1d6z405B5d60avoaHo8zqoY/xmjfFOv8z6t81O9tPate6jo1E8o6Pt92XKunfPP8DfNe+DqV/znJdd4+7uwLjncS7PYyVtNzIzYTzUNP3BFsuuAsaB8ZGRkdY7U6TKRkfTT16jo2VHFqaO8ZcRc+g2u0nAGeveyOi+t8cv+Z6Pf/pHDYvllTAKedLbzJYBV7n7UfH7h9z94MT0B929bT2GnvSWWkt5AheIWsTszaWZfH/VMf5OBy3KQ+h+Cnyy+/VPvokvb30JAFMMMMDMdTsGU1PYQPrD3HUfD+NeMzsEIP73vpLiEClOXg9mlaWO8XfbVLWXOojQ/ZTxBLdvmsSMfa/pZAEwSfq6bXQkM1nkqayE8UXg1PjvU4H/KCkOkeLk9WBWWeoaf6dNVXttWRW6nzISy6aMpHDdR29j2dqSf4M8yrVavYD1wC+B3cAW4K3AMFHrqJ/G/y4OWZcqvaX2Oq3krJq6xx8ij3qPM890HxyMlhscjN7HfvXwrxzcV7LWt9NYh7GdIV/JWgf3Vc+9Ln3dXfwG1KkOIy+qwxCRvuu1rial3mQHQ5zOGtbTeHezknVcxHmMMMkkI9EdRB/qVvKqw1DCEBFJ6rWb8YzlJxjl6TQuX9Tpt+6V3iIi1dRFXc2xB92xr5J676b0yuwRJrnvR/fvL39aW7OHIFHCEBFpFNCy6sqzv9vQkummHc/fNy2rJdPA6AhLn7ckelPTLktUJCUi0sYDP3+Q4We27/JuZHALmz55XftnP/IaXS+QiqRERFrpsT+n5B1Eq2SRbOa0ac9hYc9+5DW6XsHmlR2AiEjumlsqTRf5QGYrJAt87i2oUGZsrHVrp5GR9DuMKj8Eie4wRGQ2CuilNnkH0SpZPLz5kYa7iFzU9CFIJQwRyU9Vuj/PKNrZm+h2I8un/vz6hgTxhMOekH98eY6uVyAlDJG5Ku+Te5Va/mQU7aS1YBpgqiFBnHLJK/sdXaTio+ulUcIQmYv6cXIva7CimO/1fXcPb9q0mh00FvnsYIhziYp8kgliygcLiW82UMIQmZbHFXdVimTa6cfJvZOWPzntp2QdxMDg/nKm9YxxOmuYYJS9REU+B65dwxU+VtjT1bNSHh1SFfVS54PSN3mMJlenEen6MRJdXgMHtehcL231aa/HH328++8xC1GnEffyeilhSN/k0UNpnUak60esoQmz1bZT1pHswTXr9YVzbug+7jkgr4ShIikRyOdBqjo9jNWPZp2hLX9a7KftZ5w9o6jsQHZyEY1FZYtobOr6xouO6z5uqE9RYsmUMEQgn9Hk6jQiXb+adYa0/MnYHxM+wtD29ME3R5hsSBCPeI5NXavUuqvilDBEIJ8r7ro9jFVSs85WLZhaddzXNyW37qoTJQwRyOeKu6YPY+Umo1in+Ynq5hZME4xyOmtYN/Wm8CFI8yxCqlNRYtnyqAgp6qVKb5GSpLVcmv4M3AcGZtREt6usvv5jt4Vvq3l6nq3R6tRYoUtoiFYRKUTKkKPMnx/dLuza1XLR5ChzT583yV27cyhayrtr8LTv19wdec2pe3MRKUZaGf/u3W2TBcAy219ZnUuygPyLkOZ6UWIHSk0YZvbXZnaHmd1uZuvN7IAy4+krNduTuuqlLL8fLcT60Rqthv06laG0hGFmhwJ/BSx396OAQeCksuLpq7o026tLUqtLnDWWrKSe8B5OxNu35/871a01Wq+qdLznURHSzQs4FNgMLCYayOkq4DWtlqltpXcdKtXq0q1FXeKsmVZPUa9krW+naZ/Pn+8+ONh6weZXnr9Tu4rx2SKn453Z0DUIcBawHdgKrMuYZxUwDoyPjIx0tJMqox/99uStDknNvT5xliXwRBp6jj9y4U/T13vmme4LFnSWMPQ7dS6n4732CQN4EvANYCkwH/gCcHKrZXSH0Ud1SGruxcRZhavXbmLIuho980zfefBTfQrzjYz6P3Omb2R03/vmpq9Bso7pdq+Q36kK+78qcjreZ0PC+CPgE4n3bwY+1mqZ2iaMOhSj1CGpufc/zir8Vt3GkLFvpmg86ezNo6go60Q2/VsMD3f3O1Vh/1eJ7jD2JYjjgDuAIcCATwJ/2WqZ2iYM9+pfNdXlP2q/46xC4uwyhubE0NGr0+/XLsack17lLlyKojqMhqTxXuBO4Hbg08DCVvPXOmHUQdWT2rR+xtmqCKCo/RNYDNE8eSOj3SeMTov0Qk5k3eyvuhSNFimH425WJIxOX0oY0ndZV7jDw8XdgWUU59zHcMtzfmprplZFR71ewfcjgeoOoy/yShhtn8Mws/8ysxd23mBXpMKy2rZntfGHSvdo6g5X+BgHrm16YvmMM2Z+n2bdPsPQj4fdTjihs8/7rUrPQFRBu4wCHE3UmunfgEPyyFLdvnSHIbnoZojQAopKdu3Y5ZBdFzGF7Xvb8fdtbhJb1aLHKt1h1KVeLwBFdz5oZn8AnA98DvgHd3+sPyksmzoflFx003ld3h3excxmfraRZSwj/231ZN266G5qcjLqgmP16v50nzEwEJ2am5lFdzJF6tNvXoZCOx80MwN+DPwL8JfAT83slF43LlKKbjqvy6k7iuaxIdKcy2q8Sl1fFNm1TZVGLdQ4GTOE1GFcD/wC+DBRdx5vAVYAx5rZmn4GJ9IX3ZyU0no0PfXU6Kq7Rfl2SIIAuOy0b+0r97jCx7B+9J7abXl8kSPSVaWfqHXrov2UpopD7halXZkVcBRERVcp0zbkUS4W+lIdhuQij7LpjHV8avS84JasherlOxfd1LXs5t1p+0p1GJ3VYaQxsyPc/a5cMlcA1WFIbnotk88o304OGNSsh/9qveulPH4WleUHyfq+g4PwyU/WsuvzvOowNOKeSIfMYIoBBpj5f8eBMdaynrFyE0SzXiqT58CIdA2qVPGeE424J1KQtHqISdLLsQ24YmgVvjauH+hXO/5O19tLZXIdRqTLcz9XqeK9avIo1yrqpToMKUJI/cNK1vqe+QdkzzBd9t6PdvzdrDdtmem6ieHh6FXF5zJC5L2fZ9HzF9NQ1yBSO2VVZrbZ7jPnbwxKEgewc+Z6s2ae3lY/HkLrdr3T+yGZLGZDxW4/9nPZFe85yythqA5DilFkOXiyQnvxYnjkEdi9e9/kvQsP4OTHL2U97bfb9r9Hqwrhycn+lIX3WsaeFXNSnSq0Z2GdQ95Uh1EE9SOTn6La8jc/ZLZtW0OyABh4/FdcRPp2my9T22r13EC/ysJ7XW/Ig2d1ejhNdQ7FyeM2pahXoUVSs7AcszTtim66WV9WcUHgSHDT/TLl9v3S4qlSHUZSyD7qR99N/Srm0f/VtlAdRp9VqRO0Omv1EFQ3+zPj5DA9zGhHgwgVUTadx0kybR29rLfdb9KPk22/T+qzrM4hb0oY/VblgVzq9J+j1dVsNyeMjPVtZNShi0GEqn4l2s+7lOljqIhWUroAK5USRr9V9QDvxwmkjBHsoKPtfOlvb2x5BzFdxLSSNlfPVfxNW+nkOKzyhUSVL8DmACWMfqtquWjeiayiY2RP7Z5KXSzzDiK5vmTz0ZBXlU9aoSfaqh6v06p6ATZHKGEUoYpXbHlfqfX7P3IHJ7KQc3vqHUTWibFdWX0dTlqhv0/VT8hVT2iznBLGXJX3iaGIooKMEd+mMN/I6L4K66zXzm0pD8wlx70eHs4+8TSX1c+fX6+TVuiJtg5FPlW8AJsjZkXCAA4GrgTuBDYAL2s1vxKG53+lVuCV6fQdwnYa499O1MppJWt9I6O+lzYnlF72QR1PWiExV/0OQ0o1WxLGJ4E/i/9eABzcan4ljFjoSS9kvj4WFVz8B9fOOH9l1kEMD4fHoZPjTCrykRZqnzCAJwAbyRicKe2lhNGBTk4gOV1179i6o211QUfPSWQlgToUv5ShjndPUoi8EkZpfUmZ2YuANcCPgBcCNwNnufuOpvlWAasARkZGjtnUrg8ciRQ06E2rYUeT9h1mIf0YNW+guT+guTagj0iPZkNfUvOAo4F/cfcXAzuAdzfP5O5r3H25uy9funRp0THWV58GsA8do3rvlDdc/u+T1ffS8HD6itL6A6rKuM8ic0yZCWMLsMXdb4zfX0mUQOqtKh0W5tQhW2iC+MY/fr8hQdhAxsxZg/FcfHF4EqjDgD4is1Ee5VrdvoBvAc+J/74A+GCr+Stfh1GliscuYznrRTMrqtNeT7IH+hOzyuBFckfd6zBgXz3GpUQtpO4CTnP3B7Pmr/x4GFUrW0+OCzEyEl2tN12Fb91wP08+cknQ6ko8VESkB3nVYWgApTzVZCCXjiuqyxSQ9ESktdlQ6T37VHQgl9B6iOaCp9I1D4a0aVP0XgNZiZRCCSNPFWm9E5ogNt94d3aCqELlfVGj9IlIECWMPJXUeue0Z10flCDesfzahgRx2LFPS5+xKlf2fWoaLCLdqV/CqMKVb8l+cvVEQ4K4/GevzJw3mSA+eNOKsA1U5cq+okV8InNVvSq9jzjCx++9t/FkNjRUnTb401fmOcfnHuXH0Hl7VpXK+z7tT5G5Zm62klq40Md37Zo5oSpdQuTYrLbUlkxVah6sVlIiPZubraTSkgWUV6bdXDyW1UfSpk1ti9FCK6p3bN3Z/5ZMFam8B6LkMDER3dlMTChZiJSoXgljwYL0zwcGiq/TSKsYzjrTm82oQL548QVBCeKq99zUkCCGlgxlz5wXdb0hIinqlTAOPXTmlS/A1FS+rXlCKtbTKobdZ579zWbeCuzcyRsfvDx10y884M6GBPH6C17S7bcIl/Z9dWUvIk3qlTAWL2688h0cnDlPr615QpuUZhWDucPoKG7GBKPszSg3GmGyYZHp162PPbf72LtRlSa0IlJ59ar0bu4apB+teUIrfDPmm2CUc1nNRZzHCJPsZYB5TLVfX1mqVMEtIn0xNyu9H3igsehk8eL0+Xpppx/6sNjq1eygsXhsB0NcxQl8nFUsYxMDOPOYYkZKq9LYDXo4TkQC1SthbNrUWHTy6KMwf37jPL2ejFs8LPb2F1+3v6L65DFOZ01U7ERU/HQ6aziRL3MgjXUbBlHxWbICGarxAGLW93Wfsw9GikiGPPpIL+p1TNrADMPD+Y6hkDKOxHaGfCVrM8eG+OgfXLN/+ZDxpqs+bkZz3GeeWXxcIpIbZsN4GJ1abuYzOjfP8enj3Tt3s+DA+axk3b46iElGOJfVrGd/K6HDB3/B5J5D01cSUidQtXqD6Yfjsp4jMYNPf1otpURqam4+6Z2WMHo8yeb+RHVIdxZV6XqjWVZcoEpwkRqbm5XezR0qdVFf8byFP+/v2BAhD71VtVO9VttXJbjInFevhDE62vHTx39/wrUNCeLOXc9Ine/hzY/k1+VGu4feqtT1RtLq1dlZdGBAFeAic1y9EsbixW2fPr5l3YaGBPHur6xIXdUNl97ekCCecNgT+hp6g6p2vTE2BmeckZ40pqb0QJ/IHFevhJFi1/ZdDQnimJOflzrfe1dc05AgjnvrUeEb6ccYHFXteuNjH4squPvxFL2I1FrpCcPMBs3s+2Z2Vegyf/sb+4uZFi5K75Dw8MFfNCSI8695VXcBltV1RpkDRY2NZVe+qy5DZM4qPWEAZwEbQmc2gwuvX5E6LZkgMpu9dqqM0edaJamiEklVK+ZFpDSlJgwzOwx4PXBp6DIXveZaAA4ZuIfdj+3Jr6I6SxldZ2QlqbPOKu5up6oV8yJSmrLvMD4CvBPIfPjAzFaZ2biZjW/dupVzvroCd7h76qnMO2Be/yMs40o7Kxlt21bc3U5VK+ZFpDSlJQwzOxG4z91vbjWfu69x9+Xuvnzp0qUFRZdQxpV2p8lo06bui6haFXG1q5gvs55FRIqXR/8i3byAvwO2ABPAPcBOYG2rZY455pio76M8+44KUfQ2s/qaGh7O7vOpmz6peunTqkr9YYlIS8ymvqTMbAXwDnc/sdV8y484wsfvvbd1txuzxXT/TpOT0R3H9B1Nc7cjaUK78eilT6uq9YclIplmVV9SwQlj4UIf37Vr5oS5dJJKJpKs3y60T6pe+rSqan9YIjLDrOpLyt2vbZcsAEhLFhCdPOdKeXqyXmF0NH2e0DqQXir01exWZM6pRMIItiD9IT0WL56b41L3WiHfy/Jqdisy59QrYRx66MyTlFmxzU2rpNemr70sr2a3InNOJeowQi1fvtzH//qv9w/2Y9b6iT2Vp4uIzK46jI5Ml+GPjrZ/vFvl6SIiualfwpjWrmsOlaeLiOSqvgmj1d2DytNFRHJX34SR1Upn7dpqjS8hIjJL1DdhqJWOiEihCujutY/GxpQgREQKUt87DBERKZQShoiIBFHCEBGRIEoYIiISRAlDRESCKGGIiEgQJQwREQmihCEiIkGUMEREJIgShoiIBFHCEBGRIKUlDDM73MyuMbMNZnaHmZ1VViwiItJemZ0P7gHOdvdbzGwRcLOZfc3df1RiTCIikqG0Owx3/6W73xL//SiwATi0rHhERKS1StRhmNky4MXAjSnTVpnZuJmNb926tfDYREQkUnrCMLODgM8Cb3f3R5qnu/sad1/u7suXLl1afIAiIgKUnDDMbD5Rsljn7p8rMxYREWmtzFZSBnwC2ODu/1RWHCIiEqbMO4xXAKcA/83Mbo1fJ5QYj4iItFBas1p3vx6wsrYvIiKdKb3SW0RE6kEJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJUmrCMLPjzezHZvYzM3t3mbGIiEhrpSUMMxsE/g/wOuBIYKWZHVlWPCIi0lqZdxjHAj9z97vcfRfw78AbS4xHRERamFfitg8FNifebwGOa57JzFYBq+K3j5vZ7QXE1qslwP1lBxFAceanDjGC4sxbXeJ8Th4rKTNhWMpnPuMD9zXAGgAzG3f35f0OrFeKM191iLMOMYLizFud4sxjPWUWSW0BDk+8Pwy4u6RYRESkjTITxk3As8zs6Wa2ADgJ+GKJ8YiISAulFUm5+x4z+wvgq8AgcJm739FmsTX9jywXijNfdYizDjGC4szbnIrT3GdUG4iIiMygJ71FRCSIEoaIiASpRMJo10WIRT4aT/+BmR0dumzBcY7F8f3AzL5jZi9MTJswsx+a2a15NXHrIc4VZvZwHMutZnZ+6LIFx/k3iRhvN7MpM1scTytkf5rZZWZ2X9bzPxU6NtvFWZVjs12cVTk228VZhWPzcDO7xsw2mNkdZnZWyjz5Hp/uXuqLqML758ARwALgNuDIpnlOAL5C9OzGS4EbQ5ctOM6XA0+K/37ddJzx+wlgSUX25wrgqm6WLTLOpvnfAHyjhP35m8DRwO0Z00s/NgPjLP3YDIyz9GMzJM6KHJuHAEfHfy8CftLvc2cV7jBCugh5I/Apj9wAHGxmhwQuW1ic7v4dd38wfnsD0bMlRetln1RqfzZZCazvUyyZ3P2bwAMtZqnCsdk2zoocmyH7M0ul9meTso7NX7r7LfHfjwIbiHrQSMr1+KxCwkjrIqT5S2fNE7JsXjrd1luJMvs0B642s5st6u6kX0LjfJmZ3WZmXzGz53e4bB6Ct2VmQ8DxwGcTHxe1P9upwrHZqbKOzVBlH5vBqnJsmtky4MXAjU2Tcj0+y+waZFpIFyFZ8wR1L5KT4G2Z2auI/lO+MvHxK9z9bjN7MvA1M7szvoopI85bgFF3325mJwBfAJ4VuGxeOtnWG4Bvu3vyiq+o/dlOFY7NYCUfmyGqcGx2ovRj08wOIkpYb3f3R5onpyzS9fFZhTuMkC5CsuYpsnuRoG2Z2QuAS4E3uvu26c/d/e743/uAzxPdEpYSp7s/4u7b47+/DMw3syUhyxYZZ8JJNN3yF7g/26nCsRmkAsdmWxU5NjtR6rFpZvOJksU6d/9cyiz5Hp/9rpgJqLiZB9wFPJ39lS/Pb5rn9TRW3HwvdNmC4xwBfga8vOnzA4FFib+/AxxfYpxPZf9Dm8cCk/G+rdT+jOd7IlFZ8oFl7M94G8vIrqQt/dgMjLP0YzMwztKPzZA4q3BsxvvlU8BHWsyT6/FZepGUZ3QRYmZnxNMvAb5MVNv/M2AncFqrZUuM83xgGPiYmQHs8agny6cAn48/mwdc4e7/WWKcfwicaWZ7gMeAkzw6iqq2PwH+O3C1u+9ILF7Y/jSz9UQtd5aY2RbgPcD8RIylH5uBcZZ+bAbGWfqxGRgnlHxsAq8ATgF+aGa3xp+dS3Rx0JfjU12DiIhIkCrUYYiISA0oYYiISBAlDBERCaKEISIiQZQwREQkiBKGiIgEUcIQEZEgShgiPYjHI3h1/PeFZvbRsmMS6ZfSn/QWqbn3AO+LO5p7MfC7Jccj0jd60lukR2Z2HXAQsMKjcQlEZiUVSYn0wMx+nWjks8eVLGS2U8IQ6VI8ctk6opHKdpjZa0sOSaSvlDBEuhCPtPY54Gx33wC8H7ig1KBE+kx1GCIiEkR3GCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiAT5/6qqV3GZBKQBAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
    " + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "from random import random, seed\n", "import numpy as np\n", @@ -2538,9 +2202,7 @@ { "cell_type": "markdown", "id": "fc6d73bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2560,9 +2222,7 @@ { "cell_type": "markdown", "id": "40a565ce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Challenge yourself\n", "\n", @@ -2572,9 +2232,7 @@ { "cell_type": "markdown", "id": "13077723", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Friday October 1" ] @@ -2582,9 +2240,7 @@ { "cell_type": "markdown", "id": "3227c51a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2599,9 +2255,7 @@ { "cell_type": "markdown", "id": "d414c535", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2612,9 +2266,7 @@ { "cell_type": "markdown", "id": "0c25b0fc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Computation of gradients\n", "\n", @@ -2625,9 +2277,7 @@ { "cell_type": "markdown", "id": "23dca3ab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2638,9 +2288,7 @@ { "cell_type": "markdown", "id": "3b169c06", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2652,9 +2300,7 @@ { "cell_type": "markdown", "id": "d7ce6406", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2674,9 +2320,7 @@ { "cell_type": "markdown", "id": "0a528877", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2689,9 +2333,7 @@ { "cell_type": "markdown", "id": "0f93ca5e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The gradient step\n", "\n", @@ -2701,9 +2343,7 @@ { "cell_type": "markdown", "id": "1d87bd14", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2714,9 +2354,7 @@ { "cell_type": "markdown", "id": "4181f46b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2728,9 +2366,7 @@ { "cell_type": "markdown", "id": "af66da80", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example code" ] @@ -2739,10 +2375,7 @@ "cell_type": "code", "execution_count": 13, "id": "83e51ad8", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2764,9 +2397,7 @@ { "cell_type": "markdown", "id": "e73bde3d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2780,9 +2411,7 @@ { "cell_type": "markdown", "id": "97d463c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## When do we stop?\n", "\n", @@ -2801,9 +2430,7 @@ { "cell_type": "markdown", "id": "cc362c51", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Slightly different approach\n", "\n", @@ -2824,10 +2451,7 @@ "cell_type": "code", "execution_count": 14, "id": "0716d374", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2859,22 +2483,47 @@ { "cell_type": "markdown", "id": "f7480cca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program for stochastic gradient" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 14, "id": "440f74c2", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[3.97284805]\n", + " [3.13660672]]\n", + "sgdreg from scikit\n", + "[3.99061711] [3.20487971]\n", + "theta from own gd\n", + "[[3.97284805]\n", + " [3.13660672]]\n", + "theta from own sdg\n", + "[[3.94799953]\n", + " [3.10740003]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAArHElEQVR4nO3de5xVdb3/8ddnhpsDKlcBgRlQvGGWKUfzlhwNstKsPFmGpYUHxV+nOvmrNDppGZ0udupnZUaWl5iyrGN1jlZgDndRUEEEvMNwE7nLHWZmf35/rD04bPZl7b3X3mvPzPv5eOwH+7Iun7Vm8f2s7/e71neZuyMiIpJLVdwBiIhI+6CEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGFIh2Vmt5nZtLjjiJKZjTGztXHHIZ2TEoaUlZmtMrO9ZrbLzDaY2X1m1ivuuEQkNyUMicNl7t4LOB14J3BLvOF0TmbWJe4YpH1RwpDYuPsG4O8EiQMAM7vZzF41s51mttzMPtzmt2vNbK6Z3WFm28xspZm9r83vI8xsVnLeGUD/tuszsw+a2TIz225mM83slDa/rTKzL5nZc2a228x+aWYDzeyvyeU9ZmZ90m1HazORmd1kZhvN7HUz+3Sb32ea2XWp29Hms5vZjWb2cnJdt5vZ8Wb2hJntMLPfm1m3lHV+1cw2J+Me3+b77sn9s9rM3jCzu83siJQ4v2JmG4B7zay/mf1vcp9sNbM5ZqZyQdLSgSGxMbOhwPuAV9p8/SpwAXA08A1gmpkNbvP72cCLBMnge8AvzcySv/0GeDr52+3ANW3WdSLwW+ALwADgUeB/UgriK4CxwInAZcBfga8ml1cFfC7L5gxKxjwEmAD8NFOCyeAS4EzgXcCXganAeGAY8DbgqpR19U+u6xpgqpmdlPztu8n4TwdGJqf5esq8fYE6YCJwE7CWYJ8MTG6vxguStJQwJA5/MrOdwBpgI3Br6w/u/pC7r3f3hLv/DngZOKvNvI3u/gt3bwHuBwYDA82sFvgn4D/cfb+7zwb+p818HwMecfcZ7t4E3AEcAZzbZpofu/sb7r4OmAM86e7Puvt+4GGC5rNMmoBvunuTuz8K7AJOyjJ9qu+6+w53XwY8D0x399fc/U2CxJW67tbtnAU8AlyZTJz/Cvy7u291953At4GPt5kvAdyanHdvMu7BQF0y9jmuAeYkAyUMicOH3P1IYAxwMm2ajszsU2a2ONlEsp3g7Lpt09KG1jfuvif5thdwLLDN3Xe3mbaxzftj23529wRBwhrSZpo32rzfm+Zzts75Le7e3ObznhzTp8pn3em281iCWkIN8HSb/fe35PetNrn7vjafv09Qw5tuZq+Z2c15xCydjBKGxCZ5dnwfwdk+ZlYH/AL4LNDP3XsTnG1bhkW09TrQx8x6tvmuts379QTNMCTXZQTNPesK34LQdhMU5K0GFbm8dNu5HthMkFxOdffeydfRyQsMWh1Se3D3ne5+k7sfR9AM90Uzu7jI+KSDUsKQuP0IGGtmpwM9CQq0TQDJjuO3hVmIuzcCi4BvmFk3MzufoABs9XvgA2Z2sZl1JWi73w/Mj2g7slkMfMTMasxsJEEfR7Fat/MC4FLgoWSt6RfAD83sGAAzG2Jm7820EDO71MxGJhPoDqAl+RI5jBKGxMrdNwEPELTJLwd+ADxB0CRzGjAvj8V9gqBTfCtBv8gDbdbzInA18GOCM/HLCC7vPRDBZuTyQ+AAwTbdD9QXubwNwDaCWkU9cIO7v5D87SsETUwLzGwH8BjZ+1JOSE6zi2C/3+XuM4uMTzooU/+WiIiEoRqGiIiEooQhIiKhKGGIiEgoShgiIhJKuxp8rH///j58+PC4wxARaVeefvrpze4+IPeU2bWrhDF8+HAWLVoUdxgiIu2KmTXmnio3NUmJiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISSskThpn9ysw2mtnzaX77v2bmZta/1HGIiEhxylHDuA+4JPVLMxsGjAVWlyEGEREpUskThrvPJnjGcqofAl8G9IxYEZF2IJY+DDP7ILDO3ZeEmHaimS0ys0WbNm0qQ3QiIpJO2ROGmdUAk4Gvh5ne3ae6+2h3Hz1gQNHDuYuISIHiqGEcD4wAlpjZKmAo8IyZDYohFhERCansD1By96XAMa2fk0ljtLtvLncsIiISXjkuq/0t8ARwkpmtNbMJpV6niIhEr+Q1DHe/Ksfvw0sdg4iIFE93eouISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhlDxhmNmvzGyjmT3f5rvvm9kLZvacmT1sZr1LHYeIiBSnHDWM+4BLUr6bAbzN3d8OvATcUoY4RESkCCVPGO4+G9ia8t10d29OflwADC11HCIi7Vp9PQwfDlVVwb/19WUPoUvZ13i4zwC/y/SjmU0EJgLU1taWKyYRkcpRXw8TJ8KePcHnxsbgM8D48WULI9ZObzObDDQDGVOlu09199HuPnrAgAHlC05EpFJMnvxWsmi1Z0/wfRnFVsMws2uAS4GL3d3jikNEpOKtXp3f9yUSSw3DzC4BvgJ80N335JpeRKRDCtsvkak5vszN9OW4rPa3wBPASWa21swmAD8BjgRmmNliM7u71HGIiGQUR4dya79EYyO4v9UvkW7dU6ZATc2h39XUBN+XkbWn1qDRo0f7okWL4g5DRDqS1A5lCArjqVNL26E8fHiQJFLV1cGqVYd/X18f9FmsXh3ULKZMCR2fmT3t7qOLihclDBHp7PItuKNSVRXULFKZQSIR6aqiShgaGkREOre4OpQrpF8iH0oYItK5xVVwV0i/RD6UMESkc4ur4B4/PugnqasLmqHq6krfb1KkSrjTW0QkPq0FdIEdykWvu4ITRColDBGRdlZwx0VNUiIi7VWZ7x9RDUNEpD2KYUBC1TBEpOOqgCHBSyaGAQlVwxCRjqlChgQvmRjuH1ENQ0Q6pnKegcdRk4nh/hElDBHpmMp1Bp5uEMGrr4b+/UubOGK4f0QJQ0Q6pnKdgaeryQBs2ZJ59NkoxHDjnxKGiHRM5ToDz1ZjKfVT8caPDwZITCSCf0vcN6OEISIdU7nOwHPVWMr8VLxSUsIQkY6rHGfg6WoybVXw6LP5UsIQESlGa02mX7/Df6vw0WfzpYQhIlKs8eNh82aYNi1zE1iUl97GdEOiEoaISBhhCulMTWD5PL87TBwhluUJZ9mfX+FHH56V/zoy0CNaRURyKfa531E+BjbLsjY+spDH7nqJ6X93Zqw8nvWJwckf28kzvc3sV8ClwEZ3f1vyu77A74DhwCrgSnfflmtZShgiEotiC/won9+dYVkJjGqCZfW1rYwd+gJjL2ph7PXHUXfu0HbzTO/7gEtSvrsZ+Ie7nwD8I/lZRKQyFXvXeLE3ESabw7yqCrf0xfZGG8S3x81k4f3L2bjvaB5cfS4T7ruA2nOGhFtHCCVPGO4+G9ia8vXlwP3J9/cDHyp1HCIiBSu2wC/iJsJt376L5msnQGMj5o4lWkitX7gZg274ELf8fQyjPzWK6m7V4eLKU1yd3gPd/XWA5L/HZJrQzCaa2SIzW7Rp06ayBSgiclCxd43nuomwTYd6YlgtSz56OzeNnsnbj3iJNyd/jy7N+w9ZnKUs3tzh/vtLfrVUxV8l5e5T3X20u48eMGBA3OGISGdUyF3jqVdVQdorqBK/nkbLZ647eNVT1do1jPzDd9j89CqOOWIXdYRs9ir1MCTE9zyMN8xssLu/bmaDgY0xxSEiEk4+z/3O8SyOdYteZ8bdrzL9MeO7jbcwjH2HzN6TPdw79OtUrVkN/fsGAxmGUeJhSOJKGH8BrgG+k/z3zzHFISISvQzP4tj66Zu4YMLZLN8/EhjMwKqNDGVd2kVUrVub/3pLPAxJyZukzOy3wBPASWa21swmECSKsWb2MjA2+VlEpDAV9ihWz3Cm37tpI8N6beeOS2fy3B9e4vWmAVhdjg71ranXDGVQhmFISl7DcPerMvx0canXLSKdQIU8inX1E+uY8fPXmP54NXf4EIaRpoYwdCh/W5NyO8SUKelvCmwt/Gtr098D0q8f9OoVNEPV1gbTa3hzEZEsSvko1iw1l53rd/I///EU//b2WZzc/TXqzh3CdfdfwNx1I3j0mE/T3KX7ocuqqaHqO/95+Dpydai///3pY7vyyrI+CwM0NIiItHdR3kXdVprhQFq69eDPI2/iR2s/yhM7RtFMV2rYzZgByxl73h7GfXoIp1x6PFZlwfyTJxdfA4hgWBGzdjI0SJSUMETkMFGO0xRiuauo5aM1f2XsOzYy7srenPOZU+h+VPfD549KBAkxqoQR11VSIiLRyNUHkIc3V7/J4z9dwYxHDvCTxtVp2+zrbA0Ld48CRhUccl4y9WHE8GAm9WGISPtWxKNYm/c1M//nS7ltzEzOO+o5+tX15CPfexe/XvZONlcNTDuPlbugLtezyUNQwhDp6CrsktOSCPkoVk84r/yjkbs+PosPH7uAfkfs5rwbTuP2WRfQkqjilvPnMvvHS9i6uwfHPHBHZRTU5Xo2eQjqwxDpyIp9jkMHsG3ldv7xkxXMeLSJ6a+MYFXzMACGd1nDuJErGXdpVy668RT6jOh9+MxRdVzHTJ3eIpJbKTqEK7wQPbDrAAvuXcGM329j+rP9WbT7FBJUcxRvctHgFYy78ABjr6vj+H+uDa5m6gTU6S0iuRX7HIdUFXKTXFuecF76+0qm/3INM+b2oOGNUeziHVTTzNlHLuc/LpzDuKv6cdY1p9Clx7tiibGjUB+GSKUoRV9Dsc9xSFXKm+TysPnFLfzu8/O57qQ51HVbz8nvP47P/fFCVmwdyCdPfZaHb36SLY27mbfj7dw2cwznXn8aXXpU0PlxVH/rcvdPuXu7eZ155pku0iFNm+ZeU+MeXHEfvGpqgu+jXq6Z+8UXu9fVBe/r6sKvx+zQZbV9FRtrFvve3OeP/+AZv+WcBj+zZpkbLQ7uvW2bXzFkvv98/Cx/bdbqIIZCtqsY+a4zqr91HssBFnkEZXDsSSCflxKGdFh1dekL4bq64pc9aVL2gj6fAqtfv+KXEUKiJeHP/+ll/+GHZvr7BjzlNexycO/CAb/gqMV++8UNvuCepd68v/mtmUqVdLMpZJ1R/a3zWE5UCUOd3iKVoFTDW0Dmju9U/frB5s3Zp+nfP/uzGYroTN+4bBOP3fUSM6YnmP7aSNYnBgNwUrfXGHvSGsZdfgRjJp3CkccemX4BpbrjO5tC1hnV3zqP5ajTW6QjKeXdvGE7uLdsCdrAs3Ve5xpqO4/O9H3b9zF36nJm/HEH05cOYvHek4EB9LWtvGfoC4y76BXGXn8cteccBxyXe4FRd/CHUcg6o/pbx3AHuDq9RSpBKe/mzacAydV5nWtZWX73hPPcH17iB5fN5L39F9GnjzP2K2fww6fOpXf3vXx73EwW3r+cjfuO5nerz2XCfRdQe86Q8LEX2sFfTMdxIeuM6m8dxx3gUbRrleulPgzp0ErVYZuunT3Ty6zwZaVpu1//7AZ/4Po5fvWIOT6oasPBSUd1f9m/8M6Z/shtT/nO13eWbjtz9ScU2+9R6PxR/a1DLgd1eotIaKkFS69e6Qv9MB2vrcsC9+rqt+abNs13b9rtf/vWQv/imQ1+Wo8XDy52gG30q+rm+r0T5viap9aXbztzFcRRdEDHcWVWnqJKGOr0FunoWu/MbmyE6mpoaQk6uHfuhAMH3pqugCFDEs0Jljz0EjN+vYHpC45i7rZR7KcH3dnH+X2WM+5dOxj7yUG846MnUtWlAlvAS3mxQQVRp7eI5JZ6Z3ZLS/Dvli3QtWuQOLZuzWuIj3WLXmfG3a8y/THjsdUnsslPBk7mtB4v8dkzFzD2w7244PpR1PQ/o3TbFZUKGjq8PVDCEOnI0t2Z3aqpKXgmdI5LaXdv3M2sny1n+p/2MGPFEJbvHwkMZmDVRt47/EXGjXuR99xwAoNPPxE4MfJNKKkIn6XRGcSaMMzs34HrAAeWAp92931xxiTSoeS6pDTN7y0HWnj2wReZPm0jMxYezbztp9LEP9GDvby73zI+M3YmY685ltM+cgJWdUyJAi+T1hpVBQ+mWEli68MwsyHAXGCUu+81s98Dj7r7fZnmUR+GSJ5y3bSXvMFs9RPrmPHz15j+eDWPrT2Zrd4XgNOPeIFxp21g7BVHcf7EUfTo3aM8cUtxUkYUHtDYuHKTe4ibWbKLu0mqC3CEmTUBNcD6mOMR6VjSNbkkNXftwbSWq/lO99d48cBxwBCOrXqdDx6/jLHjqnjPjSdyzKlB/4S0I2lGFK6FuigWHdtlC+6+DrgDWA28Drzp7tNTpzOziWa2yMwWbdq0qdxhirRvyae1eW0tDrRQRQJopJZPNd3D/1l7C8cdvYUffmgWy/78CmubBnHvyxfwiZ+exzGnDsi+7M7wJL/2KE2/lUVU1seWMMysD3A5MAI4FuhpZlenTufuU919tLuPHjAgxwEsUkrtrIBcOXsNPx8/m3/5ynH0XbOEKpyuNHFWzTJ+fs79/OsPRrH1zS48uvGf+MLDFzLqgyPDP1Co9Sy2sTG4LLX1uRip+6Sd7bMOoZRDoURxM0chL+CjwC/bfP4UcFe2eXTjnsQm7jt6Q9jeuN0fvnmBTzp1lo/suvJgmMOq1/qEE2f7g5+b55te2BzNysLc8BbH6LFxqaSb99L8bc4E9yjK7SgWUtCK4WxgGUHfhQH3A/+WbR4lDIlNIXcEl7jAbNrb5PPufs5vvbDBzz1yiVfT5ODeix1+2cAFfucVM/2FR1/1REsikvUdItNw6W2HFinlkO2VpNISY5p4zoAWj6DcjvVObzP7BvAxoBl4FrjO3fdnml5XSUls8r0juL4errnmrRvl2ipwuG1POK82rGb6L1YxY3Z3Hn/9FHZwNFW0MLrnCsa9czPjPtaHs689hW69uuW9/LyEGda7k9xFHcuw6rmU6CopDQ0iEkY+hULqVSqp8igwt63czj9+soIZjzYx/ZURrGoeFoTTZQ3jRq5k7Pu7ctGNJ9P3+D7htyUK6bYxdWiRSixIS6EdJMaohgYJ03T0GPCOKKozxb7UJCWxyafZIVNTTIgmmf079/usOxf7185v8LN6LvUqmh3cj2K7f2jwE/7Tj830lx9bVZpmpnzlarevtKaaUmkHTW+Ua7Ra4AzgceBeYHAUKy30pYTRSVRSB2JbYePK9jjUlAIz0ZLwFx591e+8YqZfNnCB92JHMAgsTX7ukUv81gsbfN7dz3nT3qaybGLkKvVvGaV2kBjLljAOTghXAEuAW4Ejolh5vi8ljE4g3X++bt2CZ0m3l0In0xlndbX7tGm+6YXN/uDn5vmEE2f7sOq1B38+vssqn3TqLH/45gW+vXF73Fsh+ajwxFjWhEFwFdPbgBuAzcBa4JNRBJDPSwmjE8jVnBPF2Vup/3OnSXrNXXv4gyMn+5k1y9xocXDvbdv8iiHz/e5PzPJXGxqjjaE1jrgLsUqIQcraJDWXYMiOGcDtwKXASODHwNQoggj7UsLoBLI150TRPhxF80GOQjDRkvDV//4Df/OIgd6CeSPD/CqmeRcO+AVHLfZvXtTgC+5Z6s37mwvbhrAxxt1MUgkxiLuXN2G8jeTVVGl+WxFFEGFfShglMmnSW09Oq64OPsclTA0j9Xr/VNkK9GI7KDMUgtu/8zOvv3GuXztyth9btf7gTyd1e9U/e9pM/8vXnvQd63YUvFvyVgkdsZUQg7h7GRNG1pnhuCiCCPtSwiiBSZPS/6eOK2mEff50pkIn11ltmBvOsslQCK6kzsG9r23xK4fN83uume2N89dGsUcKU+x2lisGNVmVRUUkjHK/lDBKoLVmka6DNi6TJuV1pdEhcp3VFnjWm2hJ+JKHXvQE6eNKYL7w/mXRNjMVU5hWwtl9rhjUZFU2ShgSjWxn8XHJ1iyVq+DMdVabRyG1/tkN/sD1c/zqEXN8UNUGB/eVZIgt6oK42MK0EgrjXDFUQlLrJJQwJBqVWMMopjkl7KB4ac7cd2/a7X/71kK/6cwGP63HiwdnHWAb/aq6uX7vhDm++Zs/iaYgzlV7iKIwrYTmnmwxVEKzWSehhCHRqLQ+DPfiCss8zqxbmlr8md+s8O++r8Ev7vO0f4p7fSV13oL5hqpB/pe3T/ZnfrPCW5paDl9HMQVxmBg7Q2GqGkbZKGFIdCrpKin3aJpjMhToaxeu93snzPGr6ub6ANt4cPFf7nKH77fuha8zH2EKys5QmFZCs1knoYQh8SrHzW8RLH/XG7v8kdue8s+fPtNHdX/5YLk0sOoNv3rEHH/g+jm+/tkN5S2gw1491BkK00poNusEokoYGq1W8hdmpNKYtBxo4dkHX2T6tI3MWHg087afShPd6MFe3t1vGePO2cXYa47ltI+ccOjT5co54mjYUVxThqhmypTY96+0T2UbrbaSXqphVIhSn43nedbZOH+t33PNbL9y2Dzva1sOhnP6ESv8y2c1+IzvPu17t+2Nd5va6iy1B6kYqElKYlPKDtkQhemOdTv8L1970j972kw/qdurByc7tmq9X3P8HK+/ca5vWLox8vUWvD3pkp+aYqSMokoYapKS/JXywTgZlr2/72C+d/pvmLGoD0/sGEUzXalhNxcOWM648/Yw9tohjLrs+EObmfJ1441Bs1pLC1RXB81ud91V+PIquOlOOhc1SUl4UZ/NlrJJJUPtpQVzo8XPrFnmt5zT4I//4Bnf9+a+4tfXqhTb1BmudJJ2ATVJSSjlbmop0PbG7f7wzQt8c9eBaQvZXUcO9E0vbC4u5mxKUbh3hnsppF2IKmFUFV1FkdKqrw+aaaqqgn/r6/Obf/Lkw58tvWdP8H0xxo8Pmp8SieDfPJtYmvc1M//nS7ltzEzOO+o5+tX15MPfOZsvNU1hHz0Onbimhp4/+wH9T+pXXMzZrF4d/vuwf5Pa2vy+F6l0UWSdQl9Ab+APwAvACuCcbNN3uhpGFLWDCjnLTbQk/OXHVvlPPzbTPzT4CT+K7Q7uVTT7WT2X+tfOb/BZdy72/Tv3x9MhHLaGkc/fRFdDSYWgIzRJAfcD1yXfdwN6Z5u+ohJGOQq1KJpJYmxH3/raNn/oi/N94smzfHiX1QdXPbzLav/F4K/5riMHeqJSrhLKNKx6v37FPU9DV0NJBWj3CQM4ClhJhoczpXuVLGHk+5+6XGeOUdQOyhFrcv8lzHxvn8H++xMn+9k9l3oVzQ7uR7HdLx+0wH/6sZn+8mOrPPHrCGOKskCeNi1IEKn7O8rnaYjEoCMkjNOBp4D7gGeBe4CeaaabCCwCFtXW1ka8G72wArVcZ+1RradEZ7mJloSv+9J/+YHqQ8dg2kWNf737f/qtFzb43LuW+IHdB0q3XeW+sklXPkk71BESxmigGTg7+fn/Abdnm6ckNYxCCoBynWVWYBv4phc2+4Ofm+cTTpztw6rXFvZ8iHz3X6aEF8eVTRX4NxHJpSMkjEHAqjafLwAeyTZPSRJGIYV/uYeRiKsNfNo0Twyr9QTmW7sN9K92/a4bLQ7uvW2bXzFkfsYn0EW2/7IV0KVI3EU8T0OkUrX7hBFsA3OAk5LvbwO+n236iqlhdOCzzERLwp//08v+6JmTfS89DtnGPfTwP46a7AvuWepNe5uCGUq9/7ItvxSJuwP/baXz6igJ4/Rk/8RzwJ+APtmmL0nCKLSA6EBnmW88v9Hrb5zr146c7cdWrXfI41Gkpd5/2WoR7eSmRJG4dYiEke+rYq6SimoZbefp1y94laGQ2rttr8/47tP+5bMa/PQjVhwsa/vaFr9y2Dz/xadmB5e7hm3uKWUBm6sWocJdJCcljEpSyJlupuv+S9AMkmhJ+JKHXvQ7Lm3wcf0Weg/2OLh3Zb+P6f2MTxnb4AvvX+bN+5vfmqlSrgZSE5FI0ZQwClGqs9FCCtdM8xRSOKfZrvXPbvAHrp/jV4+Y44OqNhxc5CndXvHPnz7TH7ntKd/5+s7sy6yUglq1CJGiKGHkK4YRVrNerZNpnnyv9kmzXXvo4VcxzcG9v23yq+rm+q8+PdvXPLU+v+1SQS3SIShh5KuQIR3a3vWbOkREMct2T39HcR7ztzS1+DO/WeHbe6Qf3XV7j4H+zG9WeEtTS7j90xkpIUonoYSRr3w7cbt2PXzabt2iGWRu2rRgWdmSRZr51y5c7/dOmONX1c31AbbRwb2lkPsgChXXxQGlUElNbiIlpoSRr3xqAdn6F6IYZC7T8quqDpl/1xu7/JHbnvLPnz7TR3V/+eBkA6ve8KtHzPEHrp/jTYOH5hdnNtm2IYoCtpIK6Urp1BcpAyWMfOVTWGXrX4jizD3DshPgC+9f5t8e1+Bjej/jXdnv4N6DPT6u30L//gcafMlDL3qiJZF9u3I1oRWyf9r5yLmH0SCC0okoYRQibC0gWw2jX7/iYnB3r65Ou+wmqg9+vLnr93xrt4GewLxl6LDcl+jmGmU1l0zb3K9f9v2RTwFbaCFdimasSkpeIiWmhFFKmfowIPi+iAJrx7odnshSw6i/ca5v/87Pyj+CbpirtnItP1fBXknDsFRS85hIiSlhlFq2we3yOAtt3t/sC+5Z6t+8qMEvOGqxd+FA7mE34hhBN8x9IdlqMGEK4EobSr5SOuBFSkwJI1+FFA4FFsKvzVrtd39ill8xZL73tm3BLLT4mTXL/OZ3Nfjzn/iWJ47IUnDGMYJurjvP0y237T4Mu/58/w7qaxApmhJGPgptfghZCG5v3O4P37zAJ506y0d2XXlwsqHV6/wzJ8z2Bz83zze9sPnwmDIVnIUU/pMmHV64FnIVU9uYMt0rki6OUhXs6msQKZoSRj4KLXQmTUo7X8vEG3ze3c/5rRc2+LlHLvFqmhzce7HDLz3mSb/zipm+4pFXD72aKR+F3NeROr1ZEH8xohqGvFwxiEhaShj5KPTsN0Mh2Miwg81MZ/Vc6pPPa/BZdy72/Tv3FxZfOlHc1xHFPSNhpy1lwa6+BpGiKGHk0raQyXAZa7az362vbcv4NLkE5g99cb5veWVr+HgyxRZFAZjvXezZCvZiYlPBLlKRlDCyCdOBm3L2e2D3AZ/948X+tfMb/OyeS72K5sKeV11IbMWeiUdxF3trAa/mH5EOJ6qEYcGy2ofRo0f7okWLck84fDg0Nh7+fXU1JBJQW4t/awov9TuH6b9cw4y5PWh4YxS7OJIqWji713LGnbmFq0Y8wYm/+xa2d89by6ipgalTYfz4wjYiU2x1dbBqVWHLrK+HiRNhT4g4q6qCVJDKDGpro49NRGJnZk+7++hil9MlimAqzurVab/2RILf/9tcZvythenXHsealiHAcRzfpZGrRz3LuEu78c83nkLvutOSc4yB99TC5MnBMmtrYcqUwpNFltgyfh9Gazxh4syUFGprSxObiHQcUVRTyvUK3SSVodllJbUO7r1tm3/k2Cf87k/M8lcbGsMtM1Wh7fVxXyaardkp7thEpCSIqEmqKu6EFSVPOMv/8gp/7X81++hxyG976cEzoz7JE794nk17evHHde/i+vp3c9yY2vxX1NoE1NgYFKmNjcHn+vrc806ZEjQXtVVTE3xfDuPHB01VdXVBM1Rd3VtNV3HHJiKVLYqsU8wLqAaeBf4317TpahhvPL/R62+c69eOnO3HVq0/eFL8xaof+pauycH7huQYvC9fUdxVXalXE1VybCJSEDpKp7eZfREYDRzl7pdmm3b06NE+97G5zJ26nBl/3MH0pYNYvPdkAPraVi4e8iLjLmpm7MQR1J03tHRBZ+s4TiRKt14RkQJ0iE5vMxsKfACYAnwx1/QvL95Dnz7OPs6gKwc49+jlTDl/JmPHH8MZV51EdbdzSh4zkL3jWESkg4r7KqkfAV8Gjsw0gZlNBCYCdOdUbjj9KcZeXsOFN5xCr0GnlyXIw0yZkv4yVrX1i0gHFlvCMLNLgY3u/rSZjck0nbtPBaZC0CT1o0UXlifAbPK5jFVEpIOI8yqp84APmtkq4EHgIjObFuka6uuDG+WqqoJ/w1zFFNb48cHNbIlE8K+ShYh0cLElDHe/xd2Huvtw4OPA4+5+dWQrKObSVxEROUyHug/jEJMnH9rHAMHnyZPLG0cpazkiImUUd6c3AO4+E5gZ6UIrYZiL1DGeWms5oCYsEWl32l8NI+wZe6ZLXMt56Wul1HJERCLQvhLG1q3h+yUqYZiLSqjliIhEpH0ljHXrwp+xZxszqVwqoZYjIhKR2IcGycdoM0/7NIxKHZIjn+dUiIiUSFRDg7SvGka3bum/r9Qz9kqo5YiIRKQirpIK7eijYfPmQwf+q/QhOcaPV4IQkQ6hfdUwtmw5NFmYwTXXqEAWESmD9pUwUvsp3OHRR+OJRUSkk2lfCSMdXaIqIlIW7T9hVGqHt4hIB9O+EkZVSriV3uEtItKBtK+EUVenS1RFRGLSvi6r7dsXFqW9dU9EREqsfdUwREQkNkoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiIQSW8Iws2Fm1mBmK8xsmZl9Pq5YREQktzhv3GsGbnL3Z8zsSOBpM5vh7stjjElERDKIrYbh7q+7+zPJ9zuBFcCQuOIREZHsKqIPw8yGA+8Enkzz20QzW2RmizZt2lT22EREJBB7wjCzXsAfgS+4+47U3919qruPdvfRAwYMKH+AIiICxJwwzKwrQbKod/f/jjMWERHJLs6rpAz4JbDC3f8rrjhERCScOGsY5wGfBC4ys8XJ1/tjjEdERLKI7bJad58LWFzrFxGR/MTe6S0iIu2DEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKLEmDDO7xMxeNLNXzOzmOGMREZHsYksYZlYN/BR4HzAKuMrMRsUVj4iIZBdnDeMs4BV3f83dDwAPApfHGI+IiGTRJcZ1DwHWtPm8Fjg7dSIzmwhMTH7cb2bPlyG2YvUHNscdRAiKMzrtIUZQnFFrL3GeFMVC4kwYluY7P+wL96nAVAAzW+Tuo0sdWLEUZ7TaQ5ztIUZQnFFrT3FGsZw4m6TWAsPafB4KrI8pFhERySHOhLEQOMHMRphZN+DjwF9ijEdERLKIrUnK3ZvN7LPA34Fq4FfuvizHbFNLH1kkFGe02kOc7SFGUJxR61Rxmvth3QYiIiKH0Z3eIiISihKGiIiEUhEJI9cQIRa4M/n7c2Z2Rth5yxzn+GR8z5nZfDN7R5vfVpnZUjNbHNUlbkXEOcbM3kzGstjMvh523jLH+aU2MT5vZi1m1jf5W1n2p5n9ysw2Zrr/p4KOzVxxVsqxmSvOSjk2c8VZCcfmMDNrMLMVZrbMzD6fZppoj093j/VF0OH9KnAc0A1YAoxKmeb9wF8J7t14F/Bk2HnLHOe5QJ/k+/e1xpn8vAroXyH7cwzwv4XMW844U6a/DHg8hv35buAM4PkMv8d+bIaMM/ZjM2ScsR+bYeKskGNzMHBG8v2RwEulLjsroYYRZoiQy4EHPLAA6G1mg0POW7Y43X2+u29LflxAcG9JuRWzTypqf6a4CvhtiWLJyN1nA1uzTFIJx2bOOCvk2AyzPzOpqP2ZIq5j83V3fyb5fiewgmAEjbYiPT4rIWGkGyIkdaMzTRNm3qjku64JBJm9lQPTzexpC4Y7KZWwcZ5jZkvM7K9mdmqe80Yh9LrMrAa4BPhjm6/LtT9zqYRjM19xHZthxX1shlYpx6aZDQfeCTyZ8lOkx2ecQ4O0CjNESKZpQg0vEpHQ6zKzfyb4T3l+m6/Pc/f1ZnYMMMPMXkiexcQR5zNAnbvvMrP3A38CTgg5b1TyWddlwDx3b3vGV679mUslHJuhxXxshlEJx2Y+Yj82zawXQcL6grvvSP05zSwFH5+VUMMIM0RIpmnKObxIqHWZ2duBe4DL3X1L6/fuvj7570bgYYIqYSxxuvsOd9+VfP8o0NXM+oeZt5xxtvFxUqr8ZdyfuVTCsRlKBRybOVXIsZmPWI9NM+tKkCzq3f2/00wS7fFZ6o6ZEB03XYDXgBG81flyaso0H+DQjpunws5b5jhrgVeAc1O+7wkc2eb9fOCSGOMcxFs3bZ4FrE7u24ran8npjiZoS+4Zx/5MrmM4mTtpYz82Q8YZ+7EZMs7Yj80wcVbCsZncLw8AP8oyTaTHZ+xNUp5hiBAzuyH5+93AowS9/a8Ae4BPZ5s3xji/DvQD7jIzgGYPRrIcCDyc/K4L8Bt3/1uMcf4LMMnMmoG9wMc9OIoqbX8CfBiY7u6728xetv1pZr8luHKnv5mtBW4FuraJMfZjM2ScsR+bIeOM/dgMGSfEfGwC5wGfBJaa2eLkd18lODkoyfGpoUFERCSUSujDEBGRdkAJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBEipB8HsHY5PtvmdmdccckUiqx3+kt0s7dCnwzOdDcO4EPxhyPSMnoTm+RIpnZLKAXMMaD5xKIdEhqkhIpgpmdRvDks/1KFtLRKWGIFCj55LJ6gieV7Taz98YckkhJKWGIFCD5pLX/Bm5y9xXA7cBtsQYlUmLqwxARkVBUwxARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJ5f8Dan8nF6iTRYwAAAAASUVORK5CYII=\n", + "text/plain": [ + "
    " + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2946,9 +2595,7 @@ { "cell_type": "markdown", "id": "c3d4f269", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**Challenge**: try to write a similar code for a Logistic Regression case." ] @@ -2956,9 +2603,7 @@ { "cell_type": "markdown", "id": "21369b43", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2969,11 +2614,1037 @@ "cell_type": "code", "execution_count": 16, "id": "00fc4272", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[3.96113114]\n", + " [3.11062375]]\n", + "Eigenvalues of Hessian Matrix:[0.26820372 4.51917517]\n", + "theta from own gd\n", + "[[3.96113114]\n", + " [3.11062375]]\n", + "65\n", + "25\n", + "55\n", + "0\n", + "50\n", + "15\n", + "20\n", + "40\n", + "45\n", + "90\n", + "35\n", + "40\n", + "20\n", + "75\n", + "85\n", + "80\n", + "0\n", + "0\n", + "20\n", + "30\n", + "30\n", + "95\n", + "95\n", + "75\n", + "95\n", + "15\n", + "10\n", + "80\n", + "75\n", + "25\n", + "35\n", + "60\n", + "10\n", + "20\n", + "40\n", + "40\n", + "25\n", + "90\n", + "50\n", + "70\n", + "70\n", + "85\n", + "60\n", + "75\n", + "0\n", + "20\n", + "15\n", + "90\n", + "75\n", + "25\n", + "80\n", + "15\n", + "65\n", + "70\n", + "90\n", + "70\n", + "90\n", + "30\n", + "80\n", + "15\n", + "15\n", + "30\n", + "5\n", + "15\n", + "40\n", + "50\n", + "20\n", + "20\n", + "5\n", + "85\n", + "15\n", + "70\n", + "80\n", + "85\n", + "40\n", + "70\n", + "75\n", + "30\n", + "60\n", + "80\n", + "50\n", + "25\n", + "75\n", + "80\n", + "5\n", + "70\n", + "55\n", + "85\n", + "90\n", + "0\n", + "75\n", + "30\n", + "45\n", + "35\n", + "20\n", + "15\n", + "55\n", + "50\n", + "0\n", + "80\n", + "10\n", + "10\n", + "95\n", + "25\n", + "20\n", + "20\n", + "80\n", + "20\n", + "10\n", + "85\n", + "65\n", + "90\n", + "25\n", + "70\n", + "95\n", + "35\n", + "65\n", + "10\n", + "85\n", + "15\n", + "85\n", + "65\n", + "50\n", + "70\n", + "5\n", + "10\n", + "75\n", + "5\n", + "15\n", + "20\n", + "75\n", + "50\n", + "70\n", + "20\n", + "50\n", + "65\n", + "50\n", + "90\n", + "40\n", + "90\n", + "75\n", + "55\n", + "40\n", + "15\n", + "0\n", + "0\n", + "90\n", + "5\n", + "95\n", + "95\n", + "95\n", + "80\n", + "60\n", + "35\n", + "50\n", + "40\n", + "70\n", + "75\n", + "75\n", + "60\n", + "80\n", + "5\n", + "70\n", + "70\n", + "80\n", + "5\n", + "85\n", + "5\n", + "25\n", + "50\n", + "25\n", + "40\n", + "70\n", + "70\n", + "95\n", + "90\n", + "55\n", + "45\n", + "45\n", + "15\n", + "0\n", + "0\n", + "10\n", + "70\n", + "70\n", + "95\n", + "15\n", + "85\n", + "85\n", + "70\n", + "50\n", + "45\n", + "5\n", + "70\n", + "30\n", + "75\n", + "35\n", + "50\n", + "35\n", + "45\n", + "75\n", + "65\n", + "35\n", + "65\n", + "55\n", + "75\n", + "0\n", + "95\n", + "65\n", + "55\n", + "0\n", + "35\n", + "50\n", + "45\n", + "85\n", + "15\n", + "30\n", + "25\n", + "60\n", + "50\n", + "60\n", + "10\n", + "85\n", + "5\n", + "35\n", + "30\n", + "75\n", + "55\n", + "55\n", + "60\n", + "90\n", + "15\n", + "15\n", + "85\n", + "15\n", + "15\n", + "85\n", + "25\n", + "95\n", + "15\n", + "60\n", + "10\n", + "5\n", + "95\n", + "20\n", + "70\n", + "40\n", + "55\n", + "95\n", + "50\n", + "45\n", + "80\n", + "80\n", + "45\n", + "75\n", + "40\n", + "65\n", + "25\n", + "55\n", + "90\n", + "5\n", + "0\n", + "75\n", + "75\n", + "60\n", + "30\n", + "70\n", + "5\n", + "20\n", + "80\n", + "90\n", + "0\n", + "10\n", + "55\n", + "25\n", + "45\n", + "80\n", + "25\n", + "10\n", + "45\n", + "40\n", + "35\n", + "70\n", + "55\n", + "10\n", + "35\n", + "60\n", + "5\n", + "20\n", + "80\n", + "60\n", + "90\n", + "75\n", + "0\n", + "85\n", + "90\n", + "60\n", + "65\n", + "20\n", + "95\n", + "30\n", + "65\n", + "0\n", + "45\n", + "45\n", + "35\n", + "65\n", + "80\n", + "35\n", + "5\n", + "15\n", + "70\n", + "25\n", + "40\n", + "55\n", + "15\n", + "80\n", + "25\n", + "85\n", + "45\n", + "45\n", + "65\n", + "65\n", + "85\n", + "85\n", + "15\n", + "60\n", + "0\n", + "40\n", + "60\n", + "45\n", + "40\n", + "30\n", + "30\n", + "25\n", + "50\n", + "75\n", + "95\n", + "35\n", + "50\n", + "85\n", + "55\n", + "90\n", + "10\n", + "70\n", + "10\n", + "25\n", + "10\n", + "75\n", + "80\n", + "35\n", + "75\n", + "60\n", + "60\n", + "20\n", + "5\n", + "85\n", + "95\n", + "50\n", + "80\n", + "30\n", + "60\n", + "90\n", + "95\n", + "15\n", + "90\n", + "25\n", + "20\n", + "20\n", + "75\n", + "20\n", + "80\n", + "40\n", + "45\n", + "65\n", + "55\n", + "80\n", + "90\n", + "75\n", + "40\n", + "40\n", + "25\n", + "75\n", + "50\n", + "85\n", + "60\n", + "0\n", + "5\n", + "80\n", + "90\n", + "60\n", + "25\n", + "55\n", + "40\n", + "60\n", + "65\n", + "65\n", + "75\n", + "45\n", + "0\n", + "75\n", + "30\n", + "10\n", + "75\n", + "85\n", + "35\n", + "85\n", + "5\n", + "85\n", + "65\n", + "50\n", + "95\n", + "65\n", + "15\n", + "75\n", + "60\n", + "70\n", + "50\n", + "60\n", + "25\n", + "40\n", + "85\n", + "70\n", + "40\n", + "35\n", + "10\n", + "20\n", + "90\n", + "65\n", + "50\n", + "35\n", + "60\n", + "60\n", + "30\n", + "75\n", + "35\n", + "10\n", + "0\n", + "65\n", + "10\n", + "25\n", + "75\n", + "25\n", + "95\n", + "0\n", + "65\n", + "60\n", + "30\n", + "85\n", + "0\n", + "50\n", + "5\n", + "20\n", + "5\n", + "35\n", + "0\n", + "70\n", + "65\n", + "95\n", + "0\n", + "10\n", + "15\n", + "85\n", + "35\n", + "25\n", + "10\n", + "70\n", + "75\n", + "80\n", + "45\n", + "25\n", + "0\n", + "90\n", + "40\n", + "45\n", + "40\n", + "85\n", + "90\n", + "40\n", + "65\n", + "85\n", + "15\n", + "70\n", + "40\n", + "10\n", + "15\n", + "15\n", + "35\n", + "95\n", + "55\n", + "75\n", + "0\n", + "5\n", + "60\n", + "50\n", + "75\n", + "35\n", + "20\n", + "30\n", + "80\n", + "30\n", + "20\n", + "55\n", + "85\n", + "90\n", + "80\n", + "70\n", + "60\n", + "80\n", + "85\n", + "90\n", + "15\n", + "0\n", + "65\n", + "0\n", + "75\n", + "80\n", + "0\n", + "35\n", + "5\n", + "20\n", + "45\n", + "90\n", + "75\n", + "35\n", + "35\n", + "10\n", + "25\n", + "85\n", + "30\n", + "90\n", + "30\n", + "20\n", + "90\n", + "75\n", + "0\n", + "35\n", + "15\n", + "5\n", + "90\n", + "80\n", + "20\n", + "40\n", + "80\n", + "75\n", + "5\n", + "15\n", + "80\n", + "25\n", + "25\n", + "65\n", + "80\n", + "20\n", + "5\n", + "75\n", + "15\n", + "50\n", + "20\n", + "65\n", + "5\n", + "95\n", + "95\n", + "55\n", + "15\n", + "10\n", + "10\n", + "95\n", + "70\n", + "70\n", + "0\n", + "65\n", + "35\n", + "80\n", + "0\n", + "50\n", + "75\n", + "20\n", + "30\n", + "15\n", + "25\n", + "55\n", + "70\n", + "25\n", + "70\n", + "35\n", + "15\n", + "40\n", + "45\n", + "55\n", + "15\n", + "55\n", + "45\n", + "0\n", + "10\n", + "75\n", + "50\n", + "65\n", + "60\n", + "20\n", + "75\n", + "40\n", + "90\n", + "55\n", + "35\n", + "25\n", + "50\n", + "95\n", + "10\n", + "20\n", + "0\n", + "10\n", + "25\n", + "45\n", + "0\n", + "90\n", + "30\n", + "50\n", + "5\n", + "20\n", + "25\n", + "40\n", + "5\n", + "70\n", + "95\n", + "35\n", + "15\n", + "20\n", + "90\n", + "50\n", + "15\n", + "30\n", + "0\n", + "75\n", + "30\n", + "75\n", + "90\n", + "0\n", + "55\n", + "95\n", + "45\n", + "20\n", + "50\n", + "30\n", + "95\n", + "60\n", + "75\n", + "30\n", + "50\n", + "85\n", + "50\n", + "15\n", + "80\n", + "25\n", + "20\n", + "95\n", + "30\n", + "55\n", + "50\n", + "95\n", + "55\n", + "50\n", + "20\n", + "30\n", + "40\n", + "85\n", + "90\n", + "85\n", + "60\n", + "55\n", + "85\n", + "75\n", + "15\n", + "75\n", + "60\n", + "70\n", + "20\n", + "60\n", + "90\n", + "0\n", + "30\n", + "85\n", + "80\n", + "5\n", + "30\n", + "85\n", + "65\n", + "35\n", + "95\n", + "80\n", + "55\n", + "50\n", + "45\n", + "75\n", + "25\n", + "10\n", + "55\n", + "95\n", + "0\n", + "30\n", + "75\n", + "80\n", + "85\n", + "90\n", + "95\n", + "50\n", + "15\n", + "90\n", + "35\n", + "15\n", + "50\n", + "45\n", + "25\n", + "30\n", + "0\n", + "45\n", + "10\n", + "45\n", + "15\n", + "0\n", + "50\n", + "75\n", + "40\n", + "50\n", + "0\n", + "10\n", + "5\n", + "15\n", + "55\n", + "50\n", + "30\n", + "0\n", + "75\n", + "85\n", + "10\n", + "80\n", + "70\n", + "75\n", + "5\n", + "95\n", + "10\n", + "5\n", + "40\n", + "20\n", + "45\n", + "60\n", + "95\n", + "95\n", + "30\n", + "60\n", + "25\n", + "40\n", + "5\n", + "0\n", + "30\n", + "80\n", + "35\n", + "75\n", + "60\n", + "45\n", + "25\n", + "80\n", + "70\n", + "25\n", + "55\n", + "15\n", + "75\n", + "70\n", + "0\n", + "20\n", + "25\n", + "25\n", + "60\n", + "70\n", + "25\n", + "70\n", + "25\n", + "20\n", + "90\n", + "15\n", + "30\n", + "55\n", + "65\n", + "15\n", + "95\n", + "0\n", + "50\n", + "95\n", + "25\n", + "15\n", + "50\n", + "20\n", + "80\n", + "45\n", + "25\n", + "65\n", + "35\n", + "15\n", + "75\n", + "15\n", + "85\n", + "10\n", + "55\n", + "25\n", + "70\n", + "15\n", + "15\n", + "65\n", + "75\n", + "95\n", + "10\n", + "70\n", + "55\n", + "0\n", + "90\n", + "60\n", + "0\n", + "10\n", + "55\n", + "10\n", + "65\n", + "95\n", + "45\n", + "5\n", + "55\n", + "25\n", + "15\n", + "50\n", + "20\n", + "95\n", + "15\n", + "80\n", + "25\n", + "70\n", + "60\n", + "30\n", + "70\n", + "75\n", + "55\n", + "80\n", + "45\n", + "10\n", + "80\n", + "80\n", + "15\n", + "20\n", + "25\n", + "0\n", + "35\n", + "5\n", + "60\n", + "0\n", + "40\n", + "25\n", + "55\n", + "55\n", + "45\n", + "30\n", + "35\n", + "30\n", + "65\n", + "80\n", + "15\n", + "95\n", + "55\n", + "0\n", + "75\n", + "75\n", + "80\n", + "15\n", + "70\n", + "85\n", + "40\n", + "85\n", + "45\n", + "45\n", + "40\n", + "40\n", + "85\n", + "65\n", + "80\n", + "75\n", + "50\n", + "25\n", + "95\n", + "55\n", + "65\n", + "60\n", + "30\n", + "30\n", + "5\n", + "95\n", + "40\n", + "30\n", + "50\n", + "45\n", + "55\n", + "70\n", + "60\n", + "45\n", + "80\n", + "95\n", + "95\n", + "65\n", + "75\n", + "85\n", + "5\n", + "80\n", + "30\n", + "60\n", + "70\n", + "85\n", + "85\n", + "15\n", + "75\n", + "45\n", + "70\n", + "55\n", + "15\n", + "5\n", + "75\n", + "0\n", + "55\n", + "10\n", + "40\n", + "85\n", + "45\n", + "45\n", + "50\n", + "70\n", + "40\n", + "70\n", + "15\n", + "60\n", + "15\n", + "65\n", + "95\n", + "15\n", + "45\n", + "30\n", + "5\n", + "55\n", + "85\n", + "40\n", + "85\n", + "65\n", + "20\n", + "25\n", + "45\n", + "70\n", + "75\n", + "45\n", + "45\n", + "75\n", + "30\n", + "45\n", + "20\n", + "40\n", + "95\n", + "15\n", + "20\n", + "85\n", + "20\n", + "15\n", + "80\n", + "5\n", + "45\n", + "85\n", + "10\n", + "5\n", + "45\n", + "0\n", + "80\n", + "40\n", + "45\n", + "65\n", + "25\n", + "70\n", + "70\n", + "15\n", + "40\n", + "25\n", + "15\n", + "5\n", + "20\n", + "5\n", + "45\n", + "65\n", + "35\n", + "20\n", + "50\n", + "65\n", + "55\n", + "25\n", + "10\n", + "60\n", + "15\n", + "25\n", + "85\n", + "70\n", + "30\n", + "theta from own sdg\n", + "[[3.97500896]\n", + " [3.09893656]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqW0lEQVR4nO3dd3wc9Z3/8ddHtlyECdgyLtiWjIkxJrSAKQkmMSHiCCX1LoGYUELOiXnkUkgjcQLJcU69+6VdCGcgoViBS3IpF5JcEEEu2DSZ6kIzLhj3BrZcJX1+f8zKXq23zGpnd3al9/PxmAfa3SmfGQ/zmW+Z75i7IyIikktV3AGIiEhlUMIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMKTHMrNvmtnsuOOIkplNMbM1ccchvZMShpSUma00s91mttPM1pvZnWY2KO64RCQ3JQyJw6XuPgg4FXgr8NV4w+mdzKxv3DFIZVHCkNi4+3rgbwSJAwAzu8HMlpvZDjNbamYfSPrtajN72Mz+3cy2mdkKM3tP0u/HmNncxLJNwNDk7ZnZe81siZltN7M5ZjYx6beVZvYlM3vWzFrN7A4zG25mf02s70EzG5xuPzqriczsC2a20czWmdk1Sb/PMbNPpO5H0mc3s+vM7KXEtm42s2PN7BEze8PMfm1m/VK2+TUz25yIe2rS9/0Tx2e1mW0ws1vNbGBKnF8xs/XAL81sqJndnzgmW81svpnpuiBp6cSQ2JjZaOA9wMtJXy8HzgWOAL4FzDazkUm/nwW8QJAMvg/cYWaW+O1XwKLEbzcDVyVt6zjgXuBzwFHAX4A/pVyIPwQ0AMcBlwJ/Bb6WWF8V8JksuzMiEfMo4FrgZ5kSTAYXAqcDZwNfBmYBU4ExwInA5SnbGprY1lXALDObkPjte4n4TwXenJjnxpRlhwD1wDTgC8AagmMyPLG/Gi9I0lLCkDj8wcx2AK8CG4GbOn9w99+4+1p373D3/wZeAs5MWnaVu9/m7u3AXcBIYLiZ1QFnAN9w973uPg/4U9JyHwH+7O5N7r4f+HdgIPD2pHl+6u4b3P01YD7wmLs/5e57gd8TVJ9lsh/4V3ff7+5/AXYCE7LMn+p77v6Guy8BFgMPuPsr7v46QeJK3Xbnfs4F/gx8OJE4/xn4vLtvdfcdwLeBy5KW6wBuSiy7OxH3SKA+Eft81wBzkoEShsTh/e5+ODAFOJ6kqiMzu9LMnk5UkWwnuLtOrlpa3/mHu+9K/DkIOBrY5u6tSfOuSvr76OTP7t5BkLBGJc2zIenv3Wk+Z2uc3+LubUmfd+WYP1U+2063n0cTlBJqgEVJx+//Et932uTue5I+/4CghPeAmb1iZjfkEbP0MkoYEpvE3fGdBHf7mFk9cBvwaaDW3Y8kuNu2DKtItg4YbGaHJX1Xl/T3WoJqGBLbMoLqnte6vwehtRJcyDuNKHB96fZzLbCZILm8xd2PTExHJDoYdOpSenD3He7+BXcfR1ANd72ZnV9gfNJDKWFI3H4ENJjZqcBhBBe0TQCJhuMTw6zE3VcBLcC3zKyfmU0muAB2+jVwsZmdb2bVBHX3e4GFEe1HNk8DHzSzGjN7M0EbR6E69/Nc4BLgN4lS023AD81sGICZjTKzf8i0EjO7xMzenEigbwDtiUnkEEoYEit33wTcTVAnvxT4D+ARgiqZk4AFeazuowSN4lsJ2kXuTtrOC8AVwE8J7sQvJejeuy+C3cjlh8A+gn26C2gscH3rgW0EpYpG4FPu/nzit68QVDE9amZvAA+SvS1lfGKenQTH/RZ3n1NgfNJDmdq3REQkDJUwREQkFCUMEREJRQlDRERCUcIQEZFQKmrwsaFDh/rYsWPjDkNEpKIsWrRos7sflXvO7CoqYYwdO5aWlpa4wxARqShmtir3XLmpSkpEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQip4wzOwXZrbRzBan+e2LZuZmNrTYcYiISGFKUcK4E7gw9UszGwM0AKtLEIOIiBSo6AnD3ecRvGM51Q+BLwN6R6yISAWIpQ3DzN4LvObuz4SYd5qZtZhZy6ZNm0oQnYiIpFPyhGFmNcAM4MYw87v7LHef5O6Tjjqq4OHcRUSkm+IoYRwLHAM8Y2YrgdHAk2Y2IoZYREQkpJK/QMndnwOGdX5OJI1J7r651LGIiEh4pehWey/wCDDBzNaY2bXF3qaIiESv6CUMd788x+9jix2DiIgUTk96i4hIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEUPWGY2S/MbKOZLU767gdm9ryZPWtmvzezI4sdh4iIFKYUJYw7gQtTvmsCTnT3k4EXga+WIA4RESlA0ROGu88DtqZ894C7tyU+PgqMLnYcIiJSmHJow/g48NdMP5rZNDNrMbOWTZs2lTAsERFJFmvCMLMZQBvQmGked5/l7pPcfdJRRx1VuuBERKSLvnFt2MyuAi4Bznd3jysOEREJJ5aEYWYXAl8B3unuu+KIQURE8lOKbrX3Ao8AE8xsjZldC/wncDjQZGZPm9mtxY5DREQKU/QShrtfnubrO4q9XRERiVY59JISEZEoNDbC2LFQVRX8tzFjf6Juia3RW0REItTYCNOmwa5Es/CqVcHnCKmEISLSE8yYcTBZdNq1K/g+IkoYIiJhFLm6p2CrV+f3fTcoYYiI5NJZ3bNqFbgfrO4pp6RRV5ff992ghCEikksJqnsKNnMm1NR0/a6mJvg+IkoYIiK5lKC6J5Rs1WJTp8KsWVBfD2bBf2fNCr6PiHpJiYjkUlcXVEOl+75UsvWC6kwKU6dGmiBSqYQhIpJLCap7ciqDajElDBGRXEpQ3ZNTGVSLqUpKRCSMIlf35FQG1WIqYYiIVIIyqBZTwhARKUepPaIgfLVYyrJDYUgUIalKSkSk3GTqETVrFqxcmfeydVAfRVgqYYiIlJtCekSlWdYiutYrYYhI5Sv3cZ7yVUiPqCL2mlLCEJHKVgnjPOWrkHGhithrSglDRCpbHA+0JZdohg4NpihLN4X0iEqzrENH4UEpYYhI1EpdPVTqB9pSSzRbtgRTlKWbQh4UTLPsakjzAEf+zN2jWE9JTJo0yVtaWuIOQ0QySe2hA8HdbjGfih47Nv0DbfX1uXsURbm9Umw7D/t27mPh7Ut58LfbmbngvEXuPqnQdRa9hGFmvzCzjWa2OOm7IWbWZGYvJf47uNhxiEgJxFE9VOoH2mJueM7EO5zFv3+JH75/DhcNe4LBh+/nvM+fyncXTI5sG6WokroTuDDluxuAv7v7eODvic8iUuniGO+o1OM8xdzwnGzd0xu451MPc+WxDzOqegMnfXA81/9xCsu3D+Wak1r4w1cfY8uq1si2V5IqKTMbC9zv7icmPr8ATHH3dWY2Epjj7hNyrUdVUiJlrtTVQ3FIV+2WavbsoiSs1o2tzLt1KU1/bKVpySgW7x0PQK1t4d2jX6DhXe00fHIcdW8b1WU5M4ukSgp3L/oEjAUWJ33envL7tizLTgNagJa6ujoXKRuzZ7vX17ubBf+dPTvuiOI3e7Z7TY170AQcTDU1Pe/YzJ7t3qdP1/3snGprI9tM2942f/zOJT6zodmnHPmkf4w7fQX13o75+qoR/seTZ/ii2Uu9fX971vUALR7FtTyKleTcSAEJI3k6/fTT8zvaIsXSWy6M3dFbEmmRzoHlzav8v6bO9X8ctdAH29YDq76h7w98b1X/bm0vqoShKimR7ugNVS+SW2Nj0KC/enXQbjFzZt5VUdtWbOehW5bR9Of9NL1UzyttwbBPo6rW0TDuZRouqOL86ccx/JIzun3ORVUlFVfC+AGwxd2/a2Y3AEPc/cu51qOEIWWjqiq4x0tlBh2RPCMlPdS+nft45I6lNP1mO01PD6WldSId9GEQOzhv+FLefc4eGq4ZzfEXjcOq7OCC+Z5zScnsFPd9z7j3LzT2oo9Wa2b3AlOAoWa2BrgJ+C7wazO7FlgN/FOx4xCJVBm8zEYqg3c4S/+0nKY7X6NpQQ1zN02klVPpQxtnDlrG198xn4aPDOGsqydSXXNW5hXlc86lNMxXQ79odqYEbRhRTWrDkLKhNoxAb2mvyNO6Zzb4PZ962K8cN9+Prlp74BQZX/2KX3fiHP/9DY/69lXb81tpPudcfX2X+U4H9yjao6NYSakmJQwpK/lcLHvihVVJ84DWTa3+15uf8OtPb/aTBrxwsMOUbfYPj1ngt105z1c+/Gr4FWY6X8KeR2ZFSRgaGkSk2OIYLqMUenHDf/u+dp667wWaGjfS9PgRLNh+AvvoTz/2MnnwEhrOfIOGK4bz1ssmUNU3z+ejozhfhg4NxrdKmAS0uFvmBcJRwhAptp56Ye1uw38EPYvisGLeqzTNWsGDc6v5+2sT2OrBW09PHvACDW9ZR8MHBnHuJ0+gZmhNjjXlUOj50tgI11wD+/cf+CqqhKFXtIoUWxzDZZRCdxr+M716FMouaWxf9ToP/WwZTX/eR9OL9SxvqwfGcHTVOi4dt5SGC4x3X3ccw0+cAOR8KiC8Qs+XGTO6JIsoqYQhUmw9tYTRnaqTMj4W+3bu49FfLqPp19toemooTyR1d50ybBkNk3en7+4atUKPUZqSn0oYIpVi5sz0F9ZijaZaKp1JIZ/qpTIqbXmHs+z+zu6uA5mz8QRaOYUq2jlz0FJmnDufhg8P5qyrJ9Jv0JmlC6zQ8yVTyS8KUbScl2pSLympWMXoJVWJPa9SunsemOrrS7L59c9t9NnTH/arjp3vo5K6u765eoVPf8tc/92XH/FtK7eXJJasCvm3TdN7LapeUiphiJTC1Km56+jzaQyuoLaALkpc2tq1eRfz/2spTb/fSdOSkTy7ZwJwFENsK+ePeoGG816iYdo4xk4eSzDkXQ/Q+e//2c926SkViSiyTqkmlTCkx8r3mYaY79RDS3ennOvuuYC76/b97d5yz1L/zj80+7sGL/J+7HFw78ceP+/IJ/3bFzT7E3ct8ba9bZHtYuSier4l6RzRg3siPUm+CSDlwawDk1kpoz4oU2LI98LXjWVWzH/VZ31srn94zAKvtc0HFjtpwAt+/enN/tebn/DWTa2R73LRRHUzkHSO6ME9kZ4k32caouptFMUzEZl6Sw0cmL5KJFuMIfZr+6rXab6ls7trHS/vHwvA0VXraDhmOQ0XGOd/ajwjTh6W336Ui6gGtkw6llH1koq91JDPpBKG9Fj53lVGUW1RhKqPUFO2UlCGklMH5t84t9nPHvSsV9Hm4H4YO/ziYY/5jz4wx5f88SXvaO/IL+5yFVUJI+nfV1VSIj1Juot3v37B29uKUNfv7kWp+gg1ZVt/hphWUO9VtPlZhz3nX5/c7HN/8rTv3bE3vzgrpVdZlGN0JfZZCUOkHER5EUpeV22te3V1NBeNTKJqB8mUeGprQ1/4Nize6I3XPey3DvuGtzKwyzJ7rL8/dvE3fesr27q/r5U2UGLEyY1KekVrVJMShpSVYl6EStELqghVH4cchwwXvl1bdvnfvt3iX5zU7KcMeP7AYoNtq//oyJv8jZrh3hFlSSCOXmVlVKJRwhCJWzEvQqXoBVWEqo9MF8fU7q792e3gXs1en3Lkkz6zodkfv7OI3V1L3auszEo0USUM9ZIS6a5ivqa1VGMuFXHk2FUL1tA0awVNzX34+5oJbPFaAE4a8CINb1lLw/sHce60iRw27LBItpdVqcewKrMxs6J6p7ee9BbprmK+prVUT0SHeQI9pNdXv07zz5+n6f49NL1Qx0v7jwFGM7JqPRcfsywY3XX6eEacfBxwXCTbDO2ii+DWW7sm+GKO51VGY2ZFKopiSqkmVUlJWSl2tUOYOvAY68n3te7z+T97xm98R7O/bdCz3of9B7q7XnTU4/7D98/xxX8og+6u6f6dzNynTy/eNovdZpLnvztqwxApA3E2bBaSsLoRd0d7hy/783L/yYfm+KXDH/XDed3BvYo2P/Ow53zGOc0+50dP5d/dtdjiavAu1s1EN9bdIxIG8HlgCbAYuBcYkG1+JQyRJPlcCLvZZXfD4o3+q08v8GvGz/PRfV47MPu4viv9kxPn+m+/uLCw7q7p4os68cY1jEqx9qkbCbDiEwYwClgBDEx8/jVwdbZllDBEkoS9EKa7I81wwdm1ZZc/8J0W/9IZzX7qwGVdurt+aNRCv/Wjc31586rwMYatVivG3XjntrPsb0XqRgLsKQnjVWAIQeP7/cAF2ZZRwhBJEvZOM+TQHe1Yl+6u7zziKf+3dzf7Y79Y3L3urmETQTGqjHIlyXJ+aC+X3ljCCPaBzwI7gU1AY4Z5pgEtQEtdXV13Dq9Iz5J855x6t5nuQhhy6I7XONo/f1qz/+Vbj/uOdTsKjzPsha0YVUa5ShaVmizce2cbBjAYeAg4CqgG/gBckW0ZlTAkozJ6qraoMvX4yXIhbB81JmeyaB8wMPpjFjYRFKOEUW7Dv0ett/WSAv4JuCPp85XALdmWUcKQtMrsqdqiCnFx3de6zx++5Rm/6Z3N/vbDn/Gp3O076Xp82vtWe8eQIcVNsGETQTH+/SrlBVMl0hMSxlmJHlI1gAF3Af+SbRklDEmrHC4OpSrhZBr+28x/+o9z/L0jDnZ3Ndr9jMMW+9fe3uxLps70jjF1+ccX8bulMyaCqI9fb7qJCKHiE0awD3wLeD7RrfYeoH+2+ZUwJK24qx8yXZymT48+iWQZ/hvcj+m7yqcdP9d/c/1C3/Ly1vj3K+7nVHpDNWUIPSJh5DspYUhacZcwMm0/TIN0siwXuM7uro3HzPBdDOiy3t02wJve9nV/+e8ry3O/CqGLfiRKljCAB4FTothYoZMShhxQ6LsjorwQ5fMCoTzeoNfWb4D/78kz/N1DWnwAuxyC7q7/OnCmbxsw3Dsw76irK3n1V8mSs6qVIlPKhHFaojfTL4GRUWy0u5MShrh7995Ol2v5Qi5E+byiNFM1WZaqprf0f9E/99Y5/udvJnV3LcWddxT7VYztJycnlUBCKXmVFPAh4Bngps6ns0s9KWGIuxdeBRV1FVa2rq5ZtvH6q6/7H7/2qH/6pDneTubG7FDbK8addzf3KzK52qZUAgmtpAkj0YvpROBTwGZgDfCxKALIZ1LCEHcvvJE72/LdvWNNXW769EMuZh01Nf7CNd/xb05p9nMOf+bA6K4DafX1VSPSx9Snz6GxlLLNJsR+Fe0iXVubfj9ra4Pf4267qiClrJJ6GFgLNAE3A5cAbwZ+CsyKIoiwkxKGuHvxShh5vIM6jI57Zvu+YUd7B+Ybqkb4x7ktyEu0+6SaJf7VtzX7Q//xpO95fU+48Z46YymHXmGlqAbKlTDiPg4VpJQJ40QI3syX5rdlUQQRdlLCEHcvvCoi0/KZLlB53LFuen6z3/eZBX7tcfO8rs+rB1Yxtu9q/+fj5/qvP7/QN7+4JXNcnRfiPn0yx9Jb7qxzJYTechwiUBbdaoFxUQQRdlLCkANmz+56ga+tDb4Le/ebbr5svYIyrGv3tt3e9L1F/uUzm/2tA5cemP0ItvsHRj7it1w21196cGX+LxHKVW3Wr1/X7/v163l197kSgtowQiuLhFHqSQlDDkh3saiuPvRCms8FJFevoJoab7/7Hn/qvuf9+xc1e0NSd9e+7PN3HPGU33x+sz9y23O+f/f+wvYvUyx9+gTtCKndiKure96FMkxCUC+pUKJKGBasqzJMmjTJW1pa4g5DysHYsenfp51OfT2sXJl7vsbGQ9+jneJVRlPHqwCc0P9lGia+RsP7anjnpyYyaMSgcPGEkS0Ws+DymSrsflaSxkaYMSN4F3ZdXfAO7ojeQd6bmNkid59U8HqUMKQiVVWlv2hm06dPcBG+5Zau3ydflIYMoaPDsW1bsTSr6MC45xPzePcnj2XUpJHdDj+Uxka46ipobw83vxl0dBQ3JqlIUSWMqiiCESmJxsagZFFVFUz5am+Hn/8crruuyzp92rSgtOIOW7awe9seNlObdhVV9XVcddvk4icLCO6k80kAdXXFi0UEJYzoJV/Uxo4NPkthGhth6FC44oqDF/Z0d93V1dCvX87V+axZvNS0klsum8vGK7+IpVT7HMYujqhpwwfWdF2wpiaoEimlTEnAUso/ccQmvU8UDSGlmsq+0Vu9NqKX6/mE1AfbkhtBMyzTAQc+ZnrCuqCH+Iq9/7lGjC2HuKWsoF5SZUj9wsPJ54KWq+dShoe0dm/b7e1WlXaZNqr8lsvm+osPrAgG7yv3f7N8jpduWiQNJYxyVKlPnpbyjjTfC1quEVMTF/b2/e3+9H8/7z+4uNkvqH3CB9LqP2W6d6QpXfj06d2Pp9x156ZFJZIeTwmjHFViCaPUF8x8j1GWEkb7gIE+553f8I/WP+zDbOOBnyb2e9k/c8oc/9M3HvP9k6d0Xe7889Mfg55ywcz3pqWnJUxJSwmjHFXi/3ylTnIRXNA6wLdwpF/ObAf3YbbRP1r/sP/y2vn+6uNrsy5b9v8ehYoqIZfzTY7kLaqEoV5SUZo6FWbNCh6gMgv+O2tWeT9otHp1ft8XKlOvnzTft+1p49Hdp/C7sZ9nvY2kA2Ml9VzNL7i8tonTLh7F0//9Auv21dK48hyuvn0yo89I6u46Y8ahD77t2hU829BTe7HNnBn0mEqWrQdVqf/9pbJFkXVKNZV9CaMSlfoOM8tdf0d7h7/4wAr/2Ufm+PtHPuJHsD0ofNDupw1c6l85q9kf/P4i371td7hthXljXE8scUTRqUAljB4FVUlJJOKotkm6oLUdPdoXXniTf2LCXK9PGt21rs+rfu1x8/y+zyzwTc9v7t52wr4xrjdfHHtjtV0v1CMSBnAk8FvgeWAZ8LZs8ythFEkhjb55Lrvn9T3+939/0m84u9lPr1niRruD+5vY7u8f+Yj/7CNzgu6u+Y7umim2XO+YyNZ+0lv0pEZ/SSuqhBHrWFJmdhcw391vN7N+QI27b880v8aSKjPpBsirqenSbuMdznO/e4mmu9bS9Mgg5m05gd3U0Jf9nP2mpTScvo2Gy2o548qJ9B3Qtzgxdo4TVVWV/gnxnjhon0iSih980MzeRPCO8HEeMggljDKTYcTYthGjabzkXpoeNB5cPZ4NHcMAOL7fchqOX0PDewcyZfpEDj/68NLGGyLBifREPWHwwXHAJuCXZvaUmd1uZoelzmRm08ysxcxaNm3aVPooe7pCxr7K0JOmav1rXH37ZB5YdRzvGvMSze+4kf0jRrNs/3h+8vpVXHr8S6VPFlCZvdhEykkU9VrdmYBJQBtwVuLzj4Gbsy2jNoyIdNZZd9bf59ng2ba3zR+9/TnfNmB42jaB7QOG+1P3Pe/t+9vVqCpSBugBz2GsAda4+2OJz78FTosxnt6hs1qmsyoptTZw166gzj+Jdzgv/30VP798Hh88+lGGDtjB2Z84kdl7Pki6usQjrvkgp35kAlV9qzI/C5GyDREpf7ElDHdfD7xqZhMSX50PLI0rnlB6wtDl6S7gqVavZuvybfzm+keYNnEe4/qvYfy767nuvnfQsnEMH3zzc9z7LwuZPvJPaV8yxF/+0mVdmbYhIhUmimJKdyfgVKAFeBb4AzA42/yxVklVctVKiCG/k6e1NrJLd9f3jXjU//Of5vgL//dK1+6uYYb5yPfBsFIPhKjupNIL0BOew8h3ijVhVOoTsWGfRUhMO6nxm/p/2785pdkX3Pqs79+9P/O6wxyTfBJtKZNyJd8AiORJCaPUKnXo8hBPO7dj3gHe+qYR3vqT28KvO+xFN+ydfClLI5V6AyDSDUoYpVaBF5idG3Z6R4Y3ynWAd2C+f8Sowu6qp08P3noHwX+T3zWRr3yScqElhEq9ARDphqgShkarDSvfUUBj0L6vncfuWMzMhjlMOfJpBg+vZhXpR4e1Pn2w2ffQd92a7j+H0NgId9118Onp9vbgc3c7A+Qxkm3Bva/y2ZaIAPE+uFdZyvShrw1f+zE7DhtBh1XxWv+x/PgTz/D1B6fwxr7+fO6Mhez48CfwgQMPXbC9PeheW0hPr6i7zOaTlAvtfVUBNwAiZSeKYkqppl7x4F6OevktL2/131y/0KcdP9c/az/2nXStltnft7+//v2fH7rOTFUwhVSpFaNap1jtHYVsS6TC0RMGH8xXjx9LKs1YRz6whmUfnEHjirfT9MxRtLROxKnicN7g5aoJDOtYf+h6UgfTa2yEK65Iv00z6OjoXrwZxpIqyWB+GhdKJLSeMJaUpEpTxWO7d1HTOIvvLZxMdVU7N75zHg/f8ixbWgcyzDekX09qtUy2KqJC6uzjrNYp0ypCkR4timJKqabQVVIVVtWw9qn1fvcn52fp0WS+fdX2QxcMWy2T7YG9Qo9NhR1rkd4IVUllUAFVFa0bW5l361Ka/thK05JRLN47HoDVjGEMaw5dIFMVT9h9zVR1VFsLmzcXtC8iUv5UJZVJpp47V10VzRhQ3RhPqn1fO4//ckmX7q4X3XQGtzx5NsNrdvDdC+ewaPYyRt39nfyqeDqrZWprD36XrkdUpqqjH/84Z+wiIgdEUUwp1RSqSirMeEndHQIij4fFljev8ls/Otc/NGqhD7atB2Y/deAy/+KkZv/bt1t815Zd6bfRWdXU+UBctqqeqJ+2LpSqqETKDnrSO4MQQ2F0uztpljaDra9s899+caF/cuJcH9d35YGfRvd5za8ZP89/9ekFvmHxxnDbyecp5nJ6Al3jM4mUpagSRu9ow0gntTtp8ruf6+qCapzUNo+qquAymKIDo5r9dNCHQezgvOFLaZi8h4aPj2HChcdgVWkHAc8sn+6qGWIqqLtsd8XZzVZEMoqqDaNvFMGUlc6LfOfFv6rq4NAVyZK7k6YmmVWrgs9J6/MOZ//QkfTbtPaQVW2wEXz93Pk0fGQIZ109keqaswrbh3yeYq6rS3+RjmOIC737QqRH63mN3hBc5FeuDO6w77ord0Nyhobyti/dwOzpC7jq2IcZXb2eqzd9n1a6rssHDmTkPT/gW3OnMPm6k6muqS48/nzGOSrlsxC5Gvw1PpNIzxZFvVappm4PDZKrITZDQ3k75uBea5v9w2MW+G1XzvONN/7Evbb24Hy1tbnr6PNtCM63LaAUDc1hYlIbhkhZQo3ehWvb2+aP37nEtw8YnjZhtPZ9k++pHekdyRfi7lzMu3MRLbfeRmEb18stbhFRo3d3rZj3Kk2zVtA0p5qH1k5gqw/hchq5g08wkD0HZ6yuDhqO9+07+F1NTfCcw5Yth644U8NuT2kILqfGdRHJixq9Q9q+6nUe+tkymv68j6YX61neVg+MYVTVOt577BIaLqji/OkXMPCZ27v2ktq589DEsGtX5t5X+Tb4VlpDcDk1rotILHpcwti3cx+P/nIZTb/eRtNTQ3midSIdnM0gdjBl2DI+M3klDdeM5viLxmFVIw8ueOLUrt1oq/LsD5CtwbcnXGhnzkw/DIneHyHSa8SeMMysD9ACvObul+S7vHc4y+5fTtOdr9G0YCBzNp5AK6dQRTtnDlrKjHPn0/DhwZz98ROorjkz/IozXehra2H37vAXzp5yoU3trpzpWRUR6bFib8Mws+uBScCbciWMSWbeUl/P9k9+hftXn0zTA86DK49lbUdQUnhz9UoajltNw8X9OO+6iRxZf0T3A8s2sB/kd+EM81CgiEiRRNWGEWvCMLPRwF3ATOD6UAkDaKWGf2YWf7P3cP6oF2g4bz8N08YxdvLoaAPUhV5EeoCekjB+C3wHOBz4YrqEYWbTgGkAp8PpnX2k9taOpHr9Gqr69sxnD0VEolLxw5ub2SXARndflG0+d5/l7pNSd7b/1vVKFiIiJRTnFfcc4L1mthK4D3iXmc0OvXSl9TIqpW68s0NEJJfYEoa7f9XdR7v7WOAy4CF3vyLUwpXYy6hUOhvrV60KHrTrHEhRSUNEClR5dTr19WX1utWyk+mNgzNmxBOPiPQYsXerzUcUQ4P0eBrCQ0RSVHyjtxSJhhgXkSJRwuhpSvl+DBHpVZQwepqpU4M2nvr6oBpKbT4iEpHYx5KSIpg6VQlCRCKnEoaIiISihCEiIqEoYYiISChKGJVAQ32ISBlQo3e5S30vR+dQH6CGbREpKZUwyp2G+hCRMtF7EkalVuusXp3f9yIiRdI7EkYlj+CqoT5EpEz0joRRydU6GupDRMpE70gYlVyto6E+RKRM9I5eUnV1QTVUuu8rgYb6EJEy0DtKGKrWEREpWO9IGKrWEREpWO+okgJV64iIFKh3lDBERKRgShgiIhJKbAnDzMaYWbOZLTOzJWb22bhiERGR3OJsw2gDvuDuT5rZ4cAiM2ty96UxxiQiIhnEVsJw93Xu/mTi7x3AMmBUXPGIiEh2ZdGGYWZjgbcCj6X5bZqZtZhZy6ZNm0oem4iIBGJPGGY2CPgf4HPu/kbq7+4+y90nufuko446qvQBiogIEHPCMLNqgmTR6O6/izMWERHJLs5eUgbcASxz9/8XVxwiIhJOnCWMc4CPAe8ys6cT00UxxiMiIlnE1q3W3R8GLK7ti4hIfmJv9BYRkcqghCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISSqwJw8wuNLMXzOxlM7shzlhERCS72BKGmfUBfga8BzgBuNzMTogrHhERyS7OEsaZwMvu/oq77wPuA94XYzwiIpJF3xi3PQp4NenzGuCs1JnMbBowLfFxr5ktLkFshRoKbI47iBAUZ3QqIUZQnFGrlDgnRLGSOBOGpfnOD/nCfRYwC8DMWtx9UrEDK5TijFYlxFkJMYLijFolxRnFeuKskloDjEn6PBpYG1MsIiKSQ5wJ4wlgvJkdY2b9gMuA/40xHhERySK2Kil3bzOzTwN/A/oAv3D3JTkWm1X8yCKhOKNVCXFWQoygOKPWq+I090OaDURERA6hJ71FRCQUJQwREQmlLBJGriFCLPCTxO/PmtlpYZctcZxTE/E9a2YLzeyUpN9WmtlzZvZ0VF3cCohzipm9nojlaTO7MeyyJY7zS0kxLjazdjMbkvitJMfTzH5hZhszPf9TRudmrjjL5dzMFWe5nJu54iyHc3OMmTWb2TIzW2Jmn00zT7Tnp7vHOhE0eC8HxgH9gGeAE1LmuQj4K8GzG2cDj4VdtsRxvh0YnPj7PZ1xJj6vBIaWyfGcAtzfnWVLGWfK/JcCD8VwPN8BnAYszvB77OdmyDhjPzdDxhn7uRkmzjI5N0cCpyX+Phx4sdjXznIoYYQZIuR9wN0eeBQ40sxGhly2ZHG6+0J335b4+CjBsyWlVsgxKavjmeJy4N4ixZKRu88DtmaZpRzOzZxxlsm5GeZ4ZlJWxzNFXOfmOnd/MvH3DmAZwQgaySI9P8shYaQbIiR1pzPNE2bZqOS7rWsJMnsnBx4ws0UWDHdSLGHjfJuZPWNmfzWzt+S5bBRCb8vMaoALgf9J+rpUxzOXcjg38xXXuRlW3OdmaOVybprZWOCtwGMpP0V6fsY5NEinMEOEZJon1PAiEQm9LTM7j+B/yslJX5/j7mvNbBjQZGbPJ+5i4ojzSaDe3Xea2UXAH4DxIZeNSj7buhRY4O7Jd3ylOp65lMO5GVrM52YY5XBu5iP2c9PMBhEkrM+5+xupP6dZpNvnZzmUMMIMEZJpnlIOLxJqW2Z2MnA78D5339L5vbuvTfx3I/B7giJhLHG6+xvuvjPx91+AajMbGmbZUsaZ5DJSivwlPJ65lMO5GUoZnJs5lcm5mY9Yz00zqyZIFo3u/rs0s0R7fha7YSZEw01f4BXgGA42vrwlZZ6L6dpw83jYZUscZx3wMvD2lO8PAw5P+nshcGGMcY7g4EObZwKrE8e2rI5nYr4jCOqSD4vjeCa2MZbMjbSxn5sh44z93AwZZ+znZpg4y+HcTByXu4EfZZkn0vMz9iopzzBEiJl9KvH7rcBfCFr7XwZ2AddkWzbGOG8EaoFbzAygzYORLIcDv0981xf4lbv/X4xx/iMw3czagN3AZR6cReV2PAE+ADzg7q1Ji5fseJrZvQQ9d4aa2RrgJqA6KcbYz82QccZ+boaMM/ZzM2ScEPO5CZwDfAx4zsyeTnz3NYKbg6KcnxoaREREQimHNgwREakAShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEIVKAxPsIGhJ//5uZ/STumESKJfYnvUUq3E3AvyYGmnsr8N6Y4xEpGj3pLVIgM5sLDAKmePBeApEeSVVSIgUws5MI3ny2V8lCejolDJFuSry5rJHgTWWtZvYPMYckUlRKGCLdkHjT2u+AL7j7MuBm4JuxBiVSZGrDEBGRUFTCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQnl/wNIgRvRIRIfLQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
    " + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -3041,9 +3712,35 @@ "plt.title(r'Random numbers ')\n", "plt.show()" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ffd93055", + "metadata": {}, + "outputs": [], + "source": [] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.8" + } + }, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index bb8bfe8d0..973434693 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -2385,6 +2385,15 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that with $t$ a finite positive number. +If we keep the $1/n$ factor, the equation for the optimal $\beta$ changes to +!bt +\[ +\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}. +\] +!et + +In many textbooks the $1/n$ term is often omitted. Note that a library like _Scikit-Learn_ does not include the $1/n$ factor in the setup of the cost function. + When we compare this with the ordinary least squares result we have !bt \[ diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt index 1c4bf8556..bc078954b 100644 --- a/doc/src/week36/week36.do.txt +++ b/doc/src/week36/week36.do.txt @@ -328,6 +328,8 @@ the optimal parameters with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that + + !bt \[ \sum_{i=0}^{p-1} \beta_i^2 \leq t, @@ -336,6 +338,23 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that with $t$ a finite positive number. + +!split +===== Note on Scikit-Learn ===== + +Note well that a library like _Scikit-Learn_ does not include the $1/n$ factor in the expression for the mean-squared error. If you include it, the optimal parameter $\beta$ becomes + +!bt +\[ +\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+n\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y}. +\] +!et + +In our codes where we compare our own codes with _Scikit-Learn_, we do thus not include the $1/n$ factor in the cost function. + + +!split +===== Comparison with OLS ===== When we compare this with the ordinary least squares result we have !bt \[