From d0201bd7081f089b82297e4229a8996da24ca136 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Mon, 28 Aug 2023 08:18:04 +0200 Subject: [PATCH] update week35 --- doc/pub/week35/html/._week35-bs001.html | 3 +- doc/pub/week35/html/._week35-bs003.html | 4 +- doc/pub/week35/html/._week35-bs005.html | 8 +- doc/pub/week35/html/._week35-bs006.html | 2 +- doc/pub/week35/html/._week35-bs013.html | 4 +- doc/pub/week35/html/._week35-bs015.html | 4 +- doc/pub/week35/html/._week35-bs032.html | 2 +- doc/pub/week35/html/._week35-bs038.html | 13 +- doc/pub/week35/html/week35-reveal.html | 40 +- doc/pub/week35/html/week35-solarized.html | 40 +- doc/pub/week35/html/week35.html | 40 +- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 192 bytes doc/pub/week35/ipynb/week35.ipynb | 3150 ++++++++++-------- doc/src/week35/week35.do.txt | 18 +- 14 files changed, 1880 insertions(+), 1448 deletions(-) diff --git a/doc/pub/week35/html/._week35-bs001.html b/doc/pub/week35/html/._week35-bs001.html index 125ef36bf..208e50552 100644 --- a/doc/pub/week35/html/._week35-bs001.html +++ b/doc/pub/week35/html/._week35-bs001.html @@ -382,7 +382,8 @@ MathJax.Hub.Config({
  1. See lecture notes for week 35 at https://compphysics.github.io/MachineLearning/doc/web/course.html
    2. -
  2. CMB sections 1.1 and 3.1
    3. +
  3. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics
    4. +
  4. Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4

diff --git a/doc/pub/week35/html/._week35-bs003.html b/doc/pub/week35/html/._week35-bs003.html index 6aff0fed8..c02e8f6fd 100644 --- a/doc/pub/week35/html/._week35-bs003.html +++ b/doc/pub/week35/html/._week35-bs003.html @@ -371,7 +371,7 @@ MathJax.Hub.Config({

Our data which we want to apply a machine learning method on, consist of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the -outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). +outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). We assume that the output data can be represented (for a regression case) by a continuous function \( f \) through

@@ -392,7 +392,7 @@ value and a variance \( \sigma^2 \).

In linear regression we approximate the unknown function with another continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on some unknown parameters -\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \). +\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).

Last week we introduced the so-called design matrix in order to define diff --git a/doc/pub/week35/html/._week35-bs005.html b/doc/pub/week35/html/._week35-bs005.html index f35c1a82b..ded58820d 100644 --- a/doc/pub/week35/html/._week35-bs005.html +++ b/doc/pub/week35/html/._week35-bs005.html @@ -383,8 +383,8 @@ $$

where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) the outcome of a -numerical experiment or another type of experiment and is thus only an +response (dependent or outcome) variable \( y_i \) is the outcome of a +numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we @@ -407,9 +407,9 @@ $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ -

or in a matrix-vector form as (multiplying away the factor \( -2/n \))

+

or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)

$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$ diff --git a/doc/pub/week35/html/._week35-bs006.html b/doc/pub/week35/html/._week35-bs006.html index 68947dd3a..c923a16e4 100644 --- a/doc/pub/week35/html/._week35-bs006.html +++ b/doc/pub/week35/html/._week35-bs006.html @@ -373,7 +373,7 @@ MathJax.Hub.Config({

We can rewrite, see the derivations below,

$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$

as

diff --git a/doc/pub/week35/html/._week35-bs013.html b/doc/pub/week35/html/._week35-bs013.html index 1a557f07b..f9a0c5f39 100644 --- a/doc/pub/week35/html/._week35-bs013.html +++ b/doc/pub/week35/html/._week35-bs013.html @@ -383,7 +383,7 @@ $$ We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).

-

The mean squared error is scalar and if we use the results from the last example, we define a new vector

+

The mean squared error is a scalar and if we use the results from the last example, we define a new vector

$$ \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, $$ @@ -403,7 +403,7 @@ $$ \frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X}, $$ -

where we ued the results from example two. Inserting the last expression we obtain

+

where we used the result from example two above. Inserting the last expression we obtain

$$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X}, $$ diff --git a/doc/pub/week35/html/._week35-bs015.html b/doc/pub/week35/html/._week35-bs015.html index f84b9f759..7d75e6151 100644 --- a/doc/pub/week35/html/._week35-bs015.html +++ b/doc/pub/week35/html/._week35-bs015.html @@ -389,9 +389,9 @@ $$

For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is -(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component -Analysis as a way to reduce the dimensionality of a machine learning +Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem.

diff --git a/doc/pub/week35/html/._week35-bs032.html b/doc/pub/week35/html/._week35-bs032.html index e3d2755a2..670fccb71 100644 --- a/doc/pub/week35/html/._week35-bs032.html +++ b/doc/pub/week35/html/._week35-bs032.html @@ -384,7 +384,7 @@ target/output variables when all other predictors are set to zero. Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. -Furthermore, in for example Ridge and Lasso regression, the default solutions +Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters \( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values. diff --git a/doc/pub/week35/html/._week35-bs038.html b/doc/pub/week35/html/._week35-bs038.html index 1d11afbad..fdb47569f 100644 --- a/doc/pub/week35/html/._week35-bs038.html +++ b/doc/pub/week35/html/._week35-bs038.html @@ -486,12 +486,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a

Printing the MSE, we see first that both methods give the same MSE, as -they should. However, when we move to for example Ridge regression, +they should. However, when we move to for example Ridge regression (discussed next week), the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the regularization term does not include \( \beta_0 \). For different values -of \( \lambda \), this may lead to differeing MSE values. +of \( \lambda \), this may lead to differing MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression is given by

@@ -509,7 +509,14 @@ $$ \lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. $$ -

It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.

+

It means that, when scaling the design matrix and the outputs/targets, +by subtracting the mean values, we have an optimization problem which +is not penalized by the intercept. The MSE value can then be smaller +since it focuses only on the remaining quantities. If we however bring +back the intercept, we will get an MSE which then contains the +intercept. This becomes more important when we discuss Ridge and Lasso +regression next week. +

diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index f3558b563..de48d3b73 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -211,7 +211,8 @@ MathJax.Hub.Config({

  1. See lecture notes for week 35 at https://compphysics.github.io/MachineLearning/doc/web/course.html
    2. -

  2. CMB sections 1.1 and 3.1
    3. +

  3. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics
    4. +

  4. Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4
@@ -243,7 +244,7 @@ Similarly, Mehta et a

Our data which we want to apply a machine learning method on, consist of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the -outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). +outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). We assume that the output data can be represented (for a regression case) by a continuous function \( f \) through

@@ -268,7 +269,7 @@ value and a variance \( \sigma^2 \).

In linear regression we approximate the unknown function with another continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on some unknown parameters -\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \). +\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).

Last week we introduced the so-called design matrix in order to define @@ -342,8 +343,8 @@ $$

where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) the outcome of a -numerical experiment or another type of experiment and is thus only an +response (dependent or outcome) variable \( y_i \) is the outcome of a +numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we @@ -372,10 +373,10 @@ $$ $$

 
-

or in a matrix-vector form as (multiplying away the factor \( -2/n \))

+

or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)

 
$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$

 
@@ -388,7 +389,7 @@ $$

We can rewrite, see the derivations below,

 
$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$

 
@@ -665,7 +666,7 @@ $$ We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).

-

The mean squared error is scalar and if we use the results from the last example, we define a new vector

+

The mean squared error is a scalar and if we use the results from the last example, we define a new vector

 
$$ \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, @@ -693,7 +694,7 @@ $$ $$

 
-

where we ued the results from example two. Inserting the last expression we obtain

+

where we used the result from example two above. Inserting the last expression we obtain

 
$$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X}, @@ -758,9 +759,9 @@ $$

For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is -(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component -Analysis as a way to reduce the dimensionality of a machine learning +Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem.

@@ -1537,7 +1538,7 @@ target/output variables when all other predictors are set to zero. Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. -Furthermore, in for example Ridge and Lasso regression, the default solutions +Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters \( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values. @@ -1865,12 +1866,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a

Printing the MSE, we see first that both methods give the same MSE, as -they should. However, when we move to for example Ridge regression, +they should. However, when we move to for example Ridge regression (discussed next week), the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the regularization term does not include \( \beta_0 \). For different values -of \( \lambda \), this may lead to differeing MSE values. +of \( \lambda \), this may lead to differing MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression is given by

@@ -1894,7 +1895,14 @@ $$ $$

 
-

It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.

+

It means that, when scaling the design matrix and the outputs/targets, +by subtracting the mean values, we have an optimization problem which +is not penalized by the intercept. The MSE value can then be smaller +since it focuses only on the remaining quantities. If we however bring +back the intercept, we will get an MSE which then contains the +intercept. This becomes more important when we discuss Ridge and Lasso +regression next week. +

diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index f400d9d40..b4a7c7281 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -332,7 +332,8 @@ MathJax.Hub.Config({
  1. See lecture notes for week 35 at https://compphysics.github.io/MachineLearning/doc/web/course.html
    2. -
  2. CMB sections 1.1 and 3.1
    3. +
  3. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics
    4. +
  4. Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4










Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week

@@ -360,7 +361,7 @@ Similarly, Mehta et a

Our data which we want to apply a machine learning method on, consist of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the -outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). +outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). We assume that the output data can be represented (for a regression case) by a continuous function \( f \) through

@@ -381,7 +382,7 @@ value and a variance \( \sigma^2 \).

In linear regression we approximate the unknown function with another continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on some unknown parameters -\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \). +\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).

Last week we introduced the so-called design matrix in order to define @@ -441,8 +442,8 @@ $$

where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) the outcome of a -numerical experiment or another type of experiment and is thus only an +response (dependent or outcome) variable \( y_i \) is the outcome of a +numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we @@ -465,9 +466,9 @@ $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ -

or in a matrix-vector form as (multiplying away the factor \( -2/n \))

+

or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)

$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$ @@ -478,7 +479,7 @@ $$

We can rewrite, see the derivations below,

$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$

as

@@ -707,7 +708,7 @@ $$ We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).

-

The mean squared error is scalar and if we use the results from the last example, we define a new vector

+

The mean squared error is a scalar and if we use the results from the last example, we define a new vector

$$ \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, $$ @@ -727,7 +728,7 @@ $$ \frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X}, $$ -

where we ued the results from example two. Inserting the last expression we obtain

+

where we used the result from example two above. Inserting the last expression we obtain

$$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X}, $$ @@ -778,9 +779,9 @@ $$

For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is -(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component -Analysis as a way to reduce the dimensionality of a machine learning +Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem.

@@ -1538,7 +1539,7 @@ target/output variables when all other predictors are set to zero. Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. -Furthermore, in for example Ridge and Lasso regression, the default solutions +Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters \( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values. @@ -1837,12 +1838,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a

Printing the MSE, we see first that both methods give the same MSE, as -they should. However, when we move to for example Ridge regression, +they should. However, when we move to for example Ridge regression (discussed next week), the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the regularization term does not include \( \beta_0 \). For different values -of \( \lambda \), this may lead to differeing MSE values. +of \( \lambda \), this may lead to differing MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression is given by

@@ -1860,7 +1861,14 @@ $$ \lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. $$ -

It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.

+

It means that, when scaling the design matrix and the outputs/targets, +by subtracting the mean values, we have an optimization problem which +is not penalized by the intercept. The MSE value can then be smaller +since it focuses only on the remaining quantities. If we however bring +back the intercept, we will get an MSE which then contains the +intercept. This becomes more important when we discuss Ridge and Lasso +regression next week. +

The Boston housing data example

diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index bd967bfc0..8cf3338bd 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -409,7 +409,8 @@ MathJax.Hub.Config({
  1. See lecture notes for week 35 at https://compphysics.github.io/MachineLearning/doc/web/course.html
    2. -
  2. CMB sections 1.1 and 3.1
    3. +
  3. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics
    4. +
  4. Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4










Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week

@@ -437,7 +438,7 @@ Similarly, Mehta et a

Our data which we want to apply a machine learning method on, consist of a set of inputs \( \boldsymbol{x}^T=[x_0,x_1,x_2,\dots,x_{n-1}] \) and the -outputs we want to model \( \boldsymbol{x}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). +outputs we want to model \( \boldsymbol{y}^T=[y_0,y_1,y_2,\dots,y_{n-1}] \). We assume that the output data can be represented (for a regression case) by a continuous function \( f \) through

@@ -458,7 +459,7 @@ value and a variance \( \sigma^2 \).

In linear regression we approximate the unknown function with another continuous function \( \tilde{\boldsymbol{y}}(\boldsymbol{x}) \) which depends linearly on some unknown parameters -\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1} \). +\( \boldsymbol{\beta}^T=[\beta_0,\beta_1,\beta_2,\dots,\beta_{p-1}] \).

Last week we introduced the so-called design matrix in order to define @@ -518,8 +519,8 @@ $$

where \( \langle y_i \rangle \) is the mean value. Keep in mind also that till now we have treated \( y_i \) as the exact value. Normally, the -response (dependent or outcome) variable \( y_i \) the outcome of a -numerical experiment or another type of experiment and is thus only an +response (dependent or outcome) variable \( y_i \) is the outcome of a +numerical experiment or another type of experiment and could thus be treated itself as an approximation to the true value. It is then always accompanied by an error estimate, often limited to a statistical error estimate given by the standard deviation discussed earlier. In the discussion here we @@ -542,9 +543,9 @@ $$ \frac{\partial C(\boldsymbol{\beta})}{\partial \beta_j} = -\frac{2}{n}\left[ \sum_{i=0}^{n-1}x_{ij}\left(y_i-\beta_0x_{i,0}-\beta_1x_{i,1}-\beta_2x_{i,2}-\dots-\beta_{n-1}x_{i,n-1}\right)\right]=0, $$ -

or in a matrix-vector form as (multiplying away the factor \( -2/n \))

+

or in a matrix-vector form as (multiplying away the factor \( -2/n \), see derivation below)

$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right). $$ @@ -555,7 +556,7 @@ $$

We can rewrite, see the derivations below,

$$ -\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), +\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T} = 0 = \boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right), $$

as

@@ -784,7 +785,7 @@ $$ We are now interested in minimizing the cost function with respect to the unknown parameters \( \boldsymbol{\beta} \).

-

The mean squared error is scalar and if we use the results from the last example, we define a new vector

+

The mean squared error is a scalar and if we use the results from the last example, we define a new vector

$$ \boldsymbol{w}=\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}, $$ @@ -804,7 +805,7 @@ $$ \frac{\partial \boldsymbol{w}}{\partial \boldsymbol{\beta}}=-\boldsymbol{X}, $$ -

where we ued the results from example two. Inserting the last expression we obtain

+

where we used the result from example two above. Inserting the last expression we obtain

$$ \frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\frac{2}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)^T\boldsymbol{X}, $$ @@ -855,9 +856,9 @@ $$

For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters \( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that it is -(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +(aside the factor \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component -Analysis as a way to reduce the dimensionality of a machine learning +Analysis as a way to reduce the dimensionality of a machine learning/data analysis problem.

@@ -1615,7 +1616,7 @@ target/output variables when all other predictors are set to zero. Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. -Furthermore, in for example Ridge and Lasso regression, the default solutions +Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters \( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values. @@ -1914,12 +1915,12 @@ when all our features are zero and our function crosses the \( y \)-axis (for a

Printing the MSE, we see first that both methods give the same MSE, as -they should. However, when we move to for example Ridge regression, +they should. However, when we move to for example Ridge regression (discussed next week), the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the regularization term does not include \( \beta_0 \). For different values -of \( \lambda \), this may lead to differeing MSE values. +of \( \lambda \), this may lead to differing MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression is given by

@@ -1937,7 +1938,14 @@ $$ \lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. $$ -

It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week.

+

It means that, when scaling the design matrix and the outputs/targets, +by subtracting the mean values, we have an optimization problem which +is not penalized by the intercept. The MSE value can then be smaller +since it focuses only on the remaining quantities. If we however bring +back the intercept, we will get an MSE which then contains the +intercept. This becomes more important when we discuss Ridge and Lasso +regression next week. +

The Boston housing data example

diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 7dca4cfff0198e36bcc7ae2d7dfea577bbaa3c84..f42da0df99b9f37fd0ffdc57ea9345ae77333955 100644 GIT binary patch literal 192 zcmV;x06+g9iwFP_JnUou1MSbv3c@f92k@Qu6nTP?uHAMP^x#1d@dY~8xjNU*wnO*! z?gR9sco`z}cli?%LUP!w*1JvQ?k-piBBqSNm|2=kiSb-d2uXl2mQkcB#srYiB&Gr2 zawolX)^RhO(o|=mtWfXfhOx5zuxEY+p7|$^m9((k_pZ_kly)N5x`vw&HBBPf_9}-$ u3p=*Jh-)W}0IquAMIoKkieJLk=#$}%jly3)<9VLveeD6hb`ui-2mk=W6JJFD literal 192 zcmV;x06+g9iwFQ=Gwfsl1MSbv3c@f92k@Qu6nTQtx^_DY?%+WX@dY}TxjNU*wnO*! z?gR9sco`z}cli?%LUPE~n_U*Uy9*XW2uT=&G1G)kNmg?`p_Bt_Wl|}V#ZWt@e4}0cU;F*8oSV;@peeWu*Kxv1$)-~J^>zGHf?Ntti uMmx5^;I)$mL8u-?QAj7X5|^+w`ea07qwv?yc%J8ZUwZ&*?4u0;2mk=t@mF5} diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 2db9b17e2..a6f7e09a8 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "48814d17", - "metadata": {}, + "id": "c02138a8", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "bc61e6c3", - "metadata": {}, + "id": "919baf6d", + "metadata": { + "editable": true + }, "source": [ "# Week 35: From Ordinary Linear Regression to Ridge and Lasso Regression\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -23,8 +27,10 @@ }, { "cell_type": "markdown", - "id": "1de6c7de", - "metadata": {}, + "id": "4d5e7179", + "metadata": { + "editable": true + }, "source": [ "## Plans for week 35\n", "\n", @@ -43,20 +49,26 @@ }, { "cell_type": "markdown", - "id": "540f9b28", - "metadata": {}, + "id": "3b3f68ae", + "metadata": { + "editable": true + }, "source": [ "### Reading recommendations:\n", "\n", "1. See lecture notes for week 35 at \n", "\n", - "2. CMB sections 1.1 and 3.1" + "2. Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics\n", + "\n", + "3. Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4" ] }, { "cell_type": "markdown", - "id": "8fa69b6b", - "metadata": {}, + "id": "3dfade93", + "metadata": { + "editable": true + }, "source": [ "## Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week\n", "\n", @@ -87,22 +99,26 @@ }, { "cell_type": "markdown", - "id": "75116bcf", - "metadata": {}, + "id": "e7c3b1f5", + "metadata": { + "editable": true + }, "source": [ "## The equations for ordinary least squares\n", "\n", "Our data which we want to apply a machine learning method on, consist\n", "of a set of inputs $\\boldsymbol{x}^T=[x_0,x_1,x_2,\\dots,x_{n-1}]$ and the\n", - "outputs we want to model $\\boldsymbol{x}^T=[y_0,y_1,y_2,\\dots,y_{n-1}]$.\n", + "outputs we want to model $\\boldsymbol{y}^T=[y_0,y_1,y_2,\\dots,y_{n-1}]$.\n", "We assume that the output data can be represented (for a regression case) by a continuous function $f$\n", "through" ] }, { "cell_type": "markdown", - "id": "8cea2508", - "metadata": {}, + "id": "4b7ace0a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i=f(x_i)+\\epsilon_i,\n", @@ -111,16 +127,20 @@ }, { "cell_type": "markdown", - "id": "f1cff657", - "metadata": {}, + "id": "6fa41e5e", + "metadata": { + "editable": true + }, "source": [ "or in general" ] }, { "cell_type": "markdown", - "id": "bca208b6", - "metadata": {}, + "id": "86724624", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}=f(\\boldsymbol{x})+\\boldsymbol{\\epsilon},\n", @@ -129,8 +149,10 @@ }, { "cell_type": "markdown", - "id": "7e1e2c71", - "metadata": {}, + "id": "f4e16a64", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\epsilon}$ represents some noise which is normally assumed to\n", "be distributed via a normal probability distribution with zero mean\n", @@ -139,7 +161,7 @@ "In linear regression we approximate the unknown function with another\n", "continuous function $\\tilde{\\boldsymbol{y}}(\\boldsymbol{x})$ which depends linearly on\n", "some unknown parameters\n", - "$\\boldsymbol{\\beta}^T=[\\beta_0,\\beta_1,\\beta_2,\\dots,\\beta_{p-1}$.\n", + "$\\boldsymbol{\\beta}^T=[\\beta_0,\\beta_1,\\beta_2,\\dots,\\beta_{p-1}]$.\n", "\n", "Last week we introduced the so-called design matrix in order to define\n", "the approximation $\\boldsymbol{\\tilde{y}}$ via the unknown quantity\n", @@ -148,8 +170,10 @@ }, { "cell_type": "markdown", - "id": "aa01ecd2", - "metadata": {}, + "id": "00f70fc7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}}= \\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -158,8 +182,10 @@ }, { "cell_type": "markdown", - "id": "ef34b811", - "metadata": {}, + "id": "859d4683", + "metadata": { + "editable": true + }, "source": [ "and in order to find the optimal parameters $\\beta_i$ we defined a function which\n", "gives a measure of the spread between the values $y_i$ (which\n", @@ -169,8 +195,10 @@ }, { "cell_type": "markdown", - "id": "2c121448", - "metadata": {}, + "id": "ab473c08", + "metadata": { + "editable": true + }, "source": [ "## The cost/loss function\n", "\n", @@ -179,8 +207,10 @@ }, { "cell_type": "markdown", - "id": "6001c152", - "metadata": {}, + "id": "a3ccaf3a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", @@ -189,16 +219,20 @@ }, { "cell_type": "markdown", - "id": "3b3e380d", - "metadata": {}, + "id": "238df8ba", + "metadata": { + "editable": true + }, "source": [ "or using the matrix $\\boldsymbol{X}$ and in a more compact matrix-vector notation as" ] }, { "cell_type": "markdown", - "id": "b227c100", - "metadata": {}, + "id": "e114bb77", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -207,8 +241,10 @@ }, { "cell_type": "markdown", - "id": "0eea8ce0", - "metadata": {}, + "id": "67833b96", + "metadata": { + "editable": true + }, "source": [ "This function represents one of many possible ways to define the so-called cost function.\n", "\n", @@ -218,8 +254,10 @@ }, { "cell_type": "markdown", - "id": "77959af4", - "metadata": {}, + "id": "68d1debe", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{2n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2,\n", @@ -228,16 +266,20 @@ }, { "cell_type": "markdown", - "id": "32483693", - "metadata": {}, + "id": "bd4914a7", + "metadata": { + "editable": true + }, "source": [ "since when taking the first derivative with respect to the unknown parameters $\\beta$, the factor of $2$ cancels out." ] }, { "cell_type": "markdown", - "id": "a471dd4d", - "metadata": {}, + "id": "9d6509bf", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "\n", @@ -246,8 +288,10 @@ }, { "cell_type": "markdown", - "id": "964a050a", - "metadata": {}, + "id": "fa97bb74", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\},\n", @@ -256,8 +300,10 @@ }, { "cell_type": "markdown", - "id": "87f8eda7", - "metadata": {}, + "id": "fdc65871", + "metadata": { + "editable": true + }, "source": [ "can be linked to the variance of the quantity $y_i$ if we interpret the latter as the mean value. \n", "When linking (see the discussions next week) with the maximum likelihood approach below, we will indeed interpret $y_i$ as a mean value" @@ -265,8 +311,10 @@ }, { "cell_type": "markdown", - "id": "71f3d982", - "metadata": {}, + "id": "6fe43d59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_{i}=\\langle y_i \\rangle = \\beta_0x_{i,0}+\\beta_1x_{i,1}+\\beta_2x_{i,2}+\\dots+\\beta_{n-1}x_{i,n-1}+\\epsilon_i,\n", @@ -275,13 +323,15 @@ }, { "cell_type": "markdown", - "id": "f20e60e4", - "metadata": {}, + "id": "7f0f5fc2", + "metadata": { + "editable": true + }, "source": [ "where $\\langle y_i \\rangle$ is the mean value. Keep in mind also that\n", "till now we have treated $y_i$ as the exact value. Normally, the\n", - "response (dependent or outcome) variable $y_i$ the outcome of a\n", - "numerical experiment or another type of experiment and is thus only an\n", + "response (dependent or outcome) variable $y_i$ is the outcome of a\n", + "numerical experiment or another type of experiment and could thus be treated itself as an\n", "approximation to the true value. It is then always accompanied by an\n", "error estimate, often limited to a statistical error estimate given by\n", "the standard deviation discussed earlier. In the discussion here we\n", @@ -292,8 +342,10 @@ }, { "cell_type": "markdown", - "id": "e82ee421", - "metadata": {}, + "id": "8c7ce425", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -303,16 +355,20 @@ }, { "cell_type": "markdown", - "id": "29dafa20", - "metadata": {}, + "id": "23cd06a6", + "metadata": { + "editable": true + }, "source": [ "In practical terms it means we will require" ] }, { "cell_type": "markdown", - "id": "865b2f5d", - "metadata": {}, + "id": "d40244fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = \\frac{\\partial }{\\partial \\beta_j}\\left[ \\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)^2\\right]=0,\n", @@ -321,16 +377,20 @@ }, { "cell_type": "markdown", - "id": "d2f7d6c6", - "metadata": {}, + "id": "003814bc", + "metadata": { + "editable": true + }, "source": [ "which results in" ] }, { "cell_type": "markdown", - "id": "45903ab2", - "metadata": {}, + "id": "765cdab4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_j} = -\\frac{2}{n}\\left[ \\sum_{i=0}^{n-1}x_{ij}\\left(y_i-\\beta_0x_{i,0}-\\beta_1x_{i,1}-\\beta_2x_{i,2}-\\dots-\\beta_{n-1}x_{i,n-1}\\right)\\right]=0,\n", @@ -339,26 +399,32 @@ }, { "cell_type": "markdown", - "id": "d11e2e5c", - "metadata": {}, + "id": "c10530e6", + "metadata": { + "editable": true + }, "source": [ - "or in a matrix-vector form as (multiplying away the factor $-2/n$)" + "or in a matrix-vector form as (multiplying away the factor $-2/n$, see derivation below)" ] }, { "cell_type": "markdown", - "id": "0ebbd61d", - "metadata": {}, + "id": "b1ce0925", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", + "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "88529a18", - "metadata": {}, + "id": "18e850a5", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "We can rewrite, see the derivations below," @@ -366,26 +432,32 @@ }, { "cell_type": "markdown", - "id": "713ac93a", - "metadata": {}, + "id": "9fe90791", + "metadata": { + "editable": true + }, "source": [ "$$\n", - "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n", + "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} = 0 = \\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right),\n", "$$" ] }, { "cell_type": "markdown", - "id": "75776840", - "metadata": {}, + "id": "320e1aef", + "metadata": { + "editable": true + }, "source": [ "as" ] }, { "cell_type": "markdown", - "id": "36f9ea8c", - "metadata": {}, + "id": "b6ea67f8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{y} = \\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -394,16 +466,20 @@ }, { "cell_type": "markdown", - "id": "9390b294", - "metadata": {}, + "id": "f374c3b2", + "metadata": { + "editable": true + }, "source": [ "and if the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ is invertible we have the solution" ] }, { "cell_type": "markdown", - "id": "30c81789", - "metadata": {}, + "id": "4a879ff3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta} =\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -412,8 +488,10 @@ }, { "cell_type": "markdown", - "id": "c035d652", - "metadata": {}, + "id": "71ca39b0", + "metadata": { + "editable": true + }, "source": [ "We note also that since our design matrix is defined as $\\boldsymbol{X}\\in\n", "{\\mathbb{R}}^{n\\times p}$, the product $\\boldsymbol{X}^T\\boldsymbol{X} \\in\n", @@ -431,8 +509,10 @@ }, { "cell_type": "markdown", - "id": "ff3ad5f2", - "metadata": {}, + "id": "0228f656", + "metadata": { + "editable": true + }, "source": [ "## Some useful matrix and vector expressions\n", "\n", @@ -456,8 +536,10 @@ }, { "cell_type": "markdown", - "id": "cb65be24", - "metadata": {}, + "id": "1a6493a2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}=f(\\boldsymbol{x}).\n", @@ -466,8 +548,10 @@ }, { "cell_type": "markdown", - "id": "5bb8744e", - "metadata": {}, + "id": "48e5ce45", + "metadata": { + "editable": true + }, "source": [ "## The Jacobian\n", "\n", @@ -476,8 +560,10 @@ }, { "cell_type": "markdown", - "id": "1e452b6a", - "metadata": {}, + "id": "2025325b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{J}=\\frac{\\partial \\boldsymbol{y}}{\\partial \\boldsymbol{x}}=\\begin{bmatrix} \\frac{\\partial y_0}{\\partial x_0} & \\frac{\\partial y_0}{\\partial x_1} & \\frac{\\partial y_0}{\\partial x_2} & \\dots & \\dots & \\frac{\\partial y_0}{\\partial x_{n-1}} \\\\ \\frac{\\partial y_0}{\\partial x_0} & \\frac{\\partial y_1}{\\partial x_1} & \\frac{\\partial y_1}{\\partial x_2} & \\dots & \\dots & \\frac{\\partial y_1}{\\partial x_{n-1}} \\\\\n", @@ -490,8 +576,10 @@ }, { "cell_type": "markdown", - "id": "1c7dfcc4", - "metadata": {}, + "id": "c4c51c01", + "metadata": { + "editable": true + }, "source": [ "which is an $m\\times n$ matrix. If $\\boldsymbol{x}$ is a scalar, then the\n", "Jacobian is only a single-column vector, or an $m\\times 1$ matrix. If\n", @@ -506,8 +594,10 @@ }, { "cell_type": "markdown", - "id": "57bb103b", - "metadata": {}, + "id": "5e0ba978", + "metadata": { + "editable": true + }, "source": [ "## Derivatives, example 1\n", "\n", @@ -516,8 +606,10 @@ }, { "cell_type": "markdown", - "id": "e097d436", - "metadata": {}, + "id": "4fd652f8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "y_i = \\sum_{j=0}^{n-1}a_{ij}x_j,\n", @@ -526,8 +618,10 @@ }, { "cell_type": "markdown", - "id": "9624a15b", - "metadata": {}, + "id": "694f5db3", + "metadata": { + "editable": true + }, "source": [ "with $\\forall i=0,1,2,\\dots,m-1$. The individual matrix elements of $\\boldsymbol{A}$ are given by the symbol $a_{ij}$.\n", "It follows that the partial derivatives of $y_i$ with respect to $x_k$" @@ -535,8 +629,10 @@ }, { "cell_type": "markdown", - "id": "2fe00b2d", - "metadata": {}, + "id": "52a90b73", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial y_i }{\\partial x_k}= a_{ik} \\forall i=0,1,2,\\dots,m-1.\n", @@ -545,16 +641,20 @@ }, { "cell_type": "markdown", - "id": "2cfbed2f", - "metadata": {}, + "id": "afab9d50", + "metadata": { + "editable": true + }, "source": [ "From this we have, using the definition of the Jacobian" ] }, { "cell_type": "markdown", - "id": "7f4c4c29", - "metadata": {}, + "id": "3dd5ebc2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\boldsymbol{y} }{\\partial \\boldsymbol{x}}= \\boldsymbol{A}.\n", @@ -563,8 +663,10 @@ }, { "cell_type": "markdown", - "id": "173f1116", - "metadata": {}, + "id": "d8571fd8", + "metadata": { + "editable": true + }, "source": [ "## Example 2\n", "\n", @@ -575,8 +677,10 @@ }, { "cell_type": "markdown", - "id": "0429caf6", - "metadata": {}, + "id": "50f37b40", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{y}^T\\boldsymbol{A}\\boldsymbol{x},\n", @@ -585,8 +689,10 @@ }, { "cell_type": "markdown", - "id": "3004627d", - "metadata": {}, + "id": "b4e94db7", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{y}$ a vector of length $m$, $\\boldsymbol{A}$ an $m\\times n$ matrix and $\\boldsymbol{x}$ a vector of length $n$. We assume also that $\\boldsymbol{A}$ does not depend on any of the two vectors.\n", "In order to find the derivative of $\\alpha$ with respect to the two vectors, we define an intermediate vector $\\boldsymbol{z}$. We define first\n", @@ -595,8 +701,10 @@ }, { "cell_type": "markdown", - "id": "9be48075", - "metadata": {}, + "id": "328ce3c0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{z}^T\\boldsymbol{x},\n", @@ -605,16 +713,20 @@ }, { "cell_type": "markdown", - "id": "774d2b3c", - "metadata": {}, + "id": "3c549113", + "metadata": { + "editable": true + }, "source": [ "which means that (using our previous example) we have" ] }, { "cell_type": "markdown", - "id": "6f1d157e", - "metadata": {}, + "id": "62a61819", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{x}} = \\boldsymbol{z}^T=\\boldsymbol{y}^T\\boldsymbol{A}.\n", @@ -623,16 +735,20 @@ }, { "cell_type": "markdown", - "id": "c134c6c5", - "metadata": {}, + "id": "0d8917d2", + "metadata": { + "editable": true + }, "source": [ "Since $\\alpha$ is a scalar we have $\\alpha =\\alpha^T=\\boldsymbol{x}^T\\boldsymbol{A}^T\\boldsymbol{y}$. Defining now $\\boldsymbol{z}=\\boldsymbol{x}^T\\boldsymbol{A}^T$ we find that" ] }, { "cell_type": "markdown", - "id": "5c70ca39", - "metadata": {}, + "id": "d27089b7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{y}} = \\boldsymbol{z}^T=\\boldsymbol{x}^T\\boldsymbol{A}^T.\n", @@ -641,8 +757,10 @@ }, { "cell_type": "markdown", - "id": "d75344c2", - "metadata": {}, + "id": "efbc5228", + "metadata": { + "editable": true + }, "source": [ "## Example 3\n", "\n", @@ -653,8 +771,10 @@ }, { "cell_type": "markdown", - "id": "4135d0c7", - "metadata": {}, + "id": "1539dfd3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x},\n", @@ -663,8 +783,10 @@ }, { "cell_type": "markdown", - "id": "c99dcd9d", - "metadata": {}, + "id": "56942822", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{x}$ a vector of length $n$.\n", "\n", @@ -673,8 +795,10 @@ }, { "cell_type": "markdown", - "id": "f1cdd616", - "metadata": {}, + "id": "444a2229", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\sum_{i=0}^{n-1}\\sum_{j=0}^{n-1}x_i a_{ij}x_j,\n", @@ -683,16 +807,20 @@ }, { "cell_type": "markdown", - "id": "8eaa500d", - "metadata": {}, + "id": "c19b8e9a", + "metadata": { + "editable": true + }, "source": [ "taking the derivative of $\\alpha$ with respect to a given component $x_k$ we get the two sums" ] }, { "cell_type": "markdown", - "id": "5cbe436e", - "metadata": {}, + "id": "c3da7629", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial x_k} = \\sum_{i=0}^{n-1}a_{ik}x_i+\\sum_{j=0}^{n-1}a_{kj}x_j,\n", @@ -701,16 +829,20 @@ }, { "cell_type": "markdown", - "id": "9f60d600", - "metadata": {}, + "id": "bc0bb616", + "metadata": { + "editable": true + }, "source": [ "for $\\forall k =0,1,2,\\dots,n-1$. We identify these sums as" ] }, { "cell_type": "markdown", - "id": "7207d303", - "metadata": {}, + "id": "81f469c7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{x}} = \\boldsymbol{x}^T\\left(\\boldsymbol{A}^T+\\boldsymbol{A}\\right).\n", @@ -719,16 +851,20 @@ }, { "cell_type": "markdown", - "id": "d02f7e0b", - "metadata": {}, + "id": "e7dbc066", + "metadata": { + "editable": true + }, "source": [ "If the matrix $\\boldsymbol{A}$ is symmetric, that is $\\boldsymbol{A}=\\boldsymbol{A}^T$, we have" ] }, { "cell_type": "markdown", - "id": "3630e104", - "metadata": {}, + "id": "e15843da", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{x}} = 2\\boldsymbol{x}^T\\boldsymbol{A}.\n", @@ -737,8 +873,10 @@ }, { "cell_type": "markdown", - "id": "54bf967b", - "metadata": {}, + "id": "31a95917", + "metadata": { + "editable": true + }, "source": [ "## Example 4\n", "\n", @@ -747,8 +885,10 @@ }, { "cell_type": "markdown", - "id": "9120827b", - "metadata": {}, + "id": "a3c00979", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\boldsymbol{y}^T\\boldsymbol{x},\n", @@ -757,8 +897,10 @@ }, { "cell_type": "markdown", - "id": "3eea746b", - "metadata": {}, + "id": "03321be3", + "metadata": { + "editable": true + }, "source": [ "where both $\\boldsymbol{y}$ and $\\boldsymbol{x}$ have the same length $n$, or if we\n", "wish to think of them as column vectors, they have dimensions $n\\times\n", @@ -770,8 +912,10 @@ }, { "cell_type": "markdown", - "id": "d4283866", - "metadata": {}, + "id": "59854eb8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha = \\sum_{i=0}^{n-1}y_ix_i,\n", @@ -780,16 +924,20 @@ }, { "cell_type": "markdown", - "id": "a40bf496", - "metadata": {}, + "id": "1f475d05", + "metadata": { + "editable": true + }, "source": [ "and the partial derivative" ] }, { "cell_type": "markdown", - "id": "108364a8", - "metadata": {}, + "id": "256476a9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial z_k} = \\sum_{i=0}^{n-1}\\left(x_i\\frac{\\partial y_i}{\\partial z_k}+y_i\\frac{\\partial x_i}{\\partial z_k}\\right),\n", @@ -798,16 +946,20 @@ }, { "cell_type": "markdown", - "id": "225fdf02", - "metadata": {}, + "id": "f23c2136", + "metadata": { + "editable": true + }, "source": [ "for $\\forall k =0,1,2,\\dots,n-1$. We can rewrite the partial derivative in a more compact form as" ] }, { "cell_type": "markdown", - "id": "c254f629", - "metadata": {}, + "id": "afeafdcb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{z}} = \\boldsymbol{x}^T\\frac{\\partial \\boldsymbol{y}}{\\partial \\boldsymbol{z}}+\\boldsymbol{y}^T\\frac{\\partial \\boldsymbol{x}}{\\partial \\boldsymbol{z}},\n", @@ -816,16 +968,20 @@ }, { "cell_type": "markdown", - "id": "e00dbb5c", - "metadata": {}, + "id": "4ed3eead", + "metadata": { + "editable": true + }, "source": [ "and if $\\boldsymbol{y}=\\boldsymbol{x}$ we have" ] }, { "cell_type": "markdown", - "id": "d6ae171e", - "metadata": {}, + "id": "9674c619", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\alpha}{\\partial \\boldsymbol{z}} = 2\\boldsymbol{x}^T\\frac{\\partial \\boldsymbol{x}}{\\partial \\boldsymbol{z}}.\n", @@ -834,8 +990,10 @@ }, { "cell_type": "markdown", - "id": "64090d25", - "metadata": {}, + "id": "82ae15ae", + "metadata": { + "editable": true + }, "source": [ "## The mean squared error and its derivative\n", "\n", @@ -844,8 +1002,10 @@ }, { "cell_type": "markdown", - "id": "ca920a83", - "metadata": {}, + "id": "688a7168", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\sum_{i=0}^{n-1}\\left(y_i-\\tilde{y}_i\\right)^2=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}}\\right)\\right\\},\n", @@ -854,16 +1014,20 @@ }, { "cell_type": "markdown", - "id": "6cf46fcb", - "metadata": {}, + "id": "01f03acf", + "metadata": { + "editable": true + }, "source": [ "or using the design/feature matrix $\\boldsymbol{X}$ we have the more compact matrix-vector" ] }, { "cell_type": "markdown", - "id": "d7f69bce", - "metadata": {}, + "id": "a30bd0f2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -872,19 +1036,23 @@ }, { "cell_type": "markdown", - "id": "e96b7e86", - "metadata": {}, + "id": "5593e084", + "metadata": { + "editable": true + }, "source": [ "We note that the design matrix $\\boldsymbol{X}$ does not depend on the unknown parameters defined by the vector $\\boldsymbol{\\beta}$.\n", "We are now interested in minimizing the cost function with respect to the unknown parameters $\\boldsymbol{\\beta}$.\n", "\n", - "The mean squared error is scalar and if we use the results from the last example, we define a new vector" + "The mean squared error is a scalar and if we use the results from the last example, we define a new vector" ] }, { "cell_type": "markdown", - "id": "d128efac", - "metadata": {}, + "id": "0ecc87b7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{w}=\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -893,16 +1061,20 @@ }, { "cell_type": "markdown", - "id": "7efcf864", - "metadata": {}, + "id": "d87cf1be", + "metadata": { + "editable": true + }, "source": [ "which depends on $\\boldsymbol{\\beta}$. We rewrite the cost function as" ] }, { "cell_type": "markdown", - "id": "2d31fe86", - "metadata": {}, + "id": "c8ac2b85", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{1}{n}\\boldsymbol{w}^T\\boldsymbol{w},\n", @@ -911,16 +1083,20 @@ }, { "cell_type": "markdown", - "id": "de727a1f", - "metadata": {}, + "id": "2a1896de", + "metadata": { + "editable": true + }, "source": [ "with partial derivative" ] }, { "cell_type": "markdown", - "id": "3ae45138", - "metadata": {}, + "id": "f07be920", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=\\frac{2}{n}\\boldsymbol{w}^T\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\beta}},\n", @@ -929,16 +1105,20 @@ }, { "cell_type": "markdown", - "id": "2ce6a086", - "metadata": {}, + "id": "8351f7a8", + "metadata": { + "editable": true + }, "source": [ "and using that" ] }, { "cell_type": "markdown", - "id": "168f47a1", - "metadata": {}, + "id": "b321536f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\boldsymbol{w}}{\\partial \\boldsymbol{\\beta}}=-\\boldsymbol{X},\n", @@ -947,16 +1127,20 @@ }, { "cell_type": "markdown", - "id": "46c9bef4", - "metadata": {}, + "id": "7dea2f6e", + "metadata": { + "editable": true + }, "source": [ - "where we ued the results from example two. Inserting the last expression we obtain" + "where we used the result from example two above. Inserting the last expression we obtain" ] }, { "cell_type": "markdown", - "id": "517fe316", - "metadata": {}, + "id": "fc8335b3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\boldsymbol{X},\n", @@ -965,16 +1149,20 @@ }, { "cell_type": "markdown", - "id": "0eac480c", - "metadata": {}, + "id": "690c48cc", + "metadata": { + "editable": true + }, "source": [ "or as" ] }, { "cell_type": "markdown", - "id": "9f870277", - "metadata": {}, + "id": "dbf4e99e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T}=-\\frac{2}{n}\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right).\n", @@ -983,8 +1171,10 @@ }, { "cell_type": "markdown", - "id": "90d819f0", - "metadata": {}, + "id": "20066c64", + "metadata": { + "editable": true + }, "source": [ "## Other useful relations\n", "\n", @@ -993,8 +1183,10 @@ }, { "cell_type": "markdown", - "id": "7c054d38", - "metadata": {}, + "id": "512705d5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial (\\boldsymbol{b}^T\\boldsymbol{a})}{\\partial \\boldsymbol{a}} = \\boldsymbol{b},\n", @@ -1003,8 +1195,10 @@ }, { "cell_type": "markdown", - "id": "a74afd02", - "metadata": {}, + "id": "898589c9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial tr(\\boldsymbol{B}\\boldsymbol{A})}{\\partial \\boldsymbol{A}} = \\boldsymbol{B}^T,\n", @@ -1013,8 +1207,10 @@ }, { "cell_type": "markdown", - "id": "5daf72a9", - "metadata": {}, + "id": "04fdb8da", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\log{\\vert\\boldsymbol{A}\\vert}}{\\partial \\boldsymbol{A}} = (\\boldsymbol{A}^{-1})^T.\n", @@ -1023,8 +1219,10 @@ }, { "cell_type": "markdown", - "id": "b2cdc0f7", - "metadata": {}, + "id": "fb0d6555", + "metadata": { + "editable": true + }, "source": [ "## Meet the Hessian Matrix\n", "\n", @@ -1037,8 +1235,10 @@ }, { "cell_type": "markdown", - "id": "3840002f", - "metadata": {}, + "id": "61ca107d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -1047,16 +1247,20 @@ }, { "cell_type": "markdown", - "id": "fbb502ee", - "metadata": {}, + "id": "71be4e48", + "metadata": { + "editable": true + }, "source": [ "The Hessian matrix plays an important role and is defined here as" ] }, { "cell_type": "markdown", - "id": "94f74938", - "metadata": {}, + "id": "9fc474c3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -1065,15 +1269,17 @@ }, { "cell_type": "markdown", - "id": "0c387c69", - "metadata": {}, + "id": "23a27406", + "metadata": { + "editable": true + }, "source": [ "For ordinary least squares, it is inversely proportional (derivation\n", "next week) with the variance of the optimal parameters\n", "$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that it is\n", - "(beside $1/n$) equal to the covariance matrix. It plays also a very\n", + "(aside the factor $1/n$) equal to the covariance matrix. It plays also a very\n", "important role in optmization algorithms and Principal Component\n", - "Analysis as a way to reduce the dimensionality of a machine learning\n", + "Analysis as a way to reduce the dimensionality of a machine learning/data analysis\n", "problem.\n", "\n", "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!)." @@ -1081,8 +1287,10 @@ }, { "cell_type": "markdown", - "id": "0d804211", - "metadata": {}, + "id": "9aa37656", + "metadata": { + "editable": true + }, "source": [ "## Interpretations and optimizing our parameters\n", "\n", @@ -1091,8 +1299,10 @@ }, { "cell_type": "markdown", - "id": "53d01ed8", - "metadata": {}, + "id": "1458e79a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon} = \\boldsymbol{y}-\\boldsymbol{\\tilde{y}} = \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta},\n", @@ -1101,16 +1311,20 @@ }, { "cell_type": "markdown", - "id": "4e460081", - "metadata": {}, + "id": "c3fcfcdf", + "metadata": { + "editable": true + }, "source": [ "and with" ] }, { "cell_type": "markdown", - "id": "076effdd", - "metadata": {}, + "id": "9f169fad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", @@ -1119,16 +1333,20 @@ }, { "cell_type": "markdown", - "id": "31e4cff9", - "metadata": {}, + "id": "81e7723a", + "metadata": { + "editable": true + }, "source": [ "we have" ] }, { "cell_type": "markdown", - "id": "10f160dd", - "metadata": {}, + "id": "b2551cd1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{\\epsilon}=\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)= 0,\n", @@ -1137,16 +1355,20 @@ }, { "cell_type": "markdown", - "id": "06ad3ea3", - "metadata": {}, + "id": "601003c8", + "metadata": { + "editable": true + }, "source": [ "meaning that the solution for $\\boldsymbol{\\beta}$ is the one which minimizes the residuals." ] }, { "cell_type": "markdown", - "id": "a5f71548", - "metadata": {}, + "id": "1f2fdfa6", + "metadata": { + "editable": true + }, "source": [ "## Example relevant for the exercises\n", "\n", @@ -1157,8 +1379,10 @@ }, { "cell_type": "markdown", - "id": "421b5969", - "metadata": {}, + "id": "ea1518ad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_i = \\beta_0+\\beta_1x_i+\\beta_2x_i^2+\\beta_3x_i^3+\\beta_4x_i^4.\n", @@ -1167,8 +1391,10 @@ }, { "cell_type": "markdown", - "id": "382915d9", - "metadata": {}, + "id": "758b843b", + "metadata": { + "editable": true + }, "source": [ "we have five predictors/features. The first is the intercept $\\beta_0$. The other terms are $\\beta_i$ with $i=1,2,3,4$. Furthermore we have $n$ entries for each predictor. It means that our design matrix is a \n", "$p\\times n$ matrix $\\boldsymbol{X}$." @@ -1176,8 +1402,10 @@ }, { "cell_type": "markdown", - "id": "053fc460", - "metadata": {}, + "id": "24715db8", + "metadata": { + "editable": true + }, "source": [ "## Own code for Ordinary Least Squares\n", "\n", @@ -1187,8 +1415,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "f4596688", - "metadata": {}, + "id": "f8b57463", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# matrix inversion to find beta\n", @@ -1199,8 +1430,10 @@ }, { "cell_type": "markdown", - "id": "65c4e9b3", - "metadata": {}, + "id": "70c97572", + "metadata": { + "editable": true + }, "source": [ "Alternatively, you can use the least squares functionality in **Numpy** as" ] @@ -1208,8 +1441,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "8ef47e51", - "metadata": {}, + "id": "1aa95889", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "fit = np.linalg.lstsq(X, Energies, rcond =None)[0]\n", @@ -1218,8 +1454,10 @@ }, { "cell_type": "markdown", - "id": "f55e615f", - "metadata": {}, + "id": "e250b2e9", + "metadata": { + "editable": true + }, "source": [ "## Adding error analysis and training set up\n", "\n", @@ -1230,8 +1468,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "53b41f14", - "metadata": {}, + "id": "a86bd577", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def R2(y_data, y_model):\n", @@ -1240,8 +1481,10 @@ }, { "cell_type": "markdown", - "id": "a880dd98", - "metadata": {}, + "id": "d6ccd7b8", + "metadata": { + "editable": true + }, "source": [ "and we would be using it as" ] @@ -1249,8 +1492,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "a98298a6", - "metadata": {}, + "id": "fb6f7c19", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "print(R2(Energies,ytilde))" @@ -1258,8 +1504,10 @@ }, { "cell_type": "markdown", - "id": "ea20f812", - "metadata": {}, + "id": "eca18427", + "metadata": { + "editable": true + }, "source": [ "We can easily add our **MSE** score as" ] @@ -1267,8 +1515,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "b0c16cad", - "metadata": {}, + "id": "0fd2b39c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def MSE(y_data,y_model):\n", @@ -1280,8 +1531,10 @@ }, { "cell_type": "markdown", - "id": "575e1949", - "metadata": {}, + "id": "db3fe7d9", + "metadata": { + "editable": true + }, "source": [ "and finally the relative error as" ] @@ -1289,8 +1542,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "69dc8db3", - "metadata": {}, + "id": "9f59c14e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def RelativeError(y_data,y_model):\n", @@ -1300,8 +1556,10 @@ }, { "cell_type": "markdown", - "id": "e619e325", - "metadata": {}, + "id": "2fdd5915", + "metadata": { + "editable": true + }, "source": [ "## Splitting our Data in Training and Test data\n", "\n", @@ -1319,34 +1577,23 @@ }, { "cell_type": "markdown", - "id": "61132ee3", - "metadata": {}, + "id": "9863e063", + "metadata": { + "editable": true + }, "source": [ "## The complete code with a simple data set" ] }, { "cell_type": "code", - "execution_count": 1, - "id": "a5f35f01", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[ 2.03535465 -0.26967974 5.68946042 -1.20793418 0.83015856]\n", - "Training R2\n", - "0.9943978947069526\n", - "Training MSE\n", - "0.010497411048834778\n", - "Test R2\n", - "0.9919208828799804\n", - "Test MSE\n", - "0.012883691086663773\n" - ] - } - ], + "execution_count": 7, + "id": "85fc6a78", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -1394,17 +1641,22 @@ }, { "cell_type": "markdown", - "id": "7f317a8b", - "metadata": {}, + "id": "6671297c", + "metadata": { + "editable": true + }, "source": [ "## Making your own test-train splitting" ] }, { "cell_type": "code", - "execution_count": 2, - "id": "e9687fbf", - "metadata": {}, + "execution_count": 8, + "id": "810f32f7", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# equivalently in numpy\n", @@ -1425,8 +1677,10 @@ }, { "cell_type": "markdown", - "id": "bd7417f3", - "metadata": {}, + "id": "1ca5e45d", + "metadata": { + "editable": true + }, "source": [ "But since **scikit-learn** has its own function for doing this and since\n", "it interfaces easily with **tensorflow** and other libraries, we\n", @@ -1435,8 +1689,10 @@ }, { "cell_type": "markdown", - "id": "f586d0bb", - "metadata": {}, + "id": "b6530670", + "metadata": { + "editable": true + }, "source": [ "## Reducing the number of degrees of freedom, overarching view\n", "\n", @@ -1462,8 +1718,10 @@ }, { "cell_type": "markdown", - "id": "e60a9ab8", - "metadata": {}, + "id": "4be4158c", + "metadata": { + "editable": true + }, "source": [ "## Preprocessing our data\n", "\n", @@ -1485,8 +1743,10 @@ }, { "cell_type": "markdown", - "id": "337c20e1", - "metadata": {}, + "id": "aa6f9ff6", + "metadata": { + "editable": true + }, "source": [ "## Functionality in Scikit-Learn\n", "\n", @@ -1503,8 +1763,10 @@ }, { "cell_type": "markdown", - "id": "e1a68a5d", - "metadata": {}, + "id": "856dee18", + "metadata": { + "editable": true + }, "source": [ "## More preprocessing\n", "\n", @@ -1528,8 +1790,10 @@ }, { "cell_type": "markdown", - "id": "f8b2ab1a", - "metadata": {}, + "id": "3497598c", + "metadata": { + "editable": true + }, "source": [ "## Frequently used scaling functions\n", "\n", @@ -1539,8 +1803,10 @@ }, { "cell_type": "markdown", - "id": "7da909f4", - "metadata": {}, + "id": "fe24fd3f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_j^{(i)} \\rightarrow \\frac{x_j^{(i)} - \\overline{x}_j}{\\sigma(x_j)},\n", @@ -1549,8 +1815,10 @@ }, { "cell_type": "markdown", - "id": "811c0870", - "metadata": {}, + "id": "55bccc6e", + "metadata": { + "editable": true + }, "source": [ "where $\\overline{x}_j$ and $\\sigma(x_j)$ are the mean and standard deviation, respectively, of the feature $x_j$.\n", "This ensures that each feature has zero mean and unit standard deviation. For data sets where we do not have the standard deviation or don't wish to calculate it, it is then common to simply set it to one." @@ -1558,8 +1826,10 @@ }, { "cell_type": "markdown", - "id": "9721560a", - "metadata": {}, + "id": "e01c4d09", + "metadata": { + "editable": true + }, "source": [ "## Example of own Standard scaling\n", "\n", @@ -1571,419 +1841,13 @@ }, { "cell_type": "code", - "execution_count": 3, - "id": "65a4f0c3", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
01234
0-1.7497650.3426801.153036-0.2524360.981321
10.5142190.221180-1.070043-0.1894960.255001
2-0.4580270.435163-0.5835950.8168470.672721
3-0.104411-0.5312801.029733-0.438136-1.118318
41.6189821.541605-0.251879-0.8424360.184519
50.9370820.7310001.361556-0.3262380.055676
60.222400-1.443217-0.7563520.8164540.750445
7-0.4559471.189622-1.690617-1.356399-1.232435
8-0.544439-0.6681720.007315-0.6129391.299748
9-1.733096-0.9833100.357508-1.6135791.470714
\n", - "
" - ], - "text/plain": [ - " 0 1 2 3 4\n", - "0 -1.749765 0.342680 1.153036 -0.252436 0.981321\n", - "1 0.514219 0.221180 -1.070043 -0.189496 0.255001\n", - "2 -0.458027 0.435163 -0.583595 0.816847 0.672721\n", - "3 -0.104411 -0.531280 1.029733 -0.438136 -1.118318\n", - "4 1.618982 1.541605 -0.251879 -0.842436 0.184519\n", - "5 0.937082 0.731000 1.361556 -0.326238 0.055676\n", - "6 0.222400 -1.443217 -0.756352 0.816454 0.750445\n", - "7 -0.455947 1.189622 -1.690617 -1.356399 -1.232435\n", - "8 -0.544439 -0.668172 0.007315 -0.612939 1.299748\n", - "9 -1.733096 -0.983310 0.357508 -1.613579 1.470714" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0 -0.175300\n", - "1 0.083527\n", - "2 -0.044334\n", - "3 -0.399836\n", - "4 0.331939\n", - "dtype: float64\n", - "0 1.069584\n", - "1 0.965548\n", - "2 1.018232\n", - "3 0.793167\n", - "4 0.918992\n", - "dtype: float64\n" - ] - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
01234
0-1.5744650.2591531.1973700.1474000.649382
10.6895190.137652-1.0257090.210340-0.076938
2-0.2827270.351636-0.5392611.2166830.340782
30.070889-0.6148081.074067-0.038300-1.450257
41.7942821.458078-0.207545-0.442600-0.147420
51.1123830.6474731.4058900.073598-0.276263
60.397700-1.526744-0.7120181.2162900.418506
7-0.2806471.106095-1.646283-0.956563-1.564374
8-0.369139-0.7516990.051649-0.2131030.967809
9-1.557795-1.0668370.401842-1.2137431.138775
\n", - "
" - ], - "text/plain": [ - " 0 1 2 3 4\n", - "0 -1.574465 0.259153 1.197370 0.147400 0.649382\n", - "1 0.689519 0.137652 -1.025709 0.210340 -0.076938\n", - "2 -0.282727 0.351636 -0.539261 1.216683 0.340782\n", - "3 0.070889 -0.614808 1.074067 -0.038300 -1.450257\n", - "4 1.794282 1.458078 -0.207545 -0.442600 -0.147420\n", - "5 1.112383 0.647473 1.405890 0.073598 -0.276263\n", - "6 0.397700 -1.526744 -0.712018 1.216290 0.418506\n", - "7 -0.280647 1.106095 -1.646283 -0.956563 -1.564374\n", - "8 -0.369139 -0.751699 0.051649 -0.213103 0.967809\n", - "9 -1.557795 -1.066837 0.401842 -1.213743 1.138775" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
01234
00.00.00.00.00.0
10.00.00.00.00.0
20.00.00.00.00.0
30.00.00.00.00.0
40.00.00.00.00.0
50.00.00.00.00.0
60.00.00.00.00.0
70.00.00.00.00.0
80.00.00.00.00.0
90.00.00.00.00.0
\n", - "
" - ], - "text/plain": [ - " 0 1 2 3 4\n", - "0 0.0 0.0 0.0 0.0 0.0\n", - "1 0.0 0.0 0.0 0.0 0.0\n", - "2 0.0 0.0 0.0 0.0 0.0\n", - "3 0.0 0.0 0.0 0.0 0.0\n", - "4 0.0 0.0 0.0 0.0 0.0\n", - "5 0.0 0.0 0.0 0.0 0.0\n", - "6 0.0 0.0 0.0 0.0 0.0\n", - "7 0.0 0.0 0.0 0.0 0.0\n", - "8 0.0 0.0 0.0 0.0 0.0\n", - "9 0.0 0.0 0.0 0.0 0.0" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 9, + "id": "6098d5d9", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import sklearn.linear_model as skl\n", "from sklearn.metrics import mean_squared_error\n", @@ -2012,16 +1876,20 @@ }, { "cell_type": "markdown", - "id": "80a703b0", - "metadata": {}, + "id": "653340bf", + "metadata": { + "editable": true + }, "source": [ "Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives." ] }, { "cell_type": "markdown", - "id": "f540d5c9", - "metadata": {}, + "id": "673c8917", + "metadata": { + "editable": true + }, "source": [ "## Min-Max Scaling\n", "\n", @@ -2033,8 +1901,10 @@ }, { "cell_type": "markdown", - "id": "3637a69b", - "metadata": {}, + "id": "4f69865b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_j^{(i)} \\rightarrow (b-a)\\frac{x_j^{(i)} - \\min(x_j)}{\\max(x_j) - \\min(x_j)} - a\n", @@ -2043,16 +1913,20 @@ }, { "cell_type": "markdown", - "id": "6baf7db9", - "metadata": {}, + "id": "f39433a0", + "metadata": { + "editable": true + }, "source": [ "where $\\min(x_j)$ and $\\max(x_j)$ return the minimum and maximum value of $x_j$ over the data set, respectively." ] }, { "cell_type": "markdown", - "id": "bb3b37b8", - "metadata": {}, + "id": "f2f5cb76", + "metadata": { + "editable": true + }, "source": [ "## Testing the Means Squared Error as function of Complexity\n", "\n", @@ -2064,9 +1938,12 @@ }, { "cell_type": "code", - "execution_count": 4, - "id": "099cdcf3", - "metadata": {}, + "execution_count": 10, + "id": "12dbcdcb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "np.random.seed()\n", @@ -2079,8 +1956,10 @@ }, { "cell_type": "markdown", - "id": "a28399c3", - "metadata": {}, + "id": "38cb3698", + "metadata": { + "editable": true + }, "source": [ "where $y$ is the function we want to fit with a given polynomial.\n", "\n", @@ -2089,21 +1968,13 @@ }, { "cell_type": "code", - "execution_count": 5, - "id": "eea94fac", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAGdCAYAAAAxCSikAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABcpUlEQVR4nO3dd3wUdf7H8dfuppGQBEIgoYQQOkEIEHpRUKRjOxX1FD31PBQL8POsp4DeiZ6onAXUs93pCUFBRUQhFpooSgkgIXQIJSGEkgZpu/P7Y5NIIIQkJJndzfv5eMxjJ5Pvzn6GUfbNzHy/X4thGAYiIiIiLsxqdgEiIiIiF6LAIiIiIi5PgUVERERcngKLiIiIuDwFFhEREXF5CiwiIiLi8hRYRERExOUpsIiIiIjL8zK7gOricDg4fPgwgYGBWCwWs8sRERGRCjAMg6ysLJo1a4bVev7rKB4TWA4fPkxERITZZYiIiEgVHDhwgBYtWpz39x4TWAIDAwHnAQcFBZlcjYiIiFREZmYmERERJd/j5+MxgaX4NlBQUJACi4iIiJu50OMceuhWREREXJ4Ci4iIiLg8BRYRERFxeR7zDIuIiLgvwzAoLCzEbrebXYpUM5vNhpeX10UPOaLAIiIipsrPzyclJYVTp06ZXYrUEH9/f5o2bYqPj0+V96HAIiIipnE4HOzduxebzUazZs3w8fHR4J8exDAM8vPzOXr0KHv37qVdu3blDg5XHgUWERExTX5+Pg6Hg4iICPz9/c0uR2pAvXr18Pb2Zv/+/eTn5+Pn51el/eihWxERMV1V/9Ut7qE6zq/+CxERERGXp8AiIiIiLk+BRURERFyeAouIiEglWCyWcpc77rijyvtu1aoVs2bNqlC7sj77+eefr/Jnuzr1EhKPZhgGn64/iN1h8IfYFnjblNFF5OKkpKSUrMfFxfH000+zffv2km316tWrlTqeeeYZ/vznP5fadr4Zjw3DwG634+VV+ms/Pz+/SmOjVPV9F0N/e4vHysot4N6PNvDXTzfz2MItjPzXKlbtPGp2WSJSDsMwOJVfaMpiGEaFagwPDy9ZgoODsVgspbatXLmS2NhY/Pz8aN26NdOnT6ewsLDk/dOmTaNly5b4+vrSrFkzHnzwQQAGDx7M/v37mTx5cskVk/IEBgaW+tzw8HACAgIAWL58ORaLhaVLl9KzZ098fX1ZtWoVgwcP5v7772fKlCmEhoZy5ZVXArBixQp69+6Nr68vTZs25bHHHitV8/neV5t0hUU80o4jWUz4cD170nPwsVkJ8LWxKy2b2979hSujw3hqdDQtG2nMBxFXc7rATvTTS0357MRnhuPvc3Ffi0uXLuXWW2/l1VdfZdCgQezevZt77rkHgKlTp/Lpp5/yyiuvMG/ePDp37kxqaiqbNm0CYOHChcTExHDPPfecc+Wkqh555BFmzpxJ69atadCgAQD/+c9/uPfee/nxxx8xDINDhw4xatQo7rjjDv773/+SlJTEn//8Z/z8/Jg2bVrJvs5+X21TYBGP8+Wmwzy6YDOn8u00C/Zj9q2xRDUKYNZ3O/jvT/uJTzzCiu1H+fOlUdw3uC0BvvrfQESqxz/+8Q8ee+wxbr/9dgBat27Ns88+yyOPPMLUqVNJTk4mPDycoUOH4u3tTcuWLenduzcAISEh2Gy2kisnF/Loo4/yt7/9rdS2xYsXM3jw4JKfn3nmmXOuhrRt25Z//vOfJT8/+eSTRERE8Prrr2OxWOjYsSOHDx/m0Ucf5emnny4ZQ+Xs99U2/U0tHqPA7mDGkiTe+3EvAAPaNuLVm7rTqL4vAFPHdubm3i155stEVu9K540fdvPp+oM8PrITV3drpuHARVxAPW8bic8MN+2zL9b69ev59ddf+cc//lGyzW63k5uby6lTp7jhhhuYNWsWrVu3ZsSIEYwaNYqxY8ee82xJRfz1r3895wHf5s2bl/q5Z8+e57zv7G3btm2jX79+pf4OHDBgANnZ2Rw8eJCWLVued1+1SYFFPEJaZi4TP97Ar/tOAHDf4Db837AO2KylQ0j7sEA+vKs3yxKP8PevEjlw/DST4hL46Of9TLuqM5c0DzajfBEpYrFYLvq2jJkcDgfTp0/nuuuuO+d3fn5+REREsH37duLj4/n222+57777ePHFF1mxYgXe3t6V+qzQ0FDatm1bbpviZ1rK22YYxjn/YCu+5XPm9rL2VZvc978KkSK/7D3OxI83cDQrj0BfL166MYZhnc9/OdVisTC8cziXtW/MO6v28MYPu1m3/wRjX1/NuJ4RPDy8A6FFV2VERCqjR48ebN++vdwgUa9ePa666iquuuoqJk6cSMeOHdmyZQs9evTAx8cHu91eixVDdHQ0CxYsKBVc1qxZQ2Bg4DlXbMykwCJuyzAM3vtxH88t2YbdYdAhLJA3b4slKrRi/wrw87Zx/+Xt+ENsC57/OokvEg4z79cDfLUlhUlD2zO+X6S6QYtIpTz99NOMGTOGiIgIbrjhBqxWK5s3b2bLli38/e9/54MPPsBut9OnTx/8/f358MMPqVevHpGRkYBzfJWVK1dy00034evrS2ho6Hk/Kysri9TU1FLb/P39CQoKqlTN9913H7NmzeKBBx7g/vvvZ/v27UydOpUpU6a41BxPrlOJSCXk5BXywNyNPLs4EbvD4OpuzfhsYv8Kh5UzNQ2ux79u6s4nE/rRuVkQWbmFPLs4kZH/WsXKHeoGLSIVN3z4cBYvXkx8fDy9evWib9++vPzyyyWBpEGDBvz73/9mwIABdO3ale+++44vv/ySRo0aAc6HZPft20ebNm1o3LhxuZ/19NNP07Rp01LLI488UumamzdvzpIlS/jll1+IiYlhwoQJ3HXXXec80Gs2i2FG36QakJmZSXBwMBkZGZVOl+Jedh/NZsKH69mZlo2X1cLfRnfi9v6tquWhWbvDYP66A7y4dDvHc/IBuDI6jL+N7kRkI3Pv34p4otzcXPbu3UtUVBR+fn5mlyM1pLzzXNHvb11hEbfyzW8pXP36j+xMy6ZJoC/z7unLHQOiqq2Hj81q4ebeLfnh4cHcOSAKm9VCfOIRrnx5Jf/8JomcvMIL70RERKqdAou4hUK7gxlfb2PCRxvIziukd1QIix8cSM9WITXyecH1vHl6bDTfPDSIQe1Cybc7mL18N5e/tJzPNx4yZdAkEZG6TIFFXF56dh63vfsLb63YA8CfB0Xxv7v70CSw5i8ftwsL5L939ubt22JpGeLPkcw8JsUlcP2bP7HlYEaNf76IiDgpsIhL25B8gjGvruanPcfw97Hxxi09eHJ0dK323rFYLAzrHM6yyZfy1+EdqOdtY/3+E1z1xmoe/XQz6dl5tVaLiEhdpcAiLskwDD78aR/j3vqJ1Mxc2jQOYNH9AxjdtalpNfl525g4pC0/PDyYa7o1wzAgbt0Bhry4nHdW7aHA7jCtNhERT6fAIi7ndL6d/5u/iae+2EqB3WBUl3C+uH8gbZuUPW16bQsP9mPWTd35dEI/LmkeRFZeIX//ahsjZq1khbpBi4jUCAUWcSn7j+Vw7ewfWbjxEDarhSdHdeKNW3pQ3wUnKOzZKoQvJg7k+eu60CjAh91Hc7j9vV+4+z+/si89x+zyREQ8igKLuIxvE48w5rXVJKVmEVrfh//d3Yc/X9rapScltFkt3NS7Jd8XdYP2slr4dlsaw15ZyQvqBi0iUm0UWMR0dofBS8u2c/d/15GVW0hsZEMWPzCIvq0bmV1ahZV0g570ezfoOct3M2Tmcj7beFDdoEXkggYPHsykSZPMLsNlKbCIqU7k5HPH+7/w2ve7ALijfyvm/rkv4cHuOeJl2yalu0GnZeUxOW4Tf5izhs0HT5pdnohUA4vFUu5yxx13VGm/Cxcu5Nlnn72o2u64444yaxoxYsRF7dcVuN6DAa7EYYc1r8KWBTD+cwg4/yRUUnmbD57k3o82cOjkaep523j+D124upvrzAxaVcXdoC9t35h3V+/ljR92sSH5JFe/8SM3xLbgr8M70jhQs0GLuKuUlJSS9bi4OJ5++mm2b99esq1evXql2hcUFODt7X3B/YaEVM9AmCNGjOD9998vtc3X9/x/55RVX0Vrrsi+qouusJTHaoOtn8ORLbDlU7Or8Sjzfknm+jk/cejkaVo18uezif09Iqycqbgb9Pf/93s36PnrDnL5TGc36PxCdYMWcUfh4eElS3BwMBaLpeTn3NxcGjRowPz58xk8eDB+fn589NFHHDt2jJtvvpkWLVrg7+9Ply5dmDt3bqn9nn1LqFWrVjz33HPceeedBAYG0rJlS95+++0L1ufr61uqxvDwcBo2bFjye4vFwptvvsnVV19NQEAAf//735k2bRrdunXjvffeo3Xr1vj6+mIYBsnJyVx99dXUr1+foKAgbrzxRo4cOVKyr/O9ryYosFxIt1ucr5s+NrcOD5FbYOeRTzfx2MIt5NsdXBkdxqIHBtIx3HMnrCzuBr3g3n50aR78ezfof61k+fY0s8sTcS2GAfk55izV+EX76KOP8uCDD7Jt2zaGDx9Obm4usbGxLF68mN9++4177rmH2267jbVr15a7n5deeomePXuyceNG7rvvPu69916SkpIuur6pU6dy9dVXs2XLFu68804Adu3axfz581mwYAEJCQkAXHPNNRw/fpwVK1YQHx/P7t27GTduXKl9lfW+mqBbQhdyyfWw9ElI2QRHEiEs2uyK3NaB46e493/r+e1QJlYL/N+wDtx7WRusVtftBVSdYiND+HziAD4pmg16z9Ec7nj/V4Z2asLfRkfTKlSzQYtQcAqea2bOZz9xGHyq5//DSZMmcd1115Xa9vDDD5esP/DAA3zzzTd88skn9OnT57z7GTVqFPfddx/gDEGvvPIKy5cvp2PHjud9z+LFi6lfv36pbY8++ihPPfVUyc+33HJLSVAplp+fz4cffkjjxo0BiI+PZ/Pmzezdu5eIiAgAPvzwQzp37syvv/5Kr169ynxfTVFguZCARtB+OCQthk1zYdjFPRBVVy3fnsakuAROniogJMCHV2/qzsB2de+ZoOJu0CO7NOXV73bynzX7+HZbGit3pHPnwCjuv7ytS445IyKV07Nnz1I/2+12nn/+eeLi4jh06BB5eXnk5eUREFB+QOratWvJevGtp7S08q/MDhkyhDlz5pTadvbzMWfXBxAZGVkqdGzbto2IiIiSsAIQHR1NgwYN2LZtW0lgOft9NUV/M1ZEzM3OwLJ5PlwxFWz6Y6soh8Pg9R928cq3OzAMiGkRzOxbY2neoN6F3+zBgut589SYaG7uHcEzi7excsdR3lyxm4UbDvLYyI5c0615nbnyJFKKt7/zSodZn11Nzg4iL730Eq+88gqzZs2iS5cuBAQEMGnSJPLz88sv6awHWC0WCw5H+c+/BQQE0LZt20rVV9Y2wzDKHAfr7O0XCl3VRd+8FdFuGNQLgexU2LMc2g01uyK3kHGqgMnzE/g+yfmvgVv6tGTq2Gh8vWwmV+Y62jYJ5D9/6sV329J49qtE9h87xZT5m/jw5/1MG9uZmIgGZpcoUrsslmq7LeNKVq1axdVXX82tt94KgMPhYOfOnXTq1Mnkys4vOjqa5ORkDhw4UHKVJTExkYyMDFPq1kO3FeHlA12ud67r4dsK2Xo4g7Gvr+b7pDR8vay8eH1Xnru2i8JKGSwWC0Ojw1g2+VIeGdEBfx8bG5NPcs3sH3nk000czdJs0CLurm3btsTHx7NmzRq2bdvGX/7yF1JTU2vks/Ly8khNTS21pKenV3o/Q4cOpWvXrvzxj39kw4YN/PLLL4wfP57LLruszFtKNU2BpaJibna+Jn0FuRnm1uLiFqw/yHWz15B8/BQRIfVYcG9/bugZceE31nG+XjbuG+ycDfra7s1LdYP+90p1gxZxZ0899RQ9evRg+PDhDB48mPDwcK655poa+axvvvmGpk2blloGDhxY6f1YLBY+//xzGjZsyKWXXsrQoUNp3bo1cXFxNVB1BeoxPGTM8MzMTIKDg8nIyCAoqAa6yBoGzO4LR5Ng7L8g9o7q/ww3l1do59nFiXz0czIAgzs0Zta4bjTw9zG5Mve0fv8Jpi3aypZDzoDcunEAT42JZkiHJiZXJlJ9cnNz2bt3L1FRUfj5uecI13Jh5Z3nin5/6wpLRVksv19lSZhbfts66PDJ09z41s989HMyFgtMGtqO927vpbByEWIjG/LFxAH88w9dCa3vw56jOfzp/V+564Nf2avZoEWkjlFgqYyu48BihQM/w/E9ZlfjMn7clc6Y11az6cBJgut5894dvZg0tL16uVQDq9XCjb0i+P7hwdw90Dkb9HdJaQx7ZQUzvt5GtmaDFpE6QoGlMoKaQushzvVN88ytxQUYhsGc5bu57d21HM/Jp3OzIBY/MFC3LGpAkJ83fxsTzTeTLuXS9o0psBu8tWIPQ2YuZ8H6gzgcHnFnV0TkvBRYKqtkqP65cIG+8J4sM7eACR+t54VvknAYcENsCxbc25+IkOobx0DO1bZJff7zp168M74nkY38OZqVx/99sonr5qxh04GTZpcnIlJjFFgqq+No8A2Ck8mQvMbsakyxPTWLq1//kaVbj+BjszLjui788/qu+Hmry3JtOLMb9KMjOuLvYyPhgHM26L9+som0rFyzSxQRqXYKLJXlXQ+ir3au18GHb79IOMQ1b/zI3vQcmgX78cmEftzcu2WZoyFKzfL1snHv4Db88PBgruvunOn6k/UHuXzmCt5euVvdoMWteEiHVTmP6ji/CixVUXxbKPFz5wyfdUCB3cH0L7fy0LwEThfYGdQulMUPDtJIrC4gLMiPl8d1Y8G9/enaIpjsvEKeW5LEiFkr+UGzQYuLKx56/tSpUyZXIjWp+PyePdVAZWgclqowDHi1G5zYB9e+DTHjLvQOt5aWmct9/9vAuv0nALh/SFsmX9kem3oBuRyHw+DT9Qf559Ik0rOdc5Rc3rEJT42JJkqzQYuLSklJ4eTJkzRp0gR/f39dsfUghmFw6tQp0tLSaNCgAU2bNj2nTUW/v6sUWGbPns2LL75ISkoKnTt3ZtasWQwaNKjMtgsXLmTOnDkkJCSQl5dH586dmTZtGsOHDy9p88EHH/CnP/3pnPeePn26wgMJ1WpgAVj+PCyf4ew1NP7zmv88k6zdc4yJH28kPTuPQF8vXh7XjSujw8wuSy4gM7eA177byfs/7qPQYeBts3DnwCgeuLydZoMWl2MYBqmpqZw8edLsUqSGNGjQgPDw8DLDaEW/vyv9N1dcXByTJk1i9uzZDBgwgLfeeouRI0eSmJhIy5Ytz2m/cuVKrrzySp577jkaNGjA+++/z9ixY1m7di3du3cvaRcUFMT27dtLvdelRz2MuckZWPYsh4xDENzc7IqqlWEYvLt6LzO+TsLuMOgYHsicW2P1r3Q3EeTnzZOjoxnXqyXPLk5kxY6jvLViDws3HOLRER25rrtmgxbXYbFYaNq0KU2aNKGgoMDscqSaeXt7Y7NdfKeMSl9h6dOnDz169GDOnDkl2zp16sQ111zDjBkzKrSPzp07M27cOJ5++mnAeYVl0qRJF5Wua/0KC8D7o2D/j3DFVBg0pXY+sxZk5xXy6Keb+WpLCgDXdGvGc9d1wd9H/zJ3R4Zh8H1SGs8uTmTfMed95G4RDZh2VWe66RkkETFZjQzNn5+fz/r16xk2bFip7cOGDWPNmop18XU4HGRlZRESElJqe3Z2NpGRkbRo0YIxY8awcePGcveTl5dHZmZmqaXWFQ/Vv2mu87kWD7ArLZtr3viRr7ak4GW18MzVnXllXDeFFTdmsVi4olMYSydfymMjOxJQ1A36mjd+5GF1gxYRN1GpwJKeno7dbicsrPQzDGFhYRWeJvull14iJyeHG2+8sWRbx44d+eCDD1i0aBFz587Fz8+PAQMGsHPnzvPuZ8aMGQQHB5csEREmzAYcfTV41YP0HXBoQ+1/fjX7eksKV7++ml1p2YQF+RL3l76M79dKD8B5CF8vGxMuK+oG3cN5C/NTdYMWETdRpW7NZ3+BGYZRoS+1uXPnMm3aNOLi4mjS5Pfh2/v27cutt95KTEwMgwYNYv78+bRv357XXnvtvPt6/PHHycjIKFkOHDhQlUO5OH5B0GmMc33Tx7X/+dWk0O7guSXbuPd/G8jJt9MnKoTFDwwiNjLkwm8Wt9MkyI+Xb+zGwvv6E3N2N+gkdYMWEddUqcASGhqKzWY752pKWlraOVddzhYXF8ddd93F/PnzGTp0aPlFWa306tWr3Cssvr6+BAUFlVpMUXxbaMunUJhnTg0X4WhWHre+u5a3Vzonc7zn0tb87+4+NA70NbkyqWk9Wjbks/sG8M/ruxJa35c96Tn86YNfuVOzQYuIC6pUYPHx8SE2Npb4+PhS2+Pj4+nfv/953zd37lzuuOMOPv74Y0aPHn3BzzEMg4SEhDL7a7uc1oMhsBnknoQd35hdTaWs33+CMa+t4uc9xwnwsTH7jz14YlQnvGwaT7CusFot3Ngzgh8evox7Lm2Nl9XC98WzQS/ZRlauemyIiGuo9DfTlClTeOedd3jvvffYtm0bkydPJjk5mQkTJgDOWzXjx48vaT937lzGjx/PSy+9RN++fUlNTSU1NZWMjIySNtOnT2fp0qXs2bOHhIQE7rrrLhISEkr26dKsNuha9DyOm8zgbBgG/1mzj5ve/okjmXm0bVKfL+4fyKgubhAQpUYE+nnzxKhOLJ18KYM7FM0GvXIPl7+0gk/WHdBs0CJiukoHlnHjxjFr1iyeeeYZunXrxsqVK1myZAmRkZGAc8TC5OTkkvZvvfUWhYWFTJw4kaZNm5YsDz30UEmbkydPcs8999CpUyeGDRvGoUOHWLlyJb17966GQ6wFxUP171wGOenm1nIBp/ILmRyXwNRFWymwG4zu0pTPJw6gbZP6ZpcmLqBN4/p88KfevHdHT6JCAzialcdfP93MtXPWsDH5hNnliUgdpqH5q8vbQ+DwBhjxPPS9t/Y/vwL2pecw4aP1JKVmYbNaeHxkR+4aGKVeQFKmvEI77/+4j9e+20lOvh2AP/RowaMjOtAkyIUHdRQRt1Ij47BIOYqvsiS4Zm+h+MQjjH19NUmpWYTW9+Xju/tw96DWCityXmd2g/5DjxYALNhwkCEzl/PWCnWDFpHapSss1eXUcZjZHhwFcO8aCOtc+zWUwe4weCV+B6//sAuAnpENeeOPPQjTv5ClkjYmn2Daoq1sOuh8/iwqNICnxnTi8o6aW0pEqk5XWGqbfwi0L5rQ0UWushzPyeeO938pCSt/GtCKuff0VViRKule1A36xaJu0HvTc7jzg3Xc8f4v7D6abXZ5IuLhFFiqU/Ftoc3zwV5oaimbDpxk7GurWbUznXreNv51Uzemju2Mt7osy0WwWi3ccEY3aG+bheXbjzJi1kqeUzdoEalB+vaqTm2vBP9GkJMGu783pQTDMJj7SzI3vPkTh06eJio0gM8nDuDqbp41m7SYq6Qb9KRLGVLUDfrtlXsYMlPdoEWkZiiwVCcvH+hyg3N909xa//jcAjuPLtjM4wu3kG93MCw6jC/uH0CH8MBar0XqhtaN6/P+Gd2g07PVDVpEaoYeuq1uhxPg7cvA5gsP74B6DWrlYw8cP8WEj9az9XAmVgv8dXhHJlymXkBSe/ILHbz/415ePaMb9HU9mvPYiI7qBi0i56WHbs3SNAaaRIM9D7Z+Visf+cP2NMa8tpqthzMJCfDhw7v6cO/gNgorUqt8vKz85bI2/PDXwVwf6+wGvXDDIYbMXM6bK3aTV2g3uUIRcWcKLNXNYvl9QsQavi3kcBjM+nYHd37wKxmnC4iJaMDiBwYyoG1ojX6uSHmaBPox84YYPruvPzERDcjJt/P810kMf2Ul3ycdMbs8EXFTuiVUE7JS4eVOYDjggQ3QqE21f8TJU/lMjkvgh+1HAbi1b0ueGhONr5et2j9LpKocDoOFGw/xwjdJHM1yzmY+uENjnhoTTZvGmg5CRHRLyFyB4dDmcud6DVxl+e1QBmNfX80P24/i62Vl5g0x/P2aLgor4nKsVgvXx7bg+/+7jL9c9ns36OGvrOQfXyWqG7SIVJgCS00puS00DxzVN4T5J+sO8Ic5azhw/DQRIfVYeF//kucFRFxVoJ83j490doO+vGMTCh0G/161lyEzVzBf3aBFpAIUWGpKx9HgGwwZB2D/6oveXV6hnSc+28JfP91MXqGDyzs2YfH9g+jcLLgaihWpHa0b1+e9O3rx/h29aF3UDfqRTzdz7ewf2aBu0CJSDgWWmuJdDzpf41zfNO+idnXo5GlufPMnPl6bjMUCU65szzvjexLs733xdYqYYEjHJnwz6VKeGNWR+r5ebDqYwXWz1zAlLoEjmblmlyciLkiBpSYVD9Wf+AXk51RpF6t3pjP2tdVsOphBA39v3r+jFw9e0Q6rVV2Wxb35eFm559I2fP/wZdxQ3A164yEun7mcOcvVDVpESlNgqUkRfSCkNeRnw7YvK/VWh8PgjR92Mf69tRzPyeeS5kF8ef9ABndoUkPFipijSaAfL94Qw+cTB9CtqBv0C984u0F/m3gED+nIKCIXSYGlJp05JkslZnDOzC3gLx+t58Wl23EYMK5nBJ9O6E9EiH8NFSpivm4RDVh4b39euiGGxoG+7Dt2irv/u4473v+VXWmaDVqkrlNgqWldxzlf966EjIMXbJ6UmslVr60mPvEIPl5Wnr+uCy9c3xU/b3VZFs9ntVr4Q2wLfnh4cEk36BU7nLNBf/jzfrPLExETKbDUtIaREDkQMC748O0XCYe49o017Dt2iuYN6vHphH7c1Ltl7dQp4kLq+3rx+MhOLJt8WUk36Ge/TGT/sao9CyYi7k+BpTZ0O2Oo/jLux+cXOpi2aCsPzUvgdIGdQe1C+fKBgXRt0aB26xRxMVGhAbx7e08Gtg0l3+7g719tM7skETGJAkttiL4avP3h2C44uK7Ur45k5nLzv3/mgzX7AHjg8rZ88KfehAT4mFCoiOuxWCxMHRuNl9VCfOIRVuw4anZJImICBZba4BsIncY6188Yqv/nPccY/epq1u8/QaCfF++M78n/DeuATV2WRUppFxbI7f1bATD9y63kF1bf6NEi4h4UWGpLcW+h3xZgFOTy75V7+OM7a0nPzqNjeCBf3j+QodFh5tYo4sIeGtqO0Po+7Dmaw3+KrkiKSN2hwFJboi6FoOaQe5J33p3NP5Zsw+4wuLZ7cz67bwCtQgPMrlDEpQX5efPI8I4A/Ou7naRlaURckbpEgaW2WG0cb3stAFGHFuFts/Ds1Z15+cYY6vmoy7JIRVwf24KuLYLJzivkn99sN7scEalFCiy1ZMmWFMavaw3AYNsmFoxvy239WmGx6HkVkYqyWi1Mv6ozAJ+uP8hGTZgoUmcosNSwQruDf3yVyH3/28Bv+eHs8u6AFw66Ho83uzQRt9S9ZUOuL5p7aNqirTgcGrpfpC5QYKlBaVm5/PGdtfx71V4A/nJZa6KG3u38ZcLcct4pIuV5ZESHklmeP11/4RGkRcT9KbDUkHX7jjPm1dWs3Xuc+r5ezPljDx4f2Qlbl+vB5gNHtkDqFrPLFHFLTQL9eOiKdgC88E0SGacLTK5IRGqaAks1MwyDD37cy01v/0xaVh7tmtTni/sHMLJLU2cD/xBoP8K5foGh+kXk/G7v34rWjQM4lpPPq9/tNLscEalhCizV6FR+IZPiEpj2ZSKFDoMxXZvy+cQBtGlcv3TDbrc4XzfPB3th7Rcq4gF8vKxMG+t8APc/a/ax80iWyRWJSE1SYKkme9NzuPaNNXyRcBgvq4Wnx0Tz2s3dCfD1Ordx26HgHwo5abD7u9ovVsRDXNq+MVdGh1HoMJj+ZSJGGXN1iYhnUGCpBsu2pnLVa6vZfiSLxoG+fPznvtw5MOr8XZZt3tD1Rud6wse1V6iIB3pqdDQ+XlZW70pn6dYjZpcjIjVEgeUi2B0G//wmiXs+XE9WXiG9WjXkqwcG0jsq5MJvLh6qf/sSOK2xJESqqmUjf+4Z5Bzj6O9fJZJbYDe5IhGpCQosVXQsO4/b3/uF2ct3A3DXwCg+/nNfmgT5VWwH4V2gSWew58NvC2uwUhHPd9+QNjQN9uPgidO8vXKP2eWISA1QYKmChAMnGfvaalbvSsffx8ZrN3fnqTHReNsq8cdpsUC3oqssmzQmi8jF8Pfx4olRnQCYvXwXh06eNrkiEaluCiyVYBgG/1u7nxvf/InDGbm0Dg3g84kDGBvTrGo77HIjWGxw8FdIV7dMkYsxpmtTekeFkFvg4LmvtpldjohUMwWWCsotsPPXTzfz5Ge/kW93MLxzGF/cP4D2YYFV32lgGLS9wrmuMVlELorFYmHa2M5YLfDVlhTW7E43uyQRqUYKLBWQfOwU181ew6frD2K1wOMjO/LmrbEE+nlf/M6LH77dHAcOx8XvT6QOi24WxB/7RAIwfVEihXb9PyXiKRRYLuCHpDTGvr6axJRMGgX48NFdffjLZW2qb5blDqPALxgyDsC+VdWzT5E6bMqV7Wng7832I1n8b22y2eWISDVRYCnHqfxCHlmwmYzTBXSLaMDiBwfSv21o9X6Itx90vs65rodvRS5awwAfHh7WAYCXlm3nWHaeyRWJSHVQYCmHv48Xr97UnfH9Ion7S1+aBtermQ8qHqo/cRHkZdfMZ4jUITf3bkmnpkFk5hYyc9kOs8sRkWqgwHIB/do04pmrL8HXy1ZzH9KiF4S0gYIc2Lao5j5HpI6wWS1Mv8o5z9C8X5P57VCGyRWJyMVSYHEFFsvvD99qqH6RatE7KoSrYpphGDB10VbNMyTi5hRYXEXMOOfrvlVwUg8KilSHJ0Z1wt/Hxvr9J/gi4bDZ5YjIRVBgcRUNWkKrQc71zXHm1iLiIcKD/Zg4pC0Azy3ZRnZeockViUhVKbC4kuKHbxPmgi5fi1SLuwZGEdnIn7SsPF7/fpfZ5YhIFSmwuJJOV4F3ABzf7RyuX0Qump+3jadGRwPw7uo97E3PMbkiEakKBRZX4lsfoq9yruvhW5Fqc0WnJlzWvjEFdoNnFyeaXY6IVIECi6sp7i20dSEU5Jpbi4iHsFgsPD02Gm+bhe+T0vg+6YjZJYlIJSmwuJpWgyCoBeRmwPYlZlcj4jHaNK7PnQOiAHjmy0TyCu0mVyQilaHA4mqs1t+7OGuofpFqdf/lbWkc6Mu+Y6d4b/U+s8sRkUpQYHFFxbeFdn0HWbp0LVJdAv28eWxERwBe+34nRzJ121XEXSiwuKLQds7h+g07bPnE7GpEPMq13ZvTvWUDTuXbef7rJLPLEZEKqlJgmT17NlFRUfj5+REbG8uqVavO23bhwoVceeWVNG7cmKCgIPr168fSpUvPabdgwQKio6Px9fUlOjqazz77rCqleY7iqyy6LSRSraxF8wxZLPDZxkOs23fc7JJEpAIqHVji4uKYNGkSTz75JBs3bmTQoEGMHDmS5OSyh5NfuXIlV155JUuWLGH9+vUMGTKEsWPHsnHjxpI2P/30E+PGjeO2225j06ZN3Hbbbdx4442sXbu26kfm7i65Dmy+cOQ3SNlsdjUiHqVriwaM6xkBOOcZsjs0UKOIq7MYlZwRrE+fPvTo0YM5c+aUbOvUqRPXXHMNM2bMqNA+OnfuzLhx43j66acBGDduHJmZmXz99dclbUaMGEHDhg2ZO7diVxgyMzMJDg4mIyODoKCgShyRC5t/OyR+Dn3vgxEV+7MVkYpJz85jyMzlZOUW8ty1XbilT0uzSxKpkyr6/V2pKyz5+fmsX7+eYcOGldo+bNgw1qxZU6F9OBwOsrKyCAkJKdn2008/nbPP4cOHl7vPvLw8MjMzSy0ep3io/s3zwV5gbi0iHia0vi+Th7YH4MWlSWSc0v9jIq6sUoElPT0du91OWFhYqe1hYWGkpqZWaB8vvfQSOTk53HjjjSXbUlNTK73PGTNmEBwcXLJERERU4kjcRJvLIaAxnEqHXd+aXY2Ix7mtXyTtw+pz4lQBr3y7w+xyRKQcVXro1mKxlPrZMIxztpVl7ty5TJs2jbi4OJo0aXJR+3z88cfJyMgoWQ4cOFCJI3ATNm/oUhTsNFS/SLXztlmZOrYzAB/+vJ+kVA+8UiviISoVWEJDQ7HZbOdc+UhLSzvnCsnZ4uLiuOuuu5g/fz5Dhw4t9bvw8PBK79PX15egoKBSi0fqVtRbaMc3cEq9GUSq24C2oYy8JBy7w2Daoq1U8rE+EakllQosPj4+xMbGEh8fX2p7fHw8/fv3P+/75s6dyx133MHHH3/M6NGjz/l9v379ztnnsmXLyt1nnRHeBcK6gD3fOb+QiFS7J0Z1wtfLys97jrNkS8Vub4tI7ar0LaEpU6bwzjvv8N5777Ft2zYmT55McnIyEyZMAJy3asaPH1/Sfu7cuYwfP56XXnqJvn37kpqaSmpqKhkZGSVtHnroIZYtW8YLL7xAUlISL7zwAt9++y2TJk26+CP0BMVXWRI0JotITYgI8WfCZW0A+MdXiZzO1zxDIq6m0oFl3LhxzJo1i2eeeYZu3bqxcuVKlixZQmRkJAApKSmlxmR56623KCwsZOLEiTRt2rRkeeihh0ra9O/fn3nz5vH+++/TtWtXPvjgA+Li4ujTp081HKIH6HIDWGxwaB2k7zS7GhGPdO/gNjRvUI/DGbnMWbHb7HJE5CyVHofFVXnkOCxn+nic8zmWgVNg6FSzqxHxSF9vSeHe/23Ax8vKd1MuIyLE3+ySRDxejYzDIiaKucn5ujkOHLpcLVITRlwSTv82jcgvdPD3rxLNLkdEzqDA4i7ajwS/YMg8BHtXml2NiEeyWCxMHdsZm9XC0q1HWLXzqNkliUgRBRZ34e0Hl/zBua4JEUVqTIfwQG7r63wmb/qXiRTYHSZXJCKgwOJeYoqG6t/2JeRlmVuLiAebfGV7QgJ82JWWzX9/2m92OSKCAot7adETGrWFglOQuMjsakQ8VnA9bx4Z3gGAWfE7OJqVZ3JFIqLA4k4sFogpGpNFt4VEatQNPSPo0jyYrLxCXlyaZHY5InWeAou7ibkJsMC+VXBCl6pFaorNamHaVc55huavO0jCgZPmFiRSxymwuJvgFhB1qXN9c5y5tYh4uNjIhlzXozkA0xZtxeHwiGGrRNySAos7OvO2kGeM+yfish4b0ZEAHxsJB06yYMNBs8sRqbMUWNxRp7HgHQDH98CBtWZXI+LRmgT58eAV7QB44ZvtZOYWmFyRSN2kwOKOfOtD9NXO9YSPza1FpA7404AoWocGkJ6dx2vfaT4vETMosLir4hmct34OBadNLUXE0/l4WXlqbDQA7/+4j11p2SZXJFL3KLC4q8iBENwS8jJg+xKzqxHxeEM6NGFopyYUOgymf7kVD5k3VsRtKLC4K6sVYsY51xM0JotIbfjb6Gh8bFZW7UwnPvGI2eWI1CkKLO6suLfQ7u8gK9XcWkTqgFahAdw9KAqAZ79KJLdAM6eL1BYFFnfWqA1E9AHDAZvnm12NSJ0wcUhbwoJ8OXD8NO+s2mN2OSJ1hgKLu4u5yfmqMVlEakWArxdPjOoEwBs/7ObwST30LlIbFFjcXefrwOYLaYmQssnsakTqhKtimtGrVUNOF9iZ8bXmGRKpDQos7q5eA+g4yrmuCRFFaoXF4pxnyGqBLzcd5uc9x8wuScTjKbB4gphbnK9bPgG7RuEUqQ2dmwVzc++WgHOeoUK7w+SKRDybAosnaHM51A+DU8dgZ7zZ1YjUGQ8P60BwPW+SUrOY+0uy2eWIeDQFFk9g84IuNzjXN2mofpHa0jDAh4eHtQdg5rIdnMjJN7kiEc+lwOIpuhXdFtr+DZw6bm4tInXIzb1b0jE8kIzTBcxctt3sckQ8lgKLpwjrDOFdwVEAvy0wuxqROsPLZmXaVZ0B+PiXZH47lGFyRSKeSYHFkxSPfKsZnEVqVd/WjRjTtSmGgeYZEqkhCiyepMsNYPWCwxvgqC5Ni9SmJ0Z1op63jV/3nWDRpsNmlyPicRRYPEn9xtD2Sue6rrKI1KpmDeoxcUgbAGYsSSInr9DkikQ8iwKLp+lWdFto83xwaGI2kdp096DWRITUIzUzlzd+2GV2OSIeRYHF07QfAX4NIOsw7F1hdjUidYqft42nRkcD8M6qvexLzzG5IhHPocDiabx8ocv1zvUEDdUvUtuujA5jULtQ8u0O/v5VotnliHgMBRZPVDxU/7YvITfT3FpE6hiLxcLUsZ3xslr4dlsaP2xPM7skEY+gwOKJmveA0PZQeBoSvzC7GpE6p22T+vxpQCsAnv0ykfxCzTMkcrEUWDyRxQIxNznXNYOziCkevKIdofV92ZOew/s/7jW7HBG3p8DiqbreBFhg/49wYp/Z1YjUOYF+3jw6ogMAr363k7TMXJMrEnFvCiyeKrg5tL7Mub5pnrm1iNRRf+jRgm4RDcjJt/P8N0lmlyPi1hRYPFnxw7eb5oKGChepdVarhelF8wwt3HCI9ftPmFyRiPtSYPFkncaAT33nLaHkn82uRqROiolowI09WwAwbdFW7A7940GkKhRYPJlPAERf41zfpKH6Rczy1+EdCfT1YsuhDD5Zd8DsckTckgKLpyseqn/r51Bw2tRSROqqxoG+PDS0HQD/XLqdjNMFJlck4n4UWDxdy/7QoCXkZULSV2ZXI1Jn3d6/FW2b1Od4Tj6zvt1hdjkibkeBxdNZrUVdnNEMziIm8rZZmTrWOc/Qf3/az44jWSZXJOJeFFjqguJB5Pb8AJkp5tYiUocNateY4Z3DsDsMpi3aiqHeeyIVpsBSFzRqAxF9wXDA5jizqxGp0/42OhofLytrdh/jm99SzS5HxG0osNQVxQ/fakwWEVNFhPgz4dLWAPz9q22czrebXJGIe1BgqSs6XwtefnA0CVISzK5GpE67d3BbmgX7cejkad5audvsckTcggJLXeEXDB1HO9cTNCGiiJnq+dh4crTzAdw5y3dz8MQpkysScX0KLHVJ8VD9Wz6BwnxzaxGp40Z1Cadv6xDyCh3846ttZpcj4vIUWOqS1oOhfjicPg47l5ldjUidZrFYmHZVZ6wW+Pq3VH7clW52SSIuTYGlLrF5QdcbnOubdFtIxGwdw4O4rW8kANO/3EqB3WFyRSKuS4Glrim+LbRjKeQcM7cWEWHKlR1o6O/NjiPZfPTzfrPLEXFZCix1TVg0NI0BRwH89qnZ1YjUecH+3vx1eEcAXo7fwbHsPJMrEnFNCix1UfFVFt0WEnEJ43pF0LlZEFm5hby4dLvZ5Yi4JAWWuqjL9WD1gsMbIS3J7GpE6jyb1cL0qzoDELfuAJsPnjS3IBEXpMBSFwWEQrvhzvVNmhBRxBX0bBXCNd2aYRgwbdFWHA6NSC1ypioFltmzZxMVFYWfnx+xsbGsWrXqvG1TUlK45ZZb6NChA1arlUmTJp3T5oMPPsBisZyz5ObmVqU8qYjiofo3zweHhgYXcQWPj+qEv4+NDckn+TzhkNnliLiUSgeWuLg4Jk2axJNPPsnGjRsZNGgQI0eOJDk5ucz2eXl5NG7cmCeffJKYmJjz7jcoKIiUlJRSi5+fX2XLk4pqNxzqNYSsFOcsziJiurAgPx64vB0AM75OIiu3wOSKRFxHpQPLyy+/zF133cXdd99Np06dmDVrFhEREcyZM6fM9q1ateJf//oX48ePJzg4+Lz7tVgshIeHl1qkBnn5wCXXO9c1VL+Iy7hzYCtaNfLnaFYer3+/y+xyRFxGpQJLfn4+69evZ9iwYaW2Dxs2jDVr1lxUIdnZ2URGRtKiRQvGjBnDxo0by22fl5dHZmZmqUUqqfi2UNJiyM0wtxYRAcDXy8bTY53zDL334152H802uSIR11CpwJKeno7dbicsLKzU9rCwMFJTU6tcRMeOHfnggw9YtGgRc+fOxc/PjwEDBrBz587zvmfGjBkEBweXLBEREVX+/DqrWQ8I7QCFuZD4hdnViEiRyzuGcXnHJhTYDZ75MhHD0AO4IlV66NZisZT62TCMc7ZVRt++fbn11luJiYlh0KBBzJ8/n/bt2/Paa6+d9z2PP/44GRkZJcuBAweq/Pl1lsXy+1UW3RYScSlPjYnG22ZhxY6jfJ+UZnY5IqarVGAJDQ3FZrOdczUlLS3tnKsuF1WU1UqvXr3KvcLi6+tLUFBQqUWqoOs4sFgheQ0c32t2NSJSJCo0gLsGtgbgmcWJ5BaoN5/UbZUKLD4+PsTGxhIfH19qe3x8PP3796+2ogzDICEhgaZNm1bbPuU8gpo5Z3EG2DTP1FJEpLT7L29Lk0Bf9h87xbur9Q8KqdsqfUtoypQpvPPOO7z33nts27aNyZMnk5yczIQJEwDnrZrx48eXek9CQgIJCQlkZ2dz9OhREhISSExMLPn99OnTWbp0KXv27CEhIYG77rqLhISEkn1KDTtzqH6HZosVcRX1fb14fJRznqHXv99FSsZpkysSMY9XZd8wbtw4jh07xjPPPENKSgqXXHIJS5YsITLSOUV6SkrKOWOydO/evWR9/fr1fPzxx0RGRrJv3z4ATp48yT333ENqairBwcF0796dlStX0rt374s4NKmwjqPBJxBO7ofkn6DVALMrEpEi13Rrzkc/J7N+/wme/zqJf93U/cJvEvFAFsNDHj/PzMwkODiYjIwMPc9SFV9MhI0fQfdb4eo3zK5GRM7w26EMxr6+GsOATyb0o1erELNLEqk2Ff3+1lxC4lR8W2jrF5B/ytxaRKSUS5oHc1OvlgBM/WIrds0zJHWQAos4tewHDSIhPwuSvjK7GhE5y8PD2hPk50ViSiZzfyl7KhQRT6bAIk5WK8QUjcmiGZxFXE6j+r5MubI9ADOXbefkqXyTKxKpXQos8ruYm5yve5ZD5mFTSxGRc93aN5IOYYGcPFXAy/E7zC5HpFYpsMjvQqKgZX8wHLA5zuxqROQsXjYrU69yzjP00c/7STysOdSk7lBgkdLOHKrfMzqQiXiU/m1CGd2lKQ4Dpn25VfMMSZ2hwCKlRV8DXn6Qvh0ObzC7GhEpwxOjO+HnbeWXvcdZvDnF7HJEaoUCi5TmFwQdxzjXNSGiiEtq3qAe9w1uC8BzS7ZxKr/Q5IpEap4Ci5yr+LbQb59CYZ65tYhIme65tDUtGtYjJSOXOct3m12OSI1TYJFztR4CgU3h9AnYuczsakSkDH7eNv422vkA7lsr95B8TAM+imdTYJFzWW3Q9Ubnum4Libis4Z3DGNg2lPxCB89+lXjhN4i4MQUWKVvxUP07l0JOurm1iEiZLBYLU8dG42W1EJ94hBU7jppdkkiNUWCRsjXpCM26g6MQtnxqdjUich7twgK5vX8rAKZ/uZX8Qoe5BYnUEAUWOb/iqywaql/EpT00tB2h9X3YczSH//60z+xyRGqEAouc3yV/AKs3pGyCI7o/LuKqgvy8eWR4RwBmfbuTtKxckysSqX4KLHJ+AY2g/XDnuq6yiLi062Nb0LVFMNl5hfzzm+1mlyNS7RRYpHzFMzhvng92DU4l4qqsVgvTr+oMwKfrD7Ix+YTJFYlULwUWKV+7YVAvBLKPOGdxFhGX1b1lQ66PbQHAtEVbcTg0z5B4DgUWKZ+XD3S5wbmu20IiLu+RER2o7+vFpoMZfLr+oNnliFQbBRa5sOKh+pO+gtwMc2sRkXI1CfTjoSvaAfDCN0lknC4wuSKR6qHAIhfWtBs07gSFubD1M7OrEZELuL1/K1o3DuBYTj6vfrfT7HJEqoUCi1yYxfL7VRYN1S/i8ny8rEwb63wA9z9r9rHzSJbJFYlcPAUWqZguN4LFCgd+hmOaGVbE1V3avjFXRodR6DCY/mUihqEHcMW9KbBIxQQ1dc7iDLBpnrm1iEiFPDU6Gh8vK6t3pbN06xGzyxG5KAosUnHdiofqnwcOzVci4upaNvLnnkGtAfj7V4nkFthNrkik6hRYpOI6jgbfIMhIhuQ1ZlcjIhVw35A2NA324+CJ07y9co/Z5YhUmQKLVJx3Peh8jXNdD9+KuAV/Hy+eGNUJgNnLd3Ho5GmTKxKpGgUWqZziGZwTP4f8HFNLEZGKGdO1Kb2jQsgtcPDckm1mlyNSJQosUjkt+0LDKMjPhm2Lza5GRCrAYrEwbWxnrBb4anMKa3anm12SSKUpsEjlWCy/T4ioofpF3EZ0syD+2CcSgOmLEim068F5cS8KLFJ5MeOcr3tWQIbmKhFxF1OubE8Df2+2H8nif2uTzS5HpFIUWKTyGraCyAGAAZvjzK5GRCqoYYAPDw/rAMBLy7ZzPCff5IpEKk6BRaom5oyh+jWCpojbuLl3Szo1DSIzt5AXl243uxyRClNgkaqJvhq86sGxnXBog9nViEgF2awWpl/lnGdo3q/J/HZIM7CLe1BgkarxC4JOY53revhWxK30jgrhqphmGAZMXbRV8wyJW1BgkaornsF5y6dQmGduLSJSKU+M6oS/j431+0/wRcJhs8sRuSAFFqm6qMsgsBnknoQd35hdjYhUQniwHxOHtAXguSXbyM4rNLkikfIpsEjVWW2/d3HWUP0ibueugVFENvInLSuP17/fZXY5IuVSYJGLU9xbaFc8ZB81txYRqRQ/bxtPjY4G4N3Ve9ibruk2xHUpsMjFadwBmvUARyFs+cTsakSkkq7o1ITL2jemwG7w7OJEs8sROS8FFrl43YomRFRvIRG3Y7FYeHpsNN42C98npfF90hGzSxIpkwKLXLxL/gBWb0jdAqm/mV2NiFRSm8b1uXNAFADPLt5GXqHd5IpEzqXAIhfPPwQ6jHCub9LDtyLu6P7L29I40Je96Tm8t3qf2eWInEOBRapHTNFtoc3zwa7ukSLuJtDPm8dGdATgte93ciQz1+SKREpTYJHq0e5K8A+FnDTY/b3Z1YhIFVzbvTndWzbgVL6d579OMrsckVIUWKR62Lyhyw3OdT18K+KWrEXzDFks8NnGQ6zbd9zskkRKKLBI9Ym5yfmatAROnzC3FhGpkq4tGjCuZwQA077cit2heYbENSiwSPVpGgNNosGeB1s/M7saEamih4d3INDPi98OZRL36wGzyxEBFFikOlksv498q6H6RdxWaH1fJg9tD8CLS5PIOFVgckUiCixS3breCBYrHPwFju02uxoRqaLb+kXSPqw+J04V8Mq3O8wuR0SBRapZYDi0ucK5rjFZRNyWt83K1LGdAfjw5/0kpWaaXJHUdQosUv26Fd0W2jQPHA5zaxGRKhvQNpSRl4RjdxhMW7QVw9ADuGIeBRapfh1Gg28wZByA/avNrkZELsITozrh62Xl5z3HWbIl1exypA5TYJHq5+0Hl1zrXNfDtyJuLSLEnwmXtQHgH18lcjpf8wyJOaoUWGbPnk1UVBR+fn7ExsayatWq87ZNSUnhlltuoUOHDlitViZNmlRmuwULFhAdHY2vry/R0dF89pm6xbq14t5CiV9AXra5tYjIRbl3cBuaN6jH4Yxc5qzQw/RijkoHlri4OCZNmsSTTz7Jxo0bGTRoECNHjiQ5ObnM9nl5eTRu3Jgnn3ySmJiYMtv89NNPjBs3jttuu41NmzZx2223ceONN7J27drKlieuIqIPhLSGghzY9qXZ1YjIRfDztvG30Z0AeHPFbg4cP2VyRVIXWYxKPkXVp08fevTowZw5c0q2derUiWuuuYYZM2aU+97BgwfTrVs3Zs2aVWr7uHHjyMzM5Ouvvy7ZNmLECBo2bMjcuRW7pZCZmUlwcDAZGRkEBQVV/ICk5qz4J/zwD4i6DG5fZHY1InIRDMPgj++sZc3uYwzvHMZbt/U0uyTxEBX9/q7UFZb8/HzWr1/PsGHDSm0fNmwYa9asqVqlOK+wnL3P4cOHl7vPvLw8MjMzSy3iYrqOc77uXQkZB82tRUQuisViYerYztisFpZuPcKqnUfNLknqmEoFlvT0dOx2O2FhYaW2h4WFkZpa9afHU1NTK73PGTNmEBwcXLJERERU+fOlhjSMhFaDAMPZxVlE3FqH8EBu6xsJwPQvEymwa9gCqT1VeujWYrGU+tkwjHO21fQ+H3/8cTIyMkqWAwc034VLKn74dtNc0BgOIm5v8pXtCQnwYVdaNv/9ab/Z5UgdUqnAEhoais1mO+fKR1pa2jlXSCojPDy80vv09fUlKCio1CIuKPoq8PaHY7vg4DqzqxGRixRcz5tHhncAYFb8DtKz80yuSOqKSgUWHx8fYmNjiY+PL7U9Pj6e/v37V7mIfv36nbPPZcuWXdQ+xUX4BkKnq5zrmz42txYRqRY39IygS/NgsvIK+ec3SWaXI3VEpW8JTZkyhXfeeYf33nuPbdu2MXnyZJKTk5kwYQLgvFUzfvz4Uu9JSEggISGB7Oxsjh49SkJCAomJiSW/f+ihh1i2bBkvvPACSUlJvPDCC3z77bfnHbNF3EzMTc7X3xZAQa65tYjIRbNZLUy7yjnP0Px1B0k4cNLcgqRO8KrsG8aNG8exY8d45plnSElJ4ZJLLmHJkiVERjofxEpJSTlnTJbu3buXrK9fv56PP/6YyMhI9u3bB0D//v2ZN28ef/vb33jqqado06YNcXFx9OnT5yIOTVxG1KUQ1BwyD8GOr6HztWZXJCIXKTayIdf1aM7CDYeYtmgrC+/tj9V6cc8yipSn0uOwuCqNw+Livp0Oq1+G9iPgljizqxGRapCWmcuQmcvJybcz84YYro9tYXZJ4oZqZBwWkSrrdovzdWc8ZKeZW4uIVIsmQX48eEU7AJ7/OonM3AKTKxJPpsAitSO0HTTvCYYdtnxidjUiUk3+NCCK1qEBpGfn8dp3O80uRzyYAovUnm5FY7JoBmcRj+HjZeWpsdEAvP/jPnalabJTqRkKLFJ7Ol8HNh84sgVSt5hdjYhUkyEdmjC0UxMKHQbTv9yKhzwaKS5GgUVqj38IdBjpXNdVFhGP8rfR0fjYrKzamc632/ScmlQ/BRapXcVD9W+ZD3Y9oCfiKVqFBnD3oCgAnl2cSG6B3eSKxNMosEjtajsU/EMh5yjs+s7sakSkGk0c0pawIF+Sj5/inVV7zC5HPIwCi9Qumzd0vdG5vkm3hUQ8SYCvF0+M6gTAGz/s5vDJ0yZXJJ5EgUVqX/Ftoe1L4PQJc2sRkWp1VUwzerVqyOkCOzO+1jxDUn0UWKT2Ne0KYZeAPR9+W2h2NSJSjSwW5zxDVgt8uekwa/ccM7sk8RAKLGKO4qssui0k4nE6Nwvm5t4tAZi6aCuFdofJFYknUGARc3S9ESw2OPgrpGt0TBFP8/CwDgTX8yYpNYu5vyRf+A0iF6DAIuao38TZYwh0lUXEAzUM8OHhYe0BmLlsBydy8k2uSNydAouYJ+Ym5+umOHDokrGIp7m5d0s6hgeScbqAl+K3m12OuDkFFjFPh1HgFwyZB2HfSrOrEZFq5mWzMu2qzgB8vDaZl5dtZ83udA0qJ1XiZXYBUod5+znnF1r/PmyaB60Hm12RiFSzvq0bcVVMMxZtOsyr3+/i1e934WOzEhMRTJ+oRvSOCiE2siEBvvo6kvJZDA+ZpSozM5Pg4GAyMjIICgoyuxypqAO/wLtXgncAPLwDfOubXZGIVLP8QgcLNhzkp93HWLv3GEcy80r93stq4ZLmwfRpHUKfqBB6tgohyM/bpGqltlX0+1uBRcxlGPBaLBzfDdfMgW63mF2RiNQgwzDYf+wUa/ceY+3e46zdc5xDZ42Ia7VAdLOgkiswvVuF0DDAx6SKpaYpsIj7WPkifP93aDUI7lhsdjUiUssOnjjFL0XhZe3eY+w7duqcNh3DA+kdFVISYhoH+ppQqdQEBRZxHycPwKwugAGTtkCDlmZXJCImOpKZW3T1xXkVZlda9jlt2jQOoHdUI/q2doaY8GA/EyqV6qDAIu7lP2Nh70oY8je47K9mVyMiLiQ9O49f9x5n7d7j/LznGNuPZHH2N1dkI396twqhT+tG9IkKISLE35xipdIUWMS9JHwMn98LIW3ggfVgsZhdkYi4qJOn8vl134mSKzBbD2fgOOubrHmDekW3kJwhplUjfyz6e8UlKbCIe8nLhpntoSAH7lwGLfuYXZGIuIms3ALW7T9R8gzMloMZFJ6VYJoE+joDTNEVmHZN6ivAuAgFFnE/n01wDtMf+ycYO8vsakTETZ3KL2TD/pPOnkh7jpNw4CT5Z03AGBLgU3QLyfkMTMfwQKxWBRgzKLCI+9mzAv57lXP02//b4RxYTkTkIuUW2Ek4cJK1e47zy75jrN9/gtyC0gEmyM+rVC+kzs2C8LJpMPjaoMAi7sfhgH91hYwDcP37cMl1ZlckIh4ov9DBlkMn+XnPcX7Ze5x1+46Tk196uoD6vl7ERjYsGcyuS/MG+HgpwNQEBRZxT989C6tmQrth8MdPzK5GROqAQruDrYczWbv3GL/sdYaYzNzCUm38vK3ERjakd6tG9GkdQreIBvh520yq2LMosIh7St8Fr8eCxQZTtkFgmNkViUgdY3cYJKVmOm8h7T3OL/uOczwnv1QbHy8r3SIaOHshRTWiR2QD/H00H1JVKLCI+3rnSjj4Cwz7O/R/wOxqRKSOczgMdh3NLjWY3dGsc+dD6toimN5RziswPSMbEqj5kCpEgUXc16/vwldToElnuPdHjckiIi7FMAz2puc4pxMoCjGHM3JLtbFa4JLmwSWD2fVuFUKwvwJMWRRYxH2dPgEzO4A9D/6yEprGmF2RiMh5GYbBwROnS12BST5eej4kiwU6hAXSt2gcmN5RITSqr/mQQIHF7HLkYs2/HRI/h773wYgZZlcjIlIpKRmn+WXvcX4uGsxuz9Gcc9q0bVK/ZCTevlEhNAmqm0M5KLCIe9uxFD6+EfxD4f+SwKZLqSLivo5m5RXdQnIOZrf9SNY5bVo18qdP0TMwfVo3onmDeiZUWvsUWMS92Qvh5U6QkwY3z4MOI82uSESk2pzIyeeXfcdLphNITMk8Z0LH5g3q0ad1CH2LBrOL9ND5kBRYxP0tfRJ+eh06XQXjPjS7GhGRGpNxuoD1+4sDzHG2HMrAftZ8SGFBviUj8fZtHUKbxp4xH5ICi7i/1N/gzQFg84H/2w7+IWZXJCJSK3LyClm//0TJYHYJB05SYC/9dR1a34feUSElPZE6hLnnfEgKLOIZ3hwIqVtg1Ezo/WezqxERMUVugZ0NySdKBrPbkHyCvMLS8yE18PemV6uQksHsopsFYXODAKPAIp7hpzdg6RPQPBb+/L3Z1YiIuIS8QjubD2aUdKNev/8Ep86aDynQ14uerRqWDGbXpXkw3i44oaMCi3iG7DR4qSMYdpj4KzRub3ZFIiIup8Du4LdDGSWD2f269zhZeaXnQ6rnbXNO6FjUlTomIhhfL/PnQ1JgEc/x8TjY8Q0MnAJDp5pdjYiIy7M7DLalZJYMZvfLvuOcPFVQqo2Pl5XuEQ1KxoHp3rIh9XxqP8AosIjn2Po5fHI7BDWHSVvAav6/CERE3InDYbAjLct5BaaoK3V6dukJHb1tFrq2aFByBSY2siH1fWt+QkcFFvEchXkwsz3knoTbPoc2Q8yuSETErRmGwe6jOaUGs0vNLD0fks1q4ZJmQfQpmk6gZ6sQgutV/yCeCiziWRZPgXXvQtdxcN3bZlcjIuJRDMPgwPHT/FwUXtbuPcbBE6dLtbFY4N+39WRodFi1fnZFv79r/lqPSHXodoszsGz7EvKywDfQ7IpERDyGxWKhZSN/Wjby58aeEQAcOnmaX0oCzHH2pucQ3cy8CwIKLOIemsdCo3ZwbCckfgHdbzW7IhERj9a8QT2u7d6Ca7u3AJzzITUONG+GadfrkC1SFosFYm5yrifMNbcWEZE6yMywAgos4k5ibgIssH81nNhndjUiIlKLFFjEfQS3gKhLneub55tbi4iI1CoFFnEv3W5xvm6ayzlzsYuIiMdSYBH30mks+NSH43vgwFqzqxERkVqiwCLuxScAoq92rid8bG4tIiJSaxRYxP3E3Ox83foZFJwuv62IiHgEBRZxP5EDILgl5GVC0ldmVyMiIrVAgUXcj9UKMeOc65s0JouISF1QpcAye/ZsoqKi8PPzIzY2llWrVpXbfsWKFcTGxuLn50fr1q158803S/3+gw8+wGKxnLPk5uaeZ49S5xXfFtr9PWSlmluLiIjUuEoHlri4OCZNmsSTTz7Jxo0bGTRoECNHjiQ5ObnM9nv37mXUqFEMGjSIjRs38sQTT/Dggw+yYMGCUu2CgoJISUkptfj5+VXtqMTzNWoDEX3AcGhMFhGROqDSgeXll1/mrrvu4u6776ZTp07MmjWLiIgI5syZU2b7N998k5YtWzJr1iw6derE3XffzZ133snMmTNLtbNYLISHh5daRMpVfJVFY7KIiHi8SgWW/Px81q9fz7Bhw0ptHzZsGGvWrCnzPT/99NM57YcPH866desoKCgo2ZadnU1kZCQtWrRgzJgxbNy4sdxa8vLyyMzMLLVIHdP5WrD5QloipGwyuxoREalBlQos6enp2O12wsLCSm0PCwsjNbXs5whSU1PLbF9YWEh6ejoAHTt25IMPPmDRokXMnTsXPz8/BgwYwM6dO89by4wZMwgODi5ZIiIiKnMo4gnqNYCOo53revhWRMSjVemhW4vFUupnwzDO2Xah9mdu79u3L7feeisxMTEMGjSI+fPn0759e1577bXz7vPxxx8nIyOjZDlw4EBVDkXcXfFQ/Vs+gcJ8c2sREZEa41WZxqGhodhstnOupqSlpZ1zFaVYeHh4me29vLxo1KhRme+xWq306tWr3Cssvr6++PqaO9W1uIDWQ6B+GGQfgV3xv19xERERj1KpKyw+Pj7ExsYSHx9fant8fDz9+/cv8z39+vU7p/2yZcvo2bMn3t7eZb7HMAwSEhJo2rRpZcqTusjmBV1ucK5rqH4REY9V6VtCU6ZM4Z133uG9995j27ZtTJ48meTkZCZMmAA4b9WMHz++pP2ECRPYv38/U6ZMYdu2bbz33nu8++67PPzwwyVtpk+fztKlS9mzZw8JCQncddddJCQklOxTpFzFt4V2LIVTx82tRUREakSlbgkBjBs3jmPHjvHMM8+QkpLCJZdcwpIlS4iMjAQgJSWl1JgsUVFRLFmyhMmTJ/PGG2/QrFkzXn31Vf7whz+UtDl58iT33HMPqampBAcH0717d1auXEnv3r2r4RDF44V1hvCukLoZflsAvf9sdkUiIlLNLIbhGQNYZGZmEhwcTEZGBkFBQWaXI7Xt5znwzWPQrAfc84PZ1YiISAVV9PtbcwmJZ+hyA1i94PAGOLrd7GpERKSaKbCIZwgIhXZFAxTq4VsREY+jwCKeo3io/s1x4LCbW4uIiFQrBRbxHO2HQ72GkJUCe5abXY2IiFQjBRbxHF6+cElR7zMN1S8i4lEUWMSzxBSNybJtMeRqQkwREU+hwCKepXkPCG0Phach8QuzqxERkWqiwCKexWL5/eFb3RYSEfEYCiziebqOAyyw/0c4sc/sakREpBoosIjnCW4OrQc71zfNM7UUERGpHgos4pmKJ0TcNBc8Y/YJEZE6TYFFPFPHMeBT33lLKPkns6sREZGLpMAinsnHH6Kvca5rqH4REbenwCKeq1tRb6Gtn0PBaVNLERGRi6PAIp6rZX9o0BLysyDpK7OrERGRi+BldgEiNcZqdY7JsuIFWD0Lju8Fq61o8QKLrYyfvc6zzVp2m5Kfy9pWvA/rWT8Xt7OY/SckIuI2FFjEs8Xc5AwsR7Y4F1diqUh4sv4eiC7UprYCV3kh7ILhr6IBUYFOREpTYBHPFtIarvu3s6eQw+5cDDs4Cot+LgTDUfrnsrYZ9t/fX9bP52tjOM5fm2EHux3s+bX35+FOLNbSgapeAwgIhYDGENDEuV6/SdHPoUXbGoN/I7DprzYRT6P/q8Xzdb3RuZjBMM4KOYVlBJ/C84SpM38+X5vywtWFAlchOByUDlxnbLtQbZWq/4xX46z18/7ZOYqOrcD5c34WZByowB+6BfxDfg81AY2Lgs0ZoSagMdQvevUJqJZTLSI1S4FFpCZZLM5/7etf/GUzjIoFLkchnD4JOWmQcxSy0yAn3bmeU7SenQanjgGG8/XUMThagRq8A8oJNmdexWkC9Ro6b6+JSK3T36IiYh6L5fdnW6qDww6njp8RbI4WhZqzgk1OuvPnwlwoyIGTOXByfwXqtZ1xW+qMYFN8tebsxduveo5LRBRYRMSDWG3O8FC/8YXbGgbkZ58VbM64cnNmsMk5CqdPOK/+ZB9xLhXhG3RGsDnzas0ZS/FVHb8GetBYpBwKLCJSN1ks4BvoXEJaX7i9veDc21BlBZviqziOAsjLdC7H91x4/1bvMm5DnRlsGlPqgWOb98X/GYi4EQUWEZGKsHlDUFPnciGGAbkZZ9yOKiPYlFzVSYe8DGfAyTrsXCrCr0HFgk39Js55tXT1RtycAouISHWzWJzdsOs1gNB2F25fkAun0s//MHFxsCneZtgh96RzSd9x4f17+Z0n1JTRRdy/UfU9UyRSjRRYRETM5u0HwS2cy4U4HM7nacp8mPjouVdxCnKcDxdnHKhEt/BG53+Y+OyrOj7+F334IhWhwCIi4k6sVgho5FzoeOH2+TlnBJmjZwSbM29XFb2WdAtPdy5Ht114/94B596GKquL+JlXbizWoltUltKvFuu52878nW5r1WkKLCIinswnwLk0bHXhtvZCOH38PA8Tn9VNPDsN7HnOKzgncuDEvpo+kjOcGWYqGn6oZPsz3nfe351vX2VsK699qd9RwbrO3heVbF+BmsvaV9/7oGFkDZzTC1NgERERJ5uX88pI/SYQ1rn8tsXdws/XS+rsqzinT1RjoYbz86H80ZKl+l1yvQKLiIi4kTO7hTdqc+H2DgclQcM4Y/3s1zJ/R9FUDRVtb5zRnnJ+d6F9Ucn2Z/6ujG0V3hcV/Jzz7auSn13yOy58jBXpJVdDFFhERKTmaUoDuUj6L0hERERcngKLiIiIuDwFFhEREXF5CiwiIiLi8hRYRERExOUpsIiIiIjLU2ARERERl6fAIiIiIi5PgUVERERcngKLiIiIuDwFFhEREXF5CiwiIiLi8hRYRERExOV5zGzNRtGU3JmZmSZXIiIiIhVV/L1d/D1+Ph4TWLKysgCIiIgwuRIRERGprKysLIKDg8/7e4txoUjjJhwOB4cPHyYwMBCLxVJt+83MzCQiIoIDBw4QFBRUbft1JZ5+jDo+9+fpx6jjc3+efow1eXyGYZCVlUWzZs2wWs//pIrHXGGxWq20aNGixvYfFBTkkf8RnsnTj1HH5/48/Rh1fO7P04+xpo6vvCsrxfTQrYiIiLg8BRYRERFxeQosF+Dr68vUqVPx9fU1u5Qa4+nHqONzf55+jDo+9+fpx+gKx+cxD92KiIiI59IVFhEREXF5CiwiIiLi8hRYRERExOUpsIiIiIjLU2ABZs+eTVRUFH5+fsTGxrJq1apy269YsYLY2Fj8/Pxo3bo1b775Zi1VWjWVOb7ly5djsVjOWZKSkmqx4opbuXIlY8eOpVmzZlgsFj7//PMLvsfdzl9lj9HdzuGMGTPo1asXgYGBNGnShGuuuYbt27df8H3uch6rcnzudA7nzJlD165dSwYU69evH19//XW573GXc1esssfoTuevLDNmzMBisTBp0qRy29X2eazzgSUuLo5Jkybx5JNPsnHjRgYNGsTIkSNJTk4us/3evXsZNWoUgwYNYuPGjTzxxBM8+OCDLFiwoJYrr5jKHl+x7du3k5KSUrK0a9euliqunJycHGJiYnj99dcr1N7dzh9U/hiLucs5XLFiBRMnTuTnn38mPj6ewsJChg0bRk5Oznnf407nsSrHV8wdzmGLFi14/vnnWbduHevWrePyyy/n6quvZuvWrWW2d6dzV6yyx1jMHc7f2X799VfefvttunbtWm47U86jUcf17t3bmDBhQqltHTt2NB577LEy2z/yyCNGx44dS237y1/+YvTt27fGarwYlT2+H374wQCMEydO1EJ11QswPvvss3LbuNv5O1tFjtGdz6FhGEZaWpoBGCtWrDhvG3c+jxU5Pnc/hw0bNjTeeeedMn/nzufuTOUdo7uev6ysLKNdu3ZGfHy8cdlllxkPPfTQeduacR7r9BWW/Px81q9fz7Bhw0ptHzZsGGvWrCnzPT/99NM57YcPH866desoKCiosVqroirHV6x79+40bdqUK664gh9++KEmy6xV7nT+Lpa7nsOMjAwAQkJCztvGnc9jRY6vmLudQ7vdzrx588jJyaFfv35ltnHncwcVO8Zi7nb+Jk6cyOjRoxk6dOgF25pxHut0YElPT8dutxMWFlZqe1hYGKmpqWW+JzU1tcz2hYWFpKen11itVVGV42vatClvv/02CxYsYOHChXTo0IErrriClStX1kbJNc6dzl9VufM5NAyDKVOmMHDgQC655JLztnPX81jR43O3c7hlyxbq16+Pr68vEyZM4LPPPiM6OrrMtu567ipzjO52/gDmzZvHhg0bmDFjRoXam3EePWa25othsVhK/WwYxjnbLtS+rO2uojLH16FDBzp06FDyc79+/Thw4AAzZ87k0ksvrdE6a4u7nb/KcudzeP/997N582ZWr159wbbueB4renzudg47dOhAQkICJ0+eZMGCBdx+++2sWLHivF/o7njuKnOM7nb+Dhw4wEMPPcSyZcvw8/Or8Ptq+zzW6SssoaGh2Gy2c642pKWlnZMci4WHh5fZ3svLi0aNGtVYrVVRleMrS9++fdm5c2d1l2cKdzp/1ckdzuEDDzzAokWL+OGHH2jRokW5bd3xPFbm+MriyufQx8eHtm3b0rNnT2bMmEFMTAz/+te/ymzrjucOKneMZXHl87d+/XrS0tKIjY3Fy8sLLy8vVqxYwauvvoqXlxd2u/2c95hxHut0YPHx8SE2Npb4+PhS2+Pj4+nfv3+Z7+nXr9857ZctW0bPnj3x9vausVqroirHV5aNGzfStGnT6i7PFO50/qqTK59DwzC4//77WbhwId9//z1RUVEXfI87nceqHF9ZXPkcns0wDPLy8sr8nTudu/KUd4xlceXzd8UVV7BlyxYSEhJKlp49e/LHP/6RhIQEbDbbOe8x5TzW2OO8bmLevHmGt7e38e677xqJiYnGpEmTjICAAGPfvn2GYRjGY489Ztx2220l7ffs2WP4+/sbkydPNhITE413333X8Pb2Nj799FOzDqFclT2+V155xfjss8+MHTt2GL/99pvx2GOPGYCxYMECsw6hXFlZWcbGjRuNjRs3GoDx8ssvGxs3bjT2799vGIb7nz/DqPwxuts5vPfee43g4GBj+fLlRkpKSsly6tSpkjbufB6rcnzudA4ff/xxY+XKlcbevXuNzZs3G0888YRhtVqNZcuWGYbh3ueuWGWP0Z3O3/mc3UvIFc5jnQ8shmEYb7zxhhEZGWn4+PgYPXr0KNXd8Pbbbzcuu+yyUu2XL19udO/e3fDx8TFatWplzJkzp5YrrpzKHN8LL7xgtGnTxvDz8zMaNmxoDBw40Pjqq69MqLpiirsPnr3cfvvthmF4xvmr7DG62zks69gA4/333y9p487nsSrH507n8M477yz5+6Vx48bGFVdcUfJFbhjufe6KVfYY3en8nc/ZgcUVzqPFMIqekhERERFxUXX6GRYRERFxDwosIiIi4vIUWERERMTlKbCIiIiIy1NgEREREZenwCIiIiIuT4FFREREXJ4Ci4iIiLg8BRYRERFxeQosIiIi4vIUWERERMTlKbCIiIiIy/t/aaICYngcq68AAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 11, + "id": "a0cd549b", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", @@ -2145,8 +2016,10 @@ }, { "cell_type": "markdown", - "id": "c0ce9e6e", - "metadata": {}, + "id": "b4d2ab87", + "metadata": { + "editable": true + }, "source": [ "## More preprocessing examples, two-dimensional example, the Franke function" ] @@ -2154,8 +2027,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "f9432055", - "metadata": {}, + "id": "9f5d7996", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -2254,8 +2130,10 @@ }, { "cell_type": "markdown", - "id": "1d262f6d", - "metadata": {}, + "id": "5ea41ab9", + "metadata": { + "editable": true + }, "source": [ "## To think about, first part\n", "\n", @@ -2272,7 +2150,7 @@ "Thus, if we cannot assume that the expected outputs/targets are zero\n", "when all predictors are zero (the columns in the design matrix), it\n", "may be a bad idea to implement a model which penalizes the intercept.\n", - "Furthermore, in for example Ridge and Lasso regression, the default solutions\n", + "Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions\n", "from the library **Scikit-Learn** (when not shrinking $\\beta_0$) for the unknown parameters\n", "$\\boldsymbol{\\beta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n", "$\\boldsymbol{X}$ are zero centered, that is we subtract the mean values." @@ -2280,8 +2158,10 @@ }, { "cell_type": "markdown", - "id": "311efbd3", - "metadata": {}, + "id": "ce0afdb4", + "metadata": { + "editable": true + }, "source": [ "## More thinking\n", "\n", @@ -2313,8 +2193,10 @@ }, { "cell_type": "markdown", - "id": "9931c947", - "metadata": {}, + "id": "ad92f42b", + "metadata": { + "editable": true + }, "source": [ "## Still thinking\n", "\n", @@ -2324,22 +2206,13 @@ }, { "cell_type": "code", - "execution_count": 6, - "id": "3a03b7ba", - "metadata": {}, - "outputs": [ - { - "ename": "NameError", - "evalue": "name 'some_model' is not defined", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)", - "Input \u001b[0;32mIn [6]\u001b[0m, in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 5\u001b[0m y_train \u001b[38;5;241m=\u001b[39m y_train \u001b[38;5;241m-\u001b[39m y_train_mean\n\u001b[1;32m 7\u001b[0m \u001b[38;5;66;03m# The we fit our model with the training data\u001b[39;00m\n\u001b[0;32m----> 8\u001b[0m trained_model \u001b[38;5;241m=\u001b[39m \u001b[43msome_model\u001b[49m\u001b[38;5;241m.\u001b[39mfit(X_train,y_train)\n\u001b[1;32m 11\u001b[0m \u001b[38;5;66;03m#Model prediction, we need also to transform our data set used for the prediction.\u001b[39;00m\n\u001b[1;32m 12\u001b[0m X_test \u001b[38;5;241m=\u001b[39m X_test \u001b[38;5;241m-\u001b[39m X_train_mean \u001b[38;5;66;03m#Use mean from training data\u001b[39;00m\n", - "\u001b[0;31mNameError\u001b[0m: name 'some_model' is not defined" - ] - } - ], + "execution_count": 13, + "id": "fae66265", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "#Model training, we compute the mean value of y and X\n", "y_train_mean = np.mean(y_train)\n", @@ -2359,8 +2232,10 @@ }, { "cell_type": "markdown", - "id": "542e3a89", - "metadata": {}, + "id": "1bf672d8", + "metadata": { + "editable": true + }, "source": [ "## What does centering (subtracting the mean values) mean mathematically?\n", "\n", @@ -2373,8 +2248,10 @@ }, { "cell_type": "markdown", - "id": "81d0d338", - "metadata": {}, + "id": "30834557", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", @@ -2383,8 +2260,10 @@ }, { "cell_type": "markdown", - "id": "f970b32f", - "metadata": {}, + "id": "5c05a427", + "metadata": { + "editable": true + }, "source": [ "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n", "\n", @@ -2396,8 +2275,10 @@ }, { "cell_type": "markdown", - "id": "4517c17b", - "metadata": {}, + "id": "902391ce", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", @@ -2406,16 +2287,20 @@ }, { "cell_type": "markdown", - "id": "2e83e766", - "metadata": {}, + "id": "a68a6228", + "metadata": { + "editable": true + }, "source": [ "for all $j$. For $\\beta_0$ we have" ] }, { "cell_type": "markdown", - "id": "5573d14d", - "metadata": {}, + "id": "212cf295", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", @@ -2424,16 +2309,20 @@ }, { "cell_type": "markdown", - "id": "c03595b1", - "metadata": {}, + "id": "e7db36e1", + "metadata": { + "editable": true + }, "source": [ "Multiplying away the constant $2/n$, we obtain" ] }, { "cell_type": "markdown", - "id": "7c7103e6", - "metadata": {}, + "id": "5d7f590c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", @@ -2442,8 +2331,10 @@ }, { "cell_type": "markdown", - "id": "a6d143de", - "metadata": {}, + "id": "bbf7c89a", + "metadata": { + "editable": true + }, "source": [ "## Further Manipulations\n", "\n", @@ -2453,8 +2344,10 @@ }, { "cell_type": "markdown", - "id": "a6a497e9", - "metadata": {}, + "id": "05692612", + "metadata": { + "editable": true + }, "source": [ "$$\n", "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", @@ -2463,16 +2356,20 @@ }, { "cell_type": "markdown", - "id": "00c903d6", - "metadata": {}, + "id": "3dc53330", + "metadata": { + "editable": true + }, "source": [ "We obtain then" ] }, { "cell_type": "markdown", - "id": "a1eb68d4", - "metadata": {}, + "id": "562d655d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", @@ -2481,16 +2378,20 @@ }, { "cell_type": "markdown", - "id": "e60d2955", - "metadata": {}, + "id": "528c869d", + "metadata": { + "editable": true + }, "source": [ "If we define" ] }, { "cell_type": "markdown", - "id": "fd989568", - "metadata": {}, + "id": "82a20d9e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n", @@ -2499,16 +2400,20 @@ }, { "cell_type": "markdown", - "id": "f18e5655", - "metadata": {}, + "id": "5e37316e", + "metadata": { + "editable": true + }, "source": [ "and if we define the mean value of the outputs as" ] }, { "cell_type": "markdown", - "id": "1ae7b2a1", - "metadata": {}, + "id": "90c44e36", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n", @@ -2517,16 +2422,20 @@ }, { "cell_type": "markdown", - "id": "2e4d559e", - "metadata": {}, + "id": "be72a7d3", + "metadata": { + "editable": true + }, "source": [ "we have" ] }, { "cell_type": "markdown", - "id": "5ab9bac7", - "metadata": {}, + "id": "0a435bf5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n", @@ -2535,16 +2444,20 @@ }, { "cell_type": "markdown", - "id": "8823ed6b", - "metadata": {}, + "id": "a67864cb", + "metadata": { + "editable": true + }, "source": [ "In the general case, that is we have more parameters than $\\beta_0$ and $\\beta_1$, we have" ] }, { "cell_type": "markdown", - "id": "319ca663", - "metadata": {}, + "id": "9d8602b7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", @@ -2553,16 +2466,20 @@ }, { "cell_type": "markdown", - "id": "b58c4a86", - "metadata": {}, + "id": "32caef0d", + "metadata": { + "editable": true + }, "source": [ "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)" ] }, { "cell_type": "markdown", - "id": "167fe9fd", - "metadata": {}, + "id": "f24465a9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", @@ -2571,8 +2488,10 @@ }, { "cell_type": "markdown", - "id": "8abba21b", - "metadata": {}, + "id": "f64212da", + "metadata": { + "editable": true + }, "source": [ "## Wrapping it up\n", "\n", @@ -2581,8 +2500,10 @@ }, { "cell_type": "markdown", - "id": "a95f6f87", - "metadata": {}, + "id": "8d0bf5ae", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", @@ -2591,8 +2512,10 @@ }, { "cell_type": "markdown", - "id": "6ca18ea2", - "metadata": {}, + "id": "e8d36ff8", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", @@ -2602,8 +2525,10 @@ }, { "cell_type": "markdown", - "id": "9630f6ff", - "metadata": {}, + "id": "aea58aac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", @@ -2612,16 +2537,20 @@ }, { "cell_type": "markdown", - "id": "406b9ba8", - "metadata": {}, + "id": "bc4b6f91", + "metadata": { + "editable": true + }, "source": [ "What does this mean? And why do we insist on all this? Let us look at some examples." ] }, { "cell_type": "markdown", - "id": "5557c0e8", - "metadata": {}, + "id": "349b03e2", + "metadata": { + "editable": true + }, "source": [ "## Linear Regression code, Intercept handling first\n", "\n", @@ -2631,42 +2560,13 @@ }, { "cell_type": "code", - "execution_count": 7, - "id": "27b9d36a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "True beta: [2, 0.5, 3.7]\n", - "Fitted beta: [2.08376632 0.19569961 3.97898392]\n", - "Sklearn fitted beta: [2.08376632 0.19569961 3.97898392]\n", - "MSE with intercept column\n", - "0.004113634617443139\n", - "MSE with intercept column from SKL\n", - "0.004113634617443147\n", - "Manual intercept: 2.083766322923899\n", - "Fitted beta (wiothout intercept): [0.19569961 3.97898392]\n", - "Sklearn intercept: 2.0837663229239043\n", - "Sklearn fitted beta (without intercept): [0.19569961 3.97898392]\n", - "MSE with Manual intercept\n", - "0.00411363461744314\n", - "MSE with Sklearn intercept\n", - "0.004113634617443131\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGdCAYAAABO2DpVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABswElEQVR4nO3dd3gUVffA8e9m0yskpAGBhE4IBBBDkypNqgjSlKqgCEp5Ebs0pUkTFRSQ8gMBBSlKCaB0kF4khBoSaiCEQHrZMr8/8rIvSxJIILubcj7Ps4/u7Mzds4dN5uTeO3dUiqIoCCGEEELkAytLByCEEEKIokMKCyGEEELkGykshBBCCJFvpLAQQgghRL6RwkIIIYQQ+UYKCyGEEELkGykshBBCCJFvpLAQQgghRL6xNvcb6vV6bt26hYuLCyqVytxvL4QQQohnoCgKiYmJlC5dGiurnPslzF5Y3Lp1Cz8/P3O/rRBCCCHywfXr1ylbtmyOr5u9sHBxcQEyA3N1dc23djUaDdu3b6dNmzbY2NjkW7vCmOTZfCTX5iF5Ng/Js3mYMs8JCQn4+fkZzuM5MXth8XD4w9XVNd8LC0dHR1xdXeVLa0KSZ/ORXJuH5Nk8JM/mYY48P20ag0zeFEIIIUS+kcJCCCGEEPlGCgshhBBC5Buzz7HIDZ1Oh0ajydMxGo0Ga2tr0tLS0Ol0JopMSJ7NpyjlWq1WY21tLZeYC1EMFLjCIikpiRs3bqAoSp6OUxQFHx8frl+/Lr+8TEjybD5FLdeOjo74+vpia2tr6VCEECZUoAoLnU7HjRs3cHR0xNPTM0+/TPV6PUlJSTg7Oz9x4Q7xfCTP5lNUcq0oChkZGdy9e5fIyEgqV65cqD+PEOLJClRhodFoUBQFT09PHBwc8nSsXq8nIyMDe3t7+aVlQpJn8ylKuXZwcMDGxoarV68aPpMQomgqkL+tikK3rxDCWGEvjoQQuSM/6UIIIYTIN1JYCCGEECLfSGEhhBBCiHwjhUU+GDBgACqVCpVKhY2NDd7e3rRu3ZrFixej1+tz3c7SpUspUaKE6QIVQgghTKxIFhY6vcI/EffYeOom/0TcQ6fP25oYz6Jdu3ZER0cTFRXF1q1badGiBSNGjKBjx45otVqTv78QQojiTadX2Dl/JFYXNnLoUoxZzn3ZKXKFRWjYbV6atpPeCw8xYvUpei88xEvTdhIaFm3S97Wzs8PHx4cyZcpQt25dPv30UzZu3MjWrVtZunQpALNmzaJmzZo4OTnh5+fHe++9R1JSEgC7d+9m4MCBxMfHG3o/xo8fD8CKFSuoV68eLi4u+Pj40KdPH2JiYkz6eYQQQhQeoWHRfD56AJzYQofk31n4y3KznPuyU6QKi78v3GPYypNEx6cZbb8dn8bQFSfMnuCWLVsSHBzMunXrgMzL7ebOnUtYWBjLli1j586djB07FoBGjRoxZ84cXF1diY6OJjo6mjFjxgCQkZHBpEmTOH36NBs2bCAyMpIBAwaY9bMIIYQomELDopm8dA2d9x2h3CF7Vt6oymGCLHbuK1ALZD0PnV5h+l9XyK7jRwFUwIQ/w2kd6IPaynzrZFSrVo1///0XgJEjRxq2BwQEMGnSJIYOHcq8efOwtbXFzc0NlUqFj4+PURuDBg0y/H+FChWYO3cuISEhhlUZhRBCFE86vcK4Df/yydkFlEiGO+5g2/JdCLfcua/I9FgcjYrjTmJGjq8rQHR8Gkci48wXFJnLGT9c8GvXrl20bt2aMmXK4OLiQr9+/bh37x7JyclPbOPkyZN06dKF8uXL4+LiQvPmzQG4du2aqcMXQghRgB2JjKPX5SlUjdShUcOykLbYOXsYXrfEua/IFBYxiem53C/t6Tvlo3PnzhEQEMDVq1dp3749QUFB/P777xw/fpwffvgB4Il3ck1OTqZNmzY4OzuzYsUKjh49yvr164HMIRIhhBDF17kdi2h59A4Au+p5cdCpVbb7mfPcV2SGQrxc7HK5n/nuUbBz507OnDnDqFGjOHbsGFqtlpkzZxqWNv7tt9+M9re1tc1ye+zz588TGxvL1KlT8fPzA+DYsWPm+QBCCCEKrLiYmwQsX4qtDi76WzHb5z/kdCY057mvyPRYvOjvjreLLTmNIKkAXzd7QgLcTfL+6enp3L59m5s3b3LixAkmT55Mly5d6NixI/369aNixYpotVq+++47rly5wvLly/nxxx+N2vD39ycpKYm///6b2NhYUlJSKFeuHLa2tobj/vjjDyZNmmSSzyCEEKLw2DVrIKXuQ7wjzK4xBFTqLPuY+tyXnSJTWKitVIxtVQEgS3Hx8Pm4ToEmm7wSGhqKr68v/v7+tGvXjl27djF37lw2btyIWq2mdu3azJo1i2nTphEUFMQvv/zClClTjNpo1KgR7777Lj179sTT05Pp06fj6enJ0qVLWbNmDYGBgUydOpUZM2aY5DMIIYQoHE5uX0E3+8Oo2j0g/PWXuWpTySLnvuyoFEUx6woaCQkJuLm5ER8fj6urq9FraWlpREZGEhAQkOfbKuv1ehISEjh4LYVJm88ZXXLq62bPuE6BtAvyzZfPUJw9zLOrq6vcrdLEilqun+fn25Q0Gg1btmyhffv22NjYWDqcIkvynH/u3IjAblFTSpDEIZ83aPDuPELDopnwZzhxSalMD9Ex9ogad2eHfD33Pen8/agiM8fioXZBPrQN8uVIZBwxiWl4uWR2AZmzWhNCCCFMIT01hf0jX6WGv5a7bpWoO3AWAO2CfGkd6MOhyzHEnjvE4v4v0qCSl0XOfUWusIDMYZGGFT2evqMQQghRiKwb8Qq1w7QkRLhTYum32Nr9r/dPbaUiJMCdLeew6B/Uhb9/VQghhCgGti+dRM19mbdziGwXSJXglywcUfaksBBCCCEKuFtR53CYvxK1Auer2tB90q+WDilHUlgIIYQQBZii13NoZC9KxcM9VwiZsxq1dcGdySCFhRBCCFGArfm8J9XPZ6BTQfK7vSkTEGjpkJ5ICgshhBCigLpxOQynU2cACHvJk7aDvrRwRE8nhYUQQghRAGWkp5G6egBt60VzroU9XeeGWjqkXMlzYXHz5k3efPNNPDw8cHR0pHbt2hw/ftwUsRV6zZs3N7pVuhBCCJFbx5f+h8raSyRbOdHo0zXYOThaOqRcyVNhcf/+fRo3boyNjQ1bt24lPDycmTNnUqJECROFVzgMGDAAlUqV5TF9+nSj+3r4+/szZ84cywUqhBCiUPhz7mju7dhOik7FlYZT8PGrZOmQci1P00qnTZuGn58fS5YsMWzz9/fP75gKpXbt2hnlBcDT0xO1OutNYYQQQoicRJw5iOeyrbglW/OHc1V6te1v6ZDyJE+FxR9//EHbtm15/fXX2bNnD2XKlOG9995j8ODBOR6Tnp5Oenq64XlCQgKQuW68RqMx2lej0aAoCnq9Hr1en5fQeHjLk4fHm5OiKNja2uLl5WW0vWXLlgQHBzN79mxatmzJ1atXGTVqFKNGjQLIcov0wsCSeS5uilqu9Xo9iqKg0WgKVMH98PfQ47+PRP6SPOeOTqvl7NghVE6G2+7Q4qtVecqZKfOc2zbzVFhcuXKF+fPnM3r0aD799FOOHDnCBx98gJ2dHf369cv2mClTpjBhwoQs27dv346jo/F4kbW1NT4+PiQlJZGRkYGiKKRp8vYLNfXegzztnxN7GytUqtwth6rRaNBqtYai6SGtVktGRgYJCQksWbKEl156iQEDBhhy9fj+hUliYqKlQyg2ikquMzIySE1NZe/evWi1WkuHk8WOHTssHUKxIHl+srjNs2kQqUOjhsgubUg4fgY4k+d2TJHnlJSUXO2Xp8JCr9dTr149Jk+eDECdOnU4e/Ys8+fPz7Gw+OSTTxg9erTheUJCAn5+frRp0ybbu5tev34dZ2dn7O3tScnQUmeaZb6EYeNb42ibu/TY2Niwbds2ypYta9jWrl07rK2tsbW1xdXVFVdXV2xsbChVqhSVK1c2VdgmpygKiYmJuLi45LrwEs+mqOU6LS0NBwcHmjZtWuDubrpjxw5at24td900Icnz0x3YMA//A3cAOPuyH31Hz8hzG6bMc27/GM5TYeHr60tgoPHCHNWrV+f333/P8Rg7Ozvs7OyybLexscnyoXU6HSqVCisrK8PDUvLy/iqVihYtWjB//nzDNicnJ3r37m34PI/uW5hvgf2wS76wf47CoKjl2soqsxcwu5/9gqCgxlXUSJ6zF3fnOsqcH7HVwaUANT1mbXmu1TVNkefctpenqBs3bsyFCxeMtl28eJHy5cvnpZlcc7BREz6xba721ev1JCYk4uLqki+/hB1s8jYG7OTkRKVKhWfWrhBCiIJj/7yhlE6GeCeoMX1BgV6y+2nyFPmoUaNo1KgRkydPpkePHhw5coQFCxawYMECkwSnUqlyPRyh1+vR2qpxtLUusH/d2draFsoJm0IIIUznxLbldFbt49YrNpzyH0KDmo0sHdJzydMZ+MUXX2T9+vWsWrWKoKAgJk2axJw5c3jjjTdMFV+R4u/vz969e7l58yaxsbGWDkcIIYSF3bkRQYV/PgbgWvketH9nsoUjen557mvp2LEjHTt2NEUsRd7EiRN55513qFixIunp6YbLCYUQQhQ/6akp/DO0M17lrPD0rUzdgbMsHVK+KLyDOAXI0qVLs92+e/duo+cNGjTg9OnTpg9ICCFEgbduxCvUvqQn9aoz6Qu/wtau4Fwt9TwK5mQEIYQQogjbvnQSNffFAHC5QyBB9dtZOKL8I4WFEEIIYUY3I8NxmL8StQLnq9rQbdKvlg4pX0lhIYQQQpiJTqvl8MhelIqHe64QMmd1ob60NDtSWAghhBBm8vsXvah+QYNOBcnv9qZMQODTDypkpLAQQgghzODG5TDsI/4FIOwlT9oO+tLCEZmGFBZCCCGEiWWkp5G6egCdA+8Q0d6JrnNDLR2SyUhhIYQQQpjYsSWjqKy9RDxO1Bu1GjsHx6cfVEhJYSGEEEKY0J9zR5Owbid3MqyJaDgNH7+ifV+pojUVVQghhChAIs4cxHPZVtySrdjtGUDPtn0tHZLJSY+FEEIIYQI6rZazY4fglgy3PaD1rPWWDskspLAwE5VKxYYNG3J83d/fnzlz5pgtnmfxxRdf8M477xieK4rCkCFDcHd3R6VScerUKZo3b87IkSOf+T2WLl1KiRIlnj9Y8UQDBgzg1VdfzZe2YmJi8PT05ObNm/nSnhBFxW//6UjlSB0ZalCPGoa7t5+lQzILKSzyQUxMDO+88w7lypXDzs4OHx8f2rZtyz///GPp0PLNnTt3+Pbbb/nkk08M20JDQ1m6dCmbNm0iOjqaoKAg1q1bx6RJkwz75LVg6tmzJxcvXsxTbM9bzJjb7t27UalUPHjw4Lna8ff3R6VSGT3Kli2bP0HmgZeXF3379mXcuHFmf28hCqq9a78n8K+rAIS3Kk/T7sMtHJH5yByLfNCtWzc0Gg3Lli2jQoUK3Llzh7///pu4uDhLh2YkIyMDW1vbZzr2559/pmHDhvj7+5OQkABAREQEvr6+NGrUyLCfu7v7c8Xo4OCAg4PDc7XxrJ4nP5YyceJEBg8ebHiuVqstEsfAgQMJCQnhm2++oWTJkhaJQYiCIu7OdXSzf8BWB5cC1PSYucnSIZlVwe6xUBTISM79Q5OSt/2f9MjlLc0fPHjA/v37mTZtGi1atKB8+fKEhITwySef0KFDhxyPmzhxIt7e3pw6dSrb1+Pj4xkyZAheXl64urrSsmVLozujRkRE0KVLF7y9vXF2dubFF1/kr7/+MmrD39+fr776igEDBuDm5sbgwYMNQw3btm2jevXqODs7065dO6Kjo5/4OVevXk3nzp0NzwcOHMj777/PtWvXUKlU+Pv7A8a9B82bN+fq1auMGjXK8Bf10zw+FDJ+/Hhq167N8uXL8ff3x83NjV69epGYmAhkdunv2bOHb7/91vAeUVFRAISHh9O+fXucnZ3x9vamb9++xMbGGtpu3rw5w4cPZ/To0ZQqVYrWrVsDcPbsWTp06ICrqysuLi40adKEiIgIw3FLliyhevXq2NvbU61aNebNm2d4LSoqCpVKxerVq2nUqBH29vbUqFHDcKfbqKgoWrRoAUDJkiVRqVQMGDDgqXnJiYuLCz4+PoaHp6cnOp2Ot956i4CAABwcHKhatSrffvvtE9tZu3YtNWvWxMHBAQ8PD1q1akVycnKuPjNAzZo18fHxYf364jGGLMSTHFn0AVZ6hQdOUGP6giK3ZPfTFOxPq0mByaVztasVUCI/3/vTW2Dr9NTdnJ2dcXZ2ZsOGDTRo0AA7O7sn7q8oCiNHjmTDhg3s37+fypUrZ7tPhw4dcHd3Z8uWLbi5ufHTTz/x8ssvc/HiRdzd3UlKSqJ9+/Z89dVX2Nvbs2zZMjp16sSFCxcoV66coa1vvvmGL774gs8//xyA/fv3k5KSwowZM1i+fDlWVla8+eabjBkzhl9++SXbmO/fv09YWBj16tUzbJszZw6VKlViwYIFHD16NNu/lNetW0dwcDBDhgwx+qs6ryIiItiwYQObNm3i/v379OjRg6lTp/L111/z7bffcvHiRYKCgpg4cSIAnp6eREdH06xZMwYPHsysWbNITU3lo48+okePHuzcudPQ9rJlyxg6dCgHDhxAURRu3rxJ06ZNad68OTt37sTV1ZUDBw6g1WoBWLhwIePGjeP777+nTp06nDx5ksGDB+Pk5ET//v0N7X744YfMmTOHwMBAZs2aRefOnYmMjMTPz4/ff/+dbt26ceHCBVxdXQ09NJMnT2by5MlPzMXWrVtp0qTJE/fR6/WULVuW3377jVKlSnHw4EGGDBmCr68vPXr0yLJ/dHQ0vXv3Zvr06XTt2pXExET27duH8t/iOrefOSQkhH379jFo0KAnxidEUXZi23LaaXaS8LKaw5VH07Bmo6cfVMQU7MKiELC2tmbp0qUMHjyYH3/8kbp169KsWTN69epFrVq1jPbVarX069ePY8eOceDAgRzHw3ft2sWZM2eIiYkxFCozZsxgw4YNrF27liFDhhAcHExwcLDhmK+++or169fzxx9/MHz4/8byWrZsyZgxYwzP9+/fj0aj4ccff6RixYoADB8+3HBSzs7Vq1dRFIXSpf9X5Lm5ueHi4oJarcbHxyfb49zd3VGr1Ya/qp+VXq9n6dKluLi4ANC3b1/+/vtvvv76a9zc3LC1tcXR0dHoPebPn0/dunWNTtSLFy/Gz8+PixcvUqVKFQAqVarE9OnTDft8+umnuLm5sXr1amxsbAAM+wJMmjSJmTNn8tprrwEQEBBAeHg4P/30k9FJdvjw4XTr1s0QS2hoKD///DNjx441DBd5eXkZ9c68++67Rid+vV5PUlISzs7OWFlldi6WKVPGKDcfffSRoWiEzOLkgw8+YMKECYZtAQEBHDx4kN9++y3HwkKr1fLaa69Rvnx5ILMHIq+fuUyZMpw8eTJL+0IUF9HXLlHhn48BOOvXm9b9P3/KEUVTwS4sbBwzew5yQa/Xk5CYiKuLi+GX8HO/dy5169aNDh06sG/fPv755x9CQ0OZPn06ixYtMurmHjVqFHZ2dhw6dIhSpUrl2N7x48dJSkrCw8PDaHtqaqqhSz45OZkJEyawadMmbt26hVarJTU1lWvXrhkd82gvw0OOjo6GogLA19eXmJiYHONJTU0FwN7ePuckmJC/v7+hqICnxwuZOdy1axfOzs5ZXouIiDAUC4/n59SpUzRp0sRQVDzq7t27XL9+nbfeesuoB0ar1eLm5ma0b8OGDQ3/b21tTb169Th37twTY3Z3dzeao6LX60lISMDV1TXH7/SHH35o9B17+L368ccfWbRoEVevXiU1NZWMjAxq166dbRvBwcG8/PLL1KxZk7Zt29KmTRu6d+9OyZIl8/SZHRwcSElJeeJnFKKoSk9N4fjgV7HxtaV8xYq8MGCmpUOymIJdWKhUuRqOAECvBxtd5v75UVjkkb29Pa1bt6Z169Z8+eWXvP3224wbN87ol37r1q1ZtWoV27Zt44033sixLb1ej6+vr2Fc/lEP/8L98MMP2bZtGzNmzKBSpUo4ODjQvXt3MjIyjPZ3csqav8dPmiqVytDtnZ2HJ6v79+9nKXbMIbt49Xr9E4/R6/V06tSJadOmZXnN19fX8P+P5+dJE0cfvufChQupX7++0Wu5mTT5tDkmzzIUUqpUKSpVMl7F77fffmPUqFHMnDmThg0b4uLiwjfffMPhw4ezbVOtVrNjxw4OHjzI9u3b+e677/jss884fPgwjo6ZBXZuPnNcXByenp5PjF+IomrdiHbUvqon7ZY9KQM+w9bOMn+IFQQFu7AoxAIDA7OsW9G5c2c6depEnz59UKvV9OrVK9tj69aty+3bt7G2tjZMinzcvn37GDBgAF27dgUgKSnJMGkxv1WsWBFXV1fCw8OznMSextbWFp1OZ5K4nvQedevW5ffff8ff3x/rPEycqlWrFsuWLUOj0WQpaLy9vSlTpgxXrlx5YmEIcOjQIZo2bQpk/nV//PhxwxDVwytPHo/5WYZCsrNv3z4aNWrEe++9Z9j26OTT7KhUKho3bkzjxo358ssvKV++POvXr2f06NG5/sxhYWE0b978qfEJUdRsWzyRmvvuAnCpYyA9mnezcESWVbCvCikE7t27R8uWLVmxYgX//vsvkZGRrFmzhunTp9OlS5cs+3ft2pXly5czcOBA1q5dm22brVq1omHDhrz66qts27aNqKgoDh48yOeff86xY8eAzLkB69at49SpU5w+fZo+ffo89a/4Z2VlZUWrVq3Yv39/no/19/dn79693Lx50+iKjPzk7+/P4cOHiYqKIjY2Fr1ez7Bhw4iLi6N3794cOXKEK1eusH37dgYNGvTEQmf48OEkJCTQq1cvjh07xqVLl1i+fDkXLlwAMq9SmTJlimHS6JkzZ1iyZAmzZs0yaueHH35g/fr1nD9/nmHDhnH//n3DpMby5cujUqnYtGkTd+/eJSkpCcgcCqlUqZLRo0KFCkbPc3MpbqVKlTh27Bjbtm3j4sWLfPHFFxw9ejTH/Q8fPszkyZM5duwY165dY926ddy9e5fq1avn+jOnpKRw/Phx2rRp89T4hChKbkaG4/TjKtQKnKtqQ7dJv1o6JIuTwuI5OTs7U79+fWbPnk3Tpk0JCgriiy++YPDgwXz//ffZHtO9e3eWLVtG3759WbduXZbXVSoVW7ZsoWnTpgwaNIgqVarQq1cvoqKi8Pb2BmD27NmULFmSRo0a0alTJ9q2bUvdunVN9jmHDBnC6tWr81y8TJw4kaioKCpWrGiybvIxY8agVqsJDAzE09OTa9euUbp0aQ4cOIBOp6Nt27YEBQUxYsQI3NzcnjgHx8PDg507d5KUlESzZs144YUXWLhwoaH34u2332bRokUsXbqUmjVr0qxZM5YuXUpAQIBRO1OnTmXatGkEBwezb98+Nm7caBhSKlOmDBMmTODjjz/G29vbaLJtfnj33Xd57bXX6NmzJ/Xr1+fevXtGvRePc3V1Ze/evbRv354qVarw+eefM3PmTF555ZVcf+aNGzdSrly5p16xIkRRotNqOTyyFx4JcM8V6s9ZXewuLc2OSnnS4LoJJCQk4ObmRnx8PK6urkavpaWlERkZSUBAQJ4nCuZmopt4doqi0KBBAz744APDGg+S56yioqIICAjg5MmTOU6WzK3C9J0OCQlh5MiR9OnTJ8d9nufn25Q0Gg1btmyhffv22U7aFfmjKOb5t0+6UXN9ODoV3PqoD20GfGHpkEya5yedvx9VsH9biQJDpVKxYMECw3oOQjwUExND9+7d6d27t6VDEcJsrl8+g8udU2it4EwTzwJRVBQUUliIXAsODqZv3+e75e8rr7xiWFTs8cfTrogQBZOXlxdjx47N1cqqQhQFGelppK0eyCvlYonp4spr34ZaOqQCRQaDhFktWrTIsC7G4573PiMFgb+//xMv3RVCFH5HFo/kJe0lHuBM0LCV2Dnkft2j4kAKC2FWublcUgghCqo/vx2Fy6+7OdfInrQ206hTtuLTDypmZChECCGEyIWIMwfx/L9QvONUnLlRljpt3rR0SAWSFBZCCCHEU+i0Ws6OHYJbMtz2gFbfbrB0SAWWDIUIIYQQj9HpFY5ExhGTmIaXiz1R375B7UgdGWqw/s9w3L39LB1igSWFhRBCCPGI0LBoJvwZTnR8GgBN0v5mzN9XATjXujy9XhtmyfAKPBkKMbHmzZszcuTIZzp2586dVKtWzWRLdZvb7t27UalUPHjwIMd9VCpVlnusFAebNm2iTp06RebfWojCKjQsmqErThiKClddHIOObMVWBxf8rXAdsNDCERZ8UljkgwEDBqBSqbI8Ll++zLp165g0aZJhX39/f+bMmZOrdseOHctnn31W4FddzE/R0dGGpaRzY+nSpYY7vhYW2X0HOnbsiEqlYuXKlZYJSgiBTq8w4c9wHr1gfJTVr6Q76XngBLOC3mHS1ovo9HJJ+ZMUnzOWibVr147o6GijR0BAAO7u7ri4uOS5vYMHD3Lp0iVef/11E0RbcPn4+GBnZ2f299XpdBbvLRg4cCDfffedRWMQojg7Ehln6KkAaGt1hIEO+3i54R0WN2/HNeuKRMencSQyzoJRFnxSWOQTOzs7fHx8jB5qtdpoKKR58+ZcvXqVUaNGGXo1crJ69WratGljdE+F8ePHU7t2bRYvXky5cuVwdnZm6NCh6HQ6pk+fjo+PD15eXnz99ddGbc2aNYuaNWvi5OSEn58f7733nuGOmvC/v/q3bdtG9erVcXZ2NhRKD2U3pPPqq68yYMAAw/MVK1ZQr149XFxc8PHxoU+fPsTExOQpj48OhURFRaFSqVi3bh0tWrTA0dGR4OBg/vnnHyBzaGXgwIHEx8cb8jl+/HgAMjIyGDt2LGXKlMHJyYn69euze/fuLJ9506ZNBAYGYmdnx9WrV0lPT2fs2LH4+flhZ2dH5cqV+fnnnw3HhYeH0759e5ydnfH29qZv375Gd21t3rw5w4cPZ/jw4ZQoUQIPDw8+//xzw6JZT/oOdO7c2XAnViGE+cUk/q+oqKiPYJpN5rDHz/qO7LBple1+IqtCUVikaFJyfKTr0nO9b5o2LVf7msq6desoW7YsEydONPRq5GTv3r3Uq1cvy/aIiAi2bt1KaGgoq1atYvHixXTo0IEbN26wZ88epk2bxueff86hQ4cMx1hZWTF37lzCwsJYtmwZO3fuZOzYsUbtpqSkMGPGDJYvX87evXu5du0aY8aMydPny8jIYNKkSZw+fZoNGzYQGRlpVHg8q88++4wxY8Zw6tQpqlSpQu/evdFqtTRq1Ig5c+bg6upqyOfDmAcOHMiBAwdYvXo1//77L6+//jrt2rXj0qVLRp95ypQpLFq0iLNnz+Ll5UW/fv1YvXo1c+fO5dy5c/z44484OzsDmcM0zZo1o3bt2hw7dozQ0FDu3LlDjx49jOJdtmwZ1tbWHD58mLlz5zJ79mwWLVoEPPk7UL58eby8vNi3b99z50wIkXdeLpl/yJXQxfLpwfnsP+PEUU0AM7WvZ7ufyF6huCqk/sr6Ob7WpEwT5rWaZ3jeYk0L0nTZV5P1vOuxpN0Sw/N2v7fjfvr9LPud6X8mzzFu2rTJcAKCzHtirFmzxmgfd3d31Gq14S/6J4mKiqJ06dJZtuv1ehYvXoyLiwuBgYG0aNGCCxcusGXLFqysrKhatSrTpk1j9+7dNGjQAMCopyEgIIBJkyYxdOhQ5s37X940Gg0//vgjFStmriI3fPhwJk6cmKccDBo0yPD/FSpUYO7cuYSEhJCUlGSUm7waM2YMHTp0AGDChAnUqFGDy5cvU61aNdzc3FCpVEb5jIiIYNWqVdy4ccOQwzFjxhAaGsqSJUsM9yTRaDTMmzeP4OBgAC5evMhvv/3Gjh07aNWqleFzPDR//nzq1q1rdE+TxYsX4+fnx8WLF6lSpQoAfn5+zJ49G5VKRdWqVTlz5gyzZ89m8ODBT/0OlClThqioqGfOlRDi2YUEuOPtbM3YQzMoHQvxKdZ8GtAHjU3mqVIF+LjZExJQ+G8/YEqForAoDFq0aMH8+fMNz52cnJ6rvdTU1GxvLe3v7280Z8Pb2xu1Wm00wdPb29toCGLXrl1MnjyZ8PBwEhIS0Gq1pKWlkZycbIjT0dHRUFQA+Pr65nkY4+TJk4wfP55Tp04RFxdnmLNw7do1AgMD89TWo2rVqmUUF2TeUbNatWrZ7n/ixAkURTGc6B9KT0/Hw8PD8NzW1tao7VOnTqFWq2nWrFm27R4/fpxdu3ZlWyRFREQY3q9BgwZGQxwNGzZk5syZ6HQ61Gr1Ez+rg4MDKSmm6zUTQuRMbaViWNTXBF7WorWC/2vcmEs2NYDMogJgXKdA1FZyw70nKRSFxeE+h3N8TW1l/It61+u7cryKwkplvD20W/7dkc7JyYlKlSrlW3ulSpXi/v2svSk2NjZGz1UqVbbbHp7Ur169Svv27Xn33XeZNGkS7u7u7N+/n7feeguNRvPEdh+9mZaVlVWWm2s9enxycjJt2rShTZs2rFixAk9PT65du0bbtm3JyMjI46fP+TM/PGE/aaKlXq9HrVZz/PjxLCfyR4sCBwcHowLAwcHhiXHo9Xo6derEtGnTsrz2sOB5XnFxcXh6euZLW0KIvNkwYxgv7L8LwF+NyrLFqavhNR83e8Z1CqRdUP78rBdlhaKwcLTJ/Z3jHG0cc315Zl7azS+2trbodLqn7lenTh3Cw8Of+/2OHTuGVqtl5syZhrz89ttveW7H09PTaD6ATqcjLCyMFi1aAHD+/HliY2OZOnUqfn5+hvc2tezyWadOHXQ6HTExMTRp0iTXbdWsWRO9Xs+ePXsMQyGPqlu3Lr///jv+/v5YW+f8o/Po/JaHzytXrmwocnL6DqSlpREREUGdOnVyHbMQIn8c3rYCv+U7sQLCajnwwYLtNHhk5c2QAHfpqcilQjF5syjx9/dn79693Lx50+hqgse1bduW/fv3P/f7VaxYEa1Wy3fffceVK1dYvnw5P/74Y57badmyJZs3b2bz5s1cvHiRYcOGGS10Va5cOWxtbQ3v88cffxit32Eq/v7+JCUl8ffffxMbG0tKSgpVqlThjTfeoF+/fqxbt47IyEiOHj3KtGnT2LJlyxPb6t+/P4MGDTJMPt29e7ehEBs2bBhxcXH07t3bcPXG9u3bGTRokFGhcP36dUaPHs2FCxdYtWoV3333HSNGjDB6n+y+A4cOHcLOzo6GDRuaIFNCiJzE3b1F8qSvcUyHq6VVvLLwL9RWKhpW9KBL7TI0rOghRUUeSGFhZhMnTiQqKoqKFSs+scv7zTffJDw8nAsXLjzX+9WuXZtZs2Yxbdo0goKC+OWXX5gyZUqe2xk0aBD9+/dnwIABdOzYkYCAAENvBWT2aCxdupQ1a9YQGBjI1KlTmTFjxnPFnhuNGjXi3XffpWfPnnh6ejJ9+nQAlixZQr9+/fjPf/5D1apV6dy5M4cPHzb0puRk/vz5dO/enffee49q1aoxePBgkpOTAShdujQHDhxAp9PRtm1bgoKCGDFiBG5ubka9ZP369SM1NZWQkBCGDRvG+++/z5AhQwyv5/QdWLVqFW+88QaOjubvSROiuNLrdFxd8haOdRKJLgUBM+fh7CaTM5+HSnl84NzEEhIScHNzIz4+HldXV6PX0tLSiIyMJCAgINuJi0+i1+tJSEjA1dW1yKxUOXbsWOLj4/npp58sHYpBUcxzfmrevDm1a9fO9eqqD929e5dq1apx7NgxAgICgKKX6+f5+TYljUbDli1baN++fZa5RiL/FNQ8/7P4QxpeW0C6YsOVjr9S/cWXLR3SczFlnp90/n5U4f9tVYR99tlnlC9fPldzMkThFhkZybx58wxFhRDC9DbMHEbpC0sBOF17XKEvKgqKQjF5s7hyc3Pj008/tXQYwgxCQkIICQmxdBhCFBsH/1hIuWU7uWtVijPdq9Gx6/uWDqnIkMJCiHz06LLhQoiC6XpEGJopsyiZAZFlrWgxYqmlQypSZChECCFEsZGemsKpYT3xug9xrlB19kKcXNwsHVaRIoWFEEKIYmP90JepFKUn3RoyRg+mYs1Glg6pyJHCQgghRLGwdmI/gg89AOBi12Ba9Bpt2YCKKCkshBBCFHmX//0H571HAPi3nis9Jq22cERFlxQWQgghirT4e3dwXN+fZo2iCWtkT+efdlg6pCJNCgshhBBFVkZ6GtcW9KK0coc4tTdtvgnFwSnnxZ3E85PLTYUQQhRZ699qhrM+lvIV7cjotRw3D29Lh1TkSY+FmahUKjZs2JDj6/7+/nleBtrcvvjiC9555x3Dc0VRGDJkCO7u7qhUKk6dOkXz5s0ZOXLkM7/H0qVLKVGixPMHK0xi06ZN1KlT54m3rReioFjzeU9qHUugwglbdpV8nYAa9S0dUrEghUU+iImJ4Z133qFcuXLY2dnh4+ND27Zt+eeffywdWr65c+cO3377LZ988olhW2hoKEuXLmXTpk1ER0cTFBTEunXrjO5qmteCqWfPnly8eDFPsT1vMWNuu3fvRqVSGd0d9lmOf9Jj6dKl+RrzQx07dkSlUrFy5UqTtC9Eftm5aiZVNvwLwOkGJegy+gcLR1R8yFBIPujWrRsajYZly5ZRoUIF7ty5w99//01cXJylQzOSkZGBra3tMx37888/07BhQ/z9/UlISAAgIiICX19fGjX633Xg7u7Pd1dABwcHHBwcnquNZ/U8+TGnRo0aER0dbXg+YsQIEhISWLJkiWGbm9v/FvzR6XSoVKp8u5HZwIED+e6773jzzTfzpT0h8tul0wewm7UIWy1cDlDTdf7flg6pWCkUPRb6lJScH+npud83LS1X++bFgwcP2L9/P9OmTaNFixaUL1+ekJAQPvnkEzp06JDjcRMnTsTb25tTp05l+3p8fDxDhgzBy8sLV1dXWrZsyenTpw2vR0RE0KVLF7y9vXF2dubFF1/kr7/+MmrD39+fr776igEDBuDm5sbgwYMNQw3btm2jevXqODs7065dO6MTVXZWr15N586dDc8HDhzI+++/z7Vr11CpVPj7+wPGvQfNmzfn6tWrjBo1yvCX9NM8PhQyfvx4ateuzfLly/H398fNzY1evXqRmJgIwIABA9izZw/ffvut4T2ioqIACA8Pp3379jg7O+Pt7U3fvn2JjY01tN28eXOGDx/O6NGjKVWqFK1btwbg7NmzdOjQAVdXV1xcXGjSpAkRERGG45YsWUL16tWxt7enWrVqzJs3z/BaVFQUKpWK1atX06hRI+zt7alRo4Zhqe+oqCjD7eZLliyJSqViwIABT83Lo2xtbfHx8TE8HBwcDD1lPj4+hIaG4uvry6ZNmwgMDMTOzo6rV69m27Pz6quvGr1/RkYGY8eOpUyZMjg5OVG/fv0sy5R37tyZI0eOcOXKlTzFLYQ5JCfGc+k/g3FPhJiSUPv71dg5OFo6rGKlUPRYXKj7Qo6vOTVrSrlHbit+uUlTlNTUbPd1fPFFyi//v//t+3IrdPfvZ9mv+vlzuY7N2dkZZ2dnNmzYQIMGDbCzs3vi/oqiMHLkSDZs2MD+/fupXLlytvt06NABd3d3tmzZgpubGz/99BMvv/wyFy9exN3dnaSkJNq3b89XX32Fvb09y5Yto1OnTly4cIFy5coZ2vrmm2/44osv+PzzzwHYv38/KSkpzJgxg+XLl2NlZcWbb77JmDFj+OWXX7KN+f79+4SFhVGvXj3Dtjlz5lCpUiUWLFjA0aNHUavVWY5bt24dwcHBDBkyhMGDB+cqn9mJiIhgw4YNbNq0ifv379OjRw+mTp3K119/zbfffsvFixcJCgpi4sSJAHh6ehIdHU2zZs0YPHgws2bNIjU1lY8++ogePXqwc+dOQ9vLli1j6NChHDhwAEVRuHnzJk2bNqV58+bs3LkTV1dXDhw4gFarBWDhwoWMGzeO77//njp16nDy5EkGDx6Mk5MT/fv3N7T74YcfMmfOHAIDA5k1axadO3cmMjISPz8/fv/9d7p168aFCxdwdXU19NBMnjyZyZMnPzEXW7dupUmTJk/NWUpKClOmTGHRokV4eHjg5eWVq1wPHDiQqKgoVq9eTenSpVm/fj3t2rXjzJkzhu9q+fLl8fLyYt++fVSoUCFX7QphLpuHvEzNGwqptmD72Rj8KgZZOqRip1AUFgWZtbU1S5cuZfDgwfz444/UrVuXZs2a0atXL2rVqmW0r1arpV+/fhw7dowDBw5QtmzZbNvctWsXZ86cISYmxlCozJgxgw0bNrB27VqGDBlCcHAwwcHBhmO++uor1q9fzx9//MHw4cMN21u2bMmYMWMMz/fv349Go+HHH3+kYsWKAAwfPtxwUs7O1atXURSF0qVLG7a5ubnh4uKCWq3Gx8cn2+Pc3d1Rq9W4uLjkuE9u6PV6li5diouLCwB9+/bl77//5uuvv8bNzQ1bW1scHR2N3mP+/PnUrVvX6ES9ePFi/Pz8uHjxIlWqVAGgUqVKTJ8+3bDPp59+ipubG6tXr8bGxgbAsC/ApEmTmDlzJq+99hoAAQEBhIeH89NPPxkVFsOHD6dbt26GWEJDQ/n5558ZO3asYbjIy8vLqHfm3XffpUePHkafOykpCWdnZ8MwRpkyZXKVM41Gw7x584y+I08TERHBqlWruHHjhuHfesyYMYSGhrJkyRKjXJYpU8bQMyREQXFk/Vy8XGNIt3Hiap8mdO34lqVDKpYKRWFR9cTxnF987C/lSvv25jyW/Nj2Sn//lf1+edStWzc6dOjAvn37+OeffwgNDWX69OksWrTIqJt51KhR2NnZcejQIUqVKpVje8ePHycpKQkPDw+j7ampqYYu+eTkZCZMmMCmTZu4desWWq2W1NRUrl27ZnTMo70MDzk6OhqKCgBfX19iYmJyjCf1vz1A9vb2OSfBhPz9/Q1FBTw9XsjM4a5du3B2ds7yWkREhKFYeDw/p06dokmTJoai4lF3797l+vXrvPXWW0Y9MFqt1mhOA0DDhg0N/29tbU29evU4d+7JPWHu7u5Gc1T0ej0JCQm4urrmeX6Era1tlsL2aU6cOIGiKEaFFEB6enqW76KDgwMpeRw2FMKULhzbSe1TE7D11bJjSHu6vv+jpUMqtgpFYWHlmPvxMStHx1z/Es5Lu09jb29P69atad26NV9++SVvv/0248aNMyosWrduzapVq9i2bRtvvPFGjm3p9Xp8fX2zvQX3w79wP/zwQ7Zt28aMGTOoVKkSDg4OdO/enYyMDKP9nZycsrTx+ElTpVKhKEqO8Twsgu7fv5/lBGMO2cX7tMsd9Xo9nTp1Ytq0aVle8/X1Nfz/4/l50sTRh++5cOFC6tc3vmwtu6Ggxz1tjkl+DoU4ODhkeT8rK6ss/84ajcbw/3q9HrVazfHjx7N8nscLtLi4ODw9PZ8ahxDmcOHUHvS/v42tnZaTjo15+T25AsSSCkVhURgFBgZmWbeic+fOdOrUiT59+qBWq+nVq1e2x9atW5fbt29jbW1tmBT5uH379jFgwAC6du0KQFJSksm6pitWrIirqyvh4eFUqlQpT8fa2tqi0+lMEteT3qNu3br8/vvv+Pv7Y22d+695rVq1WLZsGRqNJktB4+3tTZkyZbhy5coTC0OAQ4cO0bRpUyCzR+P48eOGIaqHV548HnN+DoVk5+Hck4d0Oh1hYWGGyaR16tRBp9MRExPzxOIlLS2NiIgI6tSp88yxCJFfkuLjiBw9FNcEO/a0KMsLo3/BKheFvjCdQnFVSEF27949WrZsyYoVK/j333+JjIxkzZo1TJ8+nS5dumTZv2vXrixfvpyBAweydu3abNts1aoVDRs25NVXX2Xbtm1ERUVx8OBBPv/8c44dOwZkzg1Yt24dp06d4vTp0/Tp08dkixZZWVnRqlUr9u/fn+dj/f392bt3Lzdv3jS6IiM/+fv7c/jwYaKiooiNjUWv1zNs2DDi4uLo3bu34QqG7du3M2jQoCcWOsOHDychIYFevXpx7NgxLl26xPLly7lw4QKQeZXKlClTDJNGz5w5w5IlS5g1a5ZROz/88APr16/n/PnzDBs2jPv37zNo0CAgc/KjSqVi06ZN3L17l6SkJCBzKKRSpUpGjwoVKhg9f55LcVu2bMnmzZvZvHkz58+f57333jNaS6NKlSq88cYb9OvXj3Xr1hEZGcnRo0eZNm0aW7ZsMex36NAh7OzsjIZ7hLCUrYNbUf6Wgp0GHFp/grNrSUuHVOxJYfGcnJ2dqV+/PrNnz6Zp06YEBQXxxRdfMHjwYL7//vtsj+nevTvLli2jb9++rFu3LsvrKpWKLVu20LRpUwYNGkSVKlXo1asXUVFReHtnLkc7e/ZsSpYsSaNGjejUqRNt27albt26JvucQ4YMYfXq1XkuXiZOnEhUVBQVK1Y0Wdf5mDFjUKvVBAYG4unpybVr1yhdujQHDhxAp9PRtm1bgoKCGDFiBG5ubk8cKvPw8GDnzp0kJSXRrFkzXnjhBRYuXGjovXj77bdZtGgRS5cupWbNmjRr1oylS5cSEBBg1M7UqVOZNm0awcHB7Nu3j40bNxqGlMqUKcOECRP4+OOP8fb2Nppsa0qDBg2if//+9OvXj2bNmhEQEGDorXhoyZIl9OvXj//85z9UrVqVzp07c/jwYfz8/Az7rFq1ijfeeAPHfBxKFOJZ/PqfjgT9m4oeuNGvJSFtZG2VgkClPGlw3QQSEhJwc3MjPj4eV1fjG8GkpaURGRlJQEBAnicKPs9EN/F0iqLQoEEDPvjgA8MaD5LnrKKioggICODkyZPUrl37udoqiN/pu3fvUq1aNY4dO5almHqa5/n5NiWNRsOWLVto3759tpN2Rf7I7zxvWzSO0rN+w1oPp5p70/vH3c8fZBFgyu/zk87fjyoYv61EgadSqViwYIFhPQdRPEVGRjJv3rw8FxVC5Kezh7fjNi+zqDhf2YYe3+fPFX4if0hhIXItODiYvn37Plcbr7zyimFRsccfT7siQlheSEgIPXv2tHQYohhLS03m4uRRuKVAdClosmAj6jxM0BamJ/8awqwWLVpkWBfjcc97n5GCwN/f/4mX7gohnp2i13Pmx0G8XPMWu/GizAdfUcpXes8KGikshFk9z+WSQoji7fCvU2kQH4pOrSLgP9Op2TTrlXfC8gpkYSF/8QlR9MjPtXgem+d9hPrIGvTl4WiV0TSQoqLAytMci/HjxxvuIPnw8Tz3gHjcw9X+Hl89UghR+D1cAlyuvBB5dWrfRjwX/kH5Q3asiQmkfu/PLR2SeII891jUqFHD6PbcuVnKONfBWFvj6OjI3bt3sbGxydMldnq9noyMDNLS0grMpXlFkeTZfIpKrhVFISUlhZiYGEqUKJGvvzNE0Rd35zqxn31MmVS44QVtJq5FVYh/HoqDPBcW1tbW+dpL8SiVSoWvry+RkZFcvXo1T8cqikJqamq290gQ+UfybD5FLdclSpQw2e8OUTTptFp2v9OR6jGQ4AjeU6ZT0lPmaRV0eS4sLl26ROnSpbGzs6N+/fpMnjyZChUq5Lh/eno66enphucJCQlA5iIej94A6SGVSoW/vz8ajSZPY7JarZaDBw/SqFGjPN0bQuSN5Nl8ikquVSoV1tbWqNXqArkOysPfQ9n9PhL551nyvHZkO2qfz0Cngti3u9AmpJ38Oz2FKb/PuW0zTytvbt26lZSUFKpUqcKdO3f46quvOH/+PGfPns3xrpfjx49nwoQJWbavXLlSlgQWQgiRrbtHfqXhupNYKXComTfu7UdZOqRiLyUlhT59+jx15c3nWtI7OTmZihUrMnbsWEaPHp3tPtn1WPj5+REbG/vEwPJKo9GwY8cOWrduLZPDTEjybD6Sa/OQPJtHXvJ8K+oc4dNew++ADecD7ei44h9ZBCuXTPl9TkhIoFSpUk8tLJ7rX8rJyYmaNWty6dKlHPexs7PDzs4uy3YbGxuT/BCbql1hTPJsPpJr85A8m8fT8pyc+AD9qr68UvYu2zsE0OqTrdg/x119iytTfJ9z295zTa1NT0/n3Llz+Pr6Pk8zQgghBDqtltPf9yZAH0UsJag9aj0lPOT8UtjkqbAYM2YMe/bsITIyksOHD9O9e3cSEhLo37+/qeITQghRTPz23sto/ojkQrIDse0X4VVGlusujPI0FHLjxg169+5NbGwsnp6eNGjQgEOHDlG+fHlTxSeEEKIY2DBjKLX2xmCFilPUp2dIa0uHJJ5RngqL1atXmyoOIYQQxdThrcvwW74bKyAs2IGe32y0dEjiOcjyZUIIISwm+up5UiZNxTEdrpZW8cqCv55+kCjQpLAQQghhEZqMdI681x2fOLjvDAEz5+Hs5m7psMRzksJCCCGERfz+QRuqROjIUEPKiP5UrdPc0iGJfCCFhRBCCLM7uX0FLUuGcb2MwoXOgbTq+7GlQxL5RJYyE0IIYVZXzx2nyoH/4GSnxblrY9oM/9nSIYl8JD0WQgghzObmlTDOfvcmTqo0ztoG8+KQeZYOSeQzKSyEEEKYlE6vcCQyDq0mg39HvEnATthwxRfft1dhY5v1lg+icJOhECGEECYTGhbNhD/DiUtK5Yuo6QRG6Um3hgcN3sTdq4ylwxMmID0WQgghTCI0LJqhK04QHZ/GG7ELeOF4EgB/NqzIhGt1CA2LtnCEwhSksBBCCJHvdHqFCX+GowCtUzbR+WDmXbAPBTvxs8dQACb8GY5Or1gwSmEKUlgIIYTId0ci44iOT+PF9IMM3rcbOw1cLm/FNxU+BUABouPTOBIZZ9E4Rf6TwkIIIUS+i0lMw5s4+iauxSUVrntDypuj0VnZZtlPFC1SWAghhMh3riSzzHYancveJqIFTKs/FHvXUln283Kxt0B0wpTkqhAhhBD56v7dm7hu6Es1q+vcVkoytcRH3LUqBegM+6gAHzd7QgLk3iBFjRQWQggh8k1qcgL7+rfFOVFLxEsuvMdH3FQ8seV/kzRV//3vuE6BqK1U2TckCi0ZChFCCJEvdFotm/s1pfIVHR73VVyqOIhRb7yKj5vxcIePmz3z36xLuyBfC0UqTEl6LIQQQuSLNW81IfhsOnoVRLzxEq+9PRGA1oE+HLocQ+y5Qyzu/yINKnlJT0URJj0WQgghntvq99sQfPgBAGEdq/DaJwsNr6mtVIa5FCEB7lJUFHFSWAghhHgua77oTfCO6wCcauZFz282WjgiYUlSWAghhHhmJ3asxHf7KQDO1HGixw9/WzYgYXEyx0IIIcQzuXhiN9X2jySuhZ6D18vRdckB1NZyWinupMdCCCFEnkWEHcLzjzdxVKVz360mry4+KLdAF4AUFkIIIfIo/OgObgweyNkYFZesKxMwbB22drKCpsgkhYUQQohcu3bpNLdHfYDXfdCccsZ1wK84u5a0dFiiAJHCQgghRK7cv3uT8Hd74xsLD5ygxJTJeJetaOmwRAEjhYUQQoinSk1OYG//tpS/qZBiB5pPhlGnWVdLhyUKICkshBBCPJFOq2Vz/2ZUuaIjwxpihr1G0+7DLR2WKKCksBBCCPFEaz/uQI2wtP8u1d2YV4Z8bemQRAEmFxwLIYTI0aGVX9Hd6RBba3mSXL4GPT9ZZOmQRAEnPRZCCCGydWzTAhpc/Aa1Ckq17yFLdYtckR4LIYQQWfz57UjYuYWMWnDSpzv1+31l6ZBEISGFhRBCCCN/LZ9K2UXbsNeoWV+qCq+PX4DKSjq4Re7IN0UIIYTBoa3LcJ21DHsNXClnRYdZ27BSqy0dlihEpLAQQggBwLmjf6EdNxWXVLjhreKFRRtwdnO3dFiikJHCQgghBNcjwoge9T4eCRBTEgK+X4hPucqWDksUQlJYCCFEMZeSlMC/Q3sYlup2nfo1lWo2tnRYopCSwkIIIYoxrSaDC/N64lv9AXGuD5fqfs3SYYlCTK4KEUKIYkrR6zkxbwAhKQdJL2HDhRmTqdVU7v8hno/0WAghRDH169DmlLzxNzpFRXjjOVJUiHwhhYUQQhRDqz9oS/Ceu8TvcGdvhQ+o0+ZNS4ckiggpLIQQophZM643wduvAXApxJMW/SdaOCJRlEhhIYQQxcgfc0ZSbc0pAM7UdqLHvJ2WDUgUOVJYCCFEMfHX8qn4/bwNaz2cr2pD1/87gNpa5vCL/CWFhRBCFANHtq8wWqr75aV/YWNrZ+mwRBEkpaoQQhRxd29FUXLPeK54qInXqHlh0QZcS3pZOixRRElhIYQQRVj8/ViSFnWhss1dVI1Lo+qxRJbqFiYlQyFCCFFE3b97k52ftSZAH0UsJXAcsJ6KNepbOixRxElhIYQQRVBqcgJ7B7Sl2s40/rzkRXy3XykdUM3SYYliQAoLIYQoYnRaLZv7N6NKhI4MNVi91J2KNRtYOixRTEhhIYQQRcyawU2pEZaGXgWX+zSkw9Aplg5JFCMyeVMIIQoRnV7hSGQcMYlpeLnYExLgjtpKZXh91Yh21P7nPgBhHSrT87PFlgpVFFNSWAghRCERGhbNhD/DiY5PM2zzdbNnXKdA2gX5smZcH2ptuwrA6Sae9Jrxh6VCFcWYFBZCCFEIhIZFM3TFCZTHtt+OT2PoihN888I9XK8fAuw4U9uR1+fLUt3CMqSwEEKIAk6nV5jwZ3iWogJAAWqqrvBK2CScyqWz0bM2XcZvlaW6hcXIN08IIQq4I5FxRsMfj6qbcYSZDotxUqVzyjqYdhO2YmfvaOYIhfgfKSyEEKKAi0nMvqiorAln9P7fiLJ14kbjEsS8+hO1pagQFiaXmwohRAHn5WKfZZu39iYfH16MRwKotSq+0L5NaS+5/4ewPCkshBCigAsJcMfXzZ6HF5W66B8w7uQcSsdCvBPMaNSb9FJBhAS4WzROIUAKCyGEKPDUVirGdQoEwEafysSwKQTcVEixhXkvteOM7QuM6xRotJ6FEJYicyyEEKIQaBfky5yu/ugm9qTalcylupc1acAln47M/+86FkIUBFJYCCFEIZCWkoT71qGoYnVo1HDstfr0HjCD2Y+tvCmEpUlhIYQQBVxy4gOivutME05zqbkT58v04q0PZlk6LCGyJYWFEEIUYDeuhHF6bl86lLhCsmKP5tWf6dTwFUuHJUSOZPKmEEIUUFfCj3BuwOuU35bGrlgPbnb5lUApKkQBJ4WFEEIUQBdO7ubq4P6UjYFkB1A1HUGVus0tHZYQTyVDIUIIUcD8e3Az9/8zBp/78MAZGD+G5h3fsnRYQuSKFBZCCFGAHP97DamffIlXAtxzBfsp46j3ci9LhyVErklhIYQQBcTJvRvJ+OhLPJIgpiSUnDGdWo07WTosIfJE5lgIIUQBcCXsMH5/D+deOS23PcDn+/lSVIhC6bkKiylTpqBSqRg5cmQ+hSOEEMXPxRN78FjblVKqBCrXdqLCkpVUfaG5pcMS4pk881DI0aNHWbBgAbVq1crPeIQQoljZuuAztJvX4h+czAXbavgM24xbyVKWDkuIZ/ZMPRZJSUm88cYbLFy4kJIlS+Z3TEIIUSz8OXc0vnPXUemCFRuv+lPmg1ApKkSh90w9FsOGDaNDhw60atWKr7766on7pqenk56ebniekJAAgEajQaPRPMvbZ+thW/nZpshK8mw+kmvzsFSe/5z5HhWX78dWB5f9rWg+9Q/sHJyL7L+3fJ/Nw5R5zm2bKkVRlLw0vHr1ar7++muOHj2Kvb09zZs3p3bt2syZMyfb/cePH8+ECROybF+5ciWOjo55eWshhCgS7u5ZRP3Qy1jr4UJFNdq+H2Pj4GLpsIR4opSUFPr06UN8fDyurq457penwuL69evUq1eP7du3ExwcDPDUwiK7Hgs/Pz9iY2OfGFheaTQaduzYQevWrbGxscm3doUxybP5SK7Nw9x5XvdlL4I2hGOlwLlqNry8ZCeOzm4mf19Lk++zeZgyzwkJCZQqVeqphUWehkKOHz9OTEwML7zwgmGbTqdj7969fP/996Snp6NWq42OsbOzw87OLktbNjY2JvlymapdYUzybD6Sa/MwR553Lv6Mypszi4qzQfZ0Wn4AO4fi1XMr32fzMEWec9tengqLl19+mTNnzhhtGzhwINWqVeOjjz7KUlQIIYTIdOiXCbS89j1Hmzvx770yvLZkPza2Wf/oEqKwy1Nh4eLiQlBQkNE2JycnPDw8smwXQggBOq2WnT++T+vYlQBoa3Sjx+BvUVnJ+oSiaJIlvYUQwkR0Wi1r3mpCwJkHXG5lx93qA2nQf4oUFaJIe+7CYvfu3fkQhhBCFC2ajHTW929M8MlkAI5l1KXXwGkWjkoI05OyWQgh8ll6agob+zSg5n+LitPtK9BrTqiFoxLCPKSwEEKIfJSSFM/m3g2pEZaGXgVnXqtBr1mbLR2WEGYjhYUQQuSThPsxbO/1EtXPZ6C1gnO969Fj8lpLhyWEWUlhIYQQ+SAtJYnzP/TAMVmDRg2X+zel+5fLLR2WEGYnV4UIIcRzSk58QNR3nQnRnuZOQ3uO+Q2g61CZqCmKJykshBDiOVyPCOPY9Dfo6htFkuLAvc5L6dCgnaXDEsJiZChECCGe0ZXwI1wY+DrV9mSw+Zont7qsJlCKClHMSWEhhBDP4MLJ3Vwd3J8yMZDgCM6vjKRK3eaWDksIi5OhECGEyKN/D27m/n/G4HMfHjiDMv5DmnUcZOmwhCgQpMdCCFFs6fQKRyLjADgSGYdOrzz1mON//0bCyDF43Yd7rmAzbRyNpKgQwkB6LIQQxVJoWDQT/gwnLimV6SEwaNlR3J0dGNcpkHZBvtkec+7oTtI/GodHEsSUhJIzZ1CrUQczRy5EwSY9FkKIYic0LJqhK04QHZ9mtP12fBpDV5wgNCw6yzERZw7hvbk/iUFp3PYAn3nzpagQIhtSWAghihWdXmHCn+FkN+jxcNuEP8ONhkXOH9tJqd9fw50EKlYsSbWVG6hap7kZohWi8JHCQghRrByJjMvSU/EoBYiOTzPMvdi64DOiPn4PRZvKeevqeA7fjm/5qmaKVojCR+ZYCCGKlZjEnIuKx/f789uR+C3chp1WxV+Xy/LKwlCcXEqYNkAhCjkpLIQQxYqXi32u9nuw4QvqrPkHGx1c9rfi5e+kqBAiN2QoRAhRrIQEuOPrZo8qh9dVwJD4hdT9LbOouFDJmmardlHSs4w5wxSi0JLCQghRrKitVIzrFAiQpbhQAUNiv6PL7gtY6+FcNVva/noA15JeZo9TiMJKCgshRLHTLsiX+W/WxcfNeFhkoM0Wmp2+ipUCZ4Ps6bDqHxycXC0UpRCFk8yxEEIUS+2CfGkd6MOhyzHEnjvE7HIHaR+9gsst7DgeE8BrC/ZiY2tn6TCFKHSkx0IIUWyprVS84OdKyvFltI/+AYC7lXrSY/E/UlQI8Yykx0IIUWzptFo2vNOCwOPxHH3ZGW2dN2nQfwoqK/mbS4hnJT89Qohi6f7dm2zs/gLBR+Kx1cFlpQYNB06TokKI5yQ/QUKIYufsoVCOdW9N9fMZ6FTwT8vSdJ+12dJhCVEkyFCIEKJY2b5kIq7fr6JsMiQ6wJ23OuLh95KlwxKiyJAeCyFEsbH5+5H4zFiFWzJElwKbWV/R7p3Jlg5LiCJFeiyEEEWeVpPBsYXDeOXub2yp6I1Wb0ejnzbgVaYiGo3G0uEJUaRIYSGEKNKuXTjB7d9G0ED3L6jAtcMrNBw4XS4nFcJEpLAQQhRZ+zcuQD9lNknuehLr2XGp0XSathtg6bCEKNKksBBCFEnrpw6h/Mp9OGSAld6KC82+p97LPSwdlhBFnhQWQogiRafV8tt7Lam19y5WQGRZK6rNXUKFwBBLhyZEsSCFhRCiyIiNjmTfO12ofTFzQuaZ2o50WLgTJxc3C0cmRPEhl5sKIYqE29cucWhgB6pd1KC1gtMdK9Fj9XEpKoQwMykshBCFXvihUGwXt6R84H3iXODGyNfoNeNPS4clRLEkQyFCiEItdP4YXr69GBuVjvslK+D6y1LKVwm2dFhCFFtSWAghCqXkxHi2vN2SymdTuNzOhhSfJlR/9/9wdJahDyEsSYZChBCFTsSZg+zp2pCg0ynYaOF0Ri3qjl4vRYUQBYAUFkKIQmXP2u+IfustAm4opNrCxYFN6TV3h9zuXIgCQoZChBCFxu+T+lPxtyPYaeBuCVB/OoqunYdYOiwhxCOksBBCFHh6nY5147tRY80FAK6Us6Lmd8spV7WuhSMTQjxOCgshRIGWnPiAiz++wauq/ewo70NiKVc6LfgbBydXS4cmhMiGFBZCiALr5J4NOO78nDqq62SorHF7ewivvP4fS4clhHgCKSyEEAXSph8+xHvhJi5V0FKyZgniOv5MoxdbWTosIcRTSGEhhChQdFota8Z0JGjbVdQKPLhnQ3KvlVSr/qKlQxNC5IIUFkKIAiMpPo7Qt14mOCwNgHPVbGmxcAslPctYODIhRG7Jhd9CiALhwqm97O/6EjXC0tADp1v60mXtcSkqhChkpLAQQljchRO7uTn8HcrfUkixg8uDW9Jr3k7U1tKpKkRhI4WFEMKijm1eSLmNr+NcN5HbHqCZ+jFd/vODpcMSQjwj+XNACGERmox0ts8eRIfUTaAC+9LBVPhsGaW8ZehDiMJMCgshhNlFXz3P0Xe7UzpaR1RbO24HdOfFt+fK0IcQRYAMhQghzOrI9hWc792VypE6bLRw3PkVGrw7T4oKIYoI+UkWQpjNH3NGUHrJdnzS4b4zpIwcQLc3P7J0WEKIfCSFhRDC5HRaLWtGvkLNv29gpcA1XxXlZ3xPoxdaWjo0IUQ+k8JCCGFSaanJbBzThuC/4wA4W8OONou241rSy8KRCSFMQeZYCCFMJvbWVa7ObMGr7mFcL61wurUfXX89JkWFEEWY9FgIIUxiz5q5VD4zm6pWccSrnfD+4mvatOhm6bCEECYmhYUQIt/99tnrVN0QxslaVmRU98O6z2qCKwVZOiwhhBlIYSGEyDfpqSlsGNyCWscSANDFO+A27C9KevhYODIhhLnIHAshRL6IOneM7V1fNBQVpxuW5JUNx6WoEKKYkR4LIcRzWzdlMKXX7KdSCqRbw+XuL9Br/ApLhyWEsAApLIQQzyw+7i4nF7yF/6pLOGTAbQ+I7PMG1dsNRadXUFupLB2iEMLMpLAQQjyT07vW4LtnLM2J4+/6JTmd5MlMv5EkXS8BCw/h62bPuE6BtAvytXSoQggzksJCCJEnt6LOcXBMH8qVjsHLPYmrlOaHUkM44VHFaL/b8WkMXXGC+W/WleJCiGJEJm8KIXLtz7mjiXz9NWqEpaH5x5X9Ht3oZzODE0qVLPsq//3vhD/D0emVLK8LIYomKSyEEE8VczOCNT3rUmneVtwTIdYNEga/hrr9N1xNzPk4BYiOT+NIZJzZYhVCWJYMhQghnmjzvI9xW7qRoMyrSDlT25GWs9dSyjeAjadu5qqNmMQ0E0YohChIpLAQQmQrNTmRHTPfpPLKiwDcc4UH/TrQY/gMwz5eLva5aiu3+wkhCj8pLIQQWZw/sgOnrR/QWbnF5srepNo703jGSnzLVzPaLyTAHV83e27Hp5HdLAoV4ONmT0iAu1niFkJYnsyxEEIYPLgXzeoBDSm5vhd+yi1icKf0iAl0X3MiS1EBoLZSMa5TIJBZRDzq4fNxnQJlPQshihEpLIQQAPz1f5M51aklwYcecOzfUhwt8Qp2I45Sp1XvJx7XLsiX+W/WxcfNeLjDx81eLjUVohjK01DI/PnzmT9/PlFRUQDUqFGDL7/8kldeecUUsQkhzCDhfgxbRnWh5uEHWCkQ7wQZTZvz4sifct1GuyBfWgf6cCQyjpjENLxcMoc/pKdCiOInT4VF2bJlmTp1KpUqVQJg2bJldOnShZMnT1KjRg2TBCiEMJ1dq2ehfL+Q4NjM5+eq2VJ3ys80qF4vz22prVQ0rOiRzxEKIQqbPBUWnTp1Mnr+9ddfM3/+fA4dOiSFhRCFiCYjnQ3jX6fahktY6yHBEW50DaHbF8ssHZoQopB75qtCdDoda9asITk5mYYNG+a4X3p6Ounp6YbnCQmZF8NrNBo0Gs2zvn0WD9vKzzZFVpJn8zFVrqPCj6L6YxhtraI44+zDXS9rqn41j841GhTLf1f5TpuH5Nk8TJnn3LapUhQlT2vtnjlzhoYNG5KWloazszMrV66kffv2Oe4/fvx4JkyYkGX7ypUrcXR0zMtbCyGegyYjlaQ9C+jlfgp7Kx0PFGf+dOqKc8WWWFmrLR2eEKKAS0lJoU+fPsTHx+Pq6prjfnkuLDIyMrh27RoPHjzg999/Z9GiRezZs4fAwMBs98+ux8LPz4/Y2NgnBpZXGo2GHTt20Lp1a2xsbPKtXWFM8mw++ZnrQ5uXkjp7Nn53FG6+lIJH5er49pmHh49fPkVbeMl32jwkz+ZhyjwnJCRQqlSppxYWeR4KsbW1NUzerFevHkePHuXbb7/lp5+yn0FuZ2eHnZ1dlu02NjYm+XKZql1hTPJsPs+T6/TUFNb/pxPV99yilA5S7CDOtwEvf7gKlZVcbf4o+U6bh+TZPEyR59y299wrbyqKYtQjIYQoGI5sX0Hc1MkE38rslLzsb0WFCbPoUb+thSMTQhRleSosPv30U1555RX8/PxITExk9erV7N69m9DQUFPFJ4TII71Ox5rPXqPqpouU10KqLVxqV5Xuk9eitpZV/IUQppWn3zJ37tyhb9++REdH4+bmRq1atQgNDaV169amik8IkQc3r5zjwerBVM+IQK0rwZVyVpT5YjI9m3SxdGhCiGIiT4XFzz//bKo4hBDPQavJYOsPI3n5/lrKqNJJcbFj+6DGdBw5D2sbW0uHJ4QoRqRfVIgCSKdXOBIZB8CRyDgaVPLKcXnsfw/8yfUJH1Pupp5bHSC9RE1K9FrIqxWqmzNkIYQApLAQosAJDYtmwp/hxCWlMj0EBi07iruzA+M6BRrd0Eun1bL20+5UDr1AhQxIt4bj1o15/aO1WKllXQohhGXI9WZCFCChYdEMXXGC6Pg0o+2349MYuuIEoWHRAJw9vJ0tHYOp9ccFHDLgmq+K9Bmf0HPyeikqhBAWJT0WQhQQOr3ChD/DyW7FOgVQAeM3hpGwegQV/jxDpXTIUMO5ZqXpOvNP7BxkJVshhOVJYSFEAXEkMi5LT8WjSvGAiWk/o7sWhVO6Hde9VTj/ZxS9Og82Y5RCCPFkUlgIUUDEJGZfVCh6HV30OxnvsJaSqiSSy6v50/1Funy9Bgen/FsWXwgh8oMUFkIUEF4u9lm2Jd29wpQzP+NxX4tjyyTOqspzrdlMerWStWOEEAWTFBZCFBAhAe74utlzOz4NBeh172cqbT2PawroVCrm3a/PWt/R7G3ZytKhCiFEjqSwEKKAUFupGNcpkFULp9Ht/A4qX9UDEO0By19ozW7HtszvEpzjehZCCFEQSGEhRAERee44KZ8P4uPzGQBoreB0HWe+8RuNo5sX8x9bx0IIIQoiKSyEsLDY29eIWDuOunc3cj7NE7DiXFUbdL2G4+LozbzqDZ648qYQQhQkUlgIYSE3I8PZN/FtGvhEUt8uBVRg84Intwe8xmu9RqPRaNiyZQshAe5SVAghCg0pLIQws/t3b7JtXD8qHrxFcBqEBTmTXrsc2hZf0qpxB0uHJ4QQz0UKCyHMJDU5gT/G98Vv10WCkzK33fYATb3mVBn7EyorWWFfCFH4SWEhhIkpej3rpryFx+ZD1Mq8YSlxrnDj5UBe/XK5LMUthChSpLAQwoTC9m3Ebs8k7M48wDvOhkQHuPKSH+3GL6Oxh1zhIYQoeqSwEMIE/v5lOvqwdbS2OweAQ0UH9jtXotmXiwgpX83C0QkhhOlIYSFEPjq0dRnR82ZS7ZKGq3560hqqOeX9GpW7j6eXd1lLhyeEECYnhYUQ+eDs4e2cm/ER1cLSqKaAHkh2tie693YaVK9n6fCEEMJspLAQ4jlcu3CCf74aSrWTCdTQZm67VEFNybeH0u21YZYNTgghLEAKCyGeQUpSPKd/n0b8/l+pddQWgKtlVNC7O53fnmjh6IQQwnKksBAiD5IT49n7f1/y4t0NNOQBGb4QWrEM+uZN6TTqe9TW8iMlhCje5LegELmgyUhnw9cD8d52EhcFXNs84Ja1N7fqjabDl29LQSGEEP8lvw2FeAKdVsvmuaNw3PAXQTGZ2xIcYbt7H9q8N5vSdvaWDVAIIQoYKSyEyMH2pZPQrlhF5RsKAKm2cLG+Jy3HL6F+mYoWjk4IIQomKSyEeMzVc8c5v2I05X7P7KLIUMP52i68+Nn39AoMsXB0QghRsElhIcR/RZ47Tuy2b6h7P5TyNgpb/H1IcXKgyuhJ9GzcydLhCSFEoSCFhSj2roQf4ejXwwgIT8K/XQxqO4WTTi9RdcY4KgZJD4UQQuSFFBai2Lpz4zK7xg+kypFYamVkbtt7pyw135tNnXotLRucEEIUUlJYiGIn4X4Mm8f3pcK+awSnZG676QUpr7Wh6/CZcumoEEI8B/kNKooNnVbLkY3fkz7zJ2rHZW67WwJi2tWly6eLsbG1s2h8QghRFEhhIYo8nVbLmd1rcDs4hYb6q2wq64ldug3Xmlek4/hfcHJxs3SIQghRZEhhIYqsiDMHOfLTl/icvIlX/QcEuKaQgCOOL3ei6nejaejtZ+kQhRCiyJHCQhQpmox0ts4bi+7vXVSK0FBbn7k9MtyVhK6vEdj9S1p6eFs2SCGEKMKksBBFwtWLp/ln+nDK/htL5YT/bb/hrSKuXgVeGj6dMgGBlgtQCCGKCSksRKGVlppC+J7fsD61gsCUY5Q+64NHgopkO7hS3Qmvrm/SuudIS4cphBDFihQWotA5uHkJ11f9iE9EAi+2isZRrYAV3HvBkTslKtNqxCzqeZYBQKdXOBIZR0xiGl4u9oQEuKO2Uln4EwghRNElhYUoFGKjI/l77hjcjpyj/E2Fkv/dvu+uJ651X8Gv5RC6VgoyOiY0LJoJf4YTHZ9m2ObrZs+4ToG0C/I1Y/RCCFF8SGEhCixFr+fQlqXc+L/vqXA+1bA6pk4FVwLU6Jo2ot37M3Bwcs1ybGhYNENXnEB5bPvt+DSGrjjB/DfrSnEhhBAmIIWFKHDu343mwl9L8Lz8Gz6JtyjxrweQuZjVrWBvar39GZ1fbJ3j8Tq9woQ/w7MUFQAKoAIm/BlO60AfGRYRQoh8JoWFKBA0GelsW/g5GTu2Y6tPo2Pd2wCkO9iwoYETTi+2oO3gr3K1OuaRyDij4Y/HKUB0fBpHIuNoWNEjvz6CEEIIpLAQFnbu6F+cXvQVpU/foeKDzG1aKytO1SpHWo3eVG/zFj3zuO5ETGLORcWz7CeEECL3pLAQZqfVZLB13liUbX9RIVJH8H/HLFJtIaKaA+5detG85+hnvhmYl4t9vu4nhBAi96SwEGZz7eK/3Ny9iEq3/sDuqgq/K5kn9qtlVMS/WI2XR8ykrm/Ac79PSIA7vm723I5Py3aehQrwccu89FQIIUT+ksJCmNT9u7f469tROB0Kw75CEi1LPwCgUmk3jtZzwa/Xu7TrOChf31NtpWJcp0CGrjiBCoyKi4dTNcd1CpSJm0IIYQJSWAiT2L3mO2J+X0aFc8kEpWduu6534N8KldHWfpOgFr2oYGe6oYh2Qb7Mf7NulnUsfGQdCyGEMCkpLES+SYyPY8uk/rgfjaDsHYWHUy7jXOF6rVIEDhxLrcadzBZPuyBfWgf6yMqbQghhRlJYiOei6PVcOPoXif8spsb9nbj9W5Kyd6zQWsHlijZYtWxOu3en0tjB0SLxqa1UckmpEEKYkRQW4plEhB3k1KKJeJ2+RfUmd6lmpwEVqKpbc6qKLyHvTKRrzUaWDlMIIYSZSWEhcu3s4W2E/fEztqfD0VzREazP3H78Rgm8XngR10aDaFPvZVRWVpYNVAghhMVIYSFydPv6Za6f2E7s6S04bYnA8wHUeuT16z4q7terRNP3p+NbvpqlwhRCCFGASGEhDE7t28jFLf+H1cUIrJ3TeTXgFj5Aik7F5URf9Cq45aUipowjZbr2pc3rIywdshBCiAJGCotiSqfV8u++jVzc9gvWF6/gfSMdjwSo+d/Xoz0VtP5WXLGpxD3fEO4NcyK4XX8eaEvhfO4QNtUboNMrcoWFEEIII1JYFBM6rZbzx/4i+eZ5VNcOUObBCVK32FIr8X/7aK3glreK++XccKj9ImlvTaCKa0kg8zbkPX4NJy4pkukhMGjZUdydHWRNCCGEEEaksCiidFotx/5eTdSO37C5fBWfGxnYaOGFLrdQqwAVHC/rTUK8NQ/KueFYO4QGr39AzTIVs7QVGhbN0BUnUAA79f+2345PY+iKE8x/s64UF0IIIQApLIoMRa/n2qV/ObJqKtYnw/G5oaFEsvFkS60VHNBUwabCSzhXaU6L0U1x/m+PRE50eoUJf4Zne88Nhcwlsif8GU7rQB8ZFhFCCFE0CgudXuFIZBwARyLjaFDJq8Cf5B7G/KwrQuq0Wo5sXcrVXesp66lQLf0s5XlA2MVSVLhgC4BGDTd9VMSXL4lz3YY0en0ENb398hTnkcg4oyWxH6cA0fFpHImMk4WohBBCFP7CIjQsmgl/hhOXlFpoxv4fxvzoCdv3Kfew0Go0HNqymBu7NmAXcYPSN7WUSIESwINWiZQqlUi6YoN1WXdOO9jh8kJjGr3+AbU8yzxXrDGJORcVz7KfEEKIoq1QFxaFcez/0Zgf9XjMep2OqHPHiDnzN7ePbab0zjg8UuHRPoF0a7jpa0WqewjhbQdToXZT2jo45Wu8Xi65u1FYbvcTQghRtBXawqIwjv0/KWaVkkGTtD1cnvkNG+7ew750Cu1Kx1IBiFDbkZHqQbpNZiGR4O9ByReb0aj7+9Qu6WXSmEMC3PF1s+d2fFr2cZN5x9CQAHeTxiGEEKJwKLSFRWEc+8+MORVXUvDmLsEJBymXfB2/mDj8onU4PfJxrqAmxdeOyw5BJPmHkFjNgcbd3qe2m3lP4GorFeM6BTJ0xQkeL88ePh/XKbDAFG9CCCEsq9AWFgVx7D8jPY1LJ3dz68IJEm9cIj0mGqv7D7BJTMU+SUt6CT3hgfdwVKWjVyBsT2lsdP87PtUWbvhacd+vFKWbv4JNj1HUsrUzW/w5aRfky/w36xrmsjzk85R5IUIIIYqfQltYmHvs//a1S1w5s5+4K2GkRF9Ffy8Wq/hEcIDaFa0ooY2lpP4+2t99KavLvo2bGgVHVToACThxrSxordTc8HTjdIla7LZ/GY2VA6sGNygwvSwPtQvypXWgD4cuxxB77hCL+79YKK6+EUIIYV6FtrDIr7H/jLRUroQf4ua5o8Rfu0hGzC1UVloqlC2JfdodnDPuErNRj2sKlCTz8aibPgpVykdnPrGCB64K1loVSc4qUp3UZLjYoS/hitrDE2f/qnSNrsS5ZGfSsIM6WWP2LcDzFdRWKkIC3NlyjjxfHiuEEKJ4KLSFRW7G/j9q6s6Jv1YTE3EabXI83l6lIDEa25Q73N4egdsDBddksNZDaTIfADe9FV50OWxo76ZN5iuptpDgDMnOVqQ526BxdUDl483JRp/j7FmOkr7+NPrUBxtb2xzjVv33qhAVGBVEMl9BCCFEUVBoCwv439j/+I1h9E5dSdqKk3wTr8E5WY9rMjhtyNzPmcxioUGLaMOx+xJ9cU/MPIHrgQQnSHJWkeJsRVopJ/4JeA1rtzI4eJTFvmk6JSvVoXq5yvkW8+PrWMh8BSGEEEVBoS4s4H9j//cnvsW5646UemD81366DcQ7Q5KrNcdcW6Fx8kHlWpr7JW6TWsKb0tVeoGLNxjg4uZo95udZeVMIIYQoiAp9YQGZwyIXy/Umou5hbjmXwN63HB4ValA+qDFVyldDbV3wPqbaSlXgJmgKIYQQz6vgnXGf0Yt9v+Kuxxbat2+PjY2NpcMRQgghiiWrvOw8ZcoUXnzxRVxcXPDy8uLVV1/lwoULpopNCCGEEIVMngqLPXv2MGzYMA4dOsSOHTvQarW0adOG5ORkU8UnhBBCiEIkT0MhoaGhRs+XLFmCl5cXx48fp2nTpvkamBBCCCEKn+eaYxEfHw+Au3vOCzqlp6eTnp5ueJ6QkACARqNBo9E8z9sbedhWfrYpspI8m4/k2jwkz+YheTYPU+Y5t22qFEXJbuHKp1IUhS5dunD//n327duX437jx49nwoQJWbavXLkSR0fHZ3lrIYQQQphZSkoKffr0IT4+HlfXnJdoeObCYtiwYWzevJn9+/dTtmzZHPfLrsfCz8+P2NjYJwaWVxqNhh07dtC6dWu5KsSEJM/mI7k2D8mzeUiezcOUeU5ISKBUqVJPLSyeaSjk/fff548//mDv3r1PLCoA7OzssLPLeodOGxsbk3y5TNWuMCZ5Nh/JtXlIns1D8mwepshzbtvLU2GhKArvv/8+69evZ/fu3QQEBDxTcEIIIYQomvJUWAwbNoyVK1eyceNGXFxcuH37NgBubm44ODiYJEAhhBBCFB55Wsdi/vz5xMfH07x5c3x9fQ2PX3/91VTxCSGEEKIQyfNQiBBCCCFETvLUYyGEEEII8SRSWAghhBAi35j97qYPh1MersCZXzQaDSkpKSQkJMilTCYkeTYfybV5SJ7NQ/JsHqbM88Pz9tOmRZi9sEhMTATAz8/P3G8thBBCiOeUmJiIm5tbjq8/88qbz0qv13Pr1i1cXFxQqVT51u7DFT2vX7+eryt6CmOSZ/ORXJuH5Nk8JM/mYco8K4pCYmIipUuXxsoq55kUZu+xsLKyeupqnc/D1dVVvrRmIHk2H8m1eUiezUPybB6myvOTeioeksmbQgghhMg3UlgIIYQQIt8UmcLCzs6OcePGZXvDM5F/JM/mI7k2D8mzeUiezaMg5NnskzeFEEIIUXQVmR4LIYQQQlieFBZCCCGEyDdSWAghhBAi30hhIYQQQoh8U6gKi3nz5hEQEIC9vT0vvPAC+/bte+L+e/bs4YUXXsDe3p4KFSrw448/minSwi0veV63bh2tW7fG09MTV1dXGjZsyLZt28wYbeGV1+/zQwcOHMDa2pratWubNsAiJK+5Tk9P57PPPqN8+fLY2dlRsWJFFi9ebKZoC6+85vmXX34hODgYR0dHfH19GThwIPfu3TNTtIXT3r176dSpE6VLl0alUrFhw4anHmP2c6FSSKxevVqxsbFRFi5cqISHhysjRoxQnJyclKtXr2a7/5UrVxRHR0dlxIgRSnh4uLJw4ULFxsZGWbt2rZkjL1zymucRI0Yo06ZNU44cOaJcvHhR+eSTTxQbGxvlxIkTZo68cMlrnh968OCBUqFCBaVNmzZKcHCweYIt5J4l1507d1bq16+v7NixQ4mMjFQOHz6sHDhwwIxRFz55zfO+ffsUKysr5dtvv1WuXLmi7Nu3T6lRo4by6quvmjnywmXLli3KZ599pvz+++8KoKxfv/6J+1viXFhoCouQkBDl3XffNdpWrVo15eOPP852/7FjxyrVqlUz2vbOO+8oDRo0MFmMRUFe85ydwMBAZcKECfkdWpHyrHnu2bOn8vnnnyvjxo2TwiKX8prrrVu3Km5ubsq9e/fMEV6Rkdc8f/PNN0qFChWMts2dO1cpW7asyWIsanJTWFjiXFgohkIyMjI4fvw4bdq0Mdrepk0bDh48mO0x//zzT5b927Zty7Fjx9BoNCaLtTB7ljw/Tq/Xk5iYiLu7uylCLBKeNc9LliwhIiKCcePGmTrEIuNZcv3HH39Qr149pk+fTpkyZahSpQpjxowhNTXVHCEXSs+S50aNGnHjxg22bNmCoijcuXOHtWvX0qFDB3OEXGxY4lxo9puQPYvY2Fh0Oh3e3t5G2729vbl9+3a2x9y+fTvb/bVaLbGxsfj6+pos3sLqWfL8uJkzZ5KcnEyPHj1MEWKR8Cx5vnTpEh9//DH79u3D2rpQ/NgWCM+S6ytXrrB//37s7e1Zv349sbGxvPfee8TFxck8ixw8S54bNWrEL7/8Qs+ePUlLS0Or1dK5c2e+++47c4RcbFjiXFgoeiweevw264qiPPHW69ntn912YSyveX5o1apVjB8/nl9//RUvLy9ThVdk5DbPOp2OPn36MGHCBKpUqWKu8IqUvHyn9Xo9KpWKX375hZCQENq3b8+sWbNYunSp9Fo8RV7yHB4ezgcffMCXX37J8ePHCQ0NJTIyknfffdccoRYr5j4XFoo/fUqVKoVarc5S+cbExGSpxB7y8fHJdn9ra2s8PDxMFmth9ix5fujXX3/lrbfeYs2aNbRq1cqUYRZ6ec1zYmIix44d4+TJkwwfPhzIPPkpioK1tTXbt2+nZcuWZom9sHmW77Svry9lypQxuj109erVURSFGzduULlyZZPGXBg9S56nTJlC48aN+fDDDwGoVasWTk5ONGnShK+++kp6lfOJJc6FhaLHwtbWlhdeeIEdO3YYbd+xYweNGjXK9piGDRtm2X/79u3Uq1cPGxsbk8VamD1LniGzp2LAgAGsXLlSxkdzIa95dnV15cyZM5w6dcrwePfdd6latSqnTp2ifv365gq90HmW73Tjxo25desWSUlJhm0XL17EysqKsmXLmjTewupZ8pySkoKVlfEpSK1WA//7i1o8P4ucC002LTSfPbyU6eeff1bCw8OVkSNHKk5OTkpUVJSiKIry8ccfK3379jXs//ASm1GjRinh4eHKzz//LJeb5kJe87xy5UrF2tpa+eGHH5To6GjD48GDB5b6CIVCXvP8OLkqJPfymuvExESlbNmySvfu3ZWzZ88qe/bsUSpXrqy8/fbblvoIhUJe87xkyRLF2tpamTdvnhIREaHs379fqVevnhISEmKpj1AoJCYmKidPnlROnjypAMqsWbOUkydPGi7rLQjnwkJTWCiKovzwww9K+fLlFVtbW6Vu3brKnj17DK/1799fadasmdH+u3fvVurUqaPY2toq/v7+yvz5880cceGUlzw3a9ZMAbI8+vfvb/7AC5m8fp8fJYVF3uQ11+fOnVNatWqlODg4KGXLllVGjx6tpKSkmDnqwieveZ47d64SGBioODg4KL6+vsobb7yh3Lhxw8xRFy67du164u/cgnAulNumCyGEECLfFIo5FkIIIYQoHKSwEEIIIUS+kcJCCCGEEPlGCgshhBBC5BspLIQQQgiRb6SwEEIIIUS+kcJCCCGEEPlGCgshhBBC5BspLIQQQgiRb6SwEEIIIUS+kcJCCCGEEPlGCgshhBBC5Jv/B0LFhVYonimoAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "execution_count": 14, + "id": "5cfc62db", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", @@ -2758,27 +2658,31 @@ }, { "cell_type": "markdown", - "id": "36ddd15f", - "metadata": {}, + "id": "64c5fa5b", + "metadata": { + "editable": true + }, "source": [ "The intercept is the value of our output/target variable\n", "when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n", "\n", "Printing the MSE, we see first that both methods give the same MSE, as\n", - "they should. However, when we move to for example Ridge regression,\n", + "they should. However, when we move to for example Ridge regression (discussed next week),\n", "the way we treat the intercept may give a larger or smaller MSE,\n", "meaning that the MSE can be penalized by the value of the\n", "intercept. Not including the intercept in the fit, means that the\n", "regularization term does not include $\\beta_0$. For different values\n", - "of $\\lambda$, this may lead to differeing MSE values. \n", + "of $\\lambda$, this may lead to differing MSE values. \n", "\n", "To remind the reader, the regularization term, with the intercept in Ridge regression is given by" ] }, { "cell_type": "markdown", - "id": "dcf20838", - "metadata": {}, + "id": "67496f35", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", @@ -2787,16 +2691,20 @@ }, { "cell_type": "markdown", - "id": "16787ea5", - "metadata": {}, + "id": "ceae31e0", + "metadata": { + "editable": true + }, "source": [ "but when we take out the intercept, this equation becomes" ] }, { "cell_type": "markdown", - "id": "54c12897", - "metadata": {}, + "id": "743c68c8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", @@ -2805,16 +2713,20 @@ }, { "cell_type": "markdown", - "id": "f6f833bd", - "metadata": {}, + "id": "4af68378", + "metadata": { + "editable": true + }, "source": [ "For Lasso regression we have" ] }, { "cell_type": "markdown", - "id": "83afca1f", - "metadata": {}, + "id": "8228af90", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", @@ -2823,16 +2735,26 @@ }, { "cell_type": "markdown", - "id": "3025c3ef", - "metadata": {}, + "id": "0a4285bc", + "metadata": { + "editable": true + }, "source": [ - "It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week." + "It means that, when scaling the design matrix and the outputs/targets,\n", + "by subtracting the mean values, we have an optimization problem which\n", + "is not penalized by the intercept. The MSE value can then be smaller\n", + "since it focuses only on the remaining quantities. If we however bring\n", + "back the intercept, we will get an MSE which then contains the\n", + "intercept. This becomes more important when we discuss Ridge and Lasso\n", + "regression next week." ] }, { "cell_type": "markdown", - "id": "675a5b61", - "metadata": {}, + "id": "406af60a", + "metadata": { + "editable": true + }, "source": [ "## The Boston housing data example\n", "\n", @@ -2873,8 +2795,10 @@ }, { "cell_type": "markdown", - "id": "d2918e16", - "metadata": {}, + "id": "0f05d1e1", + "metadata": { + "editable": true + }, "source": [ "## Housing data, the code\n", "We start by importing the libraries" @@ -2883,8 +2807,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "aa97ca51", - "metadata": {}, + "id": "70f8a406", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -2896,8 +2823,10 @@ }, { "cell_type": "markdown", - "id": "34ffd981", - "metadata": {}, + "id": "10a6162c", + "metadata": { + "editable": true + }, "source": [ "and load the Boston Housing DataSet from **Scikit-Learn**" ] @@ -2905,8 +2834,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "e86122fb", - "metadata": {}, + "id": "be236243", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.datasets import load_boston\n", @@ -2920,8 +2852,10 @@ }, { "cell_type": "markdown", - "id": "2ab5f3f1", - "metadata": {}, + "id": "88a7376f", + "metadata": { + "editable": true + }, "source": [ "Then we invoke Pandas" ] @@ -2929,8 +2863,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "73bb029f", - "metadata": {}, + "id": "ec268779", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "boston = pd.DataFrame(boston_dataset.data, columns=boston_dataset.feature_names)\n", @@ -2940,8 +2877,10 @@ }, { "cell_type": "markdown", - "id": "04650405", - "metadata": {}, + "id": "b45f3733", + "metadata": { + "editable": true + }, "source": [ "and preprocess the data" ] @@ -2949,8 +2888,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "b1ba0529", - "metadata": {}, + "id": "8b9423c6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# check for missing values in all the columns\n", @@ -2959,8 +2901,10 @@ }, { "cell_type": "markdown", - "id": "fb86d39a", - "metadata": {}, + "id": "1061cb1f", + "metadata": { + "editable": true + }, "source": [ "We can then visualize the data" ] @@ -2968,8 +2912,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "9269bc8c", - "metadata": {}, + "id": "00d79fad", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# set the size of the figure\n", @@ -2982,8 +2929,10 @@ }, { "cell_type": "markdown", - "id": "a553bff9", - "metadata": {}, + "id": "18871b23", + "metadata": { + "editable": true + }, "source": [ "It is now useful to look at the correlation matrix" ] @@ -2991,8 +2940,11 @@ { "cell_type": "code", "execution_count": 20, - "id": "f0a35752", - "metadata": {}, + "id": "abfd55f9", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# compute the pair wise correlation for all columns \n", @@ -3004,8 +2956,10 @@ }, { "cell_type": "markdown", - "id": "f6006385", - "metadata": {}, + "id": "231f35c5", + "metadata": { + "editable": true + }, "source": [ "From the above coorelation plot we can see that **MEDV** is strongly correlated to **LSTAT** and **RM**. We see also that **RAD** and **TAX** are stronly correlated, but we don't include this in our features together to avoid multi-colinearity" ] @@ -3013,8 +2967,11 @@ { "cell_type": "code", "execution_count": 21, - "id": "a7a242b8", - "metadata": {}, + "id": "5bb68d00", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "plt.figure(figsize=(20, 5))\n", @@ -3034,8 +2991,10 @@ }, { "cell_type": "markdown", - "id": "1f0d201f", - "metadata": {}, + "id": "9c2f9f6b", + "metadata": { + "editable": true + }, "source": [ "Now we start training our model" ] @@ -3043,8 +3002,11 @@ { "cell_type": "code", "execution_count": 22, - "id": "1ebe4233", - "metadata": {}, + "id": "df3cc855", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "X = pd.DataFrame(np.c_[boston['LSTAT'], boston['RM']], columns = ['LSTAT','RM'])\n", @@ -3053,8 +3015,10 @@ }, { "cell_type": "markdown", - "id": "1e4b17f6", - "metadata": {}, + "id": "da67457b", + "metadata": { + "editable": true + }, "source": [ "We split the data into training and test sets" ] @@ -3062,8 +3026,11 @@ { "cell_type": "code", "execution_count": 23, - "id": "2d320774", - "metadata": {}, + "id": "f9784866", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -3079,8 +3046,10 @@ }, { "cell_type": "markdown", - "id": "de6cb89b", - "metadata": {}, + "id": "3429e67f", + "metadata": { + "editable": true + }, "source": [ "Then we use the linear regression functionality from **Scikit-Learn**" ] @@ -3088,8 +3057,11 @@ { "cell_type": "code", "execution_count": 24, - "id": "406f9cc7", - "metadata": {}, + "id": "9ec7ba47", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.linear_model import LinearRegression\n", @@ -3128,8 +3100,11 @@ { "cell_type": "code", "execution_count": 25, - "id": "1790d8c6", - "metadata": {}, + "id": "0c0be681", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# plotting the y_test vs y_pred\n", @@ -3140,16 +3115,20 @@ }, { "cell_type": "markdown", - "id": "a602af77", - "metadata": {}, + "id": "ed12f12d", + "metadata": { + "editable": true + }, "source": [ "## Material for lecture Thursday, August 31" ] }, { "cell_type": "markdown", - "id": "0190db0a", - "metadata": {}, + "id": "50c91ade", + "metadata": { + "editable": true + }, "source": [ "## Mathematical Interpretation of Ordinary Least Squares\n", "\n", @@ -3160,8 +3139,10 @@ }, { "cell_type": "markdown", - "id": "42ea647f", - "metadata": {}, + "id": "682a501b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -3170,8 +3151,10 @@ }, { "cell_type": "markdown", - "id": "88ef32a4", - "metadata": {}, + "id": "057bbc5d", + "metadata": { + "editable": true + }, "source": [ "The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n", "\n", @@ -3180,8 +3163,10 @@ }, { "cell_type": "markdown", - "id": "7ce1fdc1", - "metadata": {}, + "id": "cddb899f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -3190,16 +3175,20 @@ }, { "cell_type": "markdown", - "id": "d15bf8e4", - "metadata": {}, + "id": "ab325ebc", + "metadata": { + "editable": true + }, "source": [ "We now define a matrix" ] }, { "cell_type": "markdown", - "id": "e8bbed74", - "metadata": {}, + "id": "095ebc5b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T.\n", @@ -3208,16 +3197,20 @@ }, { "cell_type": "markdown", - "id": "27b4b670", - "metadata": {}, + "id": "08e642f6", + "metadata": { + "editable": true + }, "source": [ "We can rewrite" ] }, { "cell_type": "markdown", - "id": "2933e7b2", - "metadata": {}, + "id": "da5c1625", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n", @@ -3226,8 +3219,10 @@ }, { "cell_type": "markdown", - "id": "cc646006", - "metadata": {}, + "id": "2bf04890", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n", "We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix." @@ -3235,8 +3230,10 @@ }, { "cell_type": "markdown", - "id": "f0578c51", - "metadata": {}, + "id": "5ba172cc", + "metadata": { + "editable": true + }, "source": [ "## Residual Error\n", "\n", @@ -3245,8 +3242,10 @@ }, { "cell_type": "markdown", - "id": "6332e593", - "metadata": {}, + "id": "7d1f0b59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=\\left[\\boldsymbol{I}-\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\right]\\boldsymbol{y}.\n", @@ -3255,16 +3254,20 @@ }, { "cell_type": "markdown", - "id": "e5c62994", - "metadata": {}, + "id": "467e185d", + "metadata": { + "editable": true + }, "source": [ "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal component of the space defined by the column vectors of $\\boldsymbol{X}$." ] }, { "cell_type": "markdown", - "id": "dd70468f", - "metadata": {}, + "id": "74822363", + "metadata": { + "editable": true + }, "source": [ "## Simple case\n", "\n", @@ -3273,8 +3276,10 @@ }, { "cell_type": "markdown", - "id": "66fce1fe", - "metadata": {}, + "id": "5414a1d2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{X}\\boldsymbol{X}^T = \\boldsymbol{I}.\n", @@ -3283,16 +3288,20 @@ }, { "cell_type": "markdown", - "id": "4811da1c", - "metadata": {}, + "id": "08ce1402", + "metadata": { + "editable": true + }, "source": [ "In this case the matrix $\\boldsymbol{A}$ becomes" ] }, { "cell_type": "markdown", - "id": "d4127df0", - "metadata": {}, + "id": "7bbb949f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n", @@ -3301,16 +3310,20 @@ }, { "cell_type": "markdown", - "id": "5bcc3a1e", - "metadata": {}, + "id": "4007519d", + "metadata": { + "editable": true + }, "source": [ "and we have the obvious case" ] }, { "cell_type": "markdown", - "id": "fe9806bb", - "metadata": {}, + "id": "45061e33", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=0.\n", @@ -3319,16 +3332,20 @@ }, { "cell_type": "markdown", - "id": "646d3f00", - "metadata": {}, + "id": "f5afee48", + "metadata": { + "editable": true + }, "source": [ "This serves also as a useful test of our codes." ] }, { "cell_type": "markdown", - "id": "fe0220de", - "metadata": {}, + "id": "6a3d6440", + "metadata": { + "editable": true + }, "source": [ "## The singular value decomposition\n", "\n", @@ -3365,8 +3382,10 @@ }, { "cell_type": "markdown", - "id": "4ab0e0c3", - "metadata": {}, + "id": "ef8c39b7", + "metadata": { + "editable": true + }, "source": [ "## Linear Regression Problems\n", "\n", @@ -3380,8 +3399,10 @@ }, { "cell_type": "markdown", - "id": "bc43fd7f", - "metadata": {}, + "id": "6edd77ca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -3401,8 +3422,10 @@ }, { "cell_type": "markdown", - "id": "687c4056", - "metadata": {}, + "id": "cba40ade", + "metadata": { + "editable": true + }, "source": [ "The columns of $\\boldsymbol{X}$ are linearly dependent. We see this easily since the \n", "the first column is the row-wise sum of the other two columns. The rank (more correct,\n", @@ -3416,8 +3439,10 @@ }, { "cell_type": "markdown", - "id": "5dae4f20", - "metadata": {}, + "id": "20f096a2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -3433,8 +3458,10 @@ }, { "cell_type": "markdown", - "id": "39ed048b", - "metadata": {}, + "id": "49f6f27c", + "metadata": { + "editable": true + }, "source": [ "We see easily that $\\mbox{det}(\\boldsymbol{X}) = x_{11} x_{22} - x_{12} x_{21} = 1 \\times (-1) - 1 \\times (-1) = 0$. Hence, $\\mathbf{X}$ is singular and its inverse is undefined.\n", "This is equivalent to saying that the matrix $\\boldsymbol{X}$ has at least an eigenvalue which is zero." @@ -3442,8 +3469,10 @@ }, { "cell_type": "markdown", - "id": "eeca30df", - "metadata": {}, + "id": "577f3338", + "metadata": { + "editable": true + }, "source": [ "## Fixing the singularity\n", "\n", @@ -3452,8 +3481,10 @@ }, { "cell_type": "markdown", - "id": "14eebe7d", - "metadata": {}, + "id": "f456df81", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -3468,8 +3499,10 @@ }, { "cell_type": "markdown", - "id": "6d4a9b40", - "metadata": {}, + "id": "ed7db14f", + "metadata": { + "editable": true + }, "source": [ "has linearly dependent column vectors, we will not be able to compute the inverse\n", "of $\\boldsymbol{X}^T\\boldsymbol{X}$ and we cannot find the parameters (estimators) $\\beta_i$. \n", @@ -3482,8 +3515,10 @@ }, { "cell_type": "markdown", - "id": "941de2f0", - "metadata": {}, + "id": "a876de90", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^{T} \\boldsymbol{X} \\rightarrow \\boldsymbol{X}^{T} \\boldsymbol{X}+\\lambda \\boldsymbol{I},\n", @@ -3492,16 +3527,20 @@ }, { "cell_type": "markdown", - "id": "a1d055f6", - "metadata": {}, + "id": "3b2eee8d", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{I}$ is the identity matrix. When we discuss **Ridge** regression this is actually what we end up evaluating. The parameter $\\lambda$ is called a hyperparameter. More about this later." ] }, { "cell_type": "markdown", - "id": "9022b09b", - "metadata": {}, + "id": "910d3f09", + "metadata": { + "editable": true + }, "source": [ "## Basic math of the SVD\n", "\n", @@ -3513,8 +3552,10 @@ }, { "cell_type": "markdown", - "id": "c6b0e69e", - "metadata": {}, + "id": "94bd4c00", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\lambda_1,\\boldsymbol{u}_1),\\dots, (\\lambda_n,\\boldsymbol{u}_n),\n", @@ -3523,16 +3564,20 @@ }, { "cell_type": "markdown", - "id": "76452c40", - "metadata": {}, + "id": "56d91a09", + "metadata": { + "editable": true + }, "source": [ "and the eigenvalues are given by the diagonal matrix" ] }, { "cell_type": "markdown", - "id": "dadb0dad", - "metadata": {}, + "id": "8e59a435", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\\mathrm{Diag}(\\lambda_1, \\dots,\\lambda_n).\n", @@ -3541,16 +3586,20 @@ }, { "cell_type": "markdown", - "id": "d6aa2f86", - "metadata": {}, + "id": "9c2e6b9f", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ can be written in terms of an orthogonal/unitary transformation $\\boldsymbol{U}$" ] }, { "cell_type": "markdown", - "id": "483467c0", - "metadata": {}, + "id": "e7188a39", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -3559,8 +3608,10 @@ }, { "cell_type": "markdown", - "id": "ec45b064", - "metadata": {}, + "id": "daac715a", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ or $\\boldsymbol{U}\\boldsymbol{U}^{\\dagger}=\\boldsymbol{I}$.\n", "\n", @@ -3569,8 +3620,10 @@ }, { "cell_type": "markdown", - "id": "f287a968", - "metadata": {}, + "id": "66322f83", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\begin{bmatrix} \n", @@ -3582,8 +3635,10 @@ }, { "cell_type": "markdown", - "id": "ce946d04", - "metadata": {}, + "id": "a03be784", + "metadata": { + "editable": true + }, "source": [ "is not diagonalizable, it is a so-called [defective matrix](https://en.wikipedia.org/wiki/Defective_matrix). It is easy to see that the condition\n", "$\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{X}^T\\boldsymbol{X}$ is not fulfilled." @@ -3591,8 +3646,10 @@ }, { "cell_type": "markdown", - "id": "ebb8258f", - "metadata": {}, + "id": "ff3964a4", + "metadata": { + "editable": true + }, "source": [ "## The SVD, a Fantastic Algorithm\n", "\n", @@ -3609,8 +3666,10 @@ }, { "cell_type": "markdown", - "id": "57d25c28", - "metadata": {}, + "id": "04d411cc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\n", @@ -3619,16 +3678,20 @@ }, { "cell_type": "markdown", - "id": "5c63309c", - "metadata": {}, + "id": "bf2ae91a", + "metadata": { + "editable": true + }, "source": [ "As an example, the above defective matrix can be decomposed as" ] }, { "cell_type": "markdown", - "id": "531d6591", - "metadata": {}, + "id": "843d113e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X} = \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& 1 \\\\ 1& -1\\\\ \\end{bmatrix} \\begin{bmatrix} 2& 0 \\\\ 0& 0\\\\ \\end{bmatrix} \\frac{1}{\\sqrt{2}}\\begin{bmatrix} 1& -1 \\\\ 1& 1\\\\ \\end{bmatrix}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -3637,8 +3700,10 @@ }, { "cell_type": "markdown", - "id": "06ea8edf", - "metadata": {}, + "id": "f59af5f8", + "metadata": { + "editable": true + }, "source": [ "with eigenvalues $\\sigma_1=2$ and $\\sigma_2=0$. \n", "The SVD exits always! \n", @@ -3664,8 +3729,10 @@ }, { "cell_type": "markdown", - "id": "c8207881", - "metadata": {}, + "id": "22f1ed07", + "metadata": { + "editable": true + }, "source": [ "## Economy-size SVD\n", "\n", @@ -3689,8 +3756,10 @@ }, { "cell_type": "markdown", - "id": "0f679a3e", - "metadata": {}, + "id": "69efb08d", + "metadata": { + "editable": true + }, "source": [ "## Codes for the SVD" ] @@ -3698,8 +3767,11 @@ { "cell_type": "code", "execution_count": 26, - "id": "f303dd05", - "metadata": {}, + "id": "640d3bdf", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -3735,8 +3807,10 @@ }, { "cell_type": "markdown", - "id": "3b89ae6a", - "metadata": {}, + "id": "98d61790", + "metadata": { + "editable": true + }, "source": [ "The matrix $\\boldsymbol{X}$ has columns that are linearly dependent. The first\n", "column is the row-wise sum of the other two columns. The rank of a\n", @@ -3750,8 +3824,10 @@ }, { "cell_type": "markdown", - "id": "ccc2ace3", - "metadata": {}, + "id": "95b1ffa6", + "metadata": { + "editable": true + }, "source": [ "## Note about SVD Calculations\n", "\n", @@ -3771,8 +3847,10 @@ }, { "cell_type": "markdown", - "id": "ec0bc486", - "metadata": {}, + "id": "4605f137", + "metadata": { + "editable": true + }, "source": [ "## Mathematics of the SVD and implications\n", "\n", @@ -3783,8 +3861,10 @@ }, { "cell_type": "markdown", - "id": "2f25f47f", - "metadata": {}, + "id": "3125bc92", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -3800,16 +3880,20 @@ }, { "cell_type": "markdown", - "id": "950d6d71", - "metadata": {}, + "id": "32cf3a9e", + "metadata": { + "editable": true + }, "source": [ "We can SVD decompose our matrix as" ] }, { "cell_type": "markdown", - "id": "61595420", - "metadata": {}, + "id": "573acb2d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -3818,8 +3902,10 @@ }, { "cell_type": "markdown", - "id": "81dbfc66", - "metadata": {}, + "id": "22320566", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{U}$ is an orthogonal matrix of dimension $n\\times n$, meaning that $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{I}_n$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $n \\times n$.\n", "\n", @@ -3830,8 +3916,10 @@ }, { "cell_type": "markdown", - "id": "6e69a431", - "metadata": {}, + "id": "af0ee993", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sigma_0 > \\sigma_1 > \\sigma_2 > \\dots > \\sigma_{p-1} > 0.\n", @@ -3840,16 +3928,20 @@ }, { "cell_type": "markdown", - "id": "c512a377", - "metadata": {}, + "id": "0213ed87", + "metadata": { + "editable": true + }, "source": [ "All values beyond $p-1$ are all zero." ] }, { "cell_type": "markdown", - "id": "f380e36b", - "metadata": {}, + "id": "06266ee6", + "metadata": { + "editable": true + }, "source": [ "## Example Matrix\n", "\n", @@ -3858,8 +3950,10 @@ }, { "cell_type": "markdown", - "id": "d5a0b156", - "metadata": {}, + "id": "23e27fdf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\n", @@ -3873,16 +3967,20 @@ }, { "cell_type": "markdown", - "id": "c4fbe6ee", - "metadata": {}, + "id": "2539e582", + "metadata": { + "editable": true + }, "source": [ "The singular values are $\\sigma_0=2$ and $\\sigma_1=1$. It is common to rewrite the matrix $\\boldsymbol{\\Sigma}$ as" ] }, { "cell_type": "markdown", - "id": "43f9dca0", - "metadata": {}, + "id": "bef2cf95", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}=\n", @@ -3895,16 +3993,20 @@ }, { "cell_type": "markdown", - "id": "bd10fa38", - "metadata": {}, + "id": "fe385753", + "metadata": { + "editable": true + }, "source": [ "where" ] }, { "cell_type": "markdown", - "id": "197008d7", - "metadata": {}, + "id": "cc270435", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\tilde{\\Sigma}}=\n", @@ -3917,16 +4019,20 @@ }, { "cell_type": "markdown", - "id": "d7d03027", - "metadata": {}, + "id": "31932192", + "metadata": { + "editable": true + }, "source": [ "contains only the singular values. Note also (and we will use this below) that" ] }, { "cell_type": "markdown", - "id": "a545ab28", - "metadata": {}, + "id": "a972a944", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\n", @@ -3939,16 +4045,20 @@ }, { "cell_type": "markdown", - "id": "0c39be82", - "metadata": {}, + "id": "d6ef62ee", + "metadata": { + "editable": true + }, "source": [ "which is a $2\\times 2 $ matrix while" ] }, { "cell_type": "markdown", - "id": "b73a0d64", - "metadata": {}, + "id": "61e10478", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T=\n", @@ -3962,8 +4072,10 @@ }, { "cell_type": "markdown", - "id": "6aa1af58", - "metadata": {}, + "id": "9e8580da", + "metadata": { + "editable": true + }, "source": [ "is a $3\\times 3 $ matrix. The last row and column of this last matrix\n", "contain only zeros. This will have important consequences for our SVD\n", @@ -3972,8 +4084,10 @@ }, { "cell_type": "markdown", - "id": "83e55282", - "metadata": {}, + "id": "0ac00991", + "metadata": { + "editable": true + }, "source": [ "## Setting up the Matrix to be inverted\n", "\n", @@ -3982,8 +4096,10 @@ }, { "cell_type": "markdown", - "id": "f57493d4", - "metadata": {}, + "id": "05f8073b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n", @@ -3992,16 +4108,20 @@ }, { "cell_type": "markdown", - "id": "1a2c5145", - "metadata": {}, + "id": "622a8443", + "metadata": { + "editable": true + }, "source": [ "and using the orthogonality of the matrix $\\boldsymbol{U}$ we have" ] }, { "cell_type": "markdown", - "id": "7e9c396a", - "metadata": {}, + "id": "580e143c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -4010,8 +4130,10 @@ }, { "cell_type": "markdown", - "id": "530d69f7", - "metadata": {}, + "id": "30b75cea", + "metadata": { + "editable": true + }, "source": [ "We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\\tilde{\\boldsymbol{\\Sigma}}^2$ which is a diagonal matrix containing only the singular values squared. It has dimensionality $p \\times p$.\n", "\n", @@ -4020,8 +4142,10 @@ }, { "cell_type": "markdown", - "id": "99f5df84", - "metadata": {}, + "id": "cdc103a7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -4030,16 +4154,20 @@ }, { "cell_type": "markdown", - "id": "05f10d74", - "metadata": {}, + "id": "ea01abf6", + "metadata": { + "editable": true + }, "source": [ "and using our SVD decomposition of $\\boldsymbol{X}$ we have" ] }, { "cell_type": "markdown", - "id": "d2eca91e", - "metadata": {}, + "id": "e4b60d5d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\left(\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^{2}(\\boldsymbol{V}^T\\right)^{-1}\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{y},\n", @@ -4048,16 +4176,20 @@ }, { "cell_type": "markdown", - "id": "2ae2fef1", - "metadata": {}, + "id": "deb0bbd3", + "metadata": { + "editable": true + }, "source": [ "which gives us, using the orthogonality of the matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$," ] }, { "cell_type": "markdown", - "id": "71f1ea41", - "metadata": {}, + "id": "f1f15de8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}^T_i\\boldsymbol{y},\n", @@ -4066,8 +4198,10 @@ }, { "cell_type": "markdown", - "id": "f3ee9ee7", - "metadata": {}, + "id": "7c54c407", + "metadata": { + "editable": true + }, "source": [ "It means that the ordinary least square model (with the optimal\n", "parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal\n", @@ -4078,8 +4212,10 @@ }, { "cell_type": "markdown", - "id": "8b7a74f0", - "metadata": {}, + "id": "31e31b03", + "metadata": { + "editable": true + }, "source": [ "## Further properties (important for our analyses later)\n", "\n", @@ -4088,8 +4224,10 @@ }, { "cell_type": "markdown", - "id": "4250e3bf", - "metadata": {}, + "id": "7d1cef4f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -4098,16 +4236,20 @@ }, { "cell_type": "markdown", - "id": "2c4099f9", - "metadata": {}, + "id": "aa8cedef", + "metadata": { + "editable": true + }, "source": [ "If we now multiply from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "id": "be49edc5", - "metadata": {}, + "id": "022623a1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}.\n", @@ -4116,8 +4258,10 @@ }, { "cell_type": "markdown", - "id": "cd0a8bc6", - "metadata": {}, + "id": "c9885020", + "metadata": { + "editable": true + }, "source": [ "This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n", "with eigenvalues given by the singular values squared, that is" @@ -4125,8 +4269,10 @@ }, { "cell_type": "markdown", - "id": "733b034c", - "metadata": {}, + "id": "f378f76e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -4135,16 +4281,20 @@ }, { "cell_type": "markdown", - "id": "ac74afd8", - "metadata": {}, + "id": "4add7e8a", + "metadata": { + "editable": true + }, "source": [ "Similarly, if we use the SVD decomposition for the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$, we have" ] }, { "cell_type": "markdown", - "id": "ad262b2a", - "metadata": {}, + "id": "51ea1a29", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T.\n", @@ -4153,16 +4303,20 @@ }, { "cell_type": "markdown", - "id": "c569cf69", - "metadata": {}, + "id": "54a250b6", + "metadata": { + "editable": true + }, "source": [ "If we now multiply from the right with $\\boldsymbol{U}$ (using the orthogonality of $\\boldsymbol{U}$) we get" ] }, { "cell_type": "markdown", - "id": "b574522e", - "metadata": {}, + "id": "ebaf68f5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T.\n", @@ -4171,8 +4325,10 @@ }, { "cell_type": "markdown", - "id": "45586d81", - "metadata": {}, + "id": "5ff687e8", + "metadata": { + "editable": true + }, "source": [ "This means the vectors $\\boldsymbol{u}_i$ of the orthogonal matrix $\\boldsymbol{U}$ are the eigenvectors of the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "with eigenvalues given by the singular values squared, that is" @@ -4180,8 +4336,10 @@ }, { "cell_type": "markdown", - "id": "c7c9d818", - "metadata": {}, + "id": "ea61a360", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{u}_i=\\boldsymbol{u}_i\\sigma_i^2.\n", @@ -4190,8 +4348,10 @@ }, { "cell_type": "markdown", - "id": "1a9d86fc", - "metadata": {}, + "id": "bc168f34", + "metadata": { + "editable": true + }, "source": [ "**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n", "$n\\times p$ matrix. In most supervised learning cases we have that $n\n", @@ -4206,8 +4366,10 @@ }, { "cell_type": "markdown", - "id": "347a807e", - "metadata": {}, + "id": "39d89895", + "metadata": { + "editable": true + }, "source": [ "## Meet the Covariance Matrix\n", "\n", @@ -4220,8 +4382,10 @@ }, { "cell_type": "markdown", - "id": "a52e2b26", - "metadata": {}, + "id": "c229143d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -4230,8 +4394,10 @@ }, { "cell_type": "markdown", - "id": "dfbc1bb8", - "metadata": {}, + "id": "0a569456", + "metadata": { + "editable": true + }, "source": [ "This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n", "\n", @@ -4240,8 +4406,10 @@ }, { "cell_type": "markdown", - "id": "9d4a8790", - "metadata": {}, + "id": "0c0b9212", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", @@ -4250,8 +4418,10 @@ }, { "cell_type": "markdown", - "id": "15117986", - "metadata": {}, + "id": "b05a1b83", + "metadata": { + "editable": true + }, "source": [ "The Hessian matrix for ordinary least squares is also proportional to\n", "the covariance matrix. This means also that we can use the SVD to find\n", @@ -4261,8 +4431,10 @@ }, { "cell_type": "markdown", - "id": "05926fd4", - "metadata": {}, + "id": "d64818ce", + "metadata": { + "editable": true + }, "source": [ "## Introducing the Covariance and Correlation functions\n", "\n", @@ -4275,8 +4447,10 @@ }, { "cell_type": "markdown", - "id": "5bb56f45", - "metadata": {}, + "id": "32df0f89", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -4287,16 +4461,20 @@ }, { "cell_type": "markdown", - "id": "bfe1b957", - "metadata": {}, + "id": "a9d9108b", + "metadata": { + "editable": true + }, "source": [ "where for example" ] }, { "cell_type": "markdown", - "id": "f6baf9c0", - "metadata": {}, + "id": "c6d76715", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n", @@ -4305,16 +4483,20 @@ }, { "cell_type": "markdown", - "id": "ed9da1e2", - "metadata": {}, + "id": "5cc35821", + "metadata": { + "editable": true + }, "source": [ "With this definition and recalling that the variance is defined as" ] }, { "cell_type": "markdown", - "id": "68353e8c", - "metadata": {}, + "id": "27c8eac0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{var}[\\boldsymbol{x}]=\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})^2,\n", @@ -4323,16 +4505,20 @@ }, { "cell_type": "markdown", - "id": "3ea76834", - "metadata": {}, + "id": "49361e47", + "metadata": { + "editable": true + }, "source": [ "we can rewrite the covariance matrix as" ] }, { "cell_type": "markdown", - "id": "543e1fa1", - "metadata": {}, + "id": "5eb15920", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -4343,8 +4529,10 @@ }, { "cell_type": "markdown", - "id": "cf004dc8", - "metadata": {}, + "id": "2160cb07", + "metadata": { + "editable": true + }, "source": [ "**Note:** we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. \n", "What you will find in essentially all statistics texts are equations\n", @@ -4358,8 +4546,10 @@ }, { "cell_type": "markdown", - "id": "70fa6885", - "metadata": {}, + "id": "a5deaa8e", + "metadata": { + "editable": true + }, "source": [ "## Covariance and Correlation Matrix\n", "\n", @@ -4372,8 +4562,10 @@ }, { "cell_type": "markdown", - "id": "dcc0c4c1", - "metadata": {}, + "id": "f6868141", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]=\\frac{\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}]}{\\sqrt{\\mathrm{var}[\\boldsymbol{x}] \\mathrm{var}[\\boldsymbol{y}]}}.\n", @@ -4382,8 +4574,10 @@ }, { "cell_type": "markdown", - "id": "26d7e490", - "metadata": {}, + "id": "9c17b156", + "metadata": { + "editable": true + }, "source": [ "The correlation function is then given by values $\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]\n", "\\in [-1,1]$. This avoids eventual problems with too large values. We\n", @@ -4393,8 +4587,10 @@ }, { "cell_type": "markdown", - "id": "a0a36a83", - "metadata": {}, + "id": "1b9e8b7e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{K}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} 1 & \\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n", @@ -4405,16 +4601,20 @@ }, { "cell_type": "markdown", - "id": "97743931", - "metadata": {}, + "id": "c00ac20a", + "metadata": { + "editable": true + }, "source": [ "In the above example this is the function we constructed using **pandas**." ] }, { "cell_type": "markdown", - "id": "e1c9d958", - "metadata": {}, + "id": "91729a44", + "metadata": { + "editable": true + }, "source": [ "## Correlation Function and Design/Feature Matrix\n", "\n", @@ -4424,8 +4624,10 @@ }, { "cell_type": "markdown", - "id": "1c653a67", - "metadata": {}, + "id": "62403241", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -4441,8 +4643,10 @@ }, { "cell_type": "markdown", - "id": "a260bd35", - "metadata": {}, + "id": "6d0de657", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors/features $p$ refering to the column numbers and the\n", "entries $n$ being the row elements.\n", @@ -4451,8 +4655,10 @@ }, { "cell_type": "markdown", - "id": "5cc7a553", - "metadata": {}, + "id": "108adf6b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} \\boldsymbol{x}_0 & \\boldsymbol{x}_1 & \\boldsymbol{x}_2 & \\dots & \\dots & \\boldsymbol{x}_{p-1}\\end{bmatrix},\n", @@ -4461,16 +4667,20 @@ }, { "cell_type": "markdown", - "id": "6cdc22ce", - "metadata": {}, + "id": "1b2ba4f7", + "metadata": { + "editable": true + }, "source": [ "with a given vector" ] }, { "cell_type": "markdown", - "id": "be7da0d5", - "metadata": {}, + "id": "1711d54c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T = \\begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \\dots & \\dots x_{n-1,i}\\end{bmatrix}.\n", @@ -4479,8 +4689,10 @@ }, { "cell_type": "markdown", - "id": "3d4d66f2", - "metadata": {}, + "id": "f88b0c88", + "metadata": { + "editable": true + }, "source": [ "With these definitions, we can now rewrite our $2\\times 2$\n", "correlation/covariance matrix in terms of a moe general design/feature\n", @@ -4490,8 +4702,10 @@ }, { "cell_type": "markdown", - "id": "f8d2fd2c", - "metadata": {}, + "id": "d1835d40", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix}\n", @@ -4507,16 +4721,20 @@ }, { "cell_type": "markdown", - "id": "2f432d38", - "metadata": {}, + "id": "7d0f1648", + "metadata": { + "editable": true + }, "source": [ "and the correlation matrix" ] }, { "cell_type": "markdown", - "id": "64a695ad", - "metadata": {}, + "id": "d16ea5cf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{K}[\\boldsymbol{x}] = \\begin{bmatrix}\n", @@ -4532,8 +4750,10 @@ }, { "cell_type": "markdown", - "id": "8d112930", - "metadata": {}, + "id": "634130de", + "metadata": { + "editable": true + }, "source": [ "## Covariance Matrix Examples\n", "\n", @@ -4548,8 +4768,10 @@ }, { "cell_type": "markdown", - "id": "62633ba5", - "metadata": {}, + "id": "cecf6c09", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{W} = \\begin{bmatrix} x_0 & x_1 & x_2 & \\dots & x_{n-2} & x_{n-1} \\\\\n", @@ -4560,8 +4782,10 @@ }, { "cell_type": "markdown", - "id": "74f22dd0", - "metadata": {}, + "id": "1703b000", + "metadata": { + "editable": true + }, "source": [ "which in turn is converted into into the $2\\times 2$ covariance matrix\n", "$\\boldsymbol{C}$ via the Numpy function **np.cov()**. We note that we can also calculate\n", @@ -4573,8 +4797,11 @@ { "cell_type": "code", "execution_count": 27, - "id": "10208407", - "metadata": {}, + "id": "5da5890f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -4591,8 +4818,10 @@ }, { "cell_type": "markdown", - "id": "fff9f0df", - "metadata": {}, + "id": "6b10b22a", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix\n", "\n", @@ -4606,8 +4835,11 @@ { "cell_type": "code", "execution_count": 28, - "id": "72df9047", - "metadata": {}, + "id": "da699527", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -4635,8 +4867,10 @@ }, { "cell_type": "markdown", - "id": "35ba27c4", - "metadata": {}, + "id": "a40fb51e", + "metadata": { + "editable": true + }, "source": [ "We see that the matrix elements along the diagonal are one as they\n", "should be and that the matrix is symmetric. Furthermore, diagonalizing\n", @@ -4647,8 +4881,10 @@ }, { "cell_type": "markdown", - "id": "813e8edd", - "metadata": {}, + "id": "413db31e", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix with Pandas\n", "\n", @@ -4658,8 +4894,11 @@ { "cell_type": "code", "execution_count": 29, - "id": "b21385d7", - "metadata": {}, + "id": "53927e5d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -4680,16 +4919,20 @@ }, { "cell_type": "markdown", - "id": "6e4af867", - "metadata": {}, + "id": "74fc3107", + "metadata": { + "editable": true + }, "source": [ "We expand this model to the Franke function discussed above." ] }, { "cell_type": "markdown", - "id": "4eeffe9a", - "metadata": {}, + "id": "030881ab", + "metadata": { + "editable": true + }, "source": [ "## Correlation Matrix with Pandas and the Franke function" ] @@ -4697,8 +4940,11 @@ { "cell_type": "code", "execution_count": 30, - "id": "3cb27e05", - "metadata": {}, + "id": "c107aa55", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Common imports\n", @@ -4748,8 +4994,10 @@ }, { "cell_type": "markdown", - "id": "5897c0af", - "metadata": {}, + "id": "3f47378e", + "metadata": { + "editable": true + }, "source": [ "We note here that the covariance is zero for the first rows and\n", "columns since all matrix elements in the design matrix were set to one\n", @@ -4763,8 +5011,10 @@ }, { "cell_type": "markdown", - "id": "d5e30492", - "metadata": {}, + "id": "dba29ab3", + "metadata": { + "editable": true + }, "source": [ "## Rewriting the Covariance and/or Correlation Matrix\n", "\n", @@ -4773,8 +5023,10 @@ }, { "cell_type": "markdown", - "id": "2da8a6f6", - "metadata": {}, + "id": "0eb702bd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}] = \\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X}= \\mathbb{E}[\\boldsymbol{X}^T\\boldsymbol{X}].\n", @@ -4783,16 +5035,20 @@ }, { "cell_type": "markdown", - "id": "06eac8d4", - "metadata": {}, + "id": "1d355153", + "metadata": { + "editable": true + }, "source": [ "To see this let us simply look at a design matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{2\\times 2}$" ] }, { "cell_type": "markdown", - "id": "7c90bb0e", - "metadata": {}, + "id": "a2316749", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}\n", @@ -4806,16 +5062,20 @@ }, { "cell_type": "markdown", - "id": "59adda50", - "metadata": {}, + "id": "8f6a654b", + "metadata": { + "editable": true + }, "source": [ "If we then compute the expectation value (note the $1/n$ factor instead of $1/(n-1)$)" ] }, { "cell_type": "markdown", - "id": "65b5748f", - "metadata": {}, + "id": "72291085", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbb{E}[\\boldsymbol{X}^T\\boldsymbol{X}] = \\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X}=\\frac{1}{n}\\begin{bmatrix}\n", @@ -4827,16 +5087,20 @@ }, { "cell_type": "markdown", - "id": "bc3e9830", - "metadata": {}, + "id": "2b28a5fe", + "metadata": { + "editable": true + }, "source": [ "which is just" ] }, { "cell_type": "markdown", - "id": "b458de4d", - "metadata": {}, + "id": "49efe427", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] = \\boldsymbol{C}[\\boldsymbol{x}]=\\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}_0] & \\mathrm{cov}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] \\\\\n", @@ -4847,8 +5111,10 @@ }, { "cell_type": "markdown", - "id": "4a57a8e0", - "metadata": {}, + "id": "7edcb842", + "metadata": { + "editable": true + }, "source": [ "where we wrote $$\\boldsymbol{C}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] = \\boldsymbol{C}[\\boldsymbol{x}]$$ to indicate that this is the covariance of the vectors $\\boldsymbol{x}$ of the design/feature matrix $\\boldsymbol{X}$.\n", "\n", @@ -4857,8 +5123,10 @@ }, { "cell_type": "markdown", - "id": "0c426ac7", - "metadata": {}, + "id": "50ec44da", + "metadata": { + "editable": true + }, "source": [ "## Linking with the SVD\n", "\n", @@ -4867,8 +5135,10 @@ }, { "cell_type": "markdown", - "id": "a6d06a3f", - "metadata": {}, + "id": "f4010e4a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -4877,16 +5147,20 @@ }, { "cell_type": "markdown", - "id": "219f9ccb", - "metadata": {}, + "id": "c0e95eb2", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix" ] }, { "cell_type": "markdown", - "id": "e0af609e", - "metadata": {}, + "id": "46abefce", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -4895,16 +5169,20 @@ }, { "cell_type": "markdown", - "id": "721e39cb", - "metadata": {}, + "id": "30544899", + "metadata": { + "editable": true + }, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] }, { "cell_type": "markdown", - "id": "ec989cfc", - "metadata": {}, + "id": "feaeedbe", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -4918,16 +5196,20 @@ }, { "cell_type": "markdown", - "id": "57275f46", - "metadata": {}, + "id": "3a2c0f0e", + "metadata": { + "editable": true + }, "source": [ "meaning we can write" ] }, { "cell_type": "markdown", - "id": "59dfa6be", - "metadata": {}, + "id": "dc33d7ee", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -4936,16 +5218,20 @@ }, { "cell_type": "markdown", - "id": "c6e46e3a", - "metadata": {}, + "id": "0ffb1c46", + "metadata": { + "editable": true + }, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] }, { "cell_type": "markdown", - "id": "82d0b1d4", - "metadata": {}, + "id": "6cf92025", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -4954,8 +5240,10 @@ }, { "cell_type": "markdown", - "id": "2a20d514", - "metadata": {}, + "id": "c21c0c0a", + "metadata": { + "editable": true + }, "source": [ "## What does it mean?\n", "\n", @@ -4966,8 +5254,10 @@ }, { "cell_type": "markdown", - "id": "1df4e973", - "metadata": {}, + "id": "48dbf778", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -4976,8 +5266,10 @@ }, { "cell_type": "markdown", - "id": "775b345d", - "metadata": {}, + "id": "3f8f57cc", + "metadata": { + "editable": true + }, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -4996,8 +5288,10 @@ }, { "cell_type": "markdown", - "id": "ea73a27d", - "metadata": {}, + "id": "199ea962", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -5006,8 +5300,10 @@ }, { "cell_type": "markdown", - "id": "5073c271", - "metadata": {}, + "id": "f5433e15", + "metadata": { + "editable": true + }, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", @@ -5019,8 +5315,10 @@ }, { "cell_type": "markdown", - "id": "e82ec69f", - "metadata": {}, + "id": "ce1511d3", + "metadata": { + "editable": true + }, "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", @@ -5029,8 +5327,10 @@ }, { "cell_type": "markdown", - "id": "6e11ce54", - "metadata": {}, + "id": "31981527", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -5039,16 +5339,20 @@ }, { "cell_type": "markdown", - "id": "969b47a9", - "metadata": {}, + "id": "ccb547be", + "metadata": { + "editable": true + }, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] }, { "cell_type": "markdown", - "id": "93eb339d", - "metadata": {}, + "id": "c9a6f756", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -5057,16 +5361,20 @@ }, { "cell_type": "markdown", - "id": "59d3c243", - "metadata": {}, + "id": "272b8d77", + "metadata": { + "editable": true + }, "source": [ "leading to" ] }, { "cell_type": "markdown", - "id": "88905c12", - "metadata": {}, + "id": "4596a399", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -5075,16 +5383,20 @@ }, { "cell_type": "markdown", - "id": "6cc970fd", - "metadata": {}, + "id": "e15dedbf", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] }, { "cell_type": "markdown", - "id": "079ac9ba", - "metadata": {}, + "id": "fe9dab27", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -5093,8 +5405,10 @@ }, { "cell_type": "markdown", - "id": "2633ab97", - "metadata": {}, + "id": "894497fb", + "metadata": { + "editable": true + }, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -5108,8 +5422,10 @@ }, { "cell_type": "markdown", - "id": "dfeab33f", - "metadata": {}, + "id": "843092e6", + "metadata": { + "editable": true + }, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -5119,8 +5435,10 @@ }, { "cell_type": "markdown", - "id": "0f33a0d4", - "metadata": {}, + "id": "4af49ad4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -5129,16 +5447,20 @@ }, { "cell_type": "markdown", - "id": "9eeab640", - "metadata": {}, + "id": "ed536a1b", + "metadata": { + "editable": true + }, "source": [ "or we can state it as" ] }, { "cell_type": "markdown", - "id": "8fa01e32", - "metadata": {}, + "id": "9e199dbf", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -5148,16 +5470,20 @@ }, { "cell_type": "markdown", - "id": "387a69ff", - "metadata": {}, + "id": "29d15cda", + "metadata": { + "editable": true + }, "source": [ "where we have used the definition of a norm-2 vector, that is" ] }, { "cell_type": "markdown", - "id": "b53f3c66", - "metadata": {}, + "id": "a9b91956", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -5166,8 +5492,10 @@ }, { "cell_type": "markdown", - "id": "db9ef814", - "metadata": {}, + "id": "f21d533a", + "metadata": { + "editable": true + }, "source": [ "By minimizing the above equation with respect to the parameters\n", "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", @@ -5177,8 +5505,10 @@ }, { "cell_type": "markdown", - "id": "bd9eff5d", - "metadata": {}, + "id": "4f218703", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -5188,8 +5518,10 @@ }, { "cell_type": "markdown", - "id": "5ef7b233", - "metadata": {}, + "id": "6bc40dcd", + "metadata": { + "editable": true + }, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -5198,8 +5530,10 @@ }, { "cell_type": "markdown", - "id": "254828fb", - "metadata": {}, + "id": "fb27a2e6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -5208,16 +5542,20 @@ }, { "cell_type": "markdown", - "id": "535f6d77", - "metadata": {}, + "id": "d285f7c7", + "metadata": { + "editable": true + }, "source": [ "we have a new optimization equation" ] }, { "cell_type": "markdown", - "id": "1b39ea82", - "metadata": {}, + "id": "57eb55da", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -5227,8 +5565,10 @@ }, { "cell_type": "markdown", - "id": "ca9ed98e", - "metadata": {}, + "id": "94f2f142", + "metadata": { + "editable": true + }, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -5237,8 +5577,10 @@ }, { "cell_type": "markdown", - "id": "2d621f3c", - "metadata": {}, + "id": "602c806b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -5247,8 +5589,10 @@ }, { "cell_type": "markdown", - "id": "1525b1e5", - "metadata": {}, + "id": "c01a7be8", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -5257,8 +5601,10 @@ }, { "cell_type": "markdown", - "id": "566ddc35", - "metadata": {}, + "id": "6bbd89f5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -5267,8 +5613,10 @@ }, { "cell_type": "markdown", - "id": "5ec1de70", - "metadata": {}, + "id": "d5507c15", + "metadata": { + "editable": true + }, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -5279,8 +5627,10 @@ }, { "cell_type": "markdown", - "id": "7b615513", - "metadata": {}, + "id": "d5eb5c88", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -5289,16 +5639,20 @@ }, { "cell_type": "markdown", - "id": "7af6c783", - "metadata": {}, + "id": "d59c7c4d", + "metadata": { + "editable": true + }, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] }, { "cell_type": "markdown", - "id": "058050d7", - "metadata": {}, + "id": "81d0dd7f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -5307,8 +5661,10 @@ }, { "cell_type": "markdown", - "id": "7c671f5e", - "metadata": {}, + "id": "6807c376", + "metadata": { + "editable": true + }, "source": [ "with $t$ a finite positive number. \n", "\n", @@ -5317,8 +5673,10 @@ }, { "cell_type": "markdown", - "id": "e79b23a7", - "metadata": {}, + "id": "89c94248", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -5327,8 +5685,10 @@ }, { "cell_type": "markdown", - "id": "d44102d2", - "metadata": {}, + "id": "24f4e74f", + "metadata": { + "editable": true + }, "source": [ "In many textbooks the $1/n$ term is often omitted. Note that a library like **Scikit-Learn** does not include the $1/n$ factor in the setup of the cost function.\n", "\n", @@ -5337,8 +5697,10 @@ }, { "cell_type": "markdown", - "id": "b1f6463c", - "metadata": {}, + "id": "eac49311", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -5347,8 +5709,10 @@ }, { "cell_type": "markdown", - "id": "ed757eb1", - "metadata": {}, + "id": "8ad4bc0d", + "metadata": { + "editable": true + }, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", @@ -5364,8 +5728,10 @@ }, { "cell_type": "markdown", - "id": "905b7d64", - "metadata": {}, + "id": "af4dd7e2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -5374,16 +5740,20 @@ }, { "cell_type": "markdown", - "id": "9f4de44d", - "metadata": {}, + "id": "92b983c6", + "metadata": { + "editable": true + }, "source": [ "For Ridge regression this becomes" ] }, { "cell_type": "markdown", - "id": "62a13ed4", - "metadata": {}, + "id": "f9d22662", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -5392,16 +5762,20 @@ }, { "cell_type": "markdown", - "id": "3c17eb55", - "metadata": {}, + "id": "0097f163", + "metadata": { + "editable": true + }, "source": [ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." ] }, { "cell_type": "markdown", - "id": "54630506", - "metadata": {}, + "id": "073824a0", + "metadata": { + "editable": true + }, "source": [ "## Interpreting the Ridge results\n", "\n", @@ -5410,8 +5784,10 @@ }, { "cell_type": "markdown", - "id": "0d18500c", - "metadata": {}, + "id": "399bdc1b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -5420,8 +5796,10 @@ }, { "cell_type": "markdown", - "id": "2a493e6a", - "metadata": {}, + "id": "d9d9fa7a", + "metadata": { + "editable": true + }, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -5434,8 +5812,10 @@ }, { "cell_type": "markdown", - "id": "00ecc387", - "metadata": {}, + "id": "40b2c49b", + "metadata": { + "editable": true + }, "source": [ "## More interpretations\n", "\n", @@ -5444,8 +5824,10 @@ }, { "cell_type": "markdown", - "id": "1ec18d73", - "metadata": {}, + "id": "6e949d98", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -5454,16 +5836,20 @@ }, { "cell_type": "markdown", - "id": "71435169", - "metadata": {}, + "id": "5a004c53", + "metadata": { + "editable": true + }, "source": [ "In this case the standard OLS results in" ] }, { "cell_type": "markdown", - "id": "61c878a8", - "metadata": {}, + "id": "64e1d6bb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", @@ -5472,16 +5858,20 @@ }, { "cell_type": "markdown", - "id": "30f48730", - "metadata": {}, + "id": "5cec9b54", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "a79a72a8", - "metadata": {}, + "id": "71516f90", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -5490,8 +5880,10 @@ }, { "cell_type": "markdown", - "id": "eafc2e6c", - "metadata": {}, + "id": "50e288e9", + "metadata": { + "editable": true + }, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -5505,8 +5897,10 @@ }, { "cell_type": "markdown", - "id": "bebba502", - "metadata": {}, + "id": "6c82431c", + "metadata": { + "editable": true + }, "source": [ "## Deriving the Lasso Regression Equations\n", "\n", @@ -5515,8 +5909,10 @@ }, { "cell_type": "markdown", - "id": "65667551", - "metadata": {}, + "id": "5ebc1428", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -5525,16 +5921,20 @@ }, { "cell_type": "markdown", - "id": "cdba7875", - "metadata": {}, + "id": "8f198345", + "metadata": { + "editable": true + }, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] }, { "cell_type": "markdown", - "id": "8e0d7210", - "metadata": {}, + "id": "443b3080", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -5543,16 +5943,20 @@ }, { "cell_type": "markdown", - "id": "0840d7b4", - "metadata": {}, + "id": "e11ec4ea", + "metadata": { + "editable": true + }, "source": [ "we have that the derivative of the cost function is" ] }, { "cell_type": "markdown", - "id": "ba1ea89b", - "metadata": {}, + "id": "1d640caa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -5561,16 +5965,20 @@ }, { "cell_type": "markdown", - "id": "2eba4be0", - "metadata": {}, + "id": "e023f076", + "metadata": { + "editable": true + }, "source": [ "and reordering we have" ] }, { "cell_type": "markdown", - "id": "c5f3e12e", - "metadata": {}, + "id": "a4204b50", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -5579,32 +5987,16 @@ }, { "cell_type": "markdown", - "id": "4680607a", - "metadata": {}, + "id": "7c9cc257", + "metadata": { + "editable": true + }, "source": [ "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 174e3aac3..61f1cf1ee 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -19,7 +19,9 @@ o Thursday: Ridge and Lasso regression and Singular Value Decomposition === Reading recommendations: === o See lecture notes for week 35 at URL:"https://compphysics.github.io/MachineLearning/doc/web/course.html" - o CMB sections 1.1 and 3.1 + o Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra and sections 3.1-3.10 on elements of statistics + o Hastie, Tibshirani and Friedman, The elements of statistical learning, sections 3.1-3.4 + @@ -1051,7 +1053,7 @@ target/output variables when all other predictors are set to zero. Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. -Furthermore, in for example Ridge and Lasso regression, the default solutions +Furthermore, in for example Ridge and Lasso regression (to be discussed in moe detail next week), the default solutions from the library _Scikit-Learn_ (when not shrinking $\beta_0$) for the unknown parameters $\bm{\beta}$, are derived under the assumption that both $\bm{y}$ and $\bm{X}$ are zero centered, that is we subtract the mean values. @@ -1327,12 +1329,12 @@ The intercept is the value of our output/target variable when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). Printing the MSE, we see first that both methods give the same MSE, as -they should. However, when we move to for example Ridge regression, +they should. However, when we move to for example Ridge regression (discussed next week), the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the regularization term does not include $\beta_0$. For different values -of $\lambda$, this may lead to differeing MSE values. +of $\lambda$, this may lead to differing MSE values. To remind the reader, the regularization term, with the intercept in Ridge regression is given by !bt @@ -1354,7 +1356,13 @@ For Lasso regression we have \] !et -It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get an MSE which then contains the intercept. This becomes more important when we discuss Ridge and Lasso regression next week. +It means that, when scaling the design matrix and the outputs/targets, +by subtracting the mean values, we have an optimization problem which +is not penalized by the intercept. The MSE value can then be smaller +since it focuses only on the remaining quantities. If we however bring +back the intercept, we will get an MSE which then contains the +intercept. This becomes more important when we discuss Ridge and Lasso +regression next week.