From 14ed7e46cddcdef46ff362e59626577e15aaa349 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Sun, 31 Aug 2025 21:13:22 +0200 Subject: [PATCH] corrected beta--> theta --- doc/pub/week35/html/._week35-bs062.html | 108 +- doc/pub/week35/html/week35-reveal.html | 108 +- doc/pub/week35/html/week35-solarized.html | 108 +- doc/pub/week35/html/week35.html | 108 +- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 191 bytes doc/pub/week35/ipynb/week35.ipynb | 1126 ++++++++---------- doc/src/week35/week35.do.txt | 112 +- 7 files changed, 772 insertions(+), 898 deletions(-) diff --git a/doc/pub/week35/html/._week35-bs062.html b/doc/pub/week35/html/._week35-bs062.html index 0622367f7..7619ecbee 100644 --- a/doc/pub/week35/html/._week35-bs062.html +++ b/doc/pub/week35/html/._week35-bs062.html @@ -348,8 +348,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters -\( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and +from the library Scikit-Learn (when not shrinking \( \theta_0 \)) for the unknown parameters +\( \boldsymbol{\theta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.

@@ -433,7 +433,7 @@ simplicity, we will focus on ordinary regression, as done in the above example.

The cost/loss function for regression is

$$ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,. $$

Recall also that we use the squared value. This expression can lead to an @@ -441,37 +441,37 @@ increased penalty for higher differences between predicted and output/target values.

-

What we have done is to single out the \( \beta_0 \) term in the +

What we have done is to single out the \( \theta_0 \) term in the definition of the mean squared error (MSE). The design matrix \( X \) does in this case not contain any intercept column. When we take the -derivative with respect to \( \beta_0 \), we want the derivative to obey +derivative with respect to \( \theta_0 \), we want the derivative to obey

$$ -\frac{\partial C}{\partial \beta_j} = 0, +\frac{\partial C}{\partial \theta_j} = 0, $$ -

for all \( j \). For \( \beta_0 \) we have

+

for all \( j \). For \( \theta_0 \) we have

$$ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right). $$

Multiplying away the constant \( 2/n \), we obtain

$$ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j. $$ -

Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \). -Our result for \( \beta_0 \) simplifies then to +

Let us specialize first to the case where we have only two parameters \( \theta_0 \) and \( \theta_1 \). +Our result for \( \theta_0 \) simplifies then to

$$ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. +n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1. $$

We obtain then

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. $$

If we define

@@ -486,17 +486,17 @@ $$

we have

$$ -\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}. +\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}. $$ -

In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have

+

In the general case with more parameters than \( \theta_0 \) and \( \theta_1 \), we have

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j. $$

We can rewrite the latter equation as

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j, +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j, $$

where we have defined

@@ -508,22 +508,22 @@ $$

Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)

$$ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}). $$ -

If we minimize with respect to \( \boldsymbol{\beta} \) we have then

+

If we minimize with respect to \( \boldsymbol{\theta} \) we have then

$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, $$

where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \) and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).

-

For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then

+

For Ridge regression we need to add \( \lambda \boldsymbol{\theta}^T\boldsymbol{\theta} \) to the cost function and get then

$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. $$

What does this mean? And why do we insist on all this? Let us look at some examples.

@@ -552,15 +552,15 @@ np.random.return np.sum((y_data-y_model)**2)/n -def fit_beta(X, y): +def fit_theta(X, y): return np.linalg.pinv(X.T @ X) @ X.T @ y -true_beta = [2, 0.5, 3.7] +true_theta = [2, 0.5, 3.7] x = np.linspace(0, 1, 11) y = np.sum( - np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0 + np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0 ) + 0.1 * np.random.normal(size=len(x)) degree = 3 @@ -570,15 +570,15 @@ X = np.z for p in range(degree): X[:, p] = x ** p -beta = fit_beta(X, y) +theta = fit_theta(X, y) # Intercept is included in the design matrix skl = LinearRegression(fit_intercept=False).fit(X, y) -print(f"True beta: {true_beta}") -print(f"Fitted beta: {beta}") -print(f"Sklearn fitted beta: {skl.coef_}") -ypredictOwn = X @ beta +print(f"True theta: {true_theta}") +print(f"Fitted theta: {theta}") +print(f"Sklearn fitted theta: {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with intercept column") print(MSE(y,ypredictOwn)) @@ -588,7 +588,7 @@ ypredictSKL = skl.figure() plt.scatter(x, y, label="Data") -plt.plot(x, X @ beta, label="Fit") +plt.plot(x, X @ theta, label="Fit") plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)") @@ -605,21 +605,21 @@ skl = LinearRegression(fit_intercept= np.average(y, axis=0) X_offset = np.average(X, axis=0) -beta = fit_beta(X - X_offset, y - y_offset) -intercept = np.mean(y_offset - X_offset @ beta) +theta = fit_theta(X - X_offset, y - y_offset) +intercept = np.mean(y_offset - X_offset @ theta) print(f"Manual intercept: {intercept}") -print(f"Fitted beta (without intercept): {beta}") +print(f"Fitted theta (without intercept): {theta}") print(f"Sklearn intercept: {skl.intercept_}") -print(f"Sklearn fitted beta (without intercept): {skl.coef_}") -ypredictOwn = X @ beta +print(f"Sklearn fitted theta (without intercept): {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with Manual intercept") print(MSE(y,ypredictOwn+intercept)) print(f"MSE with Sklearn intercept") print(MSE(y,ypredictSKL)) -plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)") +plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)") plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)") plt.grid() plt.legend() @@ -649,23 +649,23 @@ they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the -regularization term does not include \( \beta_0 \). For different values +regularization term does not include \( \theta_0 \). For different values of \( \lambda \), this may lead to different MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression, is given by

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2, $$

but when we take out the intercept, this equation becomes

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2. $$

For Lasso regression we have

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert. $$

It means that, when scaling the design matrix and the outputs/targets, @@ -721,20 +721,20 @@ MSERidgePredict = np= np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # Note: we include the intercept column and no scaling RegRidge = linear_model.Ridge(lmb,fit_intercept=False) RegRidge.fit(X_train,y_train) # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta + ytildeOwnRidge = X_train @ OwnRidgeTheta + ypredictOwnRidge = X_test @ OwnRidgeTheta ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) @@ -825,18 +825,18 @@ MSERidgePredict = np= np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) - intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data #Add intercept to prediction - ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) #Intercept is given by mean of target variable - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) #Intercept is given by mean of target variable + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print('Intercept from own implementation:') print(intercept_) @@ -874,7 +874,7 @@ plt.show()

We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value -\( \beta_0 \) in the fitting. This applies to Lasso regularization as +\( \theta_0 \) in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure.

diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index fdeb4ebb3..67e6542ce 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -3016,8 +3016,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters -\( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and +from the library Scikit-Learn (when not shrinking \( \theta_0 \)) for the unknown parameters +\( \boldsymbol{\theta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.

@@ -3102,7 +3102,7 @@ simplicity, we will focus on ordinary regression, as done in the above example.

The cost/loss function for regression is

 
$$ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,. $$

 
@@ -3111,46 +3111,46 @@ increased penalty for higher differences between predicted and output/target values.

-

What we have done is to single out the \( \beta_0 \) term in the +

What we have done is to single out the \( \theta_0 \) term in the definition of the mean squared error (MSE). The design matrix \( X \) does in this case not contain any intercept column. When we take the -derivative with respect to \( \beta_0 \), we want the derivative to obey +derivative with respect to \( \theta_0 \), we want the derivative to obey

 
$$ -\frac{\partial C}{\partial \beta_j} = 0, +\frac{\partial C}{\partial \theta_j} = 0, $$

 
-

for all \( j \). For \( \beta_0 \) we have

+

for all \( j \). For \( \theta_0 \) we have

 
$$ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right). $$

 

Multiplying away the constant \( 2/n \), we obtain

 
$$ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j. $$

 
-

Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \). -Our result for \( \beta_0 \) simplifies then to +

Let us specialize first to the case where we have only two parameters \( \theta_0 \) and \( \theta_1 \). +Our result for \( \theta_0 \) simplifies then to

 
$$ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. +n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1. $$

 

We obtain then

 
$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. $$

 
@@ -3171,21 +3171,21 @@ $$

we have

 
$$ -\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}. +\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}. $$

 
-

In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have

+

In the general case with more parameters than \( \theta_0 \) and \( \theta_1 \), we have

 
$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j. $$

 

We can rewrite the latter equation as

 
$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j, +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j, $$

 
@@ -3201,15 +3201,15 @@ $$

Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)

 
$$ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}). $$

 
-

If we minimize with respect to \( \boldsymbol{\beta} \) we have then

+

If we minimize with respect to \( \boldsymbol{\theta} \) we have then

 
$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, $$

 
@@ -3217,10 +3217,10 @@ $$ and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).

-

For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then

+

For Ridge regression we need to add \( \lambda \boldsymbol{\theta}^T\boldsymbol{\theta} \) to the cost function and get then

 
$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. $$

 
@@ -3250,15 +3250,15 @@ np.random.seed(2021) return np.sum((y_data-y_model)**2)/n -def fit_beta(X, y): +def fit_theta(X, y): return np.linalg.pinv(X.T @ X) @ X.T @ y -true_beta = [2, 0.5, 3.7] +true_theta = [2, 0.5, 3.7] x = np.linspace(0, 1, 11) y = np.sum( - np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0 + np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0 ) + 0.1 * np.random.normal(size=len(x)) degree = 3 @@ -3268,15 +3268,15 @@ X = np.zeros((len(x), degree)) for p in range(degree): X[:, p] = x ** p -beta = fit_beta(X, y) +theta = fit_theta(X, y) # Intercept is included in the design matrix skl = LinearRegression(fit_intercept=False).fit(X, y) -print(f"True beta: {true_beta}") -print(f"Fitted beta: {beta}") -print(f"Sklearn fitted beta: {skl.coef_}") -ypredictOwn = X @ beta +print(f"True theta: {true_theta}") +print(f"Fitted theta: {theta}") +print(f"Sklearn fitted theta: {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with intercept column") print(MSE(y,ypredictOwn)) @@ -3286,7 +3286,7 @@ ypredictSKL = skl.predict(X) plt.figure() plt.scatter(x, y, label="Data") -plt.plot(x, X @ beta, label="Fit") +plt.plot(x, X @ theta, label="Fit") plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)") @@ -3303,21 +3303,21 @@ skl = LinearRegression(fit_intercept=0) X_offset = np.average(X, axis=0) -beta = fit_beta(X - X_offset, y - y_offset) -intercept = np.mean(y_offset - X_offset @ beta) +theta = fit_theta(X - X_offset, y - y_offset) +intercept = np.mean(y_offset - X_offset @ theta) print(f"Manual intercept: {intercept}") -print(f"Fitted beta (without intercept): {beta}") +print(f"Fitted theta (without intercept): {theta}") print(f"Sklearn intercept: {skl.intercept_}") -print(f"Sklearn fitted beta (without intercept): {skl.coef_}") -ypredictOwn = X @ beta +print(f"Sklearn fitted theta (without intercept): {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with Manual intercept") print(MSE(y,ypredictOwn+intercept)) print(f"MSE with Sklearn intercept") print(MSE(y,ypredictSKL)) -plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)") +plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)") plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)") plt.grid() plt.legend() @@ -3347,28 +3347,28 @@ they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the -regularization term does not include \( \beta_0 \). For different values +regularization term does not include \( \theta_0 \). For different values of \( \lambda \), this may lead to different MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression, is given by

 
$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2, $$

 

but when we take out the intercept, this equation becomes

 
$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2. $$

 

For Lasso regression we have

 
$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert. $$

 
@@ -3425,20 +3425,20 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # Note: we include the intercept column and no scaling RegRidge = linear_model.Ridge(lmb,fit_intercept=False) RegRidge.fit(X_train,y_train) # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta + ytildeOwnRidge = X_train @ OwnRidgeTheta + ypredictOwnRidge = X_test @ OwnRidgeTheta ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) @@ -3529,18 +3529,18 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) - intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data #Add intercept to prediction - ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) #Intercept is given by mean of target variable - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) #Intercept is given by mean of target variable + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print('Intercept from own implementation:') print(intercept_) @@ -3578,7 +3578,7 @@ plt.show()

We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value -\( \beta_0 \) in the fitting. This applies to Lasso regularization as +\( \theta_0 \) in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure.

diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 9dfe9620b..1ac1f4fa4 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -2775,8 +2775,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters -\( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and +from the library Scikit-Learn (when not shrinking \( \theta_0 \)) for the unknown parameters +\( \boldsymbol{\theta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.

@@ -2860,7 +2860,7 @@ simplicity, we will focus on ordinary regression, as done in the above example.

The cost/loss function for regression is

$$ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,. $$

Recall also that we use the squared value. This expression can lead to an @@ -2868,37 +2868,37 @@ increased penalty for higher differences between predicted and output/target values.

-

What we have done is to single out the \( \beta_0 \) term in the +

What we have done is to single out the \( \theta_0 \) term in the definition of the mean squared error (MSE). The design matrix \( X \) does in this case not contain any intercept column. When we take the -derivative with respect to \( \beta_0 \), we want the derivative to obey +derivative with respect to \( \theta_0 \), we want the derivative to obey

$$ -\frac{\partial C}{\partial \beta_j} = 0, +\frac{\partial C}{\partial \theta_j} = 0, $$ -

for all \( j \). For \( \beta_0 \) we have

+

for all \( j \). For \( \theta_0 \) we have

$$ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right). $$

Multiplying away the constant \( 2/n \), we obtain

$$ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j. $$ -

Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \). -Our result for \( \beta_0 \) simplifies then to +

Let us specialize first to the case where we have only two parameters \( \theta_0 \) and \( \theta_1 \). +Our result for \( \theta_0 \) simplifies then to

$$ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. +n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1. $$

We obtain then

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. $$

If we define

@@ -2913,17 +2913,17 @@ $$

we have

$$ -\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}. +\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}. $$ -

In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have

+

In the general case with more parameters than \( \theta_0 \) and \( \theta_1 \), we have

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j. $$

We can rewrite the latter equation as

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j, +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j, $$

where we have defined

@@ -2935,22 +2935,22 @@ $$

Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)

$$ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}). $$ -

If we minimize with respect to \( \boldsymbol{\beta} \) we have then

+

If we minimize with respect to \( \boldsymbol{\theta} \) we have then

$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, $$

where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \) and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).

-

For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then

+

For Ridge regression we need to add \( \lambda \boldsymbol{\theta}^T\boldsymbol{\theta} \) to the cost function and get then

$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. $$

What does this mean? And why do we insist on all this? Let us look at some examples.

@@ -2979,15 +2979,15 @@ np.random.seed(2021) return np.sum((y_data-y_model)**2)/n -def fit_beta(X, y): +def fit_theta(X, y): return np.linalg.pinv(X.T @ X) @ X.T @ y -true_beta = [2, 0.5, 3.7] +true_theta = [2, 0.5, 3.7] x = np.linspace(0, 1, 11) y = np.sum( - np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0 + np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0 ) + 0.1 * np.random.normal(size=len(x)) degree = 3 @@ -2997,15 +2997,15 @@ X = np.zeros((len(x), degree)) for p in range(degree): X[:, p] = x ** p -beta = fit_beta(X, y) +theta = fit_theta(X, y) # Intercept is included in the design matrix skl = LinearRegression(fit_intercept=False).fit(X, y) -print(f"True beta: {true_beta}") -print(f"Fitted beta: {beta}") -print(f"Sklearn fitted beta: {skl.coef_}") -ypredictOwn = X @ beta +print(f"True theta: {true_theta}") +print(f"Fitted theta: {theta}") +print(f"Sklearn fitted theta: {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with intercept column") print(MSE(y,ypredictOwn)) @@ -3015,7 +3015,7 @@ ypredictSKL = skl.predict(X) plt.figure() plt.scatter(x, y, label="Data") -plt.plot(x, X @ beta, label="Fit") +plt.plot(x, X @ theta, label="Fit") plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)") @@ -3032,21 +3032,21 @@ skl = LinearRegression(fit_intercept=0) X_offset = np.average(X, axis=0) -beta = fit_beta(X - X_offset, y - y_offset) -intercept = np.mean(y_offset - X_offset @ beta) +theta = fit_theta(X - X_offset, y - y_offset) +intercept = np.mean(y_offset - X_offset @ theta) print(f"Manual intercept: {intercept}") -print(f"Fitted beta (without intercept): {beta}") +print(f"Fitted theta (without intercept): {theta}") print(f"Sklearn intercept: {skl.intercept_}") -print(f"Sklearn fitted beta (without intercept): {skl.coef_}") -ypredictOwn = X @ beta +print(f"Sklearn fitted theta (without intercept): {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with Manual intercept") print(MSE(y,ypredictOwn+intercept)) print(f"MSE with Sklearn intercept") print(MSE(y,ypredictSKL)) -plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)") +plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)") plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)") plt.grid() plt.legend() @@ -3076,23 +3076,23 @@ they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the -regularization term does not include \( \beta_0 \). For different values +regularization term does not include \( \theta_0 \). For different values of \( \lambda \), this may lead to different MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression, is given by

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2, $$

but when we take out the intercept, this equation becomes

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2. $$

For Lasso regression we have

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert. $$

It means that, when scaling the design matrix and the outputs/targets, @@ -3148,20 +3148,20 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # Note: we include the intercept column and no scaling RegRidge = linear_model.Ridge(lmb,fit_intercept=False) RegRidge.fit(X_train,y_train) # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta + ytildeOwnRidge = X_train @ OwnRidgeTheta + ypredictOwnRidge = X_test @ OwnRidgeTheta ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) @@ -3252,18 +3252,18 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) - intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data #Add intercept to prediction - ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) #Intercept is given by mean of target variable - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) #Intercept is given by mean of target variable + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print('Intercept from own implementation:') print(intercept_) @@ -3301,7 +3301,7 @@ plt.show()

We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value -\( \beta_0 \) in the fitting. This applies to Lasso regularization as +\( \theta_0 \) in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure.

diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 2d2eed06a..92193961b 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -2852,8 +2852,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library Scikit-Learn (when not shrinking \( \beta_0 \)) for the unknown parameters -\( \boldsymbol{\beta} \), are derived under the assumption that both \( \boldsymbol{y} \) and +from the library Scikit-Learn (when not shrinking \( \theta_0 \)) for the unknown parameters +\( \boldsymbol{\theta} \), are derived under the assumption that both \( \boldsymbol{y} \) and \( \boldsymbol{X} \) are zero centered, that is we subtract the mean values.

@@ -2937,7 +2937,7 @@ simplicity, we will focus on ordinary regression, as done in the above example.

The cost/loss function for regression is

$$ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,. $$

Recall also that we use the squared value. This expression can lead to an @@ -2945,37 +2945,37 @@ increased penalty for higher differences between predicted and output/target values.

-

What we have done is to single out the \( \beta_0 \) term in the +

What we have done is to single out the \( \theta_0 \) term in the definition of the mean squared error (MSE). The design matrix \( X \) does in this case not contain any intercept column. When we take the -derivative with respect to \( \beta_0 \), we want the derivative to obey +derivative with respect to \( \theta_0 \), we want the derivative to obey

$$ -\frac{\partial C}{\partial \beta_j} = 0, +\frac{\partial C}{\partial \theta_j} = 0, $$ -

for all \( j \). For \( \beta_0 \) we have

+

for all \( j \). For \( \theta_0 \) we have

$$ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right). $$

Multiplying away the constant \( 2/n \), we obtain

$$ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j. $$ -

Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \). -Our result for \( \beta_0 \) simplifies then to +

Let us specialize first to the case where we have only two parameters \( \theta_0 \) and \( \theta_1 \). +Our result for \( \theta_0 \) simplifies then to

$$ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. +n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1. $$

We obtain then

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. $$

If we define

@@ -2990,17 +2990,17 @@ $$

we have

$$ -\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}. +\theta_0 = \mu_y - \theta_1\mu_{\boldsymbol{x}_1}. $$ -

In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have

+

In the general case with more parameters than \( \theta_0 \) and \( \theta_1 \), we have

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j. $$

We can rewrite the latter equation as

$$ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j, +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\theta_j, $$

where we have defined

@@ -3012,22 +3012,22 @@ $$

Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)

$$ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}). $$ -

If we minimize with respect to \( \boldsymbol{\beta} \) we have then

+

If we minimize with respect to \( \boldsymbol{\theta} \) we have then

$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, $$

where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \) and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).

-

For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then

+

For Ridge regression we need to add \( \lambda \boldsymbol{\theta}^T\boldsymbol{\theta} \) to the cost function and get then

$$ -\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\hat{\boldsymbol{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. $$

What does this mean? And why do we insist on all this? Let us look at some examples.

@@ -3056,15 +3056,15 @@ np.random.return np.sum((y_data-y_model)**2)/n -def fit_beta(X, y): +def fit_theta(X, y): return np.linalg.pinv(X.T @ X) @ X.T @ y -true_beta = [2, 0.5, 3.7] +true_theta = [2, 0.5, 3.7] x = np.linspace(0, 1, 11) y = np.sum( - np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0 + np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0 ) + 0.1 * np.random.normal(size=len(x)) degree = 3 @@ -3074,15 +3074,15 @@ X = np.z for p in range(degree): X[:, p] = x ** p -beta = fit_beta(X, y) +theta = fit_theta(X, y) # Intercept is included in the design matrix skl = LinearRegression(fit_intercept=False).fit(X, y) -print(f"True beta: {true_beta}") -print(f"Fitted beta: {beta}") -print(f"Sklearn fitted beta: {skl.coef_}") -ypredictOwn = X @ beta +print(f"True theta: {true_theta}") +print(f"Fitted theta: {theta}") +print(f"Sklearn fitted theta: {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with intercept column") print(MSE(y,ypredictOwn)) @@ -3092,7 +3092,7 @@ ypredictSKL = skl.figure() plt.scatter(x, y, label="Data") -plt.plot(x, X @ beta, label="Fit") +plt.plot(x, X @ theta, label="Fit") plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)") @@ -3109,21 +3109,21 @@ skl = LinearRegression(fit_intercept= np.average(y, axis=0) X_offset = np.average(X, axis=0) -beta = fit_beta(X - X_offset, y - y_offset) -intercept = np.mean(y_offset - X_offset @ beta) +theta = fit_theta(X - X_offset, y - y_offset) +intercept = np.mean(y_offset - X_offset @ theta) print(f"Manual intercept: {intercept}") -print(f"Fitted beta (without intercept): {beta}") +print(f"Fitted theta (without intercept): {theta}") print(f"Sklearn intercept: {skl.intercept_}") -print(f"Sklearn fitted beta (without intercept): {skl.coef_}") -ypredictOwn = X @ beta +print(f"Sklearn fitted theta (without intercept): {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with Manual intercept") print(MSE(y,ypredictOwn+intercept)) print(f"MSE with Sklearn intercept") print(MSE(y,ypredictSKL)) -plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)") +plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)") plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)") plt.grid() plt.legend() @@ -3153,23 +3153,23 @@ they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the -regularization term does not include \( \beta_0 \). For different values +regularization term does not include \( \theta_0 \). For different values of \( \lambda \), this may lead to different MSE values.

To remind the reader, the regularization term, with the intercept in Ridge regression, is given by

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2, $$

but when we take out the intercept, this equation becomes

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2. $$

For Lasso regression we have

$$ -\lambda \vert\vert \boldsymbol{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +\lambda \vert\vert \boldsymbol{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert. $$

It means that, when scaling the design matrix and the outputs/targets, @@ -3225,20 +3225,20 @@ MSERidgePredict = np= np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # Note: we include the intercept column and no scaling RegRidge = linear_model.Ridge(lmb,fit_intercept=False) RegRidge.fit(X_train,y_train) # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta + ytildeOwnRidge = X_train @ OwnRidgeTheta + ypredictOwnRidge = X_test @ OwnRidgeTheta ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) @@ -3329,18 +3329,18 @@ MSERidgePredict = np= np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) - intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data #Add intercept to prediction - ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) #Intercept is given by mean of target variable - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) #Intercept is given by mean of target variable + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print('Intercept from own implementation:') print(intercept_) @@ -3378,7 +3378,7 @@ plt.show()

We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value -\( \beta_0 \) in the fitting. This applies to Lasso regularization as +\( \theta_0 \) in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure.

diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 8ab1fc8da2c2e538054b08b94741921d513123b0..f63b6f1e480cc90d3321b8af7ad5b0be29d027e6 100644 GIT binary patch literal 191 zcmV;w06_mAiwFS0p0sEH1MSbv3c@f92k@Qu6nTQtxOF=V?%+WX@dYZ2uFkcy?a;lw z`v5&DUWN$$UH*iGknHm1YLiFq?thes=l6p7Sj~$O6_Drw9GylZClor<8&Xrn$(vA*VS8yY$Vjjt+QwJzC t*`Wo7uB|i#LUkaDLOQ9HxP*<-CnIVbg};8r^E}V{+5^+Ms&D`Z004O^UupmV literal 192 zcmV;x06+g9iwFQwwXSFY1MSbv3c@f92k@Qu6nTQtuIoGt?%+WX@dY}TxjNU*wnO*! z?gR9sco`z}cli?%LUPE~n_U*Uy9*XW2uYI!#$2U*N)pfYgi;Qa6V5n`c}#&&#^WRa zvfN28opr(tr!>_Wl|}V#ZWt@e4}0cU;F*8oSV;@peeWu*Kxv1$)-~J^YsMql_9}-$ uqa9md@Y+d(AXE>cD5R5GiA&fTeKMl4QTXd;JkRsIuRQ?saq$xX2mk;I9a;$h diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 5ec2cf291..0300d90b1 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "39c51fdf", + "id": "e5c865ad", "metadata": { "editable": true }, @@ -14,7 +14,7 @@ }, { "cell_type": "markdown", - "id": "07a23085", + "id": "725e6917", "metadata": { "editable": true }, @@ -27,7 +27,7 @@ }, { "cell_type": "markdown", - "id": "35e04f23", + "id": "7aab4a45", "metadata": { "editable": true }, @@ -49,7 +49,7 @@ }, { "cell_type": "markdown", - "id": "107afb83", + "id": "dd9d9be5", "metadata": { "editable": true }, @@ -71,7 +71,7 @@ }, { "cell_type": "markdown", - "id": "7e1893d1", + "id": "68a8aa98", "metadata": { "editable": true }, @@ -104,7 +104,7 @@ }, { "cell_type": "markdown", - "id": "504523f6", + "id": "b2b56f10", "metadata": { "editable": true }, @@ -120,7 +120,7 @@ }, { "cell_type": "markdown", - "id": "2e81f798", + "id": "d0ffdd72", "metadata": { "editable": true }, @@ -132,7 +132,7 @@ }, { "cell_type": "markdown", - "id": "2a107826", + "id": "f230ed73", "metadata": { "editable": true }, @@ -142,7 +142,7 @@ }, { "cell_type": "markdown", - "id": "e136942c", + "id": "47c6c18d", "metadata": { "editable": true }, @@ -154,7 +154,7 @@ }, { "cell_type": "markdown", - "id": "af14d994", + "id": "77221535", "metadata": { "editable": true }, @@ -175,7 +175,7 @@ }, { "cell_type": "markdown", - "id": "9f2028b1", + "id": "a6b3ef86", "metadata": { "editable": true }, @@ -187,7 +187,7 @@ }, { "cell_type": "markdown", - "id": "066184af", + "id": "ab12e0e1", "metadata": { "editable": true }, @@ -200,7 +200,7 @@ }, { "cell_type": "markdown", - "id": "c19196a8", + "id": "8806260c", "metadata": { "editable": true }, @@ -212,7 +212,7 @@ }, { "cell_type": "markdown", - "id": "4e143d2b", + "id": "c28506f9", "metadata": { "editable": true }, @@ -224,7 +224,7 @@ }, { "cell_type": "markdown", - "id": "515c3480", + "id": "6ee2c6f3", "metadata": { "editable": true }, @@ -234,7 +234,7 @@ }, { "cell_type": "markdown", - "id": "38bd8ec2", + "id": "9d693259", "metadata": { "editable": true }, @@ -246,7 +246,7 @@ }, { "cell_type": "markdown", - "id": "656f9030", + "id": "4c8e76fe", "metadata": { "editable": true }, @@ -259,7 +259,7 @@ }, { "cell_type": "markdown", - "id": "e56f5f4c", + "id": "6ef26cfb", "metadata": { "editable": true }, @@ -271,7 +271,7 @@ }, { "cell_type": "markdown", - "id": "9e3b8671", + "id": "a7d47918", "metadata": { "editable": true }, @@ -281,7 +281,7 @@ }, { "cell_type": "markdown", - "id": "60582fbd", + "id": "3f18a933", "metadata": { "editable": true }, @@ -293,7 +293,7 @@ }, { "cell_type": "markdown", - "id": "8d5980de", + "id": "acfc82c7", "metadata": { "editable": true }, @@ -305,7 +305,7 @@ }, { "cell_type": "markdown", - "id": "af8111d4", + "id": "6cfbfe7b", "metadata": { "editable": true }, @@ -316,7 +316,7 @@ }, { "cell_type": "markdown", - "id": "68a3d94f", + "id": "5c04831b", "metadata": { "editable": true }, @@ -328,7 +328,7 @@ }, { "cell_type": "markdown", - "id": "971ee527", + "id": "63341f8e", "metadata": { "editable": true }, @@ -347,7 +347,7 @@ }, { "cell_type": "markdown", - "id": "f1be2271", + "id": "68d06111", "metadata": { "editable": true }, @@ -360,7 +360,7 @@ }, { "cell_type": "markdown", - "id": "91369e25", + "id": "08d49b08", "metadata": { "editable": true }, @@ -370,7 +370,7 @@ }, { "cell_type": "markdown", - "id": "7bdb9123", + "id": "5799e099", "metadata": { "editable": true }, @@ -382,7 +382,7 @@ }, { "cell_type": "markdown", - "id": "5e055bc2", + "id": "cdbe32c8", "metadata": { "editable": true }, @@ -392,7 +392,7 @@ }, { "cell_type": "markdown", - "id": "8b664327", + "id": "a34a60de", "metadata": { "editable": true }, @@ -404,7 +404,7 @@ }, { "cell_type": "markdown", - "id": "c3ef56f9", + "id": "4380cb3d", "metadata": { "editable": true }, @@ -414,7 +414,7 @@ }, { "cell_type": "markdown", - "id": "53e95dc3", + "id": "0fa26221", "metadata": { "editable": true }, @@ -426,7 +426,7 @@ }, { "cell_type": "markdown", - "id": "e075450a", + "id": "d5cec27d", "metadata": { "editable": true }, @@ -437,7 +437,7 @@ }, { "cell_type": "markdown", - "id": "fe17b4a4", + "id": "5a8454a2", "metadata": { "editable": true }, @@ -449,7 +449,7 @@ }, { "cell_type": "markdown", - "id": "123fb84e", + "id": "3f2c53c6", "metadata": { "editable": true }, @@ -459,7 +459,7 @@ }, { "cell_type": "markdown", - "id": "125f6feb", + "id": "1a38e0c7", "metadata": { "editable": true }, @@ -471,7 +471,7 @@ }, { "cell_type": "markdown", - "id": "34d49d95", + "id": "f9d67e56", "metadata": { "editable": true }, @@ -481,7 +481,7 @@ }, { "cell_type": "markdown", - "id": "00a87f4a", + "id": "9605ae4f", "metadata": { "editable": true }, @@ -493,7 +493,7 @@ }, { "cell_type": "markdown", - "id": "792f952a", + "id": "de14b580", "metadata": { "editable": true }, @@ -513,7 +513,7 @@ }, { "cell_type": "markdown", - "id": "1b50e035", + "id": "4054fa5c", "metadata": { "editable": true }, @@ -540,7 +540,7 @@ }, { "cell_type": "markdown", - "id": "6cd595f3", + "id": "61bd0198", "metadata": { "editable": true }, @@ -552,7 +552,7 @@ }, { "cell_type": "markdown", - "id": "1314cb83", + "id": "bdb7e922", "metadata": { "editable": true }, @@ -564,7 +564,7 @@ }, { "cell_type": "markdown", - "id": "84735270", + "id": "54b6bbfa", "metadata": { "editable": true }, @@ -580,7 +580,7 @@ }, { "cell_type": "markdown", - "id": "8b892d07", + "id": "fc3e4f2b", "metadata": { "editable": true }, @@ -598,7 +598,7 @@ }, { "cell_type": "markdown", - "id": "e86c7b3e", + "id": "9624726e", "metadata": { "editable": true }, @@ -610,7 +610,7 @@ }, { "cell_type": "markdown", - "id": "e5deafb7", + "id": "e9579efb", "metadata": { "editable": true }, @@ -622,7 +622,7 @@ }, { "cell_type": "markdown", - "id": "e7178fd6", + "id": "d6da92cc", "metadata": { "editable": true }, @@ -633,7 +633,7 @@ }, { "cell_type": "markdown", - "id": "47a8fe32", + "id": "214a54af", "metadata": { "editable": true }, @@ -645,7 +645,7 @@ }, { "cell_type": "markdown", - "id": "4914c592", + "id": "736edceb", "metadata": { "editable": true }, @@ -655,7 +655,7 @@ }, { "cell_type": "markdown", - "id": "002e4097", + "id": "70b72395", "metadata": { "editable": true }, @@ -667,7 +667,7 @@ }, { "cell_type": "markdown", - "id": "4246b3e9", + "id": "d751f216", "metadata": { "editable": true }, @@ -681,7 +681,7 @@ }, { "cell_type": "markdown", - "id": "f73b4e8d", + "id": "46b18e4b", "metadata": { "editable": true }, @@ -693,7 +693,7 @@ }, { "cell_type": "markdown", - "id": "a995aecb", + "id": "dc835d90", "metadata": { "editable": true }, @@ -705,7 +705,7 @@ }, { "cell_type": "markdown", - "id": "6753bbaf", + "id": "9dff0ac1", "metadata": { "editable": true }, @@ -717,7 +717,7 @@ }, { "cell_type": "markdown", - "id": "d6e1fe88", + "id": "243d40fd", "metadata": { "editable": true }, @@ -727,7 +727,7 @@ }, { "cell_type": "markdown", - "id": "1c7ea98c", + "id": "022a53ae", "metadata": { "editable": true }, @@ -739,7 +739,7 @@ }, { "cell_type": "markdown", - "id": "2d02b628", + "id": "b321d425", "metadata": { "editable": true }, @@ -751,7 +751,7 @@ }, { "cell_type": "markdown", - "id": "9445a953", + "id": "edc20e3b", "metadata": { "editable": true }, @@ -763,7 +763,7 @@ }, { "cell_type": "markdown", - "id": "ca4f49b7", + "id": "a03ea760", "metadata": { "editable": true }, @@ -777,7 +777,7 @@ }, { "cell_type": "markdown", - "id": "058cf41d", + "id": "89b04349", "metadata": { "editable": true }, @@ -789,7 +789,7 @@ }, { "cell_type": "markdown", - "id": "9901b68b", + "id": "cde4e746", "metadata": { "editable": true }, @@ -801,7 +801,7 @@ }, { "cell_type": "markdown", - "id": "0dcba42c", + "id": "b4d2c178", "metadata": { "editable": true }, @@ -813,7 +813,7 @@ }, { "cell_type": "markdown", - "id": "80403936", + "id": "21a9b2ea", "metadata": { "editable": true }, @@ -823,7 +823,7 @@ }, { "cell_type": "markdown", - "id": "92ccb9c0", + "id": "b3cab956", "metadata": { "editable": true }, @@ -835,7 +835,7 @@ }, { "cell_type": "markdown", - "id": "ee1dfff5", + "id": "47fbe8fd", "metadata": { "editable": true }, @@ -845,7 +845,7 @@ }, { "cell_type": "markdown", - "id": "9407da3d", + "id": "254a6af2", "metadata": { "editable": true }, @@ -857,7 +857,7 @@ }, { "cell_type": "markdown", - "id": "b95a96ae", + "id": "f8b23553", "metadata": { "editable": true }, @@ -867,7 +867,7 @@ }, { "cell_type": "markdown", - "id": "7ab72f35", + "id": "1dbfd08f", "metadata": { "editable": true }, @@ -879,7 +879,7 @@ }, { "cell_type": "markdown", - "id": "155d3bce", + "id": "23802261", "metadata": { "editable": true }, @@ -891,7 +891,7 @@ }, { "cell_type": "markdown", - "id": "fbd50da5", + "id": "5e2f646b", "metadata": { "editable": true }, @@ -903,7 +903,7 @@ }, { "cell_type": "markdown", - "id": "963ec823", + "id": "0457dac9", "metadata": { "editable": true }, @@ -918,7 +918,7 @@ }, { "cell_type": "markdown", - "id": "c5936c14", + "id": "795da8f4", "metadata": { "editable": true }, @@ -930,7 +930,7 @@ }, { "cell_type": "markdown", - "id": "81b26f0c", + "id": "81262cf1", "metadata": { "editable": true }, @@ -940,7 +940,7 @@ }, { "cell_type": "markdown", - "id": "cf0b05ad", + "id": "fef60c8c", "metadata": { "editable": true }, @@ -952,7 +952,7 @@ }, { "cell_type": "markdown", - "id": "e9d92f53", + "id": "80836258", "metadata": { "editable": true }, @@ -962,7 +962,7 @@ }, { "cell_type": "markdown", - "id": "7019dc33", + "id": "7f130753", "metadata": { "editable": true }, @@ -974,7 +974,7 @@ }, { "cell_type": "markdown", - "id": "c74cbe6e", + "id": "9c42dc95", "metadata": { "editable": true }, @@ -984,7 +984,7 @@ }, { "cell_type": "markdown", - "id": "d724dd20", + "id": "72dc1ea4", "metadata": { "editable": true }, @@ -996,7 +996,7 @@ }, { "cell_type": "markdown", - "id": "3c33b64b", + "id": "9755f3e0", "metadata": { "editable": true }, @@ -1008,7 +1008,7 @@ }, { "cell_type": "markdown", - "id": "8030d91f", + "id": "e7505b7d", "metadata": { "editable": true }, @@ -1020,7 +1020,7 @@ }, { "cell_type": "markdown", - "id": "3a305b26", + "id": "602afb09", "metadata": { "editable": true }, @@ -1030,7 +1030,7 @@ }, { "cell_type": "markdown", - "id": "0ff16721", + "id": "73328415", "metadata": { "editable": true }, @@ -1042,7 +1042,7 @@ }, { "cell_type": "markdown", - "id": "83824d8d", + "id": "357f9fcb", "metadata": { "editable": true }, @@ -1055,7 +1055,7 @@ }, { "cell_type": "markdown", - "id": "eda8b985", + "id": "155beb6e", "metadata": { "editable": true }, @@ -1067,7 +1067,7 @@ }, { "cell_type": "markdown", - "id": "0fed8494", + "id": "14dd6c43", "metadata": { "editable": true }, @@ -1077,7 +1077,7 @@ }, { "cell_type": "markdown", - "id": "cce65600", + "id": "65041e77", "metadata": { "editable": true }, @@ -1089,7 +1089,7 @@ }, { "cell_type": "markdown", - "id": "b55e6223", + "id": "90a53e7e", "metadata": { "editable": true }, @@ -1099,7 +1099,7 @@ }, { "cell_type": "markdown", - "id": "4a7f2705", + "id": "104011e3", "metadata": { "editable": true }, @@ -1111,7 +1111,7 @@ }, { "cell_type": "markdown", - "id": "1ade676b", + "id": "046656ab", "metadata": { "editable": true }, @@ -1121,7 +1121,7 @@ }, { "cell_type": "markdown", - "id": "7ad86f07", + "id": "5d29e5be", "metadata": { "editable": true }, @@ -1133,7 +1133,7 @@ }, { "cell_type": "markdown", - "id": "4e61f362", + "id": "2da73e87", "metadata": { "editable": true }, @@ -1143,7 +1143,7 @@ }, { "cell_type": "markdown", - "id": "576e7cbb", + "id": "0f5cb23b", "metadata": { "editable": true }, @@ -1155,7 +1155,7 @@ }, { "cell_type": "markdown", - "id": "bd83a530", + "id": "3f41a473", "metadata": { "editable": true }, @@ -1165,7 +1165,7 @@ }, { "cell_type": "markdown", - "id": "c6af271b", + "id": "5e2f3e3c", "metadata": { "editable": true }, @@ -1177,7 +1177,7 @@ }, { "cell_type": "markdown", - "id": "cbb3be57", + "id": "4d58e7cb", "metadata": { "editable": true }, @@ -1193,7 +1193,7 @@ }, { "cell_type": "markdown", - "id": "028b5199", + "id": "3f37e988", "metadata": { "editable": true }, @@ -1205,7 +1205,7 @@ }, { "cell_type": "markdown", - "id": "bd92a55b", + "id": "e259bd27", "metadata": { "editable": true }, @@ -1215,7 +1215,7 @@ }, { "cell_type": "markdown", - "id": "40f6c21e", + "id": "143d5d28", "metadata": { "editable": true }, @@ -1227,7 +1227,7 @@ }, { "cell_type": "markdown", - "id": "892159ed", + "id": "af9f94d8", "metadata": { "editable": true }, @@ -1245,7 +1245,7 @@ }, { "cell_type": "markdown", - "id": "d151a69e", + "id": "387e3480", "metadata": { "editable": true }, @@ -1257,7 +1257,7 @@ }, { "cell_type": "markdown", - "id": "59f2ee34", + "id": "19eb19ee", "metadata": { "editable": true }, @@ -1269,7 +1269,7 @@ }, { "cell_type": "markdown", - "id": "89bcd15e", + "id": "dd32a1a4", "metadata": { "editable": true }, @@ -1279,7 +1279,7 @@ }, { "cell_type": "markdown", - "id": "b66fbf47", + "id": "b7fc97aa", "metadata": { "editable": true }, @@ -1291,7 +1291,7 @@ }, { "cell_type": "markdown", - "id": "8bc4abe7", + "id": "92f090a2", "metadata": { "editable": true }, @@ -1301,7 +1301,7 @@ }, { "cell_type": "markdown", - "id": "02889fdd", + "id": "a07b14b0", "metadata": { "editable": true }, @@ -1313,7 +1313,7 @@ }, { "cell_type": "markdown", - "id": "016886bc", + "id": "690f1feb", "metadata": { "editable": true }, @@ -1323,7 +1323,7 @@ }, { "cell_type": "markdown", - "id": "07004804", + "id": "98ce0e54", "metadata": { "editable": true }, @@ -1337,7 +1337,7 @@ }, { "cell_type": "markdown", - "id": "b5e824e8", + "id": "8355044d", "metadata": { "editable": true }, @@ -1349,7 +1349,7 @@ }, { "cell_type": "markdown", - "id": "7bf1e4f6", + "id": "a6b37202", "metadata": { "editable": true }, @@ -1360,7 +1360,7 @@ }, { "cell_type": "markdown", - "id": "ea671dac", + "id": "9554ba3d", "metadata": { "editable": true }, @@ -1373,13 +1373,10 @@ { "cell_type": "code", "execution_count": 1, - "id": "6c7cab59", + "id": "5996cc7a", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1403,7 +1400,7 @@ }, { "cell_type": "markdown", - "id": "a7f3256b", + "id": "05513cf5", "metadata": { "editable": true }, @@ -1414,13 +1411,10 @@ { "cell_type": "code", "execution_count": 2, - "id": "a820e68e", + "id": "6383f412", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1430,7 +1424,7 @@ }, { "cell_type": "markdown", - "id": "06d14597", + "id": "1201be66", "metadata": { "editable": true }, @@ -1444,13 +1438,10 @@ { "cell_type": "code", "execution_count": 3, - "id": "c17a94fa", + "id": "6f353ffc", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1460,7 +1451,7 @@ }, { "cell_type": "markdown", - "id": "3ab630c5", + "id": "d8571bf0", "metadata": { "editable": true }, @@ -1471,13 +1462,10 @@ { "cell_type": "code", "execution_count": 4, - "id": "0046ebdc", + "id": "203d6f02", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1486,7 +1474,7 @@ }, { "cell_type": "markdown", - "id": "2ff9e7f9", + "id": "8e158e42", "metadata": { "editable": true }, @@ -1497,13 +1485,10 @@ { "cell_type": "code", "execution_count": 5, - "id": "8dc762ac", + "id": "cc537d67", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1516,7 +1501,7 @@ }, { "cell_type": "markdown", - "id": "f9966627", + "id": "dcfdabf6", "metadata": { "editable": true }, @@ -1527,13 +1512,10 @@ { "cell_type": "code", "execution_count": 6, - "id": "467a42fb", + "id": "d9cd7659", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1544,7 +1526,7 @@ }, { "cell_type": "markdown", - "id": "689951a8", + "id": "a812e890", "metadata": { "editable": true }, @@ -1565,7 +1547,7 @@ }, { "cell_type": "markdown", - "id": "5553b42e", + "id": "c16d6b48", "metadata": { "editable": true }, @@ -1576,13 +1558,10 @@ { "cell_type": "code", "execution_count": 7, - "id": "67ef7e27", + "id": "bd392aef", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1632,7 +1611,7 @@ }, { "cell_type": "markdown", - "id": "fb0f7303", + "id": "e01305b4", "metadata": { "editable": true }, @@ -1643,13 +1622,10 @@ { "cell_type": "code", "execution_count": 8, - "id": "6eb747de", + "id": "c274659b", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1671,7 +1647,7 @@ }, { "cell_type": "markdown", - "id": "a7f7ec90", + "id": "40c22ab9", "metadata": { "editable": true }, @@ -1683,7 +1659,7 @@ }, { "cell_type": "markdown", - "id": "a04dc637", + "id": "14c0cb14", "metadata": { "editable": true }, @@ -1712,7 +1688,7 @@ }, { "cell_type": "markdown", - "id": "acc2722f", + "id": "d0adb995", "metadata": { "editable": true }, @@ -1737,7 +1713,7 @@ }, { "cell_type": "markdown", - "id": "10f204b3", + "id": "c8acfb81", "metadata": { "editable": true }, @@ -1757,7 +1733,7 @@ }, { "cell_type": "markdown", - "id": "5f13f1d3", + "id": "25284826", "metadata": { "editable": true }, @@ -1784,7 +1760,7 @@ }, { "cell_type": "markdown", - "id": "5f70140d", + "id": "a673748d", "metadata": { "editable": true }, @@ -1797,7 +1773,7 @@ }, { "cell_type": "markdown", - "id": "98541249", + "id": "82b9b725", "metadata": { "editable": true }, @@ -1809,7 +1785,7 @@ }, { "cell_type": "markdown", - "id": "21191090", + "id": "ac08b34d", "metadata": { "editable": true }, @@ -1820,7 +1796,7 @@ }, { "cell_type": "markdown", - "id": "40b4448f", + "id": "8e4281c3", "metadata": { "editable": true }, @@ -1836,13 +1812,10 @@ { "cell_type": "code", "execution_count": 9, - "id": "67e975b0", + "id": "81ebdd6e", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -1873,7 +1846,7 @@ }, { "cell_type": "markdown", - "id": "0f0d85bb", + "id": "beb0eda1", "metadata": { "editable": true }, @@ -1883,7 +1856,7 @@ }, { "cell_type": "markdown", - "id": "be74a304", + "id": "d2f901fc", "metadata": { "editable": true }, @@ -1898,7 +1871,7 @@ }, { "cell_type": "markdown", - "id": "5da0965b", + "id": "3d254833", "metadata": { "editable": true }, @@ -1910,7 +1883,7 @@ }, { "cell_type": "markdown", - "id": "fd82f58b", + "id": "4e7c28a9", "metadata": { "editable": true }, @@ -1920,7 +1893,7 @@ }, { "cell_type": "markdown", - "id": "8fd6cf7d", + "id": "763af26f", "metadata": { "editable": true }, @@ -1935,20 +1908,16 @@ }, { "cell_type": "code", - "execution_count": 3, - "id": "8e764959", + "execution_count": 10, + "id": "cea37f94", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ - "np.random.seed(2025)\n", + "np.random.seed()\n", "n = 100\n", - "from sklearn.preprocessing import MinMaxScaler, StandardScaler, Normalizer\n", "maxdegree = 14\n", "# Make data set.\n", "x = np.linspace(-3, 3, n).reshape(-1, 1)\n", @@ -1957,7 +1926,7 @@ }, { "cell_type": "markdown", - "id": "d84fde6d", + "id": "54451e45", "metadata": { "editable": true }, @@ -1969,27 +1938,13 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "80dae6ab", + "execution_count": 11, + "id": "498277ed", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiQAAAGdCAYAAAAi3mhQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAzFUlEQVR4nO3de3wU9b3/8ffsJtkkmIRLgCSSQBAqCohyEblpaBFERaytdylqj5cCKuVUkarcVOKlak7LEY/2dxT1qLRVqcd6tDlVQEtRQBEOIooEEy4xiDabBLKb7Mzvj2SXLAkhl01mL6/n4zGP7MzOznwYBvaTz/cyhmVZlgAAAGzksDsAAAAAEhIAAGA7EhIAAGA7EhIAAGA7EhIAAGA7EhIAAGA7EhIAAGA7EhIAAGC7OLsDOJZpmtq/f79SUlJkGIbd4QAAgBawLEsVFRXKysqSw9H6ekfYJST79+9Xdna23WEAAIA2KCkpUZ8+fVr9ubBLSFJSUiTV/YFSU1NtjgYAALSE2+1WdnZ24Hu8tcIuIfE306SmppKQAAAQYdra3aLVjTzr1q3TtGnTlJWVJcMwtHr16uPue8stt8gwDBUUFLQpOAAAEBtanZBUVVVp2LBhWr58ebP7rV69Wh9++KGysrLaHBwAAIgNrW6ymTp1qqZOndrsPvv27dOcOXP0zjvv6KKLLmpzcAAAIDaEvA+JaZqaMWOG7rzzTg0ePPiE+3s8Hnk8nsC62+0+4Wcsy1Jtba18Pl+7YkV4io+Pl9PptDsMAEAnCnlC8vDDDysuLk633357i/bPz8/XkiVLWnx8r9erAwcO6PDhw20NEWHOMAz16dNHJ510kt2hAAA6SUgTks2bN+vf/u3f9PHHH7e4l+2CBQs0b968wLp/2FBTTNNUUVGRnE6nsrKylJCQwORpUcayLB08eFB79+7VwIEDqZQAQIwIaULy/vvvq6ysTDk5OYFtPp9P//qv/6qCggLt2bOn0WdcLpdcLleLju/1emWaprKzs5WcnByqsBFmevbsqT179qimpoaEBABiREgTkhkzZmjSpElB26ZMmaIZM2bohhtuCNl52jIlLSIHVS8AiD2tTkgqKyu1a9euwHpRUZG2bNmi7t27KycnRz169AjaPz4+XhkZGTr11FPbHy0AAIhKrU5INm3apIkTJwbW/f0/Zs6cqeeeey5kgQEAgNjR6oQkLy9PlmW1eP+m+o0AAAA0RGeMTmAYRrPL9ddf3+Zj9+vXr0VT8/fr16/Jcz/00ENtPjcAAKESdg/Xi0YHDhwIvF61apUWLlyonTt3BrYlJSV1ShxLly7VTTfdFLTteE9ltCxLPp9PcXHBt4jX61VCQkKrz93WzwEAQqfKU6sH39qhlMQ4zZ8ySA5H+AwiiPgKiWVZOuyttWVpadNVRkZGYElLS5NhGEHb1q1bpxEjRigxMVH9+/fXkiVLVFtbG/j84sWLlZOTI5fLpaysrMCkc3l5efr666/1y1/+MlDxaE5KSkrQeTMyMtSlSxdJ0po1a2QYht555x2NHDlSLpdL77//vvLy8jRnzhzNmzdP6enpOv/88yVJa9eu1dlnny2Xy6XMzEzdfffdQTEf73MAAPt8f9irlz4s1rN/3xNWyYgUBRWSIzU+nb7wHVvO/dnSKUpOaN8lfOedd3Tdddfpt7/9rSZMmKCvvvpKN998syRp0aJF+tOf/qQnnnhCr7zyigYPHqzS0lJ9+umnkqTXXntNw4YN080339yo8tFWd911l37zm9+of//+6tq1qyRp5cqV+sUvfqG///3vsixL+/bt04UXXqjrr79ezz//vD7//HPddNNNSkxM1OLFiwPHOvZzAAB7VVTX/eKYmhh+X//hF1GMefDBB3X33Xdr5syZkqT+/fvr/vvv11133aVFixapuLhYGRkZmjRpkuLj45WTk6Ozzz5bktS9e3c5nc5A5eNE5s+fr3vvvTdo25tvvqm8vLzA+tKlSxtVMwYMGKBHHnkksH7PPfcoOztby5cvl2EYGjRokPbv36/58+dr4cKFgXlijv0cAMBe/oQkJTHe5kgai/iEJCneqc+WTrHt3O21efNmbdy4UQ8++GBgm8/nU3V1tQ4fPqzLL79cBQUF6t+/vy644AJdeOGFmjZtWqO+HS1x5513NupAe/LJJwetjxw5stHnjt22Y8cOjRkzJqiJaNy4caqsrNTevXsDM/U2dSwAgH0qqmskSSlUSELPMIx2N5vYyTRNLVmyRJdddlmj9xITE5Wdna2dO3eqsLBQ//u//6tZs2bp0Ucf1dq1axUf37oMNz09XQMGDGh2H3+fkua2WZbVqL+Kv0mm4famjgUAsM/RCkn4fW+GX0QxZvjw4dq5c2eziUJSUpIuueQSXXLJJZo9e7YGDRqkbdu2afjw4UpISJDP5+vEiKXTTz9dr776alBisn79eqWkpDSquAAAwkegQuKiyQbHWLhwoS6++GJlZ2fr8ssvl8Ph0NatW7Vt2zY98MADeu655+Tz+TR69GglJyfrhRdeUFJSkvr27Supbn6RdevW6aqrrpLL5VJ6evpxz1VRUaHS0tKgbcnJyUpNTW1VzLNmzVJBQYFuu+02zZkzRzt37tSiRYs0b948njMEAGHMHcYVEr49bDZlyhS9+eabKiws1KhRo3TOOefo8ccfDyQcXbt21TPPPKNx48bpjDPO0N/+9jf993//d+CZQUuXLtWePXt0yimnqGfPns2ea+HChcrMzAxa7rrrrlbHfPLJJ+utt97SRx99pGHDhunWW2/Vz3/+80YdZgEA4SWcO7UaVpiNx3S73UpLS1N5eXmj39yrq6tVVFSk3NxcJSYm2hQhOhp/zwDQMe55fZv+68Ni3fGjgfrl+T8I6bGb+/5uCSokAADEiHDu1EpCAgBAjPB3ak0NwyYbEhIAAGJEYKbWJCokAADAJuHcqZWEBACAGBHOM7WSkAAAECPcVEgAAICdfKalSg+jbAAAgI38yYhEQoIQy8vL09y5c+0OAwAQAfz9RxLiHHLFtf9p9aFGQtIJDMNodrn++uvbdNzXXntN999/f7tiu/7665uM6YILLmjXcQEA4SUw5DcMqyMSD9frFAcOHAi8XrVqlRYuXKidO3cGtiUlJQXtX1NTo/j4E3c46t69e0jiu+CCC/Tss88GbXO5XMfdv6n4WhpzS44FAAi9cB7yK1Eh6RQZGRmBJS0tTYZhBNarq6vVtWtX/eEPf1BeXp4SExP14osv6tChQ7r66qvVp08fJScna+jQoXr55ZeDjntsk02/fv20bNky3XjjjUpJSVFOTo6efvrpE8bncrmCYszIyFC3bt0C7xuGoaeeekrTp09Xly5d9MADD2jx4sU688wz9Z//+Z/q37+/XC6XLMtScXGxpk+frpNOOkmpqam64oor9M033wSOdbzPAQA6VjgP+ZWiISGxLMlbZc8Swi/S+fPn6/bbb9eOHTs0ZcoUVVdXa8SIEXrzzTf1f//3f7r55ps1Y8YMffjhh80e57HHHtPIkSP1ySefaNasWfrFL36hzz//vN3xLVq0SNOnT9e2bdt04403SpJ27dqlP/zhD3r11Ve1ZcsWSdKll16q7777TmvXrlVhYaG++uorXXnllUHHaupzAICOFc7PsZGiocmm5rC0LMuec/96v5TQJSSHmjt3ri677LKgbb/61a8Cr2+77Ta9/fbb+uMf/6jRo0cf9zgXXnihZs2aJakuyXniiSe0Zs0aDRo06LifefPNN3XSSScFbZs/f77uu+++wPo111wTSET8vF6vXnjhBfXs2VOSVFhYqK1bt6qoqEjZ2dmSpBdeeEGDBw/Wxo0bNWrUqCY/BwDoeIEKiSs8m2wiPyGJEiNHjgxa9/l8euihh7Rq1Srt27dPHo9HHo9HXbo0nwCdccYZgdf+pqGysrJmPzNx4kStWLEiaNux/VOOjU+S+vbtG5RU7NixQ9nZ2YFkRJJOP/10de3aVTt27AgkJMd+DgDQ8dxUSDpYfHJdpcKuc4fIsYnGY489pieeeEIFBQUaOnSounTporlz58rr9TYf0jEdRA3DkGmaJzz3gAEDWhVfU9ssy5JhGI32O3b7iZIqAEDohXun1shPSAwjZM0m4eT999/X9OnTdd1110mSTNPUl19+qdNOO83myI7v9NNPV3FxsUpKSgJVks8++0zl5eVhHTcAxIJw79QanlFBAwYM0Kuvvqr169erW7duevzxx1VaWtohX+wej0elpaVB2+Li4pSent6q40yaNElnnHGGrr32WhUUFKi2tlazZs3Seeed12STDwCg84R7p9bIH2UTpe677z4NHz5cU6ZMUV5enjIyMnTppZd2yLnefvttZWZmBi3jx49v9XEMw9Dq1avVrVs3nXvuuZo0aZL69++vVatWdUDUAIDW8FdIUsO0ycawwmwSCLfbrbS0NJWXlys1NTXoverqahUVFSk3N1eJiYk2RYiOxt8zAITeT1es16avv9eKa4dr6tDMkB+/ue/vlqBCAgBADAj3Tq0kJAAAxIBw79RKQgIAQAygUysAALCVaVqq9NJkAwAAbFTprQ08fo0KSQiF2cAghBh/vwAQWv7mmgSnQ4nxTpujaVpEJST+adEPHz5scyToSP7p8Z3O8PxHAwCRJjAHSVJ4VkekNszUum7dOj366KPavHmzDhw4oNdffz0wYVdNTY3uvfdevfXWW9q9e7fS0tI0adIkPfTQQ8rKav8TeZ1Op7p27Rp4WFxycnKTz05B5DJNUwcPHlRycrLi4sL3Hw4ARJJwH/IrtSEhqaqq0rBhw3TDDTfoJz/5SdB7hw8f1scff6z77rtPw4YN0/fff6+5c+fqkksu0aZNm0IScEZGhiSd8Am2iFwOh0M5OTkkmwAQIu4j4T3kV2pDQjJ16lRNnTq1yffS0tJUWFgYtO13v/udzj77bBUXFysnJ6dtUTZgGIYyMzPVq1cv1dTUtPt4CD8JCQlyOCKqNREAwlq4D/mVOuHheuXl5TIMQ127dm3yfY/HI4/HE1h3u90tOq7T6aSPAQAALRCYFM0Vvk02HfpraHV1te6++25dc801x53XPj8/X2lpaYHF/9h6AAAQGu4IqJB0WEJSU1Ojq666SqZp6sknnzzufgsWLFB5eXlgKSkp6aiQAACISVHZqbUlampqdMUVV6ioqEjvvvtus0/9c7lccrlcHREGAABQ+D/HRuqAhMSfjHz55Zd677331KNHj1CfAgAAtEJUdmqtrKzUrl27AutFRUXasmWLunfvrqysLP30pz/Vxx9/rDfffFM+n0+lpaWSpO7duyshISF0kQMAgBYJTIwWTU02mzZt0sSJEwPr8+bNkyTNnDlTixcv1htvvCFJOvPMM4M+99577ykvL6/tkQIAgDaJygpJXl5es88a4TkkAACEl0jo1MrsUwAARLlI6NRKQgIAQJSLhCYbEhIAAKKYaVqq9NJkAwAAbFTprZW/eycVEgAAYAt/c02C06HE+PB9BhwJCQAAUSwSOrRKJCQAAES1SOjQKpGQAAAQ1Y5WSMK3Q6tEQgIAQFSjQgIAAGznJiEBAAB2o8kGAADYzt9kE85P+pVISAAAiGoM+wUAALZzH6EPCQAAsJm/QkKTDQAAsA3DfgEAgO2OJiRUSAAAgE3o1AoAAGxHkw0AALCVaVqq9NJkAwAAbFTprZVl1b2mQgIAAGzhb65JcDqUGO+0OZrmkZAAABClIqVDq0RCAgBA1IqUDq0SCQkAAFErUp70K5GQAAAQtaiQAAAA27lJSAAAgN1osgEAALajyQYAANiOCgkAALCdv0KSSoUEAADYhSYbAABgO5psAACA7aiQAAAA2x1NSKiQAAAAm7jrm2zo1AoAAGxhmpYqPVRIAACAjSq9tbKsutf0IQEAALbw9x9JcDqUGO+0OZoTa3VCsm7dOk2bNk1ZWVkyDEOrV68Oet+yLC1evFhZWVlKSkpSXl6etm/fHqp4AQBACxwd8hv+1RGpDQlJVVWVhg0bpuXLlzf5/iOPPKLHH39cy5cv18aNG5WRkaHzzz9fFRUV7Q4WAAC0TCQN+ZWkVkc5depUTZ06tcn3LMtSQUGB7rnnHl122WWSpJUrV6p379566aWXdMstt7QvWgAA0CKRNCmaFOI+JEVFRSotLdXkyZMD21wul8477zytX7++yc94PB653e6gBQAAtE+kVUhCmpCUlpZKknr37h20vXfv3oH3jpWfn6+0tLTAkp2dHcqQAACISe5YTkj8DMMIWrcsq9E2vwULFqi8vDywlJSUdERIAADElEhrsglp2pSRkSGprlKSmZkZ2F5WVtaoauLncrnkcrlCGQYAADEvpptscnNzlZGRocLCwsA2r9ertWvXauzYsaE8FQAAaEbUV0gqKyu1a9euwHpRUZG2bNmi7t27KycnR3PnztWyZcs0cOBADRw4UMuWLVNycrKuueaakAYOAACOz18hiYTn2EhtSEg2bdqkiRMnBtbnzZsnSZo5c6aee+453XXXXTpy5IhmzZql77//XqNHj9Zf//pXpaSkhC5qAADQrEhrsml1lHl5ebL8k+M3wTAMLV68WIsXL25PXAAAoB0ircmGZ9kAABCFIq1CQkICAEAUOpqQUCEBAAA2cUf7w/UAAEB4M01LlR6abAAAgI2qvLXyjz9JpckGAADYwd9/JN5pyBUXGV/1kRElAABosYYdWo/3LLlwQ0ICAECUqYiwDq0SCQkAAFEn0uYgkUhIAACIOv4hv5HSoVUiIQEAIOq4qZAAAAC7RdpzbCQSEgAAog59SAAAgO2okAAAANv5KySpVEgAAIBdaLIBAAC2o8kGAADYjgoJAACwXcNn2UQKEhIAAKKMm2fZAAAAO5mmpUoPTTYAAMBGVd5aWVbda55lAwAAbOHvPxLvNOSKi5yv+ciJFAAAnFDDDq2GYdgcTcuRkAAAEEUqIrBDq0RCAgBAVInEOUgkEhIAAKJKYMivK3I6tEokJAAARBUqJAAAwHaROEurREICAEBUoVMrAACwnb9CkkpCAgAA7HK0QkKTDQAAsImbTq0AAMBu/gpJahIVEgAAYBOG/QIAANsx7BcAANjOzbBfAABgJ9O0VOmhyUaSVFtbq3vvvVe5ublKSkpS//79tXTpUpmmGepTAQCABqq8tbKsutepEdZkE/L06eGHH9ZTTz2llStXavDgwdq0aZNuuOEGpaWl6Y477gj16QAAQD1//5F4pyFXXGQ1goQ8IfnHP/6h6dOn66KLLpIk9evXTy+//LI2bdoU6lMBAIAGGnZoNQzD5mhaJ+Tp0/jx4/W3v/1NX3zxhSTp008/1QcffKALL7ywyf09Ho/cbnfQAgAAWi9Sn2MjdUCFZP78+SovL9egQYPkdDrl8/n04IMP6uqrr25y//z8fC1ZsiTUYQAAEHMidQ4SqQMqJKtWrdKLL76ol156SR9//LFWrlyp3/zmN1q5cmWT+y9YsEDl5eWBpaSkJNQhAQAQEwJDfl2R1aFV6oAKyZ133qm7775bV111lSRp6NCh+vrrr5Wfn6+ZM2c22t/lcsnlcoU6DAAAYg4VkgYOHz4shyP4sE6nk2G/AAB0sEidpVXqgArJtGnT9OCDDyonJ0eDBw/WJ598oscff1w33nhjqE8FAAAaoFNrA7/73e903333adasWSorK1NWVpZuueUWLVy4MNSnAgAADfgrJKkkJFJKSooKCgpUUFAQ6kMDAIBmHK2QRF6TTWRN4wYAAI6LTq0AAMB2kdyplYQEAIAo4Y7gTq0kJAAARAmabAAAgO3o1AoAAGxlmpYqPZE77JeEBACAKFDlrZVp1b2mQgIAAGzh7z8S7zSUGB95X++RFzEAAGik4ZBfwzBsjqb1SEgAAIgCkfwcG4mEBACAqBDJQ34lEhIAAKJCYFI0V+R1aJVISAAAiApUSAAAgO0i+Tk2EgkJAABRgU6tAADAdv4KSSTO0iqRkAAAEBUi+Tk2EgkJAABRgU6tAADAdnRqBQAAtnPTqRUAANiNJhsAAGA7OrUCAABbWZalSg/DfgEAgI2qvD6ZVt1rKiQAAMAW/uaaOIehxPjI/GqPzKgBAEBAww6thmHYHE3bkJAAABDhIr1Dq0RCAgBAxHNH+JBfiYQEAICIF+lzkEgkJAAARDz3EZpsAACAzaiQAAAA2/k7taZSIQEAAHbxV0gidZZWiYQEAICIx7BfAABgO/qQAAAA2x1NSKiQAAAAm7gDTTZUSAAAgE1osjmOffv26brrrlOPHj2UnJysM888U5s3b+6IUwEAEPOioVNryFOp77//XuPGjdPEiRP1P//zP+rVq5e++uorde3aNdSnAgAg5lmWpUpP5A/7DXnkDz/8sLKzs/Xss88GtvXr1y/UpwEAAJKqvD6ZVt3rSK6QhLzJ5o033tDIkSN1+eWXq1evXjrrrLP0zDPPhPo0AABAR5tr4hyGEuMjt2toyCPfvXu3VqxYoYEDB+qdd97Rrbfeqttvv13PP/98k/t7PB653e6gBQAAtEzDDq2GYdgcTduFvMnGNE2NHDlSy5YtkySdddZZ2r59u1asWKGf/exnjfbPz8/XkiVLQh0GAAAxIRo6tEodUCHJzMzU6aefHrTttNNOU3FxcZP7L1iwQOXl5YGlpKQk1CEBABC13FEw5FfqgArJuHHjtHPnzqBtX3zxhfr27dvk/i6XSy6XK9RhAAAQE6JhDhKpAyokv/zlL7VhwwYtW7ZMu3bt0ksvvaSnn35as2fPDvWpAACIeTTZHMeoUaP0+uuv6+WXX9aQIUN0//33q6CgQNdee22oTwUAQMyLlgpJh0R/8cUX6+KLL+6IQwMAgAb8FZJUKiQAAMAu0VIhISEBACCCkZAAAADb0akVAADYzn2ECgkAALCZmwoJAACwG31IAACA7Rj2CwAAbGVZlio9dRWSVCokAADADlVen0yr7jV9SAAAgC38zTVxDkOJ8ZH9lR7Z0QMAEMMadmg1DMPmaNqHhAQAgAgVLZOiSSQkAABELHeUDPmVSEgAAIhY0TIHiURCAgBAxKLJBgAA2I4KCQAAsF20zNIqkZAAABCxqJAAAADbkZAAAADb0akVAADYjnlIAACA7Y422VAhAQAANjnaZEOFBAAA2MRfIUklIQEAAHawLEuVHppsAACAjQ57ffKZliSabAAAgE3c9f1HnA5DSfFOm6NpPxISAAAiUMNJ0QzDsDma9iMhAQAgAkXTCBuJhAQAgIjkDoywifwOrRIJCQAAESmanmMjkZAAABCRouk5NhIJCQAAEYkKCQAAsJ2/QkIfEgAAYBsqJAAAwHYkJAAAwHZ0agUAALZzUyEBAAB2O9pkQ4WkRfLz82UYhubOndvRpwIAIGYwdXwrbNy4UU8//bTOOOOMjjwNAAAxpyIwdTwJSbMqKyt17bXX6plnnlG3bt066jQAAMQcy7JU6aHJpkVmz56tiy66SJMmTWp2P4/HI7fbHbQAAIDjO+z1yWdakqKnyaZD/hSvvPKKPv74Y23cuPGE++bn52vJkiUdEQYAAFHJ31zjdBhKinfaHE1ohLxCUlJSojvuuEMvvviiEhMTT7j/ggULVF5eHlhKSkpCHRIAAFGlYYdWwzBsjiY0Ql4h2bx5s8rKyjRixIjANp/Pp3Xr1mn58uXyeDxyOo9mcy6XSy6XK9RhAAAQtaJtDhKpAxKSH/3oR9q2bVvQthtuuEGDBg3S/Pnzg5IRAADQeoEKiSs6OrRKHZCQpKSkaMiQIUHbunTpoh49ejTaDgAAWi/anmMjMVMrAAARJ9pmaZU6aJTNsdasWdMZpwEAICa465tsomVSNIkKCQAAESfapo2XSEgAAIg40dhkQ0ICAECEoVMrAACwnb/JJjWJCgkAALBJNE6MRkICAECEoQ8JAACwHaNsAACA7fwVEuYhAQAAtrAsS5UemmwAAICNDnt98pmWJJpsAACATfzNNU6HoaR4p83RhA4JCQAAEaRhh1bDMGyOJnRISAAAiCDROAeJREICAEBECVRIXNHToVUiIQEAIKJE43NsJBISAAAiSjTO0iqRkAAAEFECD9ajQgIAAOxCkw0AALDd0WG/NNkAAACbUCEBAAC2c9OpFQAA2K3hTK3RhIQEAIAIwkytAADAdnRqBQAAtvN3amUeEgAAYAvLslTpoVMrAACw0WGvTz7TkkQfEgAAYBN/c43TYSg5wWlzNKFFQgIAQIRoOOTXMAybowktEhIAACJEtA75lUhIAACIGIEKiSu6OrRKJCQAAESMaH2OjURCAgBAxKiI0ufYSCQkAABEDH+TTbRNiiaRkAAAEDFosgEAALaL1ufYSCQkAABEDCokAADAdm46tbZcfn6+Ro0apZSUFPXq1UuXXnqpdu7cGerTAAAQcxrO1BptQp6QrF27VrNnz9aGDRtUWFio2tpaTZ48WVVVVaE+FQAAMSWam2xC/id6++23g9afffZZ9erVS5s3b9a5554b6tMBABAzKjzR26m1w1Os8vJySVL37t2bfN/j8cjj8QTW3W53R4cEAEBE8ldImIeklSzL0rx58zR+/HgNGTKkyX3y8/OVlpYWWLKzszsyJAAAIpJlWczU2lZz5szR1q1b9fLLLx93nwULFqi8vDywlJSUdGRIAABEpCM1PvlMSxJ9SFrltttu0xtvvKF169apT58+x93P5XLJ5XJ1VBgAAEQFf3XE6TCUnOC0OZrQC3lCYlmWbrvtNr3++utas2aNcnNzQ30KAABijvtIXYfWk1xxMgzD5mhCL+QJyezZs/XSSy/pz3/+s1JSUlRaWipJSktLU1JSUqhPBwBATHBH8ZBfqQP6kKxYsULl5eXKy8tTZmZmYFm1alWoTwUAQNg64vXp//aVh+x40fwcG6mDmmwAAIh1d/7pU7259YDuveg0/cuE/u0+XjRPiibxLBsAAEKu6Nsq/WXbAUnSw29/ru37218pieY5SCQSEgAAQu7/fbBbllU3IqbGZ+mOV7boiNfXrmNGe5MNCQkAACH0XZVXf9q8V5L0u6vPUq8Ul3aVVWrZWzvadVwqJAAAoMVe3PC1qmtMDTk5VVOHZOixK4ZJkl7Y8LX+tuObNh+XCgkAAGiR6hqfnv/HHknSTRP6yzAMTRjYUz8fXzcn111/2qqDFZ5mjnB8dGoFAAAtsvqTffq20qustERdODQzsP3OKadqUEaKDlV5deefPm3TiFR3FD/HRiIhAQAgJEzT0jPv75Yk3Tg+V/HOo1+xifFO/fbqs+SKc2jNzoN6/h9ft/r4R5tsqJAAAIDjWPNFmb46WKUUV5yuHNX4yfU/6J2iX194miTpwbd26ItvKlp1fJpsAADACT29rq46cvXonOM2q/xsTF/lndpT3lpTt7/8iTy1LR8KXOGhUysAAGjGtr3l2rD7O8U5DF0/tt9x9zMMQ4/89Az16JKgz0sr9OjbO1t8Dob9AgCAZvn7jlx8Rqayujb/INleKYl65KdnSJJ+/0GR3v/y4AmPb1lWgyYbKiQAAOAY+/55JDBNfEufWfOj03prxjl9JUn/+odP9V2Vt9n9j9T45DPrRubQhwQAADTy7AdF8pmWxg3ooSEnp7X4c7++8DQN6HWSyio8uvvVrc0OBfZXR5wOQ8kJznbHHI5ISAAAaKPyIzV6+aNiSS2vjvglJTj1b1edqXinob9+9o1WbSw57r7+Ib8nueJkGEbbAw5jJCQAALTRKx8Vq8rr08BeJynvBz1b/fnBWWm6c8qpkqQl//2Zdh+sbHI/d5QP+ZVISAAAaBNvraln/75H0tFp4tviX8b319hTeuhIjU93vLJF3lqz0T7R3qFVIiEBAKBN/rJtv0rd1Uo/yaXpZ2W1+TgOh6HHrzhTaUnx2ravXAX/+0WjfaJ9llaJhAQAgFazLEvPrCuSJN0wrp9cce3raJqRlqiHLhsqSVqx9itt2H0o6P1on4NEIiEBAKDV1n91SJ8dcCsp3qlrR+eE5JhTh2bqipF9ZFnSvFVbVH6kJvDe0QoJTTYAAKCef5r4K0b2UdfkhJAdd9G0werXI1n7y6t1z+vbAkOB3Ufo1AoAABrYWVqhtV8clMOoe6pvKHVxxangqrPkdBh6c+sBvf7JPkn0IQEAAMf4ff008VMGZ6hvjy4hP/6Z2V0190cDJUkL/7xdxYcOM8oGAAAcVeau1uotdVWLm85t3URorTFr4gCN6tdNlZ5azV31if55hAoJAACot/Ife1TjszSibzcNz+nWYedx1g8FTnHF6ePif2rNzjJJVEgAAIh5h721enFD3TTxN7Vymvi2yO6erAd+PESSVP9cPSokAADEuj9u2qvyIzXq1yNZ55/eu1POOf3Mk3XpmUcnXUulQgIAQOzymZZ+/0FdZ9afj8+V09F5D7hbeukQ9euRLFecQ/16JHfaeTtb9NZ+AAAIkXe2l6rkuyPqlhyvn47I7tRzpybG6y+3T1Clp1Y9TnJ16rk7EwkJAADNsCwrMBHadef0VVJC+6aJb4surjh1cUX3VzZNNgAANGPz199rS8k/leB06Gdj+tkdTtQiIQEAoBnP1E+Edtnwk9UzJXqbTOxGQgIAwHEUfVulv372jSTpXyaEdpp4BCMhAQDgOP7fB7tlWdIPB/XSgF4pdocT1UhIAABowndVXv1p815JVEc6Q3R32T3GPa9v08ndknTuwJ46PTNVjk4cRw4AiCwvbvha1TWmhpycqjH9e9gdTtSLmYTkUKVH3256VZvN3vrN233UtUuixg9I1/iB6ZowMF2ZaUl2hwgAaCXTtLR9v1v/2P2tuiUnaOyAdJ3ctf3/n1fX+PT8P/ZIqpsm3jD4BbajxUxCEqdaPen6dzlNr6qsRG3xnqJPtg/Q29sG6iFzgHr0ytKEgT01YWC6RvfvruSEmLk0ABBRKqpr9MGX3+rdz8u05ouDOljhCXq/b49kjT2lh8ackq4x/Xu0aWTM6k/26dtKr7LSEnXh0MxQhY5mxMy3bppVIfUbK+3drC7eCo1zbtc4bQ+8v+efvfXxRwP17oYB+q1+oC45wzT21AxNGNBTg7No3gEAu1iWpa8OVum9z8v07udl2rjnO9X6nzYnKTnBqbGn9NChKq+27i3X14cO6+tDh/XyRyWSpB/0PkljT0nXmFN66JzcHkpLbv55MKZpBYb63jg+V/FOult2BsOyLOvEu7Xek08+qUcffVQHDhzQ4MGDVVBQoAkTJpzwc263W2lpaSovL1dqamroAzN90sGd0t6N0t6PpL2bpIOfN9rtiJWgrVZ/fWIO0K6EQXLljtGw006leQcAOkF1jU8bdh/Se5+X6b2dB1X83eGg93PTu2jiqb30w0G9NCq3m1xxdbOnVlTXaOOe77R+1yGt/+qQPjvgDvqcYUhDstLqKyg9NKpf90YzoL77+Te68blNSnHFaf2CHyolih9oF0rt/f7ukIRk1apVmjFjhp588kmNGzdO//Ef/6Hf//73+uyzz5STk9PsZzs8IWnKkX9K+zZLezfJ2vuRzJKNcnrKG+2210rXFnOAipMHK6HfaKX06isj3iXD6ZIjziVnvEtxcQ7FORyKcxiKcxqKd/pfOxTvNOR0HN0W73QozmkozuFQYrxDifFOMnEAMWv/P4/UNcPsLNPfdx3SkRpf4L0Ep0Oj+3fXxFN7aeKgXspN79KiY35X5dWHu+uSk/VffauvDlYFvR/nMHRmdtdAE89ZOV11/bMfacPu73Tzuf316wtPC+mfMZqFZUIyevRoDR8+XCtWrAhsO+2003TppZcqPz+/2c/akpAcyzSl776SSj6Sr+QjeYo+VOL3O+WQecKPeqw4eRUvr+LkUby8Vt1r/zav4uW16t8LrNe9rlGcao04WY4Emc4EyRkvORMkZ4KMuAQ5nC4Z8QlyxLnkiEuQMz5RzgSX4uJcinMlKD4hUXFxrrpfAU6gJQ1QhmHI4XTKcDjlcDjkcMbV/3TK4XDK4YiTw+mQ01n302E45IyLl8PhUJwzTk6nIYdRl3A5HJLDqFv3v3bWrxsOBV4H7WeIjmTAMSzLkqfWVJWnVpX+pbpWVd5aVVTXqsrjU6WnRpUeX932+n2O1PhkSHI46v7tOR2GDENy1q87Gvx01G931O8X2LfBerzTIVecQ/HOuiUhru6XroSgdYcS4ozg9Qav45yGdpZW6N3Py/Te52X6vLQi6M/aO9WlHw7qpbxTe2n8gPSQPMvlG3e1Nuw+VFdB2f2tSr47EvR+QpxD3lpTcQ5D6+6aqKwQdJCNFWGXkHi9XiUnJ+uPf/yjfvzjHwe233HHHdqyZYvWrl3b7OfDIiFpiqdC2v+Jqos+VPmXf1fywU+VVOtWnGrtjixsmZYhnxwyZciSIbP+ddPrR1+b9dPjmJYh03DIavC+Vb/IqP8p1f9suF1B63Wv67Yd/VzdNst/fMMhSw1f153TMup+SkbgtWUc3V8N9m9RltdGTR86cpM1K8xjP+5/itbx9jl2rYV/vhMl3JalWtNSrc9Ujc9Sjc9Urc+S2cb/tkN53UP1xdEwJsOQ0k9y6eSuSTq5W5K6JSe0LuI2/AJTUV2rb9zVOlBerVJ3tY5466oy/Xt20fgB6U2dpA3nt+F+P9G1cMRJUx4M6Snb+/0d8k6t3377rXw+n3r37h20vXfv3iotLW20v8fjkcdztIe02+1utE9YcKVIuecqMfdcJf7wzqPbTVPyeSWfR6r1/6xfgrY1eM/nDfpp1VartsYrX41HvhqPams8Mms8Mms9Mmu8Mmvr9/N5JZ9Xhs8r+WpkmF4ZZo2cplcOs1ZOq0ZOsyaEf2j/178ph3U0bXC2oFIkSQ7DkkO+E+94PG35N3zs/5Id0kMKsFHnP2i2cx2pXw50zulS6pcB/g3+b8XvJW3snBhs4XSFPCFprw4bZXNsqd2yrCbL7/n5+VqyZElHhdHxHA7JkSjFJ7b5EIak+PolYpimZJmS5av/adZ1GPa/DlpvuN2SLFOWZcoyTZmmT6bpk2VaDV6bMk1TlumTzzKlhu/5TEmmTMuSZVmyzLpjBtYts/5ng3XTkurXzfr31WBffzyGZckyffXv+epjrf9pmrJ0NP4mlw77LaiJrKoFvyF3UH/1dqurTbXgWnX2L5XHhHWi0zd6v4PibdjMkRB3tJkkwemQI/B/agv+rsPsfjAtSz7TDDQNNatFsXfyn6/F19OG696S2Bzhl9mGPCFJT0+X0+lsVA0pKytrVDWRpAULFmjevHmBdbfbrezs7FCHhVBzOFT35IG23UJG/UIXXiA2+f8HAfxCfj8kJCRoxIgRKiwsDNpeWFiosWPHNtrf5XIpNTU1aAEAALGlQ5ps5s2bpxkzZmjkyJEaM2aMnn76aRUXF+vWW2/tiNMBAIAI1yEJyZVXXqlDhw5p6dKlOnDggIYMGaK33npLffv27YjTAQCACNdhM7W2VdgO+wUAAMfV3u9v+hQBAADbkZAAAADbkZAAAADbkZAAAADbkZAAAADbkZAAAADbkZAAAADbkZAAAADbkZAAAADbdcjU8e3hnzjW7XbbHAkAAGgp//d2WyeAD7uEpKKiQpKUnZ1tcyQAAKC1KioqlJaW1urPhd2zbEzT1P79+5WSkiLDMEJ6bLfbrezsbJWUlPCcnE7EdbcH190eXHd7cN3t0fC6p6SkqKKiQllZWXI4Wt8jJOwqJA6HQ3369OnQc6SmpnLD2oDrbg+uuz247vbgutvDf93bUhnxo1MrAACwHQkJAACwXUwlJC6XS4sWLZLL5bI7lJjCdbcH190eXHd7cN3tEcrrHnadWgEAQOyJqQoJAAAITyQkAADAdiQkAADAdiQkAADAdjGTkDz55JPKzc1VYmKiRowYoffff9/ukKLe4sWLZRhG0JKRkWF3WFFn3bp1mjZtmrKysmQYhlavXh30vmVZWrx4sbKyspSUlKS8vDxt377dnmCjyImu+/XXX9/o/j/nnHPsCTZK5Ofna9SoUUpJSVGvXr106aWXaufOnUH7cL+HXkuueyju95hISFatWqW5c+fqnnvu0SeffKIJEyZo6tSpKi4utju0qDd48GAdOHAgsGzbts3ukKJOVVWVhg0bpuXLlzf5/iOPPKLHH39cy5cv18aNG5WRkaHzzz8/8NwotM2JrrskXXDBBUH3/1tvvdWJEUaftWvXavbs2dqwYYMKCwtVW1uryZMnq6qqKrAP93voteS6SyG4360YcPbZZ1u33npr0LZBgwZZd999t00RxYZFixZZw4YNszuMmCLJev311wPrpmlaGRkZ1kMPPRTYVl1dbaWlpVlPPfWUDRFGp2Ovu2VZ1syZM63p06fbEk+sKCsrsyRZa9eutSyL+72zHHvdLSs093vUV0i8Xq82b96syZMnB22fPHmy1q9fb1NUsePLL79UVlaWcnNzddVVV2n37t12hxRTioqKVFpaGnT/u1wunXfeedz/nWDNmjXq1auXfvCDH+imm25SWVmZ3SFFlfLycklS9+7dJXG/d5Zjr7tfe+/3qE9Ivv32W/l8PvXu3Ttoe+/evVVaWmpTVLFh9OjRev755/XOO+/omWeeUWlpqcaOHatDhw7ZHVrM8N/j3P+db+rUqfqv//ovvfvuu3rssce0ceNG/fCHP5TH47E7tKhgWZbmzZun8ePHa8iQIZK43ztDU9ddCs39HnZP++0ohmEErVuW1WgbQmvq1KmB10OHDtWYMWN0yimnaOXKlZo3b56NkcUe7v/Od+WVVwZeDxkyRCNHjlTfvn31l7/8RZdddpmNkUWHOXPmaOvWrfrggw8avcf93nGOd91Dcb9HfYUkPT1dTqezUXZcVlbWKItGx+rSpYuGDh2qL7/80u5QYoZ/VBP3v/0yMzPVt29f7v8QuO222/TGG2/ovffeU58+fQLbud871vGue1Pacr9HfUKSkJCgESNGqLCwMGh7YWGhxo4da1NUscnj8WjHjh3KzMy0O5SYkZubq4yMjKD73+v1au3atdz/nezQoUMqKSnh/m8Hy7I0Z84cvfbaa3r33XeVm5sb9D73e8c40XVvSlvu95hospk3b55mzJihkSNHasyYMXr66adVXFysW2+91e7QotqvfvUrTZs2TTk5OSorK9MDDzwgt9utmTNn2h1aVKmsrNSuXbsC60VFRdqyZYu6d++unJwczZ07V8uWLdPAgQM1cOBALVu2TMnJybrmmmtsjDryNXfdu3fvrsWLF+snP/mJMjMztWfPHv36179Wenq6fvzjH9sYdWSbPXu2XnrpJf35z39WSkpKoBKSlpampKQkGYbB/d4BTnTdKysrQ3O/t2uMTgT593//d6tv375WQkKCNXz48KDhSugYV155pZWZmWnFx8dbWVlZ1mWXXWZt377d7rCiznvvvWdJarTMnDnTsqy6oZCLFi2yMjIyLJfLZZ177rnWtm3b7A06CjR33Q8fPmxNnjzZ6tmzpxUfH2/l5ORYM2fOtIqLi+0OO6I1db0lWc8++2xgH+730DvRdQ/V/W7UnwwAAMA2Ud+HBAAAhD8SEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYDsSEgAAYLv/Dyee7GynVUxAAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", @@ -2000,15 +1955,15 @@ "\n", "\n", "np.random.seed(2018)\n", - "n = 100\n", - "maxdegree = 25\n", + "n = 50\n", + "maxdegree = 5\n", "# Make data set.\n", "x = np.linspace(-3, 3, n).reshape(-1, 1)\n", "y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.normal(0, 0.1, x.shape)\n", "TestError = np.zeros(maxdegree)\n", "TrainError = np.zeros(maxdegree)\n", "polydegree = np.zeros(maxdegree)\n", - "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.5)\n", + "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2)\n", "scaler = StandardScaler()\n", "scaler.fit(x_train)\n", "x_train_scaled = scaler.transform(x_train)\n", @@ -2031,7 +1986,7 @@ }, { "cell_type": "markdown", - "id": "25ffcc2d", + "id": "dbabb429", "metadata": { "editable": true }, @@ -2045,7 +2000,7 @@ }, { "cell_type": "markdown", - "id": "0018d728", + "id": "fd4d35d3", "metadata": { "editable": true }, @@ -2057,7 +2012,7 @@ }, { "cell_type": "markdown", - "id": "34a3e7e2", + "id": "72475f65", "metadata": { "editable": true }, @@ -2069,7 +2024,7 @@ }, { "cell_type": "markdown", - "id": "66f377a9", + "id": "fe531c24", "metadata": { "editable": true }, @@ -2081,7 +2036,7 @@ }, { "cell_type": "markdown", - "id": "3de81002", + "id": "a3430e87", "metadata": { "editable": true }, @@ -2091,7 +2046,7 @@ }, { "cell_type": "markdown", - "id": "fc3cd88f", + "id": "02ea8f17", "metadata": { "editable": true }, @@ -2103,7 +2058,7 @@ }, { "cell_type": "markdown", - "id": "3e95f53c", + "id": "cba7fe69", "metadata": { "editable": true }, @@ -2113,7 +2068,7 @@ }, { "cell_type": "markdown", - "id": "97056993", + "id": "68a5ef20", "metadata": { "editable": true }, @@ -2125,7 +2080,7 @@ }, { "cell_type": "markdown", - "id": "70726c69", + "id": "e0e90274", "metadata": { "editable": true }, @@ -2136,7 +2091,7 @@ }, { "cell_type": "markdown", - "id": "fc38a31a", + "id": "bcc38362", "metadata": { "editable": true }, @@ -2148,7 +2103,7 @@ }, { "cell_type": "markdown", - "id": "72ad5d93", + "id": "1f2209b1", "metadata": { "editable": true }, @@ -2160,7 +2115,7 @@ }, { "cell_type": "markdown", - "id": "0a919fc8", + "id": "5b38d0d9", "metadata": { "editable": true }, @@ -2170,7 +2125,7 @@ }, { "cell_type": "markdown", - "id": "dd828ba6", + "id": "381bb9f1", "metadata": { "editable": true }, @@ -2182,7 +2137,7 @@ }, { "cell_type": "markdown", - "id": "d3a714fa", + "id": "8c247e6e", "metadata": { "editable": true }, @@ -2194,7 +2149,7 @@ }, { "cell_type": "markdown", - "id": "0cfe1576", + "id": "cb0b8048", "metadata": { "editable": true }, @@ -2204,7 +2159,7 @@ }, { "cell_type": "markdown", - "id": "78cc4884", + "id": "9dbf0514", "metadata": { "editable": true }, @@ -2216,7 +2171,7 @@ }, { "cell_type": "markdown", - "id": "8782e97d", + "id": "d5d0f41c", "metadata": { "editable": true }, @@ -2226,7 +2181,7 @@ }, { "cell_type": "markdown", - "id": "13a6373c", + "id": "a8f9a174", "metadata": { "editable": true }, @@ -2238,7 +2193,7 @@ }, { "cell_type": "markdown", - "id": "39482068", + "id": "f3ff3d0d", "metadata": { "editable": true }, @@ -2248,7 +2203,7 @@ }, { "cell_type": "markdown", - "id": "65bd5967", + "id": "b1cfea2c", "metadata": { "editable": true }, @@ -2288,7 +2243,7 @@ }, { "cell_type": "markdown", - "id": "c8a31c83", + "id": "02c449e0", "metadata": { "editable": true }, @@ -2305,7 +2260,7 @@ }, { "cell_type": "markdown", - "id": "1e578cc6", + "id": "7ccce190", "metadata": { "editable": true }, @@ -2328,7 +2283,7 @@ }, { "cell_type": "markdown", - "id": "ceb8ed26", + "id": "5cfffb71", "metadata": { "editable": true }, @@ -2345,7 +2300,7 @@ }, { "cell_type": "markdown", - "id": "d39b6280", + "id": "8c175ba9", "metadata": { "editable": true }, @@ -2364,7 +2319,7 @@ }, { "cell_type": "markdown", - "id": "1f7d68f1", + "id": "3733595c", "metadata": { "editable": true }, @@ -2375,7 +2330,7 @@ }, { "cell_type": "markdown", - "id": "017b4c07", + "id": "3d6b26a5", "metadata": { "editable": true }, @@ -2387,7 +2342,7 @@ }, { "cell_type": "markdown", - "id": "9fbb4bb4", + "id": "1c19da0b", "metadata": { "editable": true }, @@ -2405,7 +2360,7 @@ }, { "cell_type": "markdown", - "id": "31cd3041", + "id": "97d8029f", "metadata": { "editable": true }, @@ -2421,7 +2376,7 @@ }, { "cell_type": "markdown", - "id": "f2c4959c", + "id": "176790e6", "metadata": { "editable": true }, @@ -2433,7 +2388,7 @@ }, { "cell_type": "markdown", - "id": "989f4761", + "id": "5b10a10c", "metadata": { "editable": true }, @@ -2443,7 +2398,7 @@ }, { "cell_type": "markdown", - "id": "3b5d93fb", + "id": "fd6aaf02", "metadata": { "editable": true }, @@ -2456,7 +2411,7 @@ }, { "cell_type": "markdown", - "id": "070c5b35", + "id": "73287deb", "metadata": { "editable": true }, @@ -2468,7 +2423,7 @@ }, { "cell_type": "markdown", - "id": "248fb7ed", + "id": "a2748589", "metadata": { "editable": true }, @@ -2478,7 +2433,7 @@ }, { "cell_type": "markdown", - "id": "3250a8c6", + "id": "bf616cae", "metadata": { "editable": true }, @@ -2491,7 +2446,7 @@ }, { "cell_type": "markdown", - "id": "ece89382", + "id": "268336e2", "metadata": { "editable": true }, @@ -2501,7 +2456,7 @@ }, { "cell_type": "markdown", - "id": "2f766efc", + "id": "b3b06dab", "metadata": { "editable": true }, @@ -2513,7 +2468,7 @@ }, { "cell_type": "markdown", - "id": "8545e937", + "id": "c35ba695", "metadata": { "editable": true }, @@ -2526,7 +2481,7 @@ }, { "cell_type": "markdown", - "id": "71b3340a", + "id": "907b685c", "metadata": { "editable": true }, @@ -2539,7 +2494,7 @@ }, { "cell_type": "markdown", - "id": "65b61e65", + "id": "b7739af7", "metadata": { "editable": true }, @@ -2551,7 +2506,7 @@ }, { "cell_type": "markdown", - "id": "b92530fd", + "id": "28f1e119", "metadata": { "editable": true }, @@ -2563,7 +2518,7 @@ }, { "cell_type": "markdown", - "id": "c6986099", + "id": "b2eb3706", "metadata": { "editable": true }, @@ -2573,7 +2528,7 @@ }, { "cell_type": "markdown", - "id": "307f3384", + "id": "3d6e929d", "metadata": { "editable": true }, @@ -2586,7 +2541,7 @@ }, { "cell_type": "markdown", - "id": "17065042", + "id": "506c9ea1", "metadata": { "editable": true }, @@ -2598,7 +2553,7 @@ }, { "cell_type": "markdown", - "id": "eec876e3", + "id": "1f8a7545", "metadata": { "editable": true }, @@ -2610,7 +2565,7 @@ }, { "cell_type": "markdown", - "id": "d5641fff", + "id": "96e67a6c", "metadata": { "editable": true }, @@ -2622,7 +2577,7 @@ }, { "cell_type": "markdown", - "id": "681d09d1", + "id": "a8e5496a", "metadata": { "editable": true }, @@ -2634,7 +2589,7 @@ }, { "cell_type": "markdown", - "id": "0bfef4bc", + "id": "af2cdec9", "metadata": { "editable": true }, @@ -2648,7 +2603,7 @@ }, { "cell_type": "markdown", - "id": "251f1ae2", + "id": "d5d7fa3b", "metadata": { "editable": true }, @@ -2660,7 +2615,7 @@ }, { "cell_type": "markdown", - "id": "2e8643a5", + "id": "7475fc98", "metadata": { "editable": true }, @@ -2670,7 +2625,7 @@ }, { "cell_type": "markdown", - "id": "6b40de2f", + "id": "6256388d", "metadata": { "editable": true }, @@ -2682,7 +2637,7 @@ }, { "cell_type": "markdown", - "id": "a301055a", + "id": "f86d8ef4", "metadata": { "editable": true }, @@ -2694,7 +2649,7 @@ }, { "cell_type": "markdown", - "id": "0355db96", + "id": "a21b725b", "metadata": { "editable": true }, @@ -2706,7 +2661,7 @@ }, { "cell_type": "markdown", - "id": "ff3a6222", + "id": "c12315f7", "metadata": { "editable": true }, @@ -2718,7 +2673,7 @@ }, { "cell_type": "markdown", - "id": "e823a4e1", + "id": "da1bdc50", "metadata": { "editable": true }, @@ -2730,7 +2685,7 @@ }, { "cell_type": "markdown", - "id": "1d32a056", + "id": "02beeb43", "metadata": { "editable": true }, @@ -2753,13 +2708,10 @@ { "cell_type": "code", "execution_count": 12, - "id": "61d739c6", + "id": "f1d32998", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -2832,7 +2784,7 @@ }, { "cell_type": "markdown", - "id": "98992f36", + "id": "1b7bf147", "metadata": { "editable": true }, @@ -2844,7 +2796,7 @@ }, { "cell_type": "markdown", - "id": "e5242603", + "id": "91e26b57", "metadata": { "editable": true }, @@ -2859,7 +2811,7 @@ }, { "cell_type": "markdown", - "id": "2d8749d4", + "id": "4e52a538", "metadata": { "editable": true }, @@ -2871,7 +2823,7 @@ }, { "cell_type": "markdown", - "id": "cbc682b9", + "id": "f296d012", "metadata": { "editable": true }, @@ -2881,7 +2833,7 @@ }, { "cell_type": "markdown", - "id": "687bc44d", + "id": "eb96aff5", "metadata": { "editable": true }, @@ -2893,7 +2845,7 @@ }, { "cell_type": "markdown", - "id": "9b928369", + "id": "095933fc", "metadata": { "editable": true }, @@ -2903,7 +2855,7 @@ }, { "cell_type": "markdown", - "id": "09c076bd", + "id": "485459fb", "metadata": { "editable": true }, @@ -2915,7 +2867,7 @@ }, { "cell_type": "markdown", - "id": "2d1b1cc0", + "id": "dc378439", "metadata": { "editable": true }, @@ -2927,7 +2879,7 @@ }, { "cell_type": "markdown", - "id": "5b488bce", + "id": "c10033aa", "metadata": { "editable": true }, @@ -2942,7 +2894,7 @@ }, { "cell_type": "markdown", - "id": "d0dcf26d", + "id": "d710a13a", "metadata": { "editable": true }, @@ -2953,7 +2905,7 @@ }, { "cell_type": "markdown", - "id": "87dda710", + "id": "1463c042", "metadata": { "editable": true }, @@ -2973,7 +2925,7 @@ }, { "cell_type": "markdown", - "id": "a15d07ab", + "id": "0f238692", "metadata": { "editable": true }, @@ -2985,7 +2937,7 @@ }, { "cell_type": "markdown", - "id": "90349fd9", + "id": "79337cdd", "metadata": { "editable": true }, @@ -2995,7 +2947,7 @@ }, { "cell_type": "markdown", - "id": "0452fb49", + "id": "43d19beb", "metadata": { "editable": true }, @@ -3007,7 +2959,7 @@ }, { "cell_type": "markdown", - "id": "f5607768", + "id": "fc9ba98c", "metadata": { "editable": true }, @@ -3036,7 +2988,7 @@ }, { "cell_type": "markdown", - "id": "fd676026", + "id": "e1c49876", "metadata": { "editable": true }, @@ -3063,7 +3015,7 @@ }, { "cell_type": "markdown", - "id": "d5b03be9", + "id": "ab5ab11f", "metadata": { "editable": true }, @@ -3074,13 +3026,10 @@ { "cell_type": "code", "execution_count": 13, - "id": "705d044f", + "id": "2cd13083", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -3117,7 +3066,7 @@ }, { "cell_type": "markdown", - "id": "5d3ab3bc", + "id": "eabd32cc", "metadata": { "editable": true }, @@ -3134,7 +3083,7 @@ }, { "cell_type": "markdown", - "id": "b788faec", + "id": "d7f8324f", "metadata": { "editable": true }, @@ -3157,7 +3106,7 @@ }, { "cell_type": "markdown", - "id": "cd20191e", + "id": "d3ab6303", "metadata": { "editable": true }, @@ -3171,7 +3120,7 @@ }, { "cell_type": "markdown", - "id": "27989025", + "id": "e325098f", "metadata": { "editable": true }, @@ -3190,7 +3139,7 @@ }, { "cell_type": "markdown", - "id": "2bf04143", + "id": "4a3be35c", "metadata": { "editable": true }, @@ -3200,7 +3149,7 @@ }, { "cell_type": "markdown", - "id": "cae0bd78", + "id": "e6385438", "metadata": { "editable": true }, @@ -3212,7 +3161,7 @@ }, { "cell_type": "markdown", - "id": "378002e5", + "id": "fd0d8edb", "metadata": { "editable": true }, @@ -3226,7 +3175,7 @@ }, { "cell_type": "markdown", - "id": "8638801d", + "id": "c743c53c", "metadata": { "editable": true }, @@ -3238,7 +3187,7 @@ }, { "cell_type": "markdown", - "id": "61401dde", + "id": "c85f493d", "metadata": { "editable": true }, @@ -3248,7 +3197,7 @@ }, { "cell_type": "markdown", - "id": "f98ff079", + "id": "040c3b45", "metadata": { "editable": true }, @@ -3260,7 +3209,7 @@ }, { "cell_type": "markdown", - "id": "9112055e", + "id": "1909dccd", "metadata": { "editable": true }, @@ -3277,7 +3226,7 @@ }, { "cell_type": "markdown", - "id": "0b8246a5", + "id": "49352aa0", "metadata": { "editable": true }, @@ -3287,7 +3236,7 @@ }, { "cell_type": "markdown", - "id": "3d03e597", + "id": "13063947", "metadata": { "editable": true }, @@ -3303,7 +3252,7 @@ }, { "cell_type": "markdown", - "id": "a1e819ab", + "id": "9764fa95", "metadata": { "editable": true }, @@ -3313,7 +3262,7 @@ }, { "cell_type": "markdown", - "id": "a461c317", + "id": "be4c1a31", "metadata": { "editable": true }, @@ -3329,7 +3278,7 @@ }, { "cell_type": "markdown", - "id": "5e8c0b1a", + "id": "956fd3f2", "metadata": { "editable": true }, @@ -3339,7 +3288,7 @@ }, { "cell_type": "markdown", - "id": "7f29f27c", + "id": "0d96a41b", "metadata": { "editable": true }, @@ -3355,7 +3304,7 @@ }, { "cell_type": "markdown", - "id": "91ea8b5b", + "id": "1f9345dd", "metadata": { "editable": true }, @@ -3365,7 +3314,7 @@ }, { "cell_type": "markdown", - "id": "1e122594", + "id": "c28e9976", "metadata": { "editable": true }, @@ -3382,7 +3331,7 @@ }, { "cell_type": "markdown", - "id": "db9f5639", + "id": "59da3beb", "metadata": { "editable": true }, @@ -3394,7 +3343,7 @@ }, { "cell_type": "markdown", - "id": "3b5f5435", + "id": "8eb5c7ad", "metadata": { "editable": true }, @@ -3406,7 +3355,7 @@ }, { "cell_type": "markdown", - "id": "13d3efbe", + "id": "ef46c055", "metadata": { "editable": true }, @@ -3418,7 +3367,7 @@ }, { "cell_type": "markdown", - "id": "b625a6a4", + "id": "9b25659c", "metadata": { "editable": true }, @@ -3428,7 +3377,7 @@ }, { "cell_type": "markdown", - "id": "b764576d", + "id": "010f41c8", "metadata": { "editable": true }, @@ -3440,7 +3389,7 @@ }, { "cell_type": "markdown", - "id": "bfada944", + "id": "06758e16", "metadata": { "editable": true }, @@ -3452,7 +3401,7 @@ }, { "cell_type": "markdown", - "id": "c7e998f0", + "id": "bdbf298a", "metadata": { "editable": true }, @@ -3464,7 +3413,7 @@ }, { "cell_type": "markdown", - "id": "a0c1879f", + "id": "a4272077", "metadata": { "editable": true }, @@ -3474,7 +3423,7 @@ }, { "cell_type": "markdown", - "id": "f88ca01b", + "id": "cb8c73ef", "metadata": { "editable": true }, @@ -3486,7 +3435,7 @@ }, { "cell_type": "markdown", - "id": "4b357a42", + "id": "8967ae8e", "metadata": { "editable": true }, @@ -3496,7 +3445,7 @@ }, { "cell_type": "markdown", - "id": "87dc7ed8", + "id": "fb30d808", "metadata": { "editable": true }, @@ -3508,7 +3457,7 @@ }, { "cell_type": "markdown", - "id": "1ec63dc4", + "id": "53969156", "metadata": { "editable": true }, @@ -3525,7 +3474,7 @@ }, { "cell_type": "markdown", - "id": "b1cc574b", + "id": "763f5e94", "metadata": { "editable": true }, @@ -3537,7 +3486,7 @@ }, { "cell_type": "markdown", - "id": "90ec138f", + "id": "af22af28", "metadata": { "editable": true }, @@ -3549,7 +3498,7 @@ }, { "cell_type": "markdown", - "id": "1051b8ee", + "id": "63027cc0", "metadata": { "editable": true }, @@ -3559,7 +3508,7 @@ }, { "cell_type": "markdown", - "id": "83e00da0", + "id": "6d4f7a03", "metadata": { "editable": true }, @@ -3571,7 +3520,7 @@ }, { "cell_type": "markdown", - "id": "32c9490a", + "id": "7fcfc0cd", "metadata": { "editable": true }, @@ -3582,7 +3531,7 @@ }, { "cell_type": "markdown", - "id": "d306c461", + "id": "f5d44c36", "metadata": { "editable": true }, @@ -3594,7 +3543,7 @@ }, { "cell_type": "markdown", - "id": "ba097359", + "id": "623c17e3", "metadata": { "editable": true }, @@ -3604,7 +3553,7 @@ }, { "cell_type": "markdown", - "id": "9e371373", + "id": "883abde1", "metadata": { "editable": true }, @@ -3616,7 +3565,7 @@ }, { "cell_type": "markdown", - "id": "a2f7d62c", + "id": "7393a1b9", "metadata": { "editable": true }, @@ -3626,7 +3575,7 @@ }, { "cell_type": "markdown", - "id": "a74f2040", + "id": "4dd69c12", "metadata": { "editable": true }, @@ -3638,7 +3587,7 @@ }, { "cell_type": "markdown", - "id": "3dd63198", + "id": "e71fd75d", "metadata": { "editable": true }, @@ -3649,7 +3598,7 @@ }, { "cell_type": "markdown", - "id": "d381288f", + "id": "4dabe6e8", "metadata": { "editable": true }, @@ -3661,7 +3610,7 @@ }, { "cell_type": "markdown", - "id": "13fadc82", + "id": "96201b0a", "metadata": { "editable": true }, @@ -3679,7 +3628,7 @@ }, { "cell_type": "markdown", - "id": "97cd992c", + "id": "940c56fa", "metadata": { "editable": true }, @@ -3695,7 +3644,7 @@ }, { "cell_type": "markdown", - "id": "18aa6e8e", + "id": "ea6a1fb1", "metadata": { "editable": true }, @@ -3707,7 +3656,7 @@ }, { "cell_type": "markdown", - "id": "bd513114", + "id": "991316eb", "metadata": { "editable": true }, @@ -3719,7 +3668,7 @@ }, { "cell_type": "markdown", - "id": "3bf1cdc2", + "id": "93aa9e83", "metadata": { "editable": true }, @@ -3731,7 +3680,7 @@ }, { "cell_type": "markdown", - "id": "354f9690", + "id": "6a6c2628", "metadata": { "editable": true }, @@ -3744,7 +3693,7 @@ }, { "cell_type": "markdown", - "id": "0e79179e", + "id": "db4254c1", "metadata": { "editable": true }, @@ -3760,7 +3709,7 @@ }, { "cell_type": "markdown", - "id": "aac2535e", + "id": "251c4816", "metadata": { "editable": true }, @@ -3774,7 +3723,7 @@ }, { "cell_type": "markdown", - "id": "82fb4a86", + "id": "3726ee5e", "metadata": { "editable": true }, @@ -3784,7 +3733,7 @@ }, { "cell_type": "markdown", - "id": "9e695331", + "id": "e0f575a7", "metadata": { "editable": true }, @@ -3796,7 +3745,7 @@ }, { "cell_type": "markdown", - "id": "c172833d", + "id": "e548131c", "metadata": { "editable": true }, @@ -3806,7 +3755,7 @@ }, { "cell_type": "markdown", - "id": "8c8f4baf", + "id": "d60225e8", "metadata": { "editable": true }, @@ -3818,7 +3767,7 @@ }, { "cell_type": "markdown", - "id": "84c90a7d", + "id": "89d9aae5", "metadata": { "editable": true }, @@ -3828,7 +3777,7 @@ }, { "cell_type": "markdown", - "id": "fcdc67b8", + "id": "5f4ae97f", "metadata": { "editable": true }, @@ -3842,7 +3791,7 @@ }, { "cell_type": "markdown", - "id": "92b5d75b", + "id": "76b9235f", "metadata": { "editable": true }, @@ -3859,7 +3808,7 @@ }, { "cell_type": "markdown", - "id": "5b46e0ea", + "id": "290a9403", "metadata": { "editable": true }, @@ -3875,7 +3824,7 @@ }, { "cell_type": "markdown", - "id": "553faa01", + "id": "832b31ab", "metadata": { "editable": true }, @@ -3887,7 +3836,7 @@ }, { "cell_type": "markdown", - "id": "69eda40a", + "id": "10a81bbb", "metadata": { "editable": true }, @@ -3900,7 +3849,7 @@ }, { "cell_type": "markdown", - "id": "1779b003", + "id": "1dd8962b", "metadata": { "editable": true }, @@ -3914,7 +3863,7 @@ }, { "cell_type": "markdown", - "id": "ebab9cca", + "id": "47f1db97", "metadata": { "editable": true }, @@ -3924,7 +3873,7 @@ }, { "cell_type": "markdown", - "id": "83698561", + "id": "833510b2", "metadata": { "editable": true }, @@ -3937,7 +3886,7 @@ }, { "cell_type": "markdown", - "id": "56a63474", + "id": "4f43985b", "metadata": { "editable": true }, @@ -3956,7 +3905,7 @@ }, { "cell_type": "markdown", - "id": "d38a14ec", + "id": "81c592cc", "metadata": { "editable": true }, @@ -3968,7 +3917,7 @@ }, { "cell_type": "markdown", - "id": "232852dc", + "id": "d1a2a046", "metadata": { "editable": true }, @@ -3980,7 +3929,7 @@ }, { "cell_type": "markdown", - "id": "6c092be4", + "id": "e59142de", "metadata": { "editable": true }, @@ -3990,7 +3939,7 @@ }, { "cell_type": "markdown", - "id": "b8b668fe", + "id": "4729dbde", "metadata": { "editable": true }, @@ -4002,7 +3951,7 @@ }, { "cell_type": "markdown", - "id": "e47e3740", + "id": "0cbc86c1", "metadata": { "editable": true }, @@ -4015,7 +3964,7 @@ }, { "cell_type": "markdown", - "id": "2ed923ca", + "id": "ca29148b", "metadata": { "editable": true }, @@ -4034,7 +3983,7 @@ }, { "cell_type": "markdown", - "id": "74ae3bd7", + "id": "660a2ac1", "metadata": { "editable": true }, @@ -4044,7 +3993,7 @@ }, { "cell_type": "markdown", - "id": "615f3237", + "id": "5a435a0a", "metadata": { "editable": true }, @@ -4063,7 +4012,7 @@ }, { "cell_type": "markdown", - "id": "c6038c4d", + "id": "764e2e96", "metadata": { "editable": true }, @@ -4081,7 +4030,7 @@ }, { "cell_type": "markdown", - "id": "4d486ff6", + "id": "d1789d6c", "metadata": { "editable": true }, @@ -4095,7 +4044,7 @@ }, { "cell_type": "markdown", - "id": "7f77bb46", + "id": "c6d6fec5", "metadata": { "editable": true }, @@ -4110,13 +4059,10 @@ { "cell_type": "code", "execution_count": 14, - "id": "e8071f32", + "id": "e6914cd6", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -4134,7 +4080,7 @@ }, { "cell_type": "markdown", - "id": "f0945a2f", + "id": "3a53a2a1", "metadata": { "editable": true }, @@ -4151,13 +4097,10 @@ { "cell_type": "code", "execution_count": 15, - "id": "6f47d535", + "id": "ec053f1e", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -4186,7 +4129,7 @@ }, { "cell_type": "markdown", - "id": "240fa490", + "id": "521e11ff", "metadata": { "editable": true }, @@ -4200,7 +4143,7 @@ }, { "cell_type": "markdown", - "id": "faa318d1", + "id": "0aef5f95", "metadata": { "editable": true }, @@ -4213,13 +4156,10 @@ { "cell_type": "code", "execution_count": 16, - "id": "7b8bd392", + "id": "3dbd8c2a", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -4241,7 +4181,7 @@ }, { "cell_type": "markdown", - "id": "89b76e07", + "id": "45dc0cd3", "metadata": { "editable": true }, @@ -4253,7 +4193,7 @@ }, { "cell_type": "markdown", - "id": "865776f5", + "id": "85285784", "metadata": { "editable": true }, @@ -4265,7 +4205,7 @@ }, { "cell_type": "markdown", - "id": "e5b1501e", + "id": "1acb278d", "metadata": { "editable": true }, @@ -4275,7 +4215,7 @@ }, { "cell_type": "markdown", - "id": "8ce4ac5d", + "id": "061cf852", "metadata": { "editable": true }, @@ -4292,7 +4232,7 @@ }, { "cell_type": "markdown", - "id": "a5eeb5c4", + "id": "255b8070", "metadata": { "editable": true }, @@ -4302,7 +4242,7 @@ }, { "cell_type": "markdown", - "id": "a167c7f9", + "id": "cfff2c26", "metadata": { "editable": true }, @@ -4317,7 +4257,7 @@ }, { "cell_type": "markdown", - "id": "3b537b6d", + "id": "2d8af4a2", "metadata": { "editable": true }, @@ -4327,7 +4267,7 @@ }, { "cell_type": "markdown", - "id": "b3872efa", + "id": "333e3603", "metadata": { "editable": true }, @@ -4341,7 +4281,7 @@ }, { "cell_type": "markdown", - "id": "c954dedd", + "id": "e2eb99a0", "metadata": { "editable": true }, @@ -4353,7 +4293,7 @@ }, { "cell_type": "markdown", - "id": "4905e610", + "id": "e4d380d0", "metadata": { "editable": true }, @@ -4365,7 +4305,7 @@ }, { "cell_type": "markdown", - "id": "ccd19e7e", + "id": "28a2dd25", "metadata": { "editable": true }, @@ -4377,7 +4317,7 @@ }, { "cell_type": "markdown", - "id": "e843eedf", + "id": "cceb8b2c", "metadata": { "editable": true }, @@ -4387,7 +4327,7 @@ }, { "cell_type": "markdown", - "id": "ceebbf65", + "id": "6faf5750", "metadata": { "editable": true }, @@ -4399,7 +4339,7 @@ }, { "cell_type": "markdown", - "id": "98520fcd", + "id": "d60d582e", "metadata": { "editable": true }, @@ -4409,7 +4349,7 @@ }, { "cell_type": "markdown", - "id": "87129903", + "id": "84c6c9a6", "metadata": { "editable": true }, @@ -4426,7 +4366,7 @@ }, { "cell_type": "markdown", - "id": "a493112c", + "id": "42c17867", "metadata": { "editable": true }, @@ -4436,7 +4376,7 @@ }, { "cell_type": "markdown", - "id": "fe9e685b", + "id": "5abca958", "metadata": { "editable": true }, @@ -4448,7 +4388,7 @@ }, { "cell_type": "markdown", - "id": "3fa486a8", + "id": "8840539e", "metadata": { "editable": true }, @@ -4458,7 +4398,7 @@ }, { "cell_type": "markdown", - "id": "5bf80f3f", + "id": "bb62c206", "metadata": { "editable": true }, @@ -4470,7 +4410,7 @@ }, { "cell_type": "markdown", - "id": "b56829bb", + "id": "b07a0443", "metadata": { "editable": true }, @@ -4484,7 +4424,7 @@ }, { "cell_type": "markdown", - "id": "5e44a2f3", + "id": "21dbb160", "metadata": { "editable": true }, @@ -4496,7 +4436,7 @@ }, { "cell_type": "markdown", - "id": "f57c9780", + "id": "94e2b39d", "metadata": { "editable": true }, @@ -4518,7 +4458,7 @@ }, { "cell_type": "markdown", - "id": "12f75bab", + "id": "c0c57304", "metadata": { "editable": true }, @@ -4530,7 +4470,7 @@ }, { "cell_type": "markdown", - "id": "87c1bfb8", + "id": "fba325af", "metadata": { "editable": true }, @@ -4545,7 +4485,7 @@ }, { "cell_type": "markdown", - "id": "7b5f03d6", + "id": "2f93e108", "metadata": { "editable": true }, @@ -4557,7 +4497,7 @@ }, { "cell_type": "markdown", - "id": "76f0262c", + "id": "0980890f", "metadata": { "editable": true }, @@ -4569,7 +4509,7 @@ }, { "cell_type": "markdown", - "id": "25d9f78c", + "id": "083e0c4f", "metadata": { "editable": true }, @@ -4579,7 +4519,7 @@ }, { "cell_type": "markdown", - "id": "06f7c207", + "id": "018f833f", "metadata": { "editable": true }, @@ -4591,7 +4531,7 @@ }, { "cell_type": "markdown", - "id": "dda74f50", + "id": "e24970b9", "metadata": { "editable": true }, @@ -4601,7 +4541,7 @@ }, { "cell_type": "markdown", - "id": "28a122c4", + "id": "717244d4", "metadata": { "editable": true }, @@ -4613,7 +4553,7 @@ }, { "cell_type": "markdown", - "id": "2b44b951", + "id": "4af271cc", "metadata": { "editable": true }, @@ -4623,7 +4563,7 @@ }, { "cell_type": "markdown", - "id": "9a7f9681", + "id": "fd37dbba", "metadata": { "editable": true }, @@ -4635,7 +4575,7 @@ }, { "cell_type": "markdown", - "id": "128ebe6e", + "id": "4994c663", "metadata": { "editable": true }, @@ -4652,7 +4592,7 @@ }, { "cell_type": "markdown", - "id": "148b103f", + "id": "619fc071", "metadata": { "editable": true }, @@ -4665,7 +4605,7 @@ }, { "cell_type": "markdown", - "id": "8b15c65e", + "id": "4aa4b3cd", "metadata": { "editable": true }, @@ -4677,7 +4617,7 @@ }, { "cell_type": "markdown", - "id": "0bdac1f3", + "id": "30146bc3", "metadata": { "editable": true }, @@ -4687,7 +4627,7 @@ }, { "cell_type": "markdown", - "id": "dec96c67", + "id": "a6931d07", "metadata": { "editable": true }, @@ -4700,7 +4640,7 @@ }, { "cell_type": "markdown", - "id": "1d1ec2a3", + "id": "28849f6b", "metadata": { "editable": true }, @@ -4710,7 +4650,7 @@ }, { "cell_type": "markdown", - "id": "932564cc", + "id": "189c09b8", "metadata": { "editable": true }, @@ -4722,7 +4662,7 @@ }, { "cell_type": "markdown", - "id": "754083f9", + "id": "2cc85459", "metadata": { "editable": true }, @@ -4735,7 +4675,7 @@ }, { "cell_type": "markdown", - "id": "81b6d9c0", + "id": "63cc18f2", "metadata": { "editable": true }, @@ -4748,7 +4688,7 @@ }, { "cell_type": "markdown", - "id": "6746454b", + "id": "26ec85d7", "metadata": { "editable": true }, @@ -4760,7 +4700,7 @@ }, { "cell_type": "markdown", - "id": "db1e44ec", + "id": "d10bf29f", "metadata": { "editable": true }, @@ -4772,7 +4712,7 @@ }, { "cell_type": "markdown", - "id": "825d33de", + "id": "d80c1950", "metadata": { "editable": true }, @@ -4782,7 +4722,7 @@ }, { "cell_type": "markdown", - "id": "fbef6c01", + "id": "8c4ef7d9", "metadata": { "editable": true }, @@ -4795,7 +4735,7 @@ }, { "cell_type": "markdown", - "id": "0433c256", + "id": "d220a4fa", "metadata": { "editable": true }, @@ -4807,7 +4747,7 @@ }, { "cell_type": "markdown", - "id": "c9e04623", + "id": "a754dfcb", "metadata": { "editable": true }, @@ -4819,7 +4759,7 @@ }, { "cell_type": "markdown", - "id": "7eb7e396", + "id": "84ce95ee", "metadata": { "editable": true }, @@ -4836,7 +4776,7 @@ }, { "cell_type": "markdown", - "id": "91d378d9", + "id": "eb70c3ba", "metadata": { "editable": true }, @@ -4848,7 +4788,7 @@ }, { "cell_type": "markdown", - "id": "29f2189d", + "id": "b78198ec", "metadata": { "editable": true }, @@ -4858,7 +4798,7 @@ }, { "cell_type": "markdown", - "id": "b953d8b5", + "id": "42500af7", "metadata": { "editable": true }, @@ -4870,7 +4810,7 @@ }, { "cell_type": "markdown", - "id": "738c94fc", + "id": "c27c4731", "metadata": { "editable": true }, @@ -4880,7 +4820,7 @@ }, { "cell_type": "markdown", - "id": "8298e6ef", + "id": "3a231959", "metadata": { "editable": true }, @@ -4892,7 +4832,7 @@ }, { "cell_type": "markdown", - "id": "9d8a12d4", + "id": "708b2568", "metadata": { "editable": true }, @@ -4904,7 +4844,7 @@ }, { "cell_type": "markdown", - "id": "0dbf3801", + "id": "3f444e5e", "metadata": { "editable": true }, @@ -4920,7 +4860,7 @@ }, { "cell_type": "markdown", - "id": "4c85457b", + "id": "28a39069", "metadata": { "editable": true }, @@ -4932,7 +4872,7 @@ }, { "cell_type": "markdown", - "id": "446d975e", + "id": "3b7f1ec3", "metadata": { "editable": true }, @@ -4944,7 +4884,7 @@ }, { "cell_type": "markdown", - "id": "3efe9a50", + "id": "58c302ac", "metadata": { "editable": true }, @@ -4954,7 +4894,7 @@ }, { "cell_type": "markdown", - "id": "4d1cc63b", + "id": "dfbc40a2", "metadata": { "editable": true }, @@ -4966,7 +4906,7 @@ }, { "cell_type": "markdown", - "id": "5a2e33a2", + "id": "27a3dfa6", "metadata": { "editable": true }, @@ -4976,7 +4916,7 @@ }, { "cell_type": "markdown", - "id": "d976df6e", + "id": "eab4118d", "metadata": { "editable": true }, @@ -4988,7 +4928,7 @@ }, { "cell_type": "markdown", - "id": "bac1c3ac", + "id": "1b6c78a5", "metadata": { "editable": true }, @@ -5005,7 +4945,7 @@ }, { "cell_type": "markdown", - "id": "06425611", + "id": "0c7d1aa4", "metadata": { "editable": true }, @@ -5017,7 +4957,7 @@ }, { "cell_type": "markdown", - "id": "0aa9954c", + "id": "66be627d", "metadata": { "editable": true }, @@ -5029,7 +4969,7 @@ }, { "cell_type": "markdown", - "id": "9cdcf4b9", + "id": "8890e9c1", "metadata": { "editable": true }, @@ -5039,7 +4979,7 @@ }, { "cell_type": "markdown", - "id": "33ebeaa6", + "id": "dd17e6a0", "metadata": { "editable": true }, @@ -5051,7 +4991,7 @@ }, { "cell_type": "markdown", - "id": "c09bd783", + "id": "e4901607", "metadata": { "editable": true }, @@ -5061,7 +5001,7 @@ }, { "cell_type": "markdown", - "id": "aabc4acf", + "id": "2eb73708", "metadata": { "editable": true }, @@ -5073,7 +5013,7 @@ }, { "cell_type": "markdown", - "id": "d96cfdd8", + "id": "1e01a932", "metadata": { "editable": true }, @@ -5083,7 +5023,7 @@ }, { "cell_type": "markdown", - "id": "77395e8b", + "id": "32808d90", "metadata": { "editable": true }, @@ -5095,7 +5035,7 @@ }, { "cell_type": "markdown", - "id": "4006cd46", + "id": "55548599", "metadata": { "editable": true }, @@ -5105,7 +5045,7 @@ }, { "cell_type": "markdown", - "id": "7a00b933", + "id": "b5750cba", "metadata": { "editable": true }, @@ -5117,7 +5057,7 @@ }, { "cell_type": "markdown", - "id": "67eb791e", + "id": "22aeaf24", "metadata": { "editable": true }, @@ -5127,7 +5067,7 @@ }, { "cell_type": "markdown", - "id": "d42d1709", + "id": "5b553c45", "metadata": { "editable": true }, @@ -5137,7 +5077,7 @@ }, { "cell_type": "markdown", - "id": "1378b008", + "id": "9b726931", "metadata": { "editable": true }, @@ -5158,8 +5098,8 @@ "when all predictors are zero (the columns in the design matrix), it\n", "may be a bad idea to implement a model which penalizes the intercept.\n", "Furthermore, in for example Ridge and Lasso regression, the default solutions\n", - "from the library **Scikit-Learn** (when not shrinking $\\beta_0$) for the unknown parameters\n", - "$\\boldsymbol{\\beta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n", + "from the library **Scikit-Learn** (when not shrinking $\\theta_0$) for the unknown parameters\n", + "$\\boldsymbol{\\theta}$, are derived under the assumption that both $\\boldsymbol{y}$ and\n", "$\\boldsymbol{X}$ are zero centered, that is we subtract the mean values.\n", "\n", "If our predictors represent different scales, then it is important to\n", @@ -5196,13 +5136,10 @@ { "cell_type": "code", "execution_count": 17, - "id": "047c9fe6", + "id": "767365c7", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -5226,7 +5163,7 @@ }, { "cell_type": "markdown", - "id": "05573c93", + "id": "3682c045", "metadata": { "editable": true }, @@ -5240,19 +5177,19 @@ }, { "cell_type": "markdown", - "id": "5fdb2a81", + "id": "d31a8d9d", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", + "C(\\theta_0, \\theta_1, ... , \\theta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij}\\theta_j\\right)^2,.\n", "$$" ] }, { "cell_type": "markdown", - "id": "d34da7b7", + "id": "16680525", "metadata": { "editable": true }, @@ -5261,49 +5198,49 @@ "increased penalty for higher differences between predicted and\n", "output/target values.\n", "\n", - "What we have done is to single out the $\\beta_0$ term in the\n", + "What we have done is to single out the $\\theta_0$ term in the\n", "definition of the mean squared error (MSE). The design matrix $X$\n", "does in this case not contain any intercept column. When we take the\n", - "derivative with respect to $\\beta_0$, we want the derivative to obey" + "derivative with respect to $\\theta_0$, we want the derivative to obey" ] }, { "cell_type": "markdown", - "id": "913166a1", + "id": "e2b809f7", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", + "\\frac{\\partial C}{\\partial \\theta_j} = 0,\n", "$$" ] }, { "cell_type": "markdown", - "id": "e8d5194d", + "id": "58a1d22e", "metadata": { "editable": true }, "source": [ - "for all $j$. For $\\beta_0$ we have" + "for all $j$. For $\\theta_0$ we have" ] }, { "cell_type": "markdown", - "id": "cef3d8b6", + "id": "409ab1ff", "metadata": { "editable": true }, "source": [ "$$\n", - "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", + "\\frac{\\partial C}{\\partial \\theta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\theta_0 - \\sum_{j=1}^{p-1} X_{ij} \\theta_j\\right).\n", "$$" ] }, { "cell_type": "markdown", - "id": "1779c0a7", + "id": "caebebfa", "metadata": { "editable": true }, @@ -5313,42 +5250,42 @@ }, { "cell_type": "markdown", - "id": "1e515abc", + "id": "8f1afa7f", "metadata": { "editable": true }, "source": [ "$$\n", - "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", + "\\sum_{i=0}^{n-1} \\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\theta_j.\n", "$$" ] }, { "cell_type": "markdown", - "id": "3f956af0", + "id": "85775b22", "metadata": { "editable": true }, "source": [ - "Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n", - "Our result for $\\beta_0$ simplifies then to" + "Let us specialize first to the case where we have only two parameters $\\theta_0$ and $\\theta_1$.\n", + "Our result for $\\theta_0$ simplifies then to" ] }, { "cell_type": "markdown", - "id": "ff21009c", + "id": "08dd11f6", "metadata": { "editable": true }, "source": [ "$$\n", - "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", + "n\\theta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\theta_1.\n", "$$" ] }, { "cell_type": "markdown", - "id": "379a0219", + "id": "5d2e98fe", "metadata": { "editable": true }, @@ -5358,19 +5295,19 @@ }, { "cell_type": "markdown", - "id": "7da4a38e", + "id": "42da16f4", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", + "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\theta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "2f5b0bec", + "id": "8c434ec5", "metadata": { "editable": true }, @@ -5380,7 +5317,7 @@ }, { "cell_type": "markdown", - "id": "37d8c686", + "id": "23217d9c", "metadata": { "editable": true }, @@ -5392,7 +5329,7 @@ }, { "cell_type": "markdown", - "id": "18de9121", + "id": "85202a44", "metadata": { "editable": true }, @@ -5402,7 +5339,7 @@ }, { "cell_type": "markdown", - "id": "981c514c", + "id": "595571b3", "metadata": { "editable": true }, @@ -5414,7 +5351,7 @@ }, { "cell_type": "markdown", - "id": "b1ec54e1", + "id": "507ad25e", "metadata": { "editable": true }, @@ -5424,41 +5361,41 @@ }, { "cell_type": "markdown", - "id": "3a172fc7", + "id": "c1af1141", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n", + "\\theta_0 = \\mu_y - \\theta_1\\mu_{\\boldsymbol{x}_1}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "2c115121", + "id": "22b1a4c2", "metadata": { "editable": true }, "source": [ - "In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have" + "In the general case with more parameters than $\\theta_0$ and $\\theta_1$, we have" ] }, { "cell_type": "markdown", - "id": "47003d34", + "id": "344d7aaa", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", + "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\theta_j.\n", "$$" ] }, { "cell_type": "markdown", - "id": "dabbcaa4", + "id": "d1d8e7a0", "metadata": { "editable": true }, @@ -5468,19 +5405,19 @@ }, { "cell_type": "markdown", - "id": "7b34c620", + "id": "9a0b634c", "metadata": { "editable": true }, "source": [ "$$\n", - "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n", + "\\theta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\theta_j,\n", "$$" ] }, { "cell_type": "markdown", - "id": "ca2d47f5", + "id": "aad7ad10", "metadata": { "editable": true }, @@ -5490,7 +5427,7 @@ }, { "cell_type": "markdown", - "id": "4789c005", + "id": "b0c4c38f", "metadata": { "editable": true }, @@ -5502,7 +5439,7 @@ }, { "cell_type": "markdown", - "id": "58c60577", + "id": "b52a1b70", "metadata": { "editable": true }, @@ -5514,41 +5451,41 @@ }, { "cell_type": "markdown", - "id": "f2073926", + "id": "90e55e2a", "metadata": { "editable": true }, "source": [ "$$\n", - "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", + "C(\\boldsymbol{\\theta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\theta}).\n", "$$" ] }, { "cell_type": "markdown", - "id": "fbea29ab", + "id": "01b093ef", "metadata": { "editable": true }, "source": [ - "If we minimize with respect to $\\boldsymbol{\\beta}$ we have then" + "If we minimize with respect to $\\boldsymbol{\\theta}$ we have then" ] }, { "cell_type": "markdown", - "id": "df6c68bf", + "id": "18810909", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", + "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", "$$" ] }, { "cell_type": "markdown", - "id": "57191282", + "id": "112af953", "metadata": { "editable": true }, @@ -5556,24 +5493,24 @@ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", "\n", - "For Ridge regression we need to add $\\lambda \\boldsymbol{\\beta}^T\\boldsymbol{\\beta}$ to the cost function and get then" + "For Ridge regression we need to add $\\lambda \\boldsymbol{\\theta}^T\\boldsymbol{\\theta}$ to the cost function and get then" ] }, { "cell_type": "markdown", - "id": "1b09c81d", + "id": "dc42bdf3", "metadata": { "editable": true }, "source": [ "$$\n", - "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", + "\\hat{\\boldsymbol{\\theta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", "$$" ] }, { "cell_type": "markdown", - "id": "d4a64271", + "id": "9c9fd729", "metadata": { "editable": true }, @@ -5586,48 +5523,13 @@ }, { "cell_type": "code", - "execution_count": 1, - "id": "59c77c8b", + "execution_count": 18, + "id": "94867b33", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "True beta: [2, 0.5, 3.7]\n", - "Fitted beta: [2.08376632 0.19569961 3.97898392]\n", - "Sklearn fitted beta: [2.08376632 0.19569961 3.97898392]\n", - "MSE with intercept column\n", - "0.004113634617443139\n", - "MSE with intercept column from SKL\n", - "0.004113634617443147\n", - "Manual intercept: 2.083766322923899\n", - "Fitted beta (without intercept): [0.19569961 3.97898392]\n", - "Sklearn intercept: 2.0837663229239043\n", - "Sklearn fitted beta (without intercept): [0.19569961 3.97898392]\n", - "MSE with Manual intercept\n", - "0.00411363461744314\n", - "MSE with Sklearn intercept\n", - "0.004113634617443131\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAGdCAYAAABO2DpVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAABswElEQVR4nO3dd3gUVffA8e9m0yskpAGBhE4IBBBDkypNqgjSlKqgCEp5Ebs0pUkTFRSQ8gMBBSlKCaB0kF4khBoSaiCEQHrZMr8/8rIvSxJIILubcj7Ps4/u7Mzds4dN5uTeO3dUiqIoCCGEEELkAytLByCEEEKIokMKCyGEEELkGykshBBCCJFvpLAQQgghRL6RwkIIIYQQ+UYKCyGEEELkGykshBBCCJFvpLAQQgghRL6xNvcb6vV6bt26hYuLCyqVytxvL4QQQohnoCgKiYmJlC5dGiurnPslzF5Y3Lp1Cz8/P3O/rRBCCCHywfXr1ylbtmyOr5u9sHBxcQEyA3N1dc23djUaDdu3b6dNmzbY2NjkW7vCmOTZfCTX5iF5Ng/Js3mYMs8JCQn4+fkZzuM5MXth8XD4w9XVNd8LC0dHR1xdXeVLa0KSZ/ORXJuH5Nk8JM/mYY48P20ag0zeFEIIIUS+kcJCCCGEEPlGCgshhBBC5Buzz7HIDZ1Oh0ajydMxGo0Ga2tr0tLS0Ol0JopMSJ7NpyjlWq1WY21tLZeYC1EMFLjCIikpiRs3bqAoSp6OUxQFHx8frl+/Lr+8TEjybD5FLdeOjo74+vpia2tr6VCEECZUoAoLnU7HjRs3cHR0xNPTM0+/TPV6PUlJSTg7Oz9x4Q7xfCTP5lNUcq0oChkZGdy9e5fIyEgqV65cqD+PEOLJClRhodFoUBQFT09PHBwc8nSsXq8nIyMDe3t7+aVlQpJn8ylKuXZwcMDGxoarV68aPpMQomgqkL+tikK3rxDCWGEvjoQQuSM/6UIIIYTIN1JYCCGEECLfSGEhhBBCiHwjhUU+GDBgACqVCpVKhY2NDd7e3rRu3ZrFixej1+tz3c7SpUspUaKE6QIVQgghTKxIFhY6vcI/EffYeOom/0TcQ6fP25oYz6Jdu3ZER0cTFRXF1q1badGiBSNGjKBjx45otVqTv78QQojiTadX2Dl/JFYXNnLoUoxZzn3ZKXKFRWjYbV6atpPeCw8xYvUpei88xEvTdhIaFm3S97Wzs8PHx4cyZcpQt25dPv30UzZu3MjWrVtZunQpALNmzaJmzZo4OTnh5+fHe++9R1JSEgC7d+9m4MCBxMfHG3o/xo8fD8CKFSuoV68eLi4u+Pj40KdPH2JiYkz6eYQQQhQeoWHRfD56AJzYQofk31n4y3KznPuyU6QKi78v3GPYypNEx6cZbb8dn8bQFSfMnuCWLVsSHBzMunXrgMzL7ebOnUtYWBjLli1j586djB07FoBGjRoxZ84cXF1diY6OJjo6mjFjxgCQkZHBpEmTOH36NBs2bCAyMpIBAwaY9bMIIYQomELDopm8dA2d9x2h3CF7Vt6oymGCLHbuK1ALZD0PnV5h+l9XyK7jRwFUwIQ/w2kd6IPaynzrZFSrVo1///0XgJEjRxq2BwQEMGnSJIYOHcq8efOwtbXFzc0NlUqFj4+PURuDBg0y/H+FChWYO3cuISEhhlUZhRBCFE86vcK4Df/yydkFlEiGO+5g2/JdCLfcua/I9FgcjYrjTmJGjq8rQHR8Gkci48wXFJnLGT9c8GvXrl20bt2aMmXK4OLiQr9+/bh37x7JyclPbOPkyZN06dKF8uXL4+LiQvPmzQG4du2aqcMXQghRgB2JjKPX5SlUjdShUcOykLbYOXsYXrfEua/IFBYxiem53C/t6Tvlo3PnzhEQEMDVq1dp3749QUFB/P777xw/fpwffvgB4Il3ck1OTqZNmzY4OzuzYsUKjh49yvr164HMIRIhhBDF17kdi2h59A4Au+p5cdCpVbb7mfPcV2SGQrxc7HK5n/nuUbBz507OnDnDqFGjOHbsGFqtlpkzZxqWNv7tt9+M9re1tc1ye+zz588TGxvL1KlT8fPzA+DYsWPm+QBCCCEKrLiYmwQsX4qtDi76WzHb5z/kdCY057mvyPRYvOjvjreLLTmNIKkAXzd7QgLcTfL+6enp3L59m5s3b3LixAkmT55Mly5d6NixI/369aNixYpotVq+++47rly5wvLly/nxxx+N2vD39ycpKYm///6b2NhYUlJSKFeuHLa2tobj/vjjDyZNmmSSzyCEEKLw2DVrIKXuQ7wjzK4xBFTqLPuY+tyXnSJTWKitVIxtVQEgS3Hx8Pm4ToEmm7wSGhqKr68v/v7+tGvXjl27djF37lw2btyIWq2mdu3azJo1i2nTphEUFMQvv/zClClTjNpo1KgR7777Lj179sTT05Pp06fj6enJ0qVLWbNmDYGBgUydOpUZM2aY5DMIIYQoHE5uX0E3+8Oo2j0g/PWXuWpTySLnvuyoFEUx6woaCQkJuLm5ER8fj6urq9FraWlpREZGEhAQkOfbKuv1ehISEjh4LYVJm88ZXXLq62bPuE6BtAvyzZfPUJw9zLOrq6vcrdLEilqun+fn25Q0Gg1btmyhffv22NjYWDqcIkvynH/u3IjAblFTSpDEIZ83aPDuPELDopnwZzhxSalMD9Ex9ogad2eHfD33Pen8/agiM8fioXZBPrQN8uVIZBwxiWl4uWR2AZmzWhNCCCFMIT01hf0jX6WGv5a7bpWoO3AWAO2CfGkd6MOhyzHEnjvE4v4v0qCSl0XOfUWusIDMYZGGFT2evqMQQghRiKwb8Qq1w7QkRLhTYum32Nr9r/dPbaUiJMCdLeew6B/Uhb9/VQghhCgGti+dRM19mbdziGwXSJXglywcUfaksBBCCCEKuFtR53CYvxK1Auer2tB90q+WDilHUlgIIYQQBZii13NoZC9KxcM9VwiZsxq1dcGdySCFhRBCCFGArfm8J9XPZ6BTQfK7vSkTEGjpkJ5ICgshhBCigLpxOQynU2cACHvJk7aDvrRwRE8nhYUQQghRAGWkp5G6egBt60VzroU9XeeGWjqkXMlzYXHz5k3efPNNPDw8cHR0pHbt2hw/ftwUsRV6zZs3N7pVuhBCCJFbx5f+h8raSyRbOdHo0zXYOThaOqRcyVNhcf/+fRo3boyNjQ1bt24lPDycmTNnUqJECROFVzgMGDAAlUqV5TF9+nSj+3r4+/szZ84cywUqhBCiUPhz7mju7dhOik7FlYZT8PGrZOmQci1P00qnTZuGn58fS5YsMWzz9/fP75gKpXbt2hnlBcDT0xO1OutNYYQQQoicRJw5iOeyrbglW/OHc1V6te1v6ZDyJE+FxR9//EHbtm15/fXX2bNnD2XKlOG9995j8ODBOR6Tnp5Oenq64XlCQgKQuW68RqMx2lej0aAoCnq9Hr1en5fQeHjLk4fHm5OiKNja2uLl5WW0vWXLlgQHBzN79mxatmzJ1atXGTVqFKNGjQLIcov0wsCSeS5uilqu9Xo9iqKg0WgKVMH98PfQ47+PRP6SPOeOTqvl7NghVE6G2+7Q4qtVecqZKfOc2zbzVFhcuXKF+fPnM3r0aD799FOOHDnCBx98gJ2dHf369cv2mClTpjBhwoQs27dv346jo/F4kbW1NT4+PiQlJZGRkYGiKKRp8vYLNfXegzztnxN7GytUqtwth6rRaNBqtYai6SGtVktGRgYJCQksWbKEl156iQEDBhhy9fj+hUliYqKlQyg2ikquMzIySE1NZe/evWi1WkuHk8WOHTssHUKxIHl+srjNs2kQqUOjhsgubUg4fgY4k+d2TJHnlJSUXO2Xp8JCr9dTr149Jk+eDECdOnU4e/Ys8+fPz7Gw+OSTTxg9erTheUJCAn5+frRp0ybbu5tev34dZ2dn7O3tScnQUmeaZb6EYeNb42ibu/TY2Niwbds2ypYta9jWrl07rK2tsbW1xdXVFVdXV2xsbChVqhSVK1c2VdgmpygKiYmJuLi45LrwEs+mqOU6LS0NBwcHmjZtWuDubrpjxw5at24td900Icnz0x3YMA//A3cAOPuyH31Hz8hzG6bMc27/GM5TYeHr60tgoPHCHNWrV+f333/P8Rg7Ozvs7OyybLexscnyoXU6HSqVCisrK8PDUvLy/iqVihYtWjB//nzDNicnJ3r37m34PI/uW5hvgf2wS76wf47CoKjl2soqsxcwu5/9gqCgxlXUSJ6zF3fnOsqcH7HVwaUANT1mbXmu1TVNkefctpenqBs3bsyFCxeMtl28eJHy5cvnpZlcc7BREz6xba721ev1JCYk4uLqki+/hB1s8jYG7OTkRKVKhWfWrhBCiIJj/7yhlE6GeCeoMX1BgV6y+2nyFPmoUaNo1KgRkydPpkePHhw5coQFCxawYMECkwSnUqlyPRyh1+vR2qpxtLUusH/d2draFsoJm0IIIUznxLbldFbt49YrNpzyH0KDmo0sHdJzydMZ+MUXX2T9+vWsWrWKoKAgJk2axJw5c3jjjTdMFV+R4u/vz969e7l58yaxsbGWDkcIIYSF3bkRQYV/PgbgWvketH9nsoUjen557mvp2LEjHTt2NEUsRd7EiRN55513qFixIunp6YbLCYUQQhQ/6akp/DO0M17lrPD0rUzdgbMsHVK+KLyDOAXI0qVLs92+e/duo+cNGjTg9OnTpg9ICCFEgbduxCvUvqQn9aoz6Qu/wtau4Fwt9TwK5mQEIYQQogjbvnQSNffFAHC5QyBB9dtZOKL8I4WFEEIIYUY3I8NxmL8StQLnq9rQbdKvlg4pX0lhIYQQQpiJTqvl8MhelIqHe64QMmd1ob60NDtSWAghhBBm8vsXvah+QYNOBcnv9qZMQODTDypkpLAQQgghzODG5TDsI/4FIOwlT9oO+tLCEZmGFBZCCCGEiWWkp5G6egCdA+8Q0d6JrnNDLR2SyUhhIYQQQpjYsSWjqKy9RDxO1Bu1GjsHx6cfVEhJYSGEEEKY0J9zR5Owbid3MqyJaDgNH7+ifV+pojUVVQghhChAIs4cxHPZVtySrdjtGUDPtn0tHZLJSY+FEEIIYQI6rZazY4fglgy3PaD1rPWWDskspLAwE5VKxYYNG3J83d/fnzlz5pgtnmfxxRdf8M477xieK4rCkCFDcHd3R6VScerUKZo3b87IkSOf+T2WLl1KiRIlnj9Y8UQDBgzg1VdfzZe2YmJi8PT05ObNm/nSnhBFxW//6UjlSB0ZalCPGoa7t5+lQzILKSzyQUxMDO+88w7lypXDzs4OHx8f2rZtyz///GPp0PLNnTt3+Pbbb/nkk08M20JDQ1m6dCmbNm0iOjqaoKAg1q1bx6RJkwz75LVg6tmzJxcvXsxTbM9bzJjb7t27UalUPHjw4Lna8ff3R6VSGT3Kli2bP0HmgZeXF3379mXcuHFmf28hCqq9a78n8K+rAIS3Kk/T7sMtHJH5yByLfNCtWzc0Gg3Lli2jQoUK3Llzh7///pu4uDhLh2YkIyMDW1vbZzr2559/pmHDhvj7+5OQkABAREQEvr6+NGrUyLCfu7v7c8Xo4OCAg4PDc7XxrJ4nP5YyceJEBg8ebHiuVqstEsfAgQMJCQnhm2++oWTJkhaJQYiCIu7OdXSzf8BWB5cC1PSYucnSIZlVwe6xUBTISM79Q5OSt/2f9MjlLc0fPHjA/v37mTZtGi1atKB8+fKEhITwySef0KFDhxyPmzhxIt7e3pw6dSrb1+Pj4xkyZAheXl64urrSsmVLozujRkRE0KVLF7y9vXF2dubFF1/kr7/+MmrD39+fr776igEDBuDm5sbgwYMNQw3btm2jevXqODs7065dO6Kjo5/4OVevXk3nzp0NzwcOHMj777/PtWvXUKlU+Pv7A8a9B82bN+fq1auMGjXK8Bf10zw+FDJ+/Hhq167N8uXL8ff3x83NjV69epGYmAhkdunv2bOHb7/91vAeUVFRAISHh9O+fXucnZ3x9vamb9++xMbGGtpu3rw5w4cPZ/To0ZQqVYrWrVsDcPbsWTp06ICrqysuLi40adKEiIgIw3FLliyhevXq2NvbU61aNebNm2d4LSoqCpVKxerVq2nUqBH29vbUqFHDcKfbqKgoWrRoAUDJkiVRqVQMGDDgqXnJiYuLCz4+PoaHp6cnOp2Ot956i4CAABwcHKhatSrffvvtE9tZu3YtNWvWxMHBAQ8PD1q1akVycnKuPjNAzZo18fHxYf364jGGLMSTHFn0AVZ6hQdOUGP6giK3ZPfTFOxPq0mByaVztasVUCI/3/vTW2Dr9NTdnJ2dcXZ2ZsOGDTRo0AA7O7sn7q8oCiNHjmTDhg3s37+fypUrZ7tPhw4dcHd3Z8uWLbi5ufHTTz/x8ssvc/HiRdzd3UlKSqJ9+/Z89dVX2Nvbs2zZMjp16sSFCxcoV66coa1vvvmGL774gs8//xyA/fv3k5KSwowZM1i+fDlWVla8+eabjBkzhl9++SXbmO/fv09YWBj16tUzbJszZw6VKlViwYIFHD16NNu/lNetW0dwcDBDhgwx+qs6ryIiItiwYQObNm3i/v379OjRg6lTp/L111/z7bffcvHiRYKCgpg4cSIAnp6eREdH06xZMwYPHsysWbNITU3lo48+okePHuzcudPQ9rJlyxg6dCgHDhxAURRu3rxJ06ZNad68OTt37sTV1ZUDBw6g1WoBWLhwIePGjeP777+nTp06nDx5ksGDB+Pk5ET//v0N7X744YfMmTOHwMBAZs2aRefOnYmMjMTPz4/ff/+dbt26ceHCBVxdXQ09NJMnT2by5MlPzMXWrVtp0qTJE/fR6/WULVuW3377jVKlSnHw4EGGDBmCr68vPXr0yLJ/dHQ0vXv3Zvr06XTt2pXExET27duH8t/iOrefOSQkhH379jFo0KAnxidEUXZi23LaaXaS8LKaw5VH07Bmo6cfVMQU7MKiELC2tmbp0qUMHjyYH3/8kbp169KsWTN69epFrVq1jPbVarX069ePY8eOceDAgRzHw3ft2sWZM2eIiYkxFCozZsxgw4YNrF27liFDhhAcHExwcLDhmK+++or169fzxx9/MHz4/8byWrZsyZgxYwzP9+/fj0aj4ccff6RixYoADB8+3HBSzs7Vq1dRFIXSpf9X5Lm5ueHi4oJarcbHxyfb49zd3VGr1Ya/qp+VXq9n6dKluLi4ANC3b1/+/vtvvv76a9zc3LC1tcXR0dHoPebPn0/dunWNTtSLFy/Gz8+PixcvUqVKFQAqVarE9OnTDft8+umnuLm5sXr1amxsbAAM+wJMmjSJmTNn8tprrwEQEBBAeHg4P/30k9FJdvjw4XTr1s0QS2hoKD///DNjx441DBd5eXkZ9c68++67Rid+vV5PUlISzs7OWFlldi6WKVPGKDcfffSRoWiEzOLkgw8+YMKECYZtAQEBHDx4kN9++y3HwkKr1fLaa69Rvnx5ILMHIq+fuUyZMpw8eTJL+0IUF9HXLlHhn48BOOvXm9b9P3/KEUVTwS4sbBwzew5yQa/Xk5CYiKuLi+GX8HO/dy5169aNDh06sG/fPv755x9CQ0OZPn06ixYtMurmHjVqFHZ2dhw6dIhSpUrl2N7x48dJSkrCw8PDaHtqaqqhSz45OZkJEyawadMmbt26hVarJTU1lWvXrhkd82gvw0OOjo6GogLA19eXmJiYHONJTU0FwN7ePuckmJC/v7+hqICnxwuZOdy1axfOzs5ZXouIiDAUC4/n59SpUzRp0sRQVDzq7t27XL9+nbfeesuoB0ar1eLm5ma0b8OGDQ3/b21tTb169Th37twTY3Z3dzeao6LX60lISMDV1TXH7/SHH35o9B17+L368ccfWbRoEVevXiU1NZWMjAxq166dbRvBwcG8/PLL1KxZk7Zt29KmTRu6d+9OyZIl8/SZHRwcSElJeeJnFKKoSk9N4fjgV7HxtaV8xYq8MGCmpUOymIJdWKhUuRqOAECvBxtd5v75UVjkkb29Pa1bt6Z169Z8+eWXvP3224wbN87ol37r1q1ZtWoV27Zt44033sixLb1ej6+vr2Fc/lEP/8L98MMP2bZtGzNmzKBSpUo4ODjQvXt3MjIyjPZ3csqav8dPmiqVytDtnZ2HJ6v79+9nKXbMIbt49Xr9E4/R6/V06tSJadOmZXnN19fX8P+P5+dJE0cfvufChQupX7++0Wu5mTT5tDkmzzIUUqpUKSpVMl7F77fffmPUqFHMnDmThg0b4uLiwjfffMPhw4ezbVOtVrNjxw4OHjzI9u3b+e677/jss884fPgwjo6ZBXZuPnNcXByenp5PjF+IomrdiHbUvqon7ZY9KQM+w9bOMn+IFQQFu7AoxAIDA7OsW9G5c2c6depEnz59UKvV9OrVK9tj69aty+3bt7G2tjZMinzcvn37GDBgAF27dgUgKSnJMGkxv1WsWBFXV1fCw8OznMSextbWFp1OZ5K4nvQedevW5ffff8ff3x/rPEycqlWrFsuWLUOj0WQpaLy9vSlTpgxXrlx5YmEIcOjQIZo2bQpk/nV//PhxwxDVwytPHo/5WYZCsrNv3z4aNWrEe++9Z9j26OTT7KhUKho3bkzjxo358ssvKV++POvXr2f06NG5/sxhYWE0b978qfEJUdRsWzyRmvvuAnCpYyA9mnezcESWVbCvCikE7t27R8uWLVmxYgX//vsvkZGRrFmzhunTp9OlS5cs+3ft2pXly5czcOBA1q5dm22brVq1omHDhrz66qts27aNqKgoDh48yOeff86xY8eAzLkB69at49SpU5w+fZo+ffo89a/4Z2VlZUWrVq3Yv39/no/19/dn79693Lx50+iKjPzk7+/P4cOHiYqKIjY2Fr1ez7Bhw4iLi6N3794cOXKEK1eusH37dgYNGvTEQmf48OEkJCTQq1cvjh07xqVLl1i+fDkXLlwAMq9SmTJlimHS6JkzZ1iyZAmzZs0yaueHH35g/fr1nD9/nmHDhnH//n3DpMby5cujUqnYtGkTd+/eJSkpCcgcCqlUqZLRo0KFCkbPc3MpbqVKlTh27Bjbtm3j4sWLfPHFFxw9ejTH/Q8fPszkyZM5duwY165dY926ddy9e5fq1avn+jOnpKRw/Phx2rRp89T4hChKbkaG4/TjKtQKnKtqQ7dJv1o6JIuTwuI5OTs7U79+fWbPnk3Tpk0JCgriiy++YPDgwXz//ffZHtO9e3eWLVtG3759WbduXZbXVSoVW7ZsoWnTpgwaNIgqVarQq1cvoqKi8Pb2BmD27NmULFmSRo0a0alTJ9q2bUvdunVN9jmHDBnC6tWr81y8TJw4kaioKCpWrGiybvIxY8agVqsJDAzE09OTa9euUbp0aQ4cOIBOp6Nt27YEBQUxYsQI3NzcnjgHx8PDg507d5KUlESzZs144YUXWLhwoaH34u2332bRokUsXbqUmjVr0qxZM5YuXUpAQIBRO1OnTmXatGkEBwezb98+Nm7caBhSKlOmDBMmTODjjz/G29vbaLJtfnj33Xd57bXX6NmzJ/Xr1+fevXtGvRePc3V1Ze/evbRv354qVarw+eefM3PmTF555ZVcf+aNGzdSrly5p16xIkRRotNqOTyyFx4JcM8V6s9ZXewuLc2OSnnS4LoJJCQk4ObmRnx8PK6urkavpaWlERkZSUBAQJ4nCuZmopt4doqi0KBBAz744APDGg+S56yioqIICAjg5MmTOU6WzK3C9J0OCQlh5MiR9OnTJ8d9nufn25Q0Gg1btmyhffv22U7aFfmjKOb5t0+6UXN9ODoV3PqoD20GfGHpkEya5yedvx9VsH9biQJDpVKxYMECw3oOQjwUExND9+7d6d27t6VDEcJsrl8+g8udU2it4EwTzwJRVBQUUliIXAsODqZv3+e75e8rr7xiWFTs8cfTrogQBZOXlxdjx47N1cqqQhQFGelppK0eyCvlYonp4spr34ZaOqQCRQaDhFktWrTIsC7G4573PiMFgb+//xMv3RVCFH5HFo/kJe0lHuBM0LCV2Dnkft2j4kAKC2FWublcUgghCqo/vx2Fy6+7OdfInrQ206hTtuLTDypmZChECCGEyIWIMwfx/L9QvONUnLlRljpt3rR0SAWSFBZCCCHEU+i0Ws6OHYJbMtz2gFbfbrB0SAWWDIUIIYQQj9HpFY5ExhGTmIaXiz1R375B7UgdGWqw/s9w3L39LB1igSWFhRBCCPGI0LBoJvwZTnR8GgBN0v5mzN9XATjXujy9XhtmyfAKPBkKMbHmzZszcuTIZzp2586dVKtWzWRLdZvb7t27UalUPHjwIMd9VCpVlnusFAebNm2iTp06RebfWojCKjQsmqErThiKClddHIOObMVWBxf8rXAdsNDCERZ8UljkgwEDBqBSqbI8Ll++zLp165g0aZJhX39/f+bMmZOrdseOHctnn31W4FddzE/R0dGGpaRzY+nSpYY7vhYW2X0HOnbsiEqlYuXKlZYJSgiBTq8w4c9wHr1gfJTVr6Q76XngBLOC3mHS1ovo9HJJ+ZMUnzOWibVr147o6GijR0BAAO7u7ri4uOS5vYMHD3Lp0iVef/11E0RbcPn4+GBnZ2f299XpdBbvLRg4cCDfffedRWMQojg7Ehln6KkAaGt1hIEO+3i54R0WN2/HNeuKRMencSQyzoJRFnxSWOQTOzs7fHx8jB5qtdpoKKR58+ZcvXqVUaNGGXo1crJ69WratGljdE+F8ePHU7t2bRYvXky5cuVwdnZm6NCh6HQ6pk+fjo+PD15eXnz99ddGbc2aNYuaNWvi5OSEn58f7733nuGOmvC/v/q3bdtG9erVcXZ2NhRKD2U3pPPqq68yYMAAw/MVK1ZQr149XFxc8PHxoU+fPsTExOQpj48OhURFRaFSqVi3bh0tWrTA0dGR4OBg/vnnHyBzaGXgwIHEx8cb8jl+/HgAMjIyGDt2LGXKlMHJyYn69euze/fuLJ9506ZNBAYGYmdnx9WrV0lPT2fs2LH4+flhZ2dH5cqV+fnnnw3HhYeH0759e5ydnfH29qZv375Gd21t3rw5w4cPZ/jw4ZQoUQIPDw8+//xzw6JZT/oOdO7c2XAnViGE+cUk/q+oqKiPYJpN5rDHz/qO7LBple1+IqtCUVikaFJyfKTr0nO9b5o2LVf7msq6desoW7YsEydONPRq5GTv3r3Uq1cvy/aIiAi2bt1KaGgoq1atYvHixXTo0IEbN26wZ88epk2bxueff86hQ4cMx1hZWTF37lzCwsJYtmwZO3fuZOzYsUbtpqSkMGPGDJYvX87evXu5du0aY8aMydPny8jIYNKkSZw+fZoNGzYQGRlpVHg8q88++4wxY8Zw6tQpqlSpQu/evdFqtTRq1Ig5c+bg6upqyOfDmAcOHMiBAwdYvXo1//77L6+//jrt2rXj0qVLRp95ypQpLFq0iLNnz+Ll5UW/fv1YvXo1c+fO5dy5c/z44484OzsDmcM0zZo1o3bt2hw7dozQ0FDu3LlDjx49jOJdtmwZ1tbWHD58mLlz5zJ79mwWLVoEPPk7UL58eby8vNi3b99z50wIkXdeLpl/yJXQxfLpwfnsP+PEUU0AM7WvZ7ufyF6huCqk/sr6Ob7WpEwT5rWaZ3jeYk0L0nTZV5P1vOuxpN0Sw/N2v7fjfvr9LPud6X8mzzFu2rTJcAKCzHtirFmzxmgfd3d31Gq14S/6J4mKiqJ06dJZtuv1ehYvXoyLiwuBgYG0aNGCCxcusGXLFqysrKhatSrTpk1j9+7dNGjQAMCopyEgIIBJkyYxdOhQ5s37X940Gg0//vgjFStmriI3fPhwJk6cmKccDBo0yPD/FSpUYO7cuYSEhJCUlGSUm7waM2YMHTp0AGDChAnUqFGDy5cvU61aNdzc3FCpVEb5jIiIYNWqVdy4ccOQwzFjxhAaGsqSJUsM9yTRaDTMmzeP4OBgAC5evMhvv/3Gjh07aNWqleFzPDR//nzq1q1rdE+TxYsX4+fnx8WLF6lSpQoAfn5+zJ49G5VKRdWqVTlz5gyzZ89m8ODBT/0OlClThqioqGfOlRDi2YUEuOPtbM3YQzMoHQvxKdZ8GtAHjU3mqVIF+LjZExJQ+G8/YEqForAoDFq0aMH8+fMNz52cnJ6rvdTU1GxvLe3v7280Z8Pb2xu1Wm00wdPb29toCGLXrl1MnjyZ8PBwEhIS0Gq1pKWlkZycbIjT0dHRUFQA+Pr65nkY4+TJk4wfP55Tp04RFxdnmLNw7do1AgMD89TWo2rVqmUUF2TeUbNatWrZ7n/ixAkURTGc6B9KT0/Hw8PD8NzW1tao7VOnTqFWq2nWrFm27R4/fpxdu3ZlWyRFREQY3q9BgwZGQxwNGzZk5syZ6HQ61Gr1Ez+rg4MDKSmm6zUTQuRMbaViWNTXBF7WorWC/2vcmEs2NYDMogJgXKdA1FZyw70nKRSFxeE+h3N8TW1l/It61+u7cryKwkplvD20W/7dkc7JyYlKlSrlW3ulSpXi/v2svSk2NjZGz1UqVbbbHp7Ur169Svv27Xn33XeZNGkS7u7u7N+/n7feeguNRvPEdh+9mZaVlVWWm2s9enxycjJt2rShTZs2rFixAk9PT65du0bbtm3JyMjI46fP+TM/PGE/aaKlXq9HrVZz/PjxLCfyR4sCBwcHowLAwcHhiXHo9Xo6derEtGnTsrz2sOB5XnFxcXh6euZLW0KIvNkwYxgv7L8LwF+NyrLFqavhNR83e8Z1CqRdUP78rBdlhaKwcLTJ/Z3jHG0cc315Zl7azS+2trbodLqn7lenTh3Cw8Of+/2OHTuGVqtl5syZhrz89ttveW7H09PTaD6ATqcjLCyMFi1aAHD+/HliY2OZOnUqfn5+hvc2tezyWadOHXQ6HTExMTRp0iTXbdWsWRO9Xs+ePXsMQyGPqlu3Lr///jv+/v5YW+f8o/Po/JaHzytXrmwocnL6DqSlpREREUGdOnVyHbMQIn8c3rYCv+U7sQLCajnwwYLtNHhk5c2QAHfpqcilQjF5syjx9/dn79693Lx50+hqgse1bduW/fv3P/f7VaxYEa1Wy3fffceVK1dYvnw5P/74Y57badmyJZs3b2bz5s1cvHiRYcOGGS10Va5cOWxtbQ3v88cffxit32Eq/v7+JCUl8ffffxMbG0tKSgpVqlThjTfeoF+/fqxbt47IyEiOHj3KtGnT2LJlyxPb6t+/P4MGDTJMPt29e7ehEBs2bBhxcXH07t3bcPXG9u3bGTRokFGhcP36dUaPHs2FCxdYtWoV3333HSNGjDB6n+y+A4cOHcLOzo6GDRuaIFNCiJzE3b1F8qSvcUyHq6VVvLLwL9RWKhpW9KBL7TI0rOghRUUeSGFhZhMnTiQqKoqKFSs+scv7zTffJDw8nAsXLjzX+9WuXZtZs2Yxbdo0goKC+OWXX5gyZUqe2xk0aBD9+/dnwIABdOzYkYCAAENvBWT2aCxdupQ1a9YQGBjI1KlTmTFjxnPFnhuNGjXi3XffpWfPnnh6ejJ9+nQAlixZQr9+/fjPf/5D1apV6dy5M4cPHzb0puRk/vz5dO/enffee49q1aoxePBgkpOTAShdujQHDhxAp9PRtm1bgoKCGDFiBG5ubka9ZP369SM1NZWQkBCGDRvG+++/z5AhQwyv5/QdWLVqFW+88QaOjubvSROiuNLrdFxd8haOdRKJLgUBM+fh7CaTM5+HSnl84NzEEhIScHNzIz4+HldXV6PX0tLSiIyMJCAgINuJi0+i1+tJSEjA1dW1yKxUOXbsWOLj4/npp58sHYpBUcxzfmrevDm1a9fO9eqqD929e5dq1apx7NgxAgICgKKX6+f5+TYljUbDli1baN++fZa5RiL/FNQ8/7P4QxpeW0C6YsOVjr9S/cWXLR3SczFlnp90/n5U4f9tVYR99tlnlC9fPldzMkThFhkZybx58wxFhRDC9DbMHEbpC0sBOF17XKEvKgqKQjF5s7hyc3Pj008/tXQYwgxCQkIICQmxdBhCFBsH/1hIuWU7uWtVijPdq9Gx6/uWDqnIkMJCiHz06LLhQoiC6XpEGJopsyiZAZFlrWgxYqmlQypSZChECCFEsZGemsKpYT3xug9xrlB19kKcXNwsHVaRIoWFEEKIYmP90JepFKUn3RoyRg+mYs1Glg6pyJHCQgghRLGwdmI/gg89AOBi12Ba9Bpt2YCKKCkshBBCFHmX//0H571HAPi3nis9Jq22cERFlxQWQgghirT4e3dwXN+fZo2iCWtkT+efdlg6pCJNCgshhBBFVkZ6GtcW9KK0coc4tTdtvgnFwSnnxZ3E85PLTYUQQhRZ699qhrM+lvIV7cjotRw3D29Lh1TkSY+FmahUKjZs2JDj6/7+/nleBtrcvvjiC9555x3Dc0VRGDJkCO7u7qhUKk6dOkXz5s0ZOXLkM7/H0qVLKVGixPMHK0xi06ZN1KlT54m3rReioFjzeU9qHUugwglbdpV8nYAa9S0dUrEghUU+iImJ4Z133qFcuXLY2dnh4+ND27Zt+eeffywdWr65c+cO3377LZ988olhW2hoKEuXLmXTpk1ER0cTFBTEunXrjO5qmteCqWfPnly8eDFPsT1vMWNuu3fvRqVSGd0d9lmOf9Jj6dKl+RrzQx07dkSlUrFy5UqTtC9Eftm5aiZVNvwLwOkGJegy+gcLR1R8yFBIPujWrRsajYZly5ZRoUIF7ty5w99//01cXJylQzOSkZGBra3tMx37888/07BhQ/z9/UlISAAgIiICX19fGjX633Xg7u7Pd1dABwcHHBwcnquNZ/U8+TGnRo0aER0dbXg+YsQIEhISWLJkiWGbm9v/FvzR6XSoVKp8u5HZwIED+e6773jzzTfzpT0h8tul0wewm7UIWy1cDlDTdf7flg6pWCkUPRb6lJScH+npud83LS1X++bFgwcP2L9/P9OmTaNFixaUL1+ekJAQPvnkEzp06JDjcRMnTsTb25tTp05l+3p8fDxDhgzBy8sLV1dXWrZsyenTpw2vR0RE0KVLF7y9vXF2dubFF1/kr7/+MmrD39+fr776igEDBuDm5sbgwYMNQw3btm2jevXqODs7065dO6MTVXZWr15N586dDc8HDhzI+++/z7Vr11CpVPj7+wPGvQfNmzfn6tWrjBo1yvCX9NM8PhQyfvx4ateuzfLly/H398fNzY1evXqRmJgIwIABA9izZw/ffvut4T2ioqIACA8Pp3379jg7O+Pt7U3fvn2JjY01tN28eXOGDx/O6NGjKVWqFK1btwbg7NmzdOjQAVdXV1xcXGjSpAkRERGG45YsWUL16tWxt7enWrVqzJs3z/BaVFQUKpWK1atX06hRI+zt7alRo4Zhqe+oqCjD7eZLliyJSqViwIABT83Lo2xtbfHx8TE8HBwcDD1lPj4+hIaG4uvry6ZNmwgMDMTOzo6rV69m27Pz6quvGr1/RkYGY8eOpUyZMjg5OVG/fv0sy5R37tyZI0eOcOXKlTzFLYQ5JCfGc+k/g3FPhJiSUPv71dg5OFo6rGKlUPRYXKj7Qo6vOTVrSrlHbit+uUlTlNTUbPd1fPFFyi//v//t+3IrdPfvZ9mv+vlzuY7N2dkZZ2dnNmzYQIMGDbCzs3vi/oqiMHLkSDZs2MD+/fupXLlytvt06NABd3d3tmzZgpubGz/99BMvv/wyFy9exN3dnaSkJNq3b89XX32Fvb09y5Yto1OnTly4cIFy5coZ2vrmm2/44osv+PzzzwHYv38/KSkpzJgxg+XLl2NlZcWbb77JmDFj+OWXX7KN+f79+4SFhVGvXj3Dtjlz5lCpUiUWLFjA0aNHUavVWY5bt24dwcHBDBkyhMGDB+cqn9mJiIhgw4YNbNq0ifv379OjRw+mTp3K119/zbfffsvFixcJCgpi4sSJAHh6ehIdHU2zZs0YPHgws2bNIjU1lY8++ogePXqwc+dOQ9vLli1j6NChHDhwAEVRuHnzJk2bNqV58+bs3LkTV1dXDhw4gFarBWDhwoWMGzeO77//njp16nDy5EkGDx6Mk5MT/fv3N7T74YcfMmfOHAIDA5k1axadO3cmMjISPz8/fv/9d7p168aFCxdwdXU19NBMnjyZyZMnPzEXW7dupUmTJk/NWUpKClOmTGHRokV4eHjg5eWVq1wPHDiQqKgoVq9eTenSpVm/fj3t2rXjzJkzhu9q+fLl8fLyYt++fVSoUCFX7QphLpuHvEzNGwqptmD72Rj8KgZZOqRip1AUFgWZtbU1S5cuZfDgwfz444/UrVuXZs2a0atXL2rVqmW0r1arpV+/fhw7dowDBw5QtmzZbNvctWsXZ86cISYmxlCozJgxgw0bNrB27VqGDBlCcHAwwcHBhmO++uor1q9fzx9//MHw4cMN21u2bMmYMWMMz/fv349Go+HHH3+kYsWKAAwfPtxwUs7O1atXURSF0qVLG7a5ubnh4uKCWq3Gx8cn2+Pc3d1Rq9W4uLjkuE9u6PV6li5diouLCwB9+/bl77//5uuvv8bNzQ1bW1scHR2N3mP+/PnUrVvX6ES9ePFi/Pz8uHjxIlWqVAGgUqVKTJ8+3bDPp59+ipubG6tXr8bGxgbAsC/ApEmTmDlzJq+99hoAAQEBhIeH89NPPxkVFsOHD6dbt26GWEJDQ/n5558ZO3asYbjIy8vLqHfm3XffpUePHkafOykpCWdnZ8MwRpkyZXKVM41Gw7x584y+I08TERHBqlWruHHjhuHfesyYMYSGhrJkyRKjXJYpU8bQMyREQXFk/Vy8XGNIt3Hiap8mdO34lqVDKpYKRWFR9cTxnF987C/lSvv25jyW/Nj2Sn//lf1+edStWzc6dOjAvn37+OeffwgNDWX69OksWrTIqJt51KhR2NnZcejQIUqVKpVje8ePHycpKQkPDw+j7ampqYYu+eTkZCZMmMCmTZu4desWWq2W1NRUrl27ZnTMo70MDzk6OhqKCgBfX19iYmJyjCf1vz1A9vb2OSfBhPz9/Q1FBTw9XsjM4a5du3B2ds7yWkREhKFYeDw/p06dokmTJoai4lF3797l+vXrvPXWW0Y9MFqt1mhOA0DDhg0N/29tbU29evU4d+7JPWHu7u5Gc1T0ej0JCQm4urrmeX6Era1tlsL2aU6cOIGiKEaFFEB6enqW76KDgwMpeRw2FMKULhzbSe1TE7D11bJjSHu6vv+jpUMqtgpFYWHlmPvxMStHx1z/Es5Lu09jb29P69atad26NV9++SVvv/0248aNMyosWrduzapVq9i2bRtvvPFGjm3p9Xp8fX2zvQX3w79wP/zwQ7Zt28aMGTOoVKkSDg4OdO/enYyMDKP9nZycsrTx+ElTpVKhKEqO8Twsgu7fv5/lBGMO2cX7tMsd9Xo9nTp1Ytq0aVle8/X1Nfz/4/l50sTRh++5cOFC6tc3vmwtu6Ggxz1tjkl+DoU4ODhkeT8rK6ss/84ajcbw/3q9HrVazfHjx7N8nscLtLi4ODw9PZ8ahxDmcOHUHvS/v42tnZaTjo15+T25AsSSCkVhURgFBgZmWbeic+fOdOrUiT59+qBWq+nVq1e2x9atW5fbt29jbW1tmBT5uH379jFgwAC6du0KQFJSksm6pitWrIirqyvh4eFUqlQpT8fa2tqi0+lMEteT3qNu3br8/vvv+Pv7Y22d+695rVq1WLZsGRqNJktB4+3tTZkyZbhy5coTC0OAQ4cO0bRpUyCzR+P48eOGIaqHV548HnN+DoVk5+Hck4d0Oh1hYWGGyaR16tRBp9MRExPzxOIlLS2NiIgI6tSp88yxCJFfkuLjiBw9FNcEO/a0KMsLo3/BKheFvjCdQnFVSEF27949WrZsyYoVK/j333+JjIxkzZo1TJ8+nS5dumTZv2vXrixfvpyBAweydu3abNts1aoVDRs25NVXX2Xbtm1ERUVx8OBBPv/8c44dOwZkzg1Yt24dp06d4vTp0/Tp08dkixZZWVnRqlUr9u/fn+dj/f392bt3Lzdv3jS6IiM/+fv7c/jwYaKiooiNjUWv1zNs2DDi4uLo3bu34QqG7du3M2jQoCcWOsOHDychIYFevXpx7NgxLl26xPLly7lw4QKQeZXKlClTDJNGz5w5w5IlS5g1a5ZROz/88APr16/n/PnzDBs2jPv37zNo0CAgc/KjSqVi06ZN3L17l6SkJCBzKKRSpUpGjwoVKhg9f55LcVu2bMnmzZvZvHkz58+f57333jNaS6NKlSq88cYb9OvXj3Xr1hEZGcnRo0eZNm0aW7ZsMex36NAh7OzsjIZ7hLCUrYNbUf6Wgp0GHFp/grNrSUuHVOxJYfGcnJ2dqV+/PrNnz6Zp06YEBQXxxRdfMHjwYL7//vtsj+nevTvLli2jb9++rFu3LsvrKpWKLVu20LRpUwYNGkSVKlXo1asXUVFReHtnLkc7e/ZsSpYsSaNGjejUqRNt27albt26JvucQ4YMYfXq1XkuXiZOnEhUVBQVK1Y0Wdf5mDFjUKvVBAYG4unpybVr1yhdujQHDhxAp9PRtm1bgoKCGDFiBG5ubk8cKvPw8GDnzp0kJSXRrFkzXnjhBRYuXGjovXj77bdZtGgRS5cupWbNmjRr1oylS5cSEBBg1M7UqVOZNm0awcHB7Nu3j40bNxqGlMqUKcOECRP4+OOP8fb2Nppsa0qDBg2if//+9OvXj2bNmhEQEGDorXhoyZIl9OvXj//85z9UrVqVzp07c/jwYfz8/Az7rFq1ijfeeAPHfBxKFOJZ/PqfjgT9m4oeuNGvJSFtZG2VgkClPGlw3QQSEhJwc3MjPj4eV1fjG8GkpaURGRlJQEBAnicKPs9EN/F0iqLQoEEDPvjgA8MaD5LnrKKioggICODkyZPUrl37udoqiN/pu3fvUq1aNY4dO5almHqa5/n5NiWNRsOWLVto3759tpN2Rf7I7zxvWzSO0rN+w1oPp5p70/vH3c8fZBFgyu/zk87fjyoYv61EgadSqViwYIFhPQdRPEVGRjJv3rw8FxVC5Kezh7fjNi+zqDhf2YYe3+fPFX4if0hhIXItODiYvn37Plcbr7zyimFRsccfT7siQlheSEgIPXv2tHQYohhLS03m4uRRuKVAdClosmAj6jxM0BamJ/8awqwWLVpkWBfjcc97n5GCwN/f/4mX7gohnp2i13Pmx0G8XPMWu/GizAdfUcpXes8KGikshFk9z+WSQoji7fCvU2kQH4pOrSLgP9Op2TTrlXfC8gpkYSF/8QlR9MjPtXgem+d9hPrIGvTl4WiV0TSQoqLAytMci/HjxxvuIPnw8Tz3gHjcw9X+Hl89UghR+D1cAlyuvBB5dWrfRjwX/kH5Q3asiQmkfu/PLR2SeII891jUqFHD6PbcuVnKONfBWFvj6OjI3bt3sbGxydMldnq9noyMDNLS0grMpXlFkeTZfIpKrhVFISUlhZiYGEqUKJGvvzNE0Rd35zqxn31MmVS44QVtJq5FVYh/HoqDPBcW1tbW+dpL8SiVSoWvry+RkZFcvXo1T8cqikJqamq290gQ+UfybD5FLdclSpQw2e8OUTTptFp2v9OR6jGQ4AjeU6ZT0lPmaRV0eS4sLl26ROnSpbGzs6N+/fpMnjyZChUq5Lh/eno66enphucJCQlA5iIej94A6SGVSoW/vz8ajSZPY7JarZaDBw/SqFGjPN0bQuSN5Nl8ikquVSoV1tbWqNXqArkOysPfQ9n9PhL551nyvHZkO2qfz0Cngti3u9AmpJ38Oz2FKb/PuW0zTytvbt26lZSUFKpUqcKdO3f46quvOH/+PGfPns3xrpfjx49nwoQJWbavXLlSlgQWQgiRrbtHfqXhupNYKXComTfu7UdZOqRiLyUlhT59+jx15c3nWtI7OTmZihUrMnbsWEaPHp3tPtn1WPj5+REbG/vEwPJKo9GwY8cOWrduLZPDTEjybD6Sa/OQPJtHXvJ8K+oc4dNew++ADecD7ei44h9ZBCuXTPl9TkhIoFSpUk8tLJ7rX8rJyYmaNWty6dKlHPexs7PDzs4uy3YbGxuT/BCbql1hTPJsPpJr85A8m8fT8pyc+AD9qr68UvYu2zsE0OqTrdg/x119iytTfJ9z295zTa1NT0/n3Llz+Pr6Pk8zQgghBDqtltPf9yZAH0UsJag9aj0lPOT8UtjkqbAYM2YMe/bsITIyksOHD9O9e3cSEhLo37+/qeITQghRTPz23sto/ojkQrIDse0X4VVGlusujPI0FHLjxg169+5NbGwsnp6eNGjQgEOHDlG+fHlTxSeEEKIY2DBjKLX2xmCFilPUp2dIa0uHJJ5RngqL1atXmyoOIYQQxdThrcvwW74bKyAs2IGe32y0dEjiOcjyZUIIISwm+up5UiZNxTEdrpZW8cqCv55+kCjQpLAQQghhEZqMdI681x2fOLjvDAEz5+Hs5m7psMRzksJCCCGERfz+QRuqROjIUEPKiP5UrdPc0iGJfCCFhRBCCLM7uX0FLUuGcb2MwoXOgbTq+7GlQxL5RJYyE0IIYVZXzx2nyoH/4GSnxblrY9oM/9nSIYl8JD0WQgghzObmlTDOfvcmTqo0ztoG8+KQeZYOSeQzKSyEEEKYlE6vcCQyDq0mg39HvEnATthwxRfft1dhY5v1lg+icJOhECGEECYTGhbNhD/DiUtK5Yuo6QRG6Um3hgcN3sTdq4ylwxMmID0WQgghTCI0LJqhK04QHZ/GG7ELeOF4EgB/NqzIhGt1CA2LtnCEwhSksBBCCJHvdHqFCX+GowCtUzbR+WDmXbAPBTvxs8dQACb8GY5Or1gwSmEKUlgIIYTId0ci44iOT+PF9IMM3rcbOw1cLm/FNxU+BUABouPTOBIZZ9E4Rf6TwkIIIUS+i0lMw5s4+iauxSUVrntDypuj0VnZZtlPFC1SWAghhMh3riSzzHYancveJqIFTKs/FHvXUln283Kxt0B0wpTkqhAhhBD56v7dm7hu6Es1q+vcVkoytcRH3LUqBegM+6gAHzd7QgLk3iBFjRQWQggh8k1qcgL7+rfFOVFLxEsuvMdH3FQ8seV/kzRV//3vuE6BqK1U2TckCi0ZChFCCJEvdFotm/s1pfIVHR73VVyqOIhRb7yKj5vxcIePmz3z36xLuyBfC0UqTEl6LIQQQuSLNW81IfhsOnoVRLzxEq+9PRGA1oE+HLocQ+y5Qyzu/yINKnlJT0URJj0WQgghntvq99sQfPgBAGEdq/DaJwsNr6mtVIa5FCEB7lJUFHFSWAghhHgua77oTfCO6wCcauZFz282WjgiYUlSWAghhHhmJ3asxHf7KQDO1HGixw9/WzYgYXEyx0IIIcQzuXhiN9X2jySuhZ6D18vRdckB1NZyWinupMdCCCFEnkWEHcLzjzdxVKVz360mry4+KLdAF4AUFkIIIfIo/OgObgweyNkYFZesKxMwbB22drKCpsgkhYUQQohcu3bpNLdHfYDXfdCccsZ1wK84u5a0dFiiAJHCQgghRK7cv3uT8Hd74xsLD5ygxJTJeJetaOmwRAEjhYUQQoinSk1OYG//tpS/qZBiB5pPhlGnWVdLhyUKICkshBBCPJFOq2Vz/2ZUuaIjwxpihr1G0+7DLR2WKKCksBBCCPFEaz/uQI2wtP8u1d2YV4Z8bemQRAEmFxwLIYTI0aGVX9Hd6RBba3mSXL4GPT9ZZOmQRAEnPRZCCCGydWzTAhpc/Aa1Ckq17yFLdYtckR4LIYQQWfz57UjYuYWMWnDSpzv1+31l6ZBEISGFhRBCCCN/LZ9K2UXbsNeoWV+qCq+PX4DKSjq4Re7IN0UIIYTBoa3LcJ21DHsNXClnRYdZ27BSqy0dlihEpLAQQggBwLmjf6EdNxWXVLjhreKFRRtwdnO3dFiikJHCQgghBNcjwoge9T4eCRBTEgK+X4hPucqWDksUQlJYCCFEMZeSlMC/Q3sYlup2nfo1lWo2tnRYopCSwkIIIYoxrSaDC/N64lv9AXGuD5fqfs3SYYlCTK4KEUKIYkrR6zkxbwAhKQdJL2HDhRmTqdVU7v8hno/0WAghRDH169DmlLzxNzpFRXjjOVJUiHwhhYUQQhRDqz9oS/Ceu8TvcGdvhQ+o0+ZNS4ckiggpLIQQophZM643wduvAXApxJMW/SdaOCJRlEhhIYQQxcgfc0ZSbc0pAM7UdqLHvJ2WDUgUOVJYCCFEMfHX8qn4/bwNaz2cr2pD1/87gNpa5vCL/CWFhRBCFANHtq8wWqr75aV/YWNrZ+mwRBEkpaoQQhRxd29FUXLPeK54qInXqHlh0QZcS3pZOixRRElhIYQQRVj8/ViSFnWhss1dVI1Lo+qxRJbqFiYlQyFCCFFE3b97k52ftSZAH0UsJXAcsJ6KNepbOixRxElhIYQQRVBqcgJ7B7Sl2s40/rzkRXy3XykdUM3SYYliQAoLIYQoYnRaLZv7N6NKhI4MNVi91J2KNRtYOixRTEhhIYQQRcyawU2pEZaGXgWX+zSkw9Aplg5JFCMyeVMIIQoRnV7hSGQcMYlpeLnYExLgjtpKZXh91Yh21P7nPgBhHSrT87PFlgpVFFNSWAghRCERGhbNhD/DiY5PM2zzdbNnXKdA2gX5smZcH2ptuwrA6Sae9Jrxh6VCFcWYFBZCCFEIhIZFM3TFCZTHtt+OT2PoihN888I9XK8fAuw4U9uR1+fLUt3CMqSwEEKIAk6nV5jwZ3iWogJAAWqqrvBK2CScyqWz0bM2XcZvlaW6hcXIN08IIQq4I5FxRsMfj6qbcYSZDotxUqVzyjqYdhO2YmfvaOYIhfgfKSyEEKKAi0nMvqiorAln9P7fiLJ14kbjEsS8+hO1pagQFiaXmwohRAHn5WKfZZu39iYfH16MRwKotSq+0L5NaS+5/4ewPCkshBCigAsJcMfXzZ6HF5W66B8w7uQcSsdCvBPMaNSb9FJBhAS4WzROIUAKCyGEKPDUVirGdQoEwEafysSwKQTcVEixhXkvteOM7QuM6xRotJ6FEJYicyyEEKIQaBfky5yu/ugm9qTalcylupc1acAln47M/+86FkIUBFJYCCFEIZCWkoT71qGoYnVo1HDstfr0HjCD2Y+tvCmEpUlhIYQQBVxy4gOivutME05zqbkT58v04q0PZlk6LCGyJYWFEEIUYDeuhHF6bl86lLhCsmKP5tWf6dTwFUuHJUSOZPKmEEIUUFfCj3BuwOuU35bGrlgPbnb5lUApKkQBJ4WFEEIUQBdO7ubq4P6UjYFkB1A1HUGVus0tHZYQTyVDIUIIUcD8e3Az9/8zBp/78MAZGD+G5h3fsnRYQuSKFBZCCFGAHP97DamffIlXAtxzBfsp46j3ci9LhyVErklhIYQQBcTJvRvJ+OhLPJIgpiSUnDGdWo07WTosIfJE5lgIIUQBcCXsMH5/D+deOS23PcDn+/lSVIhC6bkKiylTpqBSqRg5cmQ+hSOEEMXPxRN78FjblVKqBCrXdqLCkpVUfaG5pcMS4pk881DI0aNHWbBgAbVq1crPeIQQoljZuuAztJvX4h+czAXbavgM24xbyVKWDkuIZ/ZMPRZJSUm88cYbLFy4kJIlS+Z3TEIIUSz8OXc0vnPXUemCFRuv+lPmg1ApKkSh90w9FsOGDaNDhw60atWKr7766on7pqenk56ebniekJAAgEajQaPRPMvbZ+thW/nZpshK8mw+kmvzsFSe/5z5HhWX78dWB5f9rWg+9Q/sHJyL7L+3fJ/Nw5R5zm2bKkVRlLw0vHr1ar7++muOHj2Kvb09zZs3p3bt2syZMyfb/cePH8+ECROybF+5ciWOjo55eWshhCgS7u5ZRP3Qy1jr4UJFNdq+H2Pj4GLpsIR4opSUFPr06UN8fDyurq457penwuL69evUq1eP7du3ExwcDPDUwiK7Hgs/Pz9iY2OfGFheaTQaduzYQevWrbGxscm3doUxybP5SK7Nw9x5XvdlL4I2hGOlwLlqNry8ZCeOzm4mf19Lk++zeZgyzwkJCZQqVeqphUWehkKOHz9OTEwML7zwgmGbTqdj7969fP/996Snp6NWq42OsbOzw87OLktbNjY2JvlymapdYUzybD6Sa/MwR553Lv6Mypszi4qzQfZ0Wn4AO4fi1XMr32fzMEWec9tengqLl19+mTNnzhhtGzhwINWqVeOjjz7KUlQIIYTIdOiXCbS89j1Hmzvx770yvLZkPza2Wf/oEqKwy1Nh4eLiQlBQkNE2JycnPDw8smwXQggBOq2WnT++T+vYlQBoa3Sjx+BvUVnJ+oSiaJIlvYUQwkR0Wi1r3mpCwJkHXG5lx93qA2nQf4oUFaJIe+7CYvfu3fkQhhBCFC2ajHTW929M8MlkAI5l1KXXwGkWjkoI05OyWQgh8ll6agob+zSg5n+LitPtK9BrTqiFoxLCPKSwEEKIfJSSFM/m3g2pEZaGXgVnXqtBr1mbLR2WEGYjhYUQQuSThPsxbO/1EtXPZ6C1gnO969Fj8lpLhyWEWUlhIYQQ+SAtJYnzP/TAMVmDRg2X+zel+5fLLR2WEGYnV4UIIcRzSk58QNR3nQnRnuZOQ3uO+Q2g61CZqCmKJykshBDiOVyPCOPY9Dfo6htFkuLAvc5L6dCgnaXDEsJiZChECCGe0ZXwI1wY+DrV9mSw+Zont7qsJlCKClHMSWEhhBDP4MLJ3Vwd3J8yMZDgCM6vjKRK3eaWDksIi5OhECGEyKN/D27m/n/G4HMfHjiDMv5DmnUcZOmwhCgQpMdCCFFs6fQKRyLjADgSGYdOrzz1mON//0bCyDF43Yd7rmAzbRyNpKgQwkB6LIQQxVJoWDQT/gwnLimV6SEwaNlR3J0dGNcpkHZBvtkec+7oTtI/GodHEsSUhJIzZ1CrUQczRy5EwSY9FkKIYic0LJqhK04QHZ9mtP12fBpDV5wgNCw6yzERZw7hvbk/iUFp3PYAn3nzpagQIhtSWAghihWdXmHCn+FkN+jxcNuEP8ONhkXOH9tJqd9fw50EKlYsSbWVG6hap7kZohWi8JHCQghRrByJjMvSU/EoBYiOTzPMvdi64DOiPn4PRZvKeevqeA7fjm/5qmaKVojCR+ZYCCGKlZjEnIuKx/f789uR+C3chp1WxV+Xy/LKwlCcXEqYNkAhCjkpLIQQxYqXi32u9nuw4QvqrPkHGx1c9rfi5e+kqBAiN2QoRAhRrIQEuOPrZo8qh9dVwJD4hdT9LbOouFDJmmardlHSs4w5wxSi0JLCQghRrKitVIzrFAiQpbhQAUNiv6PL7gtY6+FcNVva/noA15JeZo9TiMJKCgshRLHTLsiX+W/WxcfNeFhkoM0Wmp2+ipUCZ4Ps6bDqHxycXC0UpRCFk8yxEEIUS+2CfGkd6MOhyzHEnjvE7HIHaR+9gsst7DgeE8BrC/ZiY2tn6TCFKHSkx0IIUWyprVS84OdKyvFltI/+AYC7lXrSY/E/UlQI8Yykx0IIUWzptFo2vNOCwOPxHH3ZGW2dN2nQfwoqK/mbS4hnJT89Qohi6f7dm2zs/gLBR+Kx1cFlpQYNB06TokKI5yQ/QUKIYufsoVCOdW9N9fMZ6FTwT8vSdJ+12dJhCVEkyFCIEKJY2b5kIq7fr6JsMiQ6wJ23OuLh95KlwxKiyJAeCyFEsbH5+5H4zFiFWzJElwKbWV/R7p3Jlg5LiCJFeiyEEEWeVpPBsYXDeOXub2yp6I1Wb0ejnzbgVaYiGo3G0uEJUaRIYSGEKNKuXTjB7d9G0ED3L6jAtcMrNBw4XS4nFcJEpLAQQhRZ+zcuQD9lNknuehLr2XGp0XSathtg6bCEKNKksBBCFEnrpw6h/Mp9OGSAld6KC82+p97LPSwdlhBFnhQWQogiRafV8tt7Lam19y5WQGRZK6rNXUKFwBBLhyZEsSCFhRCiyIiNjmTfO12ofTFzQuaZ2o50WLgTJxc3C0cmRPEhl5sKIYqE29cucWhgB6pd1KC1gtMdK9Fj9XEpKoQwMykshBCFXvihUGwXt6R84H3iXODGyNfoNeNPS4clRLEkQyFCiEItdP4YXr69GBuVjvslK+D6y1LKVwm2dFhCFFtSWAghCqXkxHi2vN2SymdTuNzOhhSfJlR/9/9wdJahDyEsSYZChBCFTsSZg+zp2pCg0ynYaOF0Ri3qjl4vRYUQBYAUFkKIQmXP2u+IfustAm4opNrCxYFN6TV3h9zuXIgCQoZChBCFxu+T+lPxtyPYaeBuCVB/OoqunYdYOiwhxCOksBBCFHh6nY5147tRY80FAK6Us6Lmd8spV7WuhSMTQjxOCgshRIGWnPiAiz++wauq/ewo70NiKVc6LfgbBydXS4cmhMiGFBZCiALr5J4NOO78nDqq62SorHF7ewivvP4fS4clhHgCKSyEEAXSph8+xHvhJi5V0FKyZgniOv5MoxdbWTosIcRTSGEhhChQdFota8Z0JGjbVdQKPLhnQ3KvlVSr/qKlQxNC5IIUFkKIAiMpPo7Qt14mOCwNgHPVbGmxcAslPctYODIhRG7Jhd9CiALhwqm97O/6EjXC0tADp1v60mXtcSkqhChkpLAQQljchRO7uTn8HcrfUkixg8uDW9Jr3k7U1tKpKkRhI4WFEMKijm1eSLmNr+NcN5HbHqCZ+jFd/vODpcMSQjwj+XNACGERmox0ts8eRIfUTaAC+9LBVPhsGaW8ZehDiMJMCgshhNlFXz3P0Xe7UzpaR1RbO24HdOfFt+fK0IcQRYAMhQghzOrI9hWc792VypE6bLRw3PkVGrw7T4oKIYoI+UkWQpjNH3NGUHrJdnzS4b4zpIwcQLc3P7J0WEKIfCSFhRDC5HRaLWtGvkLNv29gpcA1XxXlZ3xPoxdaWjo0IUQ+k8JCCGFSaanJbBzThuC/4wA4W8OONou241rSy8KRCSFMQeZYCCFMJvbWVa7ObMGr7mFcL61wurUfXX89JkWFEEWY9FgIIUxiz5q5VD4zm6pWccSrnfD+4mvatOhm6bCEECYmhYUQIt/99tnrVN0QxslaVmRU98O6z2qCKwVZOiwhhBlIYSGEyDfpqSlsGNyCWscSANDFO+A27C9KevhYODIhhLnIHAshRL6IOneM7V1fNBQVpxuW5JUNx6WoEKKYkR4LIcRzWzdlMKXX7KdSCqRbw+XuL9Br/ApLhyWEsAApLIQQzyw+7i4nF7yF/6pLOGTAbQ+I7PMG1dsNRadXUFupLB2iEMLMpLAQQjyT07vW4LtnLM2J4+/6JTmd5MlMv5EkXS8BCw/h62bPuE6BtAvytXSoQggzksJCCJEnt6LOcXBMH8qVjsHLPYmrlOaHUkM44VHFaL/b8WkMXXGC+W/WleJCiGJEJm8KIXLtz7mjiXz9NWqEpaH5x5X9Ht3oZzODE0qVLPsq//3vhD/D0emVLK8LIYomKSyEEE8VczOCNT3rUmneVtwTIdYNEga/hrr9N1xNzPk4BYiOT+NIZJzZYhVCWJYMhQghnmjzvI9xW7qRoMyrSDlT25GWs9dSyjeAjadu5qqNmMQ0E0YohChIpLAQQmQrNTmRHTPfpPLKiwDcc4UH/TrQY/gMwz5eLva5aiu3+wkhCj8pLIQQWZw/sgOnrR/QWbnF5srepNo703jGSnzLVzPaLyTAHV83e27Hp5HdLAoV4ONmT0iAu1niFkJYnsyxEEIYPLgXzeoBDSm5vhd+yi1icKf0iAl0X3MiS1EBoLZSMa5TIJBZRDzq4fNxnQJlPQshihEpLIQQAPz1f5M51aklwYcecOzfUhwt8Qp2I45Sp1XvJx7XLsiX+W/WxcfNeLjDx81eLjUVohjK01DI/PnzmT9/PlFRUQDUqFGDL7/8kldeecUUsQkhzCDhfgxbRnWh5uEHWCkQ7wQZTZvz4sifct1GuyBfWgf6cCQyjpjENLxcMoc/pKdCiOInT4VF2bJlmTp1KpUqVQJg2bJldOnShZMnT1KjRg2TBCiEMJ1dq2ehfL+Q4NjM5+eq2VJ3ys80qF4vz22prVQ0rOiRzxEKIQqbPBUWnTp1Mnr+9ddfM3/+fA4dOiSFhRCFiCYjnQ3jX6fahktY6yHBEW50DaHbF8ssHZoQopB75qtCdDoda9asITk5mYYNG+a4X3p6Ounp6YbnCQmZF8NrNBo0Gs2zvn0WD9vKzzZFVpJn8zFVrqPCj6L6YxhtraI44+zDXS9rqn41j841GhTLf1f5TpuH5Nk8TJnn3LapUhQlT2vtnjlzhoYNG5KWloazszMrV66kffv2Oe4/fvx4JkyYkGX7ypUrcXR0zMtbCyGegyYjlaQ9C+jlfgp7Kx0PFGf+dOqKc8WWWFmrLR2eEKKAS0lJoU+fPsTHx+Pq6prjfnkuLDIyMrh27RoPHjzg999/Z9GiRezZs4fAwMBs98+ux8LPz4/Y2NgnBpZXGo2GHTt20Lp1a2xsbPKtXWFM8mw++ZnrQ5uXkjp7Nn53FG6+lIJH5er49pmHh49fPkVbeMl32jwkz+ZhyjwnJCRQqlSppxYWeR4KsbW1NUzerFevHkePHuXbb7/lp5+yn0FuZ2eHnZ1dlu02NjYm+XKZql1hTPJsPs+T6/TUFNb/pxPV99yilA5S7CDOtwEvf7gKlZVcbf4o+U6bh+TZPEyR59y299wrbyqKYtQjIYQoGI5sX0Hc1MkE38rslLzsb0WFCbPoUb+thSMTQhRleSosPv30U1555RX8/PxITExk9erV7N69m9DQUFPFJ4TII71Ox5rPXqPqpouU10KqLVxqV5Xuk9eitpZV/IUQppWn3zJ37tyhb9++REdH4+bmRq1atQgNDaV169amik8IkQc3r5zjwerBVM+IQK0rwZVyVpT5YjI9m3SxdGhCiGIiT4XFzz//bKo4hBDPQavJYOsPI3n5/lrKqNJJcbFj+6DGdBw5D2sbW0uHJ4QoRqRfVIgCSKdXOBIZB8CRyDgaVPLKcXnsfw/8yfUJH1Pupp5bHSC9RE1K9FrIqxWqmzNkIYQApLAQosAJDYtmwp/hxCWlMj0EBi07iruzA+M6BRrd0Eun1bL20+5UDr1AhQxIt4bj1o15/aO1WKllXQohhGXI9WZCFCChYdEMXXGC6Pg0o+2349MYuuIEoWHRAJw9vJ0tHYOp9ccFHDLgmq+K9Bmf0HPyeikqhBAWJT0WQhQQOr3ChD/DyW7FOgVQAeM3hpGwegQV/jxDpXTIUMO5ZqXpOvNP7BxkJVshhOVJYSFEAXEkMi5LT8WjSvGAiWk/o7sWhVO6Hde9VTj/ZxS9Og82Y5RCCPFkUlgIUUDEJGZfVCh6HV30OxnvsJaSqiSSy6v50/1Funy9Bgen/FsWXwgh8oMUFkIUEF4u9lm2Jd29wpQzP+NxX4tjyyTOqspzrdlMerWStWOEEAWTFBZCFBAhAe74utlzOz4NBeh172cqbT2PawroVCrm3a/PWt/R7G3ZytKhCiFEjqSwEKKAUFupGNcpkFULp9Ht/A4qX9UDEO0By19ozW7HtszvEpzjehZCCFEQSGEhRAERee44KZ8P4uPzGQBoreB0HWe+8RuNo5sX8x9bx0IIIQoiKSyEsLDY29eIWDuOunc3cj7NE7DiXFUbdL2G4+LozbzqDZ648qYQQhQkUlgIYSE3I8PZN/FtGvhEUt8uBVRg84Intwe8xmu9RqPRaNiyZQshAe5SVAghCg0pLIQws/t3b7JtXD8qHrxFcBqEBTmTXrsc2hZf0qpxB0uHJ4QQz0UKCyHMJDU5gT/G98Vv10WCkzK33fYATb3mVBn7EyorWWFfCFH4SWEhhIkpej3rpryFx+ZD1Mq8YSlxrnDj5UBe/XK5LMUthChSpLAQwoTC9m3Ebs8k7M48wDvOhkQHuPKSH+3GL6Oxh1zhIYQoeqSwEMIE/v5lOvqwdbS2OweAQ0UH9jtXotmXiwgpX83C0QkhhOlIYSFEPjq0dRnR82ZS7ZKGq3560hqqOeX9GpW7j6eXd1lLhyeEECYnhYUQ+eDs4e2cm/ER1cLSqKaAHkh2tie693YaVK9n6fCEEMJspLAQ4jlcu3CCf74aSrWTCdTQZm67VEFNybeH0u21YZYNTgghLEAKCyGeQUpSPKd/n0b8/l+pddQWgKtlVNC7O53fnmjh6IQQwnKksBAiD5IT49n7f1/y4t0NNOQBGb4QWrEM+uZN6TTqe9TW8iMlhCje5LegELmgyUhnw9cD8d52EhcFXNs84Ja1N7fqjabDl29LQSGEEP8lvw2FeAKdVsvmuaNw3PAXQTGZ2xIcYbt7H9q8N5vSdvaWDVAIIQoYKSyEyMH2pZPQrlhF5RsKAKm2cLG+Jy3HL6F+mYoWjk4IIQomKSyEeMzVc8c5v2I05X7P7KLIUMP52i68+Nn39AoMsXB0QghRsElhIcR/RZ47Tuy2b6h7P5TyNgpb/H1IcXKgyuhJ9GzcydLhCSFEoSCFhSj2roQf4ejXwwgIT8K/XQxqO4WTTi9RdcY4KgZJD4UQQuSFFBai2Lpz4zK7xg+kypFYamVkbtt7pyw135tNnXotLRucEEIUUlJYiGIn4X4Mm8f3pcK+awSnZG676QUpr7Wh6/CZcumoEEI8B/kNKooNnVbLkY3fkz7zJ2rHZW67WwJi2tWly6eLsbG1s2h8QghRFEhhIYo8nVbLmd1rcDs4hYb6q2wq64ldug3Xmlek4/hfcHJxs3SIQghRZEhhIYqsiDMHOfLTl/icvIlX/QcEuKaQgCOOL3ei6nejaejtZ+kQhRCiyJHCQhQpmox0ts4bi+7vXVSK0FBbn7k9MtyVhK6vEdj9S1p6eFs2SCGEKMKksBBFwtWLp/ln+nDK/htL5YT/bb/hrSKuXgVeGj6dMgGBlgtQCCGKCSksRKGVlppC+J7fsD61gsCUY5Q+64NHgopkO7hS3Qmvrm/SuudIS4cphBDFihQWotA5uHkJ11f9iE9EAi+2isZRrYAV3HvBkTslKtNqxCzqeZYBQKdXOBIZR0xiGl4u9oQEuKO2Uln4EwghRNElhYUoFGKjI/l77hjcjpyj/E2Fkv/dvu+uJ651X8Gv5RC6VgoyOiY0LJoJf4YTHZ9m2ObrZs+4ToG0C/I1Y/RCCFF8SGEhCixFr+fQlqXc+L/vqXA+1bA6pk4FVwLU6Jo2ot37M3Bwcs1ybGhYNENXnEB5bPvt+DSGrjjB/DfrSnEhhBAmIIWFKHDu343mwl9L8Lz8Gz6JtyjxrweQuZjVrWBvar39GZ1fbJ3j8Tq9woQ/w7MUFQAKoAIm/BlO60AfGRYRQoh8JoWFKBA0GelsW/g5GTu2Y6tPo2Pd2wCkO9iwoYETTi+2oO3gr3K1OuaRyDij4Y/HKUB0fBpHIuNoWNEjvz6CEEIIpLAQFnbu6F+cXvQVpU/foeKDzG1aKytO1SpHWo3eVG/zFj3zuO5ETGLORcWz7CeEECL3pLAQZqfVZLB13liUbX9RIVJH8H/HLFJtIaKaA+5detG85+hnvhmYl4t9vu4nhBAi96SwEGZz7eK/3Ny9iEq3/sDuqgq/K5kn9qtlVMS/WI2XR8ykrm/Ac79PSIA7vm723I5Py3aehQrwccu89FQIIUT+ksJCmNT9u7f469tROB0Kw75CEi1LPwCgUmk3jtZzwa/Xu7TrOChf31NtpWJcp0CGrjiBCoyKi4dTNcd1CpSJm0IIYQJSWAiT2L3mO2J+X0aFc8kEpWduu6534N8KldHWfpOgFr2oYGe6oYh2Qb7Mf7NulnUsfGQdCyGEMCkpLES+SYyPY8uk/rgfjaDsHYWHUy7jXOF6rVIEDhxLrcadzBZPuyBfWgf6yMqbQghhRlJYiOei6PVcOPoXif8spsb9nbj9W5Kyd6zQWsHlijZYtWxOu3en0tjB0SLxqa1UckmpEEKYkRQW4plEhB3k1KKJeJ2+RfUmd6lmpwEVqKpbc6qKLyHvTKRrzUaWDlMIIYSZSWEhcu3s4W2E/fEztqfD0VzREazP3H78Rgm8XngR10aDaFPvZVRWVpYNVAghhMVIYSFydPv6Za6f2E7s6S04bYnA8wHUeuT16z4q7terRNP3p+NbvpqlwhRCCFGASGEhDE7t28jFLf+H1cUIrJ3TeTXgFj5Aik7F5URf9Cq45aUipowjZbr2pc3rIywdshBCiAJGCotiSqfV8u++jVzc9gvWF6/gfSMdjwSo+d/Xoz0VtP5WXLGpxD3fEO4NcyK4XX8eaEvhfO4QNtUboNMrcoWFEEIII1JYFBM6rZbzx/4i+eZ5VNcOUObBCVK32FIr8X/7aK3glreK++XccKj9ImlvTaCKa0kg8zbkPX4NJy4pkukhMGjZUdydHWRNCCGEEEaksCiidFotx/5eTdSO37C5fBWfGxnYaOGFLrdQqwAVHC/rTUK8NQ/KueFYO4QGr39AzTIVs7QVGhbN0BUnUAA79f+2345PY+iKE8x/s64UF0IIIQApLIoMRa/n2qV/ObJqKtYnw/G5oaFEsvFkS60VHNBUwabCSzhXaU6L0U1x/m+PRE50eoUJf4Zne88Nhcwlsif8GU7rQB8ZFhFCCFE0CgudXuFIZBwARyLjaFDJq8Cf5B7G/KwrQuq0Wo5sXcrVXesp66lQLf0s5XlA2MVSVLhgC4BGDTd9VMSXL4lz3YY0en0ENb398hTnkcg4oyWxH6cA0fFpHImMk4WohBBCFP7CIjQsmgl/hhOXlFpoxv4fxvzoCdv3Kfew0Go0HNqymBu7NmAXcYPSN7WUSIESwINWiZQqlUi6YoN1WXdOO9jh8kJjGr3+AbU8yzxXrDGJORcVz7KfEEKIoq1QFxaFcez/0Zgf9XjMep2OqHPHiDnzN7ePbab0zjg8UuHRPoF0a7jpa0WqewjhbQdToXZT2jo45Wu8Xi65u1FYbvcTQghRtBXawqIwjv0/KWaVkkGTtD1cnvkNG+7ew750Cu1Kx1IBiFDbkZHqQbpNZiGR4O9ByReb0aj7+9Qu6WXSmEMC3PF1s+d2fFr2cZN5x9CQAHeTxiGEEKJwKLSFRWEc+8+MORVXUvDmLsEJBymXfB2/mDj8onU4PfJxrqAmxdeOyw5BJPmHkFjNgcbd3qe2m3lP4GorFeM6BTJ0xQkeL88ePh/XKbDAFG9CCCEsq9AWFgVx7D8jPY1LJ3dz68IJEm9cIj0mGqv7D7BJTMU+SUt6CT3hgfdwVKWjVyBsT2lsdP87PtUWbvhacd+vFKWbv4JNj1HUsrUzW/w5aRfky/w36xrmsjzk85R5IUIIIYqfQltYmHvs//a1S1w5s5+4K2GkRF9Ffy8Wq/hEcIDaFa0ooY2lpP4+2t99KavLvo2bGgVHVToACThxrSxordTc8HTjdIla7LZ/GY2VA6sGNygwvSwPtQvypXWgD4cuxxB77hCL+79YKK6+EUIIYV6FtrDIr7H/jLRUroQf4ua5o8Rfu0hGzC1UVloqlC2JfdodnDPuErNRj2sKlCTz8aibPgpVykdnPrGCB64K1loVSc4qUp3UZLjYoS/hitrDE2f/qnSNrsS5ZGfSsIM6WWP2LcDzFdRWKkIC3NlyjjxfHiuEEKJ4KLSFRW7G/j9q6s6Jv1YTE3EabXI83l6lIDEa25Q73N4egdsDBddksNZDaTIfADe9FV50OWxo76ZN5iuptpDgDMnOVqQ526BxdUDl483JRp/j7FmOkr7+NPrUBxtb2xzjVv33qhAVGBVEMl9BCCFEUVBoCwv439j/+I1h9E5dSdqKk3wTr8E5WY9rMjhtyNzPmcxioUGLaMOx+xJ9cU/MPIHrgQQnSHJWkeJsRVopJ/4JeA1rtzI4eJTFvmk6JSvVoXq5yvkW8+PrWMh8BSGEEEVBoS4s4H9j//cnvsW5646UemD81366DcQ7Q5KrNcdcW6Fx8kHlWpr7JW6TWsKb0tVeoGLNxjg4uZo95udZeVMIIYQoiAp9YQGZwyIXy/Umou5hbjmXwN63HB4ValA+qDFVyldDbV3wPqbaSlXgJmgKIYQQz6vgnXGf0Yt9v+Kuxxbat2+PjY2NpcMRQgghiiWrvOw8ZcoUXnzxRVxcXPDy8uLVV1/lwoULpopNCCGEEIVMngqLPXv2MGzYMA4dOsSOHTvQarW0adOG5ORkU8UnhBBCiEIkT0MhoaGhRs+XLFmCl5cXx48fp2nTpvkamBBCCCEKn+eaYxEfHw+Au3vOCzqlp6eTnp5ueJ6QkACARqNBo9E8z9sbedhWfrYpspI8m4/k2jwkz+YheTYPU+Y5t22qFEXJbuHKp1IUhS5dunD//n327duX437jx49nwoQJWbavXLkSR0fHZ3lrIYQQQphZSkoKffr0IT4+HlfXnJdoeObCYtiwYWzevJn9+/dTtmzZHPfLrsfCz8+P2NjYJwaWVxqNhh07dtC6dWu5KsSEJM/mI7k2D8mzeUiezcOUeU5ISKBUqVJPLSyeaSjk/fff548//mDv3r1PLCoA7OzssLPLeodOGxsbk3y5TNWuMCZ5Nh/JtXlIns1D8mwepshzbtvLU2GhKArvv/8+69evZ/fu3QQEBDxTcEIIIYQomvJUWAwbNoyVK1eyceNGXFxcuH37NgBubm44ODiYJEAhhBBCFB55Wsdi/vz5xMfH07x5c3x9fQ2PX3/91VTxCSGEEKIQyfNQiBBCCCFETvLUYyGEEEII8SRSWAghhBAi35j97qYPh1MersCZXzQaDSkpKSQkJMilTCYkeTYfybV5SJ7NQ/JsHqbM88Pz9tOmRZi9sEhMTATAz8/P3G8thBBCiOeUmJiIm5tbjq8/88qbz0qv13Pr1i1cXFxQqVT51u7DFT2vX7+eryt6CmOSZ/ORXJuH5Nk8JM/mYco8K4pCYmIipUuXxsoq55kUZu+xsLKyeupqnc/D1dVVvrRmIHk2H8m1eUiezUPybB6myvOTeioeksmbQgghhMg3UlgIIYQQIt8UmcLCzs6OcePGZXvDM5F/JM/mI7k2D8mzeUiezaMg5NnskzeFEEIIUXQVmR4LIYQQQlieFBZCCCGEyDdSWAghhBAi30hhIYQQQoh8U6gKi3nz5hEQEIC9vT0vvPAC+/bte+L+e/bs4YUXXsDe3p4KFSrw448/minSwi0veV63bh2tW7fG09MTV1dXGjZsyLZt28wYbeGV1+/zQwcOHMDa2pratWubNsAiJK+5Tk9P57PPPqN8+fLY2dlRsWJFFi9ebKZoC6+85vmXX34hODgYR0dHfH19GThwIPfu3TNTtIXT3r176dSpE6VLl0alUrFhw4anHmP2c6FSSKxevVqxsbFRFi5cqISHhysjRoxQnJyclKtXr2a7/5UrVxRHR0dlxIgRSnh4uLJw4ULFxsZGWbt2rZkjL1zymucRI0Yo06ZNU44cOaJcvHhR+eSTTxQbGxvlxIkTZo68cMlrnh968OCBUqFCBaVNmzZKcHCweYIt5J4l1507d1bq16+v7NixQ4mMjFQOHz6sHDhwwIxRFz55zfO+ffsUKysr5dtvv1WuXLmi7Nu3T6lRo4by6quvmjnywmXLli3KZ599pvz+++8KoKxfv/6J+1viXFhoCouQkBDl3XffNdpWrVo15eOPP852/7FjxyrVqlUz2vbOO+8oDRo0MFmMRUFe85ydwMBAZcKECfkdWpHyrHnu2bOn8vnnnyvjxo2TwiKX8prrrVu3Km5ubsq9e/fMEV6Rkdc8f/PNN0qFChWMts2dO1cpW7asyWIsanJTWFjiXFgohkIyMjI4fvw4bdq0Mdrepk0bDh48mO0x//zzT5b927Zty7Fjx9BoNCaLtTB7ljw/Tq/Xk5iYiLu7uylCLBKeNc9LliwhIiKCcePGmTrEIuNZcv3HH39Qr149pk+fTpkyZahSpQpjxowhNTXVHCEXSs+S50aNGnHjxg22bNmCoijcuXOHtWvX0qFDB3OEXGxY4lxo9puQPYvY2Fh0Oh3e3t5G2729vbl9+3a2x9y+fTvb/bVaLbGxsfj6+pos3sLqWfL8uJkzZ5KcnEyPHj1MEWKR8Cx5vnTpEh9//DH79u3D2rpQ/NgWCM+S6ytXrrB//37s7e1Zv349sbGxvPfee8TFxck8ixw8S54bNWrEL7/8Qs+ePUlLS0Or1dK5c2e+++47c4RcbFjiXFgoeiweevw264qiPPHW69ntn912YSyveX5o1apVjB8/nl9//RUvLy9ThVdk5DbPOp2OPn36MGHCBKpUqWKu8IqUvHyn9Xo9KpWKX375hZCQENq3b8+sWbNYunSp9Fo8RV7yHB4ezgcffMCXX37J8ePHCQ0NJTIyknfffdccoRYr5j4XFoo/fUqVKoVarc5S+cbExGSpxB7y8fHJdn9ra2s8PDxMFmth9ix5fujXX3/lrbfeYs2aNbRq1cqUYRZ6ec1zYmIix44d4+TJkwwfPhzIPPkpioK1tTXbt2+nZcuWZom9sHmW77Svry9lypQxuj109erVURSFGzduULlyZZPGXBg9S56nTJlC48aN+fDDDwGoVasWTk5ONGnShK+++kp6lfOJJc6FhaLHwtbWlhdeeIEdO3YYbd+xYweNGjXK9piGDRtm2X/79u3Uq1cPGxsbk8VamD1LniGzp2LAgAGsXLlSxkdzIa95dnV15cyZM5w6dcrwePfdd6latSqnTp2ifv365gq90HmW73Tjxo25desWSUlJhm0XL17EysqKsmXLmjTewupZ8pySkoKVlfEpSK1WA//7i1o8P4ucC002LTSfPbyU6eeff1bCw8OVkSNHKk5OTkpUVJSiKIry8ccfK3379jXs//ASm1GjRinh4eHKzz//LJeb5kJe87xy5UrF2tpa+eGHH5To6GjD48GDB5b6CIVCXvP8OLkqJPfymuvExESlbNmySvfu3ZWzZ88qe/bsUSpXrqy8/fbblvoIhUJe87xkyRLF2tpamTdvnhIREaHs379fqVevnhISEmKpj1AoJCYmKidPnlROnjypAMqsWbOUkydPGi7rLQjnwkJTWCiKovzwww9K+fLlFVtbW6Vu3brKnj17DK/1799fadasmdH+u3fvVurUqaPY2toq/v7+yvz5880cceGUlzw3a9ZMAbI8+vfvb/7AC5m8fp8fJYVF3uQ11+fOnVNatWqlODg4KGXLllVGjx6tpKSkmDnqwieveZ47d64SGBioODg4KL6+vsobb7yh3Lhxw8xRFy67du164u/cgnAulNumCyGEECLfFIo5FkIIIYQoHKSwEEIIIUS+kcJCCCGEEPlGCgshhBBC5BspLIQQQgiRb6SwEEIIIUS+kcJCCCGEEPlGCgshhBBC5BspLIQQQgiRb6SwEEIIIUS+kcJCCCGEEPlGCgshhBBC5Jv/B0LFhVYonimoAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", @@ -5642,15 +5544,15 @@ " return np.sum((y_data-y_model)**2)/n\n", "\n", "\n", - "def fit_beta(X, y):\n", + "def fit_theta(X, y):\n", " return np.linalg.pinv(X.T @ X) @ X.T @ y\n", "\n", "\n", - "true_beta = [2, 0.5, 3.7]\n", + "true_theta = [2, 0.5, 3.7]\n", "\n", "x = np.linspace(0, 1, 11)\n", "y = np.sum(\n", - " np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0\n", + " np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0\n", ") + 0.1 * np.random.normal(size=len(x))\n", "\n", "degree = 3\n", @@ -5660,15 +5562,15 @@ "for p in range(degree):\n", " X[:, p] = x ** p\n", "\n", - "beta = fit_beta(X, y)\n", + "theta = fit_theta(X, y)\n", "\n", "# Intercept is included in the design matrix\n", "skl = LinearRegression(fit_intercept=False).fit(X, y)\n", "\n", - "print(f\"True beta: {true_beta}\")\n", - "print(f\"Fitted beta: {beta}\")\n", - "print(f\"Sklearn fitted beta: {skl.coef_}\")\n", - "ypredictOwn = X @ beta\n", + "print(f\"True theta: {true_theta}\")\n", + "print(f\"Fitted theta: {theta}\")\n", + "print(f\"Sklearn fitted theta: {skl.coef_}\")\n", + "ypredictOwn = X @ theta\n", "ypredictSKL = skl.predict(X)\n", "print(f\"MSE with intercept column\")\n", "print(MSE(y,ypredictOwn))\n", @@ -5678,7 +5580,7 @@ "\n", "plt.figure()\n", "plt.scatter(x, y, label=\"Data\")\n", - "plt.plot(x, X @ beta, label=\"Fit\")\n", + "plt.plot(x, X @ theta, label=\"Fit\")\n", "plt.plot(x, skl.predict(X), label=\"Sklearn (fit_intercept=False)\")\n", "\n", "\n", @@ -5695,21 +5597,21 @@ "y_offset = np.average(y, axis=0)\n", "X_offset = np.average(X, axis=0)\n", "\n", - "beta = fit_beta(X - X_offset, y - y_offset)\n", - "intercept = np.mean(y_offset - X_offset @ beta)\n", + "theta = fit_theta(X - X_offset, y - y_offset)\n", + "intercept = np.mean(y_offset - X_offset @ theta)\n", "\n", "print(f\"Manual intercept: {intercept}\")\n", - "print(f\"Fitted beta (without intercept): {beta}\")\n", + "print(f\"Fitted theta (without intercept): {theta}\")\n", "print(f\"Sklearn intercept: {skl.intercept_}\")\n", - "print(f\"Sklearn fitted beta (without intercept): {skl.coef_}\")\n", - "ypredictOwn = X @ beta\n", + "print(f\"Sklearn fitted theta (without intercept): {skl.coef_}\")\n", + "ypredictOwn = X @ theta\n", "ypredictSKL = skl.predict(X)\n", "print(f\"MSE with Manual intercept\")\n", "print(MSE(y,ypredictOwn+intercept))\n", "print(f\"MSE with Sklearn intercept\")\n", "print(MSE(y,ypredictSKL))\n", "\n", - "plt.plot(x, X @ beta + intercept, \"--\", label=\"Fit (manual intercept)\")\n", + "plt.plot(x, X @ theta + intercept, \"--\", label=\"Fit (manual intercept)\")\n", "plt.plot(x, skl.predict(X), \"--\", label=\"Sklearn (fit_intercept=True)\")\n", "plt.grid()\n", "plt.legend()\n", @@ -5719,7 +5621,7 @@ }, { "cell_type": "markdown", - "id": "df21278e", + "id": "5bd0d190", "metadata": { "editable": true }, @@ -5732,7 +5634,7 @@ "the way we treat the intercept may give a larger or smaller MSE,\n", "meaning that the MSE can be penalized by the value of the\n", "intercept. Not including the intercept in the fit, means that the\n", - "regularization term does not include $\\beta_0$. For different values\n", + "regularization term does not include $\\theta_0$. For different values\n", "of $\\lambda$, this may lead to different MSE values. \n", "\n", "To remind the reader, the regularization term, with the intercept in Ridge regression, is given by" @@ -5740,19 +5642,19 @@ }, { "cell_type": "markdown", - "id": "35baa0d5", + "id": "a06d4663", "metadata": { "editable": true }, "source": [ "$$\n", - "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", + "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\theta_j^2,\n", "$$" ] }, { "cell_type": "markdown", - "id": "e5cd5856", + "id": "01ab58d2", "metadata": { "editable": true }, @@ -5762,19 +5664,19 @@ }, { "cell_type": "markdown", - "id": "991247e4", + "id": "dca995a5", "metadata": { "editable": true }, "source": [ "$$\n", - "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", + "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\theta_j^2.\n", "$$" ] }, { "cell_type": "markdown", - "id": "a2a95901", + "id": "3694dc1d", "metadata": { "editable": true }, @@ -5784,19 +5686,19 @@ }, { "cell_type": "markdown", - "id": "71096629", + "id": "5104f166", "metadata": { "editable": true }, "source": [ "$$\n", - "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", + "\\lambda \\vert\\vert \\boldsymbol{\\theta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\theta_j\\vert.\n", "$$" ] }, { "cell_type": "markdown", - "id": "21c6f842", + "id": "91921fb2", "metadata": { "editable": true }, @@ -5814,13 +5716,10 @@ { "cell_type": "code", "execution_count": 19, - "id": "9f5e69c9", + "id": "9af3375b", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -5860,20 +5759,20 @@ "lambdas = np.logspace(-4, 2, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", + " OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n", " # Note: we include the intercept column and no scaling\n", " RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n", " RegRidge.fit(X_train,y_train)\n", " # and then make the prediction\n", - " ytildeOwnRidge = X_train @ OwnRidgeBeta\n", - " ypredictOwnRidge = X_test @ OwnRidgeBeta\n", + " ytildeOwnRidge = X_train @ OwnRidgeTheta\n", + " ypredictOwnRidge = X_test @ OwnRidgeTheta\n", " ytildeRidge = RegRidge.predict(X_train)\n", " ypredictRidge = RegRidge.predict(X_test)\n", " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", - " print(\"Beta values for own Ridge implementation\")\n", - " print(OwnRidgeBeta)\n", - " print(\"Beta values for Scikit-Learn Ridge implementation\")\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta)\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", " print(RegRidge.coef_)\n", " print(\"MSE values for own Ridge implementation\")\n", " print(MSEOwnRidgePredict[i])\n", @@ -5893,7 +5792,7 @@ }, { "cell_type": "markdown", - "id": "85305149", + "id": "e4a67fbf", "metadata": { "editable": true }, @@ -5907,13 +5806,10 @@ { "cell_type": "code", "execution_count": 20, - "id": "806a0fde", + "id": "07d02e73", "metadata": { "collapsed": false, - "editable": true, - "jupyter": { - "outputs_hidden": false - } + "editable": true }, "outputs": [], "source": [ @@ -5964,18 +5860,18 @@ "lambdas = np.logspace(-4, 2, nlambdas)\n", "for i in range(nlambdas):\n", " lmb = lambdas[i]\n", - " OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n", - " intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data\n", + " OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled)\n", + " intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data\n", " #Add intercept to prediction\n", - " ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler \n", + " ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler \n", " RegRidge = linear_model.Ridge(lmb)\n", " RegRidge.fit(X_train,y_train)\n", " ypredictRidge = RegRidge.predict(X_test)\n", " MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge)\n", " MSERidgePredict[i] = MSE(y_test,ypredictRidge)\n", - " print(\"Beta values for own Ridge implementation\")\n", - " print(OwnRidgeBeta) #Intercept is given by mean of target variable\n", - " print(\"Beta values for Scikit-Learn Ridge implementation\")\n", + " print(\"Theta values for own Ridge implementation\")\n", + " print(OwnRidgeTheta) #Intercept is given by mean of target variable\n", + " print(\"Theta values for Scikit-Learn Ridge implementation\")\n", " print(RegRidge.coef_)\n", " print('Intercept from own implementation:')\n", " print(intercept_)\n", @@ -5999,7 +5895,7 @@ }, { "cell_type": "markdown", - "id": "d8aa3ac8", + "id": "20eff4f1", "metadata": { "editable": true }, @@ -6007,31 +5903,13 @@ "We see here, when compared to the code which includes explicitely the\n", "intercept column, that our MSE value is actually smaller. This is\n", "because the regularization term does not include the intercept value\n", - "$\\beta_0$ in the fitting. This applies to Lasso regularization as\n", + "$\\theta_0$ in the fitting. This applies to Lasso regularization as\n", "well. It means that our optimization is now done only with the\n", "centered matrix and/or vector that enter the fitting procedure." ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.15" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 4309600e7..6e9d1f149 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -23,10 +23,6 @@ o Goodfellow, Bengio and Courville, Deep Learning, chapter 2 on linear algebra o Raschka et al on preprocessing of data, relevant for exercise 3 this week, see chapter 4. o For exercise 1 of week 35, the book by A. Aldo Faisal, Cheng Soon Ong, and Marc Peter Deisenroth on the Mathematics of Machine Learning, may be very relevant. In particular chapter 5 at URL"https://mml-book.github.io/" (section 5.5 on derivatives) is very useful for exercise 1 this coming week. - - - - !split ===== Reminder from last week ===== @@ -2342,8 +2338,8 @@ Thus, if we cannot assume that the expected outputs/targets are zero when all predictors are zero (the columns in the design matrix), it may be a bad idea to implement a model which penalizes the intercept. Furthermore, in for example Ridge and Lasso regression, the default solutions -from the library _Scikit-Learn_ (when not shrinking $\beta_0$) for the unknown parameters -$\bm{\beta}$, are derived under the assumption that both $\bm{y}$ and +from the library _Scikit-Learn_ (when not shrinking $\theta_0$) for the unknown parameters +$\bm{\theta}$, are derived under the assumption that both $\bm{y}$ and $\bm{X}$ are zero centered, that is we subtract the mean values. @@ -2407,7 +2403,7 @@ simplicity, we will focus on ordinary regression, as done in the above example. The cost/loss function for regression is !bt \[ -C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,. +C(\theta_0, \theta_1, ... , \theta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij}\theta_j\right)^2,. \] !et @@ -2415,42 +2411,42 @@ Recall also that we use the squared value. This expression can lead to an increased penalty for higher differences between predicted and output/target values. -What we have done is to single out the $\beta_0$ term in the +What we have done is to single out the $\theta_0$ term in the definition of the mean squared error (MSE). The design matrix $X$ does in this case not contain any intercept column. When we take the -derivative with respect to $\beta_0$, we want the derivative to obey +derivative with respect to $\theta_0$, we want the derivative to obey !bt \[ -\frac{\partial C}{\partial \beta_j} = 0, +\frac{\partial C}{\partial \theta_j} = 0, \] !et -for all $j$. For $\beta_0$ we have +for all $j$. For $\theta_0$ we have !bt \[ -\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right). +\frac{\partial C}{\partial \theta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \theta_0 - \sum_{j=1}^{p-1} X_{ij} \theta_j\right). \] !et Multiplying away the constant $2/n$, we obtain !bt \[ -\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j. +\sum_{i=0}^{n-1} \theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \theta_j. \] !et -Let us specialize first to the case where we have only two parameters $\beta_0$ and $\beta_1$. -Our result for $\beta_0$ simplifies then to +Let us specialize first to the case where we have only two parameters $\theta_0$ and $\theta_1$. +Our result for $\theta_0$ simplifies then to !bt \[ -n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1. +n\theta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \theta_1. \] !et We obtain then !bt \[ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \theta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}. \] !et If we define @@ -2468,20 +2464,20 @@ and the mean value of the outputs as we have !bt \[ -\beta_0 = \mu_y - \beta_1\mu_{\bm{x}_1}. +\theta_0 = \mu_y - \theta_1\mu_{\bm{x}_1}. \] !et -In the general case with more parameters than $\beta_0$ and $\beta_1$, we have +In the general case with more parameters than $\theta_0$ and $\theta_1$, we have !bt \[ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j. +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\theta_j. \] !et We can rewrite the latter equation as !bt \[ -\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\bm{x}_j}\beta_j, +\theta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\bm{x}_j}\theta_j, \] !et where we have defined @@ -2497,27 +2493,27 @@ the mean value for all elements of the column vector $\bm{x}_j$. Replacing $y_i$ with $y_i - y_i - \overline{\bm{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise) !bt \[ -C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}). +C(\boldsymbol{\theta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\theta}). \] !et -If we minimize with respect to $\bm{\beta}$ we have then +If we minimize with respect to $\bm{\theta}$ we have then !bt \[ -\hat{\bm{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, +\hat{\bm{\theta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}, \] !et where $\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\bm{y}}$ and $\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}$. -For Ridge regression we need to add $\lambda \boldsymbol{\beta}^T\boldsymbol{\beta}$ to the cost function and get then +For Ridge regression we need to add $\lambda \boldsymbol{\theta}^T\boldsymbol{\theta}$ to the cost function and get then !bt \[ -\hat{\bm{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. +\hat{\bm{\theta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}. \] !et @@ -2541,15 +2537,15 @@ def MSE(y_data,y_model): return np.sum((y_data-y_model)**2)/n -def fit_beta(X, y): +def fit_theta(X, y): return np.linalg.pinv(X.T @ X) @ X.T @ y -true_beta = [2, 0.5, 3.7] +true_theta = [2, 0.5, 3.7] x = np.linspace(0, 1, 11) y = np.sum( - np.asarray([x ** p * b for p, b in enumerate(true_beta)]), axis=0 + np.asarray([x ** p * b for p, b in enumerate(true_theta)]), axis=0 ) + 0.1 * np.random.normal(size=len(x)) degree = 3 @@ -2559,15 +2555,15 @@ X = np.zeros((len(x), degree)) for p in range(degree): X[:, p] = x ** p -beta = fit_beta(X, y) +theta = fit_theta(X, y) # Intercept is included in the design matrix skl = LinearRegression(fit_intercept=False).fit(X, y) -print(f"True beta: {true_beta}") -print(f"Fitted beta: {beta}") -print(f"Sklearn fitted beta: {skl.coef_}") -ypredictOwn = X @ beta +print(f"True theta: {true_theta}") +print(f"Fitted theta: {theta}") +print(f"Sklearn fitted theta: {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with intercept column") print(MSE(y,ypredictOwn)) @@ -2577,7 +2573,7 @@ print(MSE(y,ypredictSKL)) plt.figure() plt.scatter(x, y, label="Data") -plt.plot(x, X @ beta, label="Fit") +plt.plot(x, X @ theta, label="Fit") plt.plot(x, skl.predict(X), label="Sklearn (fit_intercept=False)") @@ -2594,21 +2590,21 @@ skl = LinearRegression(fit_intercept=True).fit(X, y) y_offset = np.average(y, axis=0) X_offset = np.average(X, axis=0) -beta = fit_beta(X - X_offset, y - y_offset) -intercept = np.mean(y_offset - X_offset @ beta) +theta = fit_theta(X - X_offset, y - y_offset) +intercept = np.mean(y_offset - X_offset @ theta) print(f"Manual intercept: {intercept}") -print(f"Fitted beta (without intercept): {beta}") +print(f"Fitted theta (without intercept): {theta}") print(f"Sklearn intercept: {skl.intercept_}") -print(f"Sklearn fitted beta (without intercept): {skl.coef_}") -ypredictOwn = X @ beta +print(f"Sklearn fitted theta (without intercept): {skl.coef_}") +ypredictOwn = X @ theta ypredictSKL = skl.predict(X) print(f"MSE with Manual intercept") print(MSE(y,ypredictOwn+intercept)) print(f"MSE with Sklearn intercept") print(MSE(y,ypredictSKL)) -plt.plot(x, X @ beta + intercept, "--", label="Fit (manual intercept)") +plt.plot(x, X @ theta + intercept, "--", label="Fit (manual intercept)") plt.plot(x, skl.predict(X), "--", label="Sklearn (fit_intercept=True)") plt.grid() plt.legend() @@ -2625,26 +2621,26 @@ they should. However, when we move to for example Ridge regression, the way we treat the intercept may give a larger or smaller MSE, meaning that the MSE can be penalized by the value of the intercept. Not including the intercept in the fit, means that the -regularization term does not include $\beta_0$. For different values +regularization term does not include $\theta_0$. For different values of $\lambda$, this may lead to different MSE values. To remind the reader, the regularization term, with the intercept in Ridge regression, is given by !bt \[ -\lambda \vert\vert \bm{\beta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\beta_j^2, +\lambda \vert\vert \bm{\theta} \vert\vert_2^2 = \lambda \sum_{j=0}^{p-1}\theta_j^2, \] !et but when we take out the intercept, this equation becomes !bt \[ -\lambda \vert\vert \bm{\beta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\beta_j^2. +\lambda \vert\vert \bm{\theta} \vert\vert_2^2 = \lambda \sum_{j=1}^{p-1}\theta_j^2. \] !et For Lasso regression we have !bt \[ -\lambda \vert\vert \bm{\beta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\beta_j\vert. +\lambda \vert\vert \bm{\theta} \vert\vert_1 = \lambda \sum_{j=1}^{p-1}\vert\theta_j\vert. \] !et @@ -2695,20 +2691,20 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train + OwnRidgeTheta = np.linalg.pinv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train # Note: we include the intercept column and no scaling RegRidge = linear_model.Ridge(lmb,fit_intercept=False) RegRidge.fit(X_train,y_train) # and then make the prediction - ytildeOwnRidge = X_train @ OwnRidgeBeta - ypredictOwnRidge = X_test @ OwnRidgeBeta + ytildeOwnRidge = X_train @ OwnRidgeTheta + ypredictOwnRidge = X_test @ OwnRidgeTheta ytildeRidge = RegRidge.predict(X_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print("MSE values for own Ridge implementation") print(MSEOwnRidgePredict[i]) @@ -2780,18 +2776,18 @@ MSERidgePredict = np.zeros(nlambdas) lambdas = np.logspace(-4, 2, nlambdas) for i in range(nlambdas): lmb = lambdas[i] - OwnRidgeBeta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) - intercept_ = y_scaler - X_train_mean@OwnRidgeBeta #The intercept can be shifted so the model can predict on uncentered data + OwnRidgeTheta = np.linalg.pinv(X_train_scaled.T @ X_train_scaled+lmb*I) @ X_train_scaled.T @ (y_train_scaled) + intercept_ = y_scaler - X_train_mean@OwnRidgeTheta #The intercept can be shifted so the model can predict on uncentered data #Add intercept to prediction - ypredictOwnRidge = X_test_scaled @ OwnRidgeBeta + y_scaler + ypredictOwnRidge = X_test_scaled @ OwnRidgeTheta + y_scaler RegRidge = linear_model.Ridge(lmb) RegRidge.fit(X_train,y_train) ypredictRidge = RegRidge.predict(X_test) MSEOwnRidgePredict[i] = MSE(y_test,ypredictOwnRidge) MSERidgePredict[i] = MSE(y_test,ypredictRidge) - print("Beta values for own Ridge implementation") - print(OwnRidgeBeta) #Intercept is given by mean of target variable - print("Beta values for Scikit-Learn Ridge implementation") + print("Theta values for own Ridge implementation") + print(OwnRidgeTheta) #Intercept is given by mean of target variable + print("Theta values for Scikit-Learn Ridge implementation") print(RegRidge.coef_) print('Intercept from own implementation:') print(intercept_) @@ -2815,7 +2811,7 @@ plt.show() We see here, when compared to the code which includes explicitely the intercept column, that our MSE value is actually smaller. This is because the regularization term does not include the intercept value -$\beta_0$ in the fitting. This applies to Lasso regularization as +$\theta_0$ in the fitting. This applies to Lasso regularization as well. It means that our optimization is now done only with the centered matrix and/or vector that enter the fitting procedure.