diff --git a/doc/pub/week36/html/._week36-bs001.html b/doc/pub/week36/html/._week36-bs001.html
index 75ce3a4a2..0085b05c6 100644
--- a/doc/pub/week36/html/._week36-bs001.html
+++ b/doc/pub/week36/html/._week36-bs001.html
@@ -278,6 +278,7 @@ MathJax.Hub.Config({
Linear Regression and links with Statistics
Recommended Reading: Goodfellow et al chapter 3 on probability theory
See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)
+ Video of lecture
diff --git a/doc/pub/week36/html/._week36-bs002.html b/doc/pub/week36/html/._week36-bs002.html
index 6ced8eb00..1b330255d 100644
--- a/doc/pub/week36/html/._week36-bs002.html
+++ b/doc/pub/week36/html/._week36-bs002.html
@@ -265,7 +265,7 @@ MathJax.Hub.Config({
Material for the active learning sessions Tuesday and Wednesday
-The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
+The material here contains a summary from last week and discussion of SVD, Ridge and Lasso regression with examples
diff --git a/doc/pub/week36/html/week36-reveal.html b/doc/pub/week36/html/week36-reveal.html
index 40817776a..4433dc228 100644
--- a/doc/pub/week36/html/week36-reveal.html
+++ b/doc/pub/week36/html/week36-reveal.html
@@ -218,6 +218,8 @@ MathJax.Hub.Config({
Recommended Reading: Goodfellow et al chapter 3 on probability theory
See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)
+
+ Video of lecture
@@ -226,7 +228,7 @@ MathJax.Hub.Config({
Material for the active learning sessions Tuesday and Wednesday
-The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
+The material here contains a summary from last week and discussion of SVD, Ridge and Lasso regression with examples
diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html
index 24579e5da..e88970504 100644
--- a/doc/pub/week36/html/week36-solarized.html
+++ b/doc/pub/week36/html/week36-solarized.html
@@ -253,12 +253,13 @@ MathJax.Hub.Config({
Linear Regression and links with Statistics
Recommended Reading: Goodfellow et al chapter 3 on probability theory
See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)
+ Video of lecture
Material for the active learning sessions Tuesday and Wednesday
-The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
+The material here contains a summary from last week and discussion of SVD, Ridge and Lasso regression with examples
Linear Regression and the SVD
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 626ad64eb..1bd4a7e72 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -330,12 +330,13 @@ MathJax.Hub.Config({
Linear Regression and links with Statistics
Recommended Reading: Goodfellow et al chapter 3 on probability theory
See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)
+ Video of lecture
Material for the active learning sessions Tuesday and Wednesday
-The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
+The material here contains a summary from last week and discussion of SVD, Ridge and Lasso regression with examples
Linear Regression and the SVD
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index 69318c705..6fdcdeebf 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index 9c48124a3..06eaec29a 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
- "id": "ff34ab4f",
- "metadata": {},
+ "id": "15bccd1f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
- "id": "a88eeeaf",
- "metadata": {},
+ "id": "98c44981",
+ "metadata": {
+ "editable": true
+ },
"source": [
"# Week 36: Statistical interpretation of Linear Regression and Resampling techniques\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
@@ -23,8 +27,10 @@
},
{
"cell_type": "markdown",
- "id": "49b54282",
- "metadata": {},
+ "id": "bc3d21f0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Plans for week 36\n",
"\n",
@@ -44,23 +50,29 @@
"\n",
" * [Recommended Reading: Goodfellow et al chapter 3 on probability theory](https://www.deeplearningbook.org/)\n",
"\n",
- " * See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)"
+ " * See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)\n",
+ "\n",
+ " * [Video of lecture](https://youtu.be/Kc20CFK0z7Y)"
]
},
{
"cell_type": "markdown",
- "id": "cb61e293",
- "metadata": {},
+ "id": "67516210",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Material for the active learning sessions Tuesday and Wednesday\n",
"\n",
- "The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples"
+ "The material here contains a summary from last week and discussion of SVD, Ridge and Lasso regression with examples"
]
},
{
"cell_type": "markdown",
- "id": "0b12f4e7",
- "metadata": {},
+ "id": "79dd04a7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Linear Regression and the SVD\n",
"\n",
@@ -69,8 +81,10 @@
},
{
"cell_type": "markdown",
- "id": "b77ff400",
- "metadata": {},
+ "id": "d31c0b5b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n",
@@ -79,16 +93,20 @@
},
{
"cell_type": "markdown",
- "id": "94203cb0",
- "metadata": {},
+ "id": "8e335214",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix"
]
},
{
"cell_type": "markdown",
- "id": "9adad1a7",
- "metadata": {},
+ "id": "4024520b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n",
@@ -97,16 +115,20 @@
},
{
"cell_type": "markdown",
- "id": "8d44ef1e",
- "metadata": {},
+ "id": "a329d7a5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is"
]
},
{
"cell_type": "markdown",
- "id": "2b65e6c1",
- "metadata": {},
+ "id": "e8260d34",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n",
@@ -120,16 +142,20 @@
},
{
"cell_type": "markdown",
- "id": "4bec5357",
- "metadata": {},
+ "id": "7728e31e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"meaning we can write"
]
},
{
"cell_type": "markdown",
- "id": "4240b8b5",
- "metadata": {},
+ "id": "10264b7f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n",
@@ -138,16 +164,20 @@
},
{
"cell_type": "markdown",
- "id": "bf4a51f1",
- "metadata": {},
+ "id": "39337b25",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get"
]
},
{
"cell_type": "markdown",
- "id": "fbec17ea",
- "metadata": {},
+ "id": "c2b1d980",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n",
@@ -156,8 +186,10 @@
},
{
"cell_type": "markdown",
- "id": "722612e1",
- "metadata": {},
+ "id": "7df9c6d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## What does it mean?\n",
"\n",
@@ -168,8 +200,10 @@
},
{
"cell_type": "markdown",
- "id": "916f14e8",
- "metadata": {},
+ "id": "87666c78",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n",
@@ -178,8 +212,10 @@
},
{
"cell_type": "markdown",
- "id": "34de5c7a",
- "metadata": {},
+ "id": "916acaf0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n",
"square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n",
@@ -198,8 +234,10 @@
},
{
"cell_type": "markdown",
- "id": "4db4d16d",
- "metadata": {},
+ "id": "720b9f19",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n",
@@ -208,8 +246,10 @@
},
{
"cell_type": "markdown",
- "id": "4fcefcd0",
- "metadata": {},
+ "id": "92bb492f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n",
"the number of samples) are the eigenvalues of the covariance\n",
@@ -221,8 +261,10 @@
},
{
"cell_type": "markdown",
- "id": "3f3197db",
- "metadata": {},
+ "id": "6a01fe6a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n",
"\n",
@@ -231,8 +273,10 @@
},
{
"cell_type": "markdown",
- "id": "0a6eaa09",
- "metadata": {},
+ "id": "fcf63d98",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n",
@@ -241,16 +285,20 @@
},
{
"cell_type": "markdown",
- "id": "c7d43a29",
- "metadata": {},
+ "id": "02f5423c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Since the matrices here have dimension $n\\times n$, we have"
]
},
{
"cell_type": "markdown",
- "id": "33e14cc4",
- "metadata": {},
+ "id": "06f8ed68",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n",
@@ -259,16 +307,20 @@
},
{
"cell_type": "markdown",
- "id": "8ad076ff",
- "metadata": {},
+ "id": "3b2cb558",
+ "metadata": {
+ "editable": true
+ },
"source": [
"leading to"
]
},
{
"cell_type": "markdown",
- "id": "b2b1aeac",
- "metadata": {},
+ "id": "12586257",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n",
@@ -277,16 +329,20 @@
},
{
"cell_type": "markdown",
- "id": "8deb30d4",
- "metadata": {},
+ "id": "e682d3ee",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem"
]
},
{
"cell_type": "markdown",
- "id": "f1c7bce6",
- "metadata": {},
+ "id": "428ad724",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n",
@@ -295,8 +351,10 @@
},
{
"cell_type": "markdown",
- "id": "d90876f4",
- "metadata": {},
+ "id": "c6d3b301",
+ "metadata": {
+ "editable": true
+ },
"source": [
"It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n",
"the non-zero singular values plus now a series of zeros. The column\n",
@@ -310,8 +368,10 @@
},
{
"cell_type": "markdown",
- "id": "85930202",
- "metadata": {},
+ "id": "520ba455",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Code for SVD and Inversion of Matrices\n",
"\n",
@@ -322,8 +382,11 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "540328fb",
- "metadata": {},
+ "id": "94e98151",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -333,8 +396,10 @@
},
{
"cell_type": "markdown",
- "id": "58c2ee4a",
- "metadata": {},
+ "id": "b2c96d9b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD."
]
@@ -342,8 +407,11 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "ec5c2fe8",
- "metadata": {},
+ "id": "e1ec5fde",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -379,8 +447,10 @@
},
{
"cell_type": "markdown",
- "id": "15898f2d",
- "metadata": {},
+ "id": "fe39aea1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Inverse of Rectangular Matrix\n",
"\n",
@@ -397,8 +467,10 @@
},
{
"cell_type": "markdown",
- "id": "82a70c91",
- "metadata": {},
+ "id": "f3d30ae1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n",
@@ -407,8 +479,10 @@
},
{
"cell_type": "markdown",
- "id": "c60476a5",
- "metadata": {},
+ "id": "484ce06a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{\\Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD."
]
@@ -416,8 +490,11 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "114372da",
- "metadata": {},
+ "id": "555daa13",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -447,16 +524,20 @@
},
{
"cell_type": "markdown",
- "id": "f08b1538",
- "metadata": {},
+ "id": "346ce362",
+ "metadata": {
+ "editable": true
+ },
"source": [
"As you can see from this example, our own decomposition based on the SVD agrees with the pseudoinverse algorithm provided by **Numpy**."
]
},
{
"cell_type": "markdown",
- "id": "515bf446",
- "metadata": {},
+ "id": "65a4bfc0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Ridge and LASSO Regression\n",
"\n",
@@ -466,8 +547,10 @@
},
{
"cell_type": "markdown",
- "id": "8e63f91c",
- "metadata": {},
+ "id": "a996dbac",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
@@ -476,16 +559,20 @@
},
{
"cell_type": "markdown",
- "id": "37cc5c42",
- "metadata": {},
+ "id": "00dda8a9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or we can state it as"
]
},
{
"cell_type": "markdown",
- "id": "3cb8a97a",
- "metadata": {},
+ "id": "cad89651",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
@@ -495,16 +582,20 @@
},
{
"cell_type": "markdown",
- "id": "d00c65cc",
- "metadata": {},
+ "id": "df182246",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have used the definition of a norm-2 vector, that is"
]
},
{
"cell_type": "markdown",
- "id": "1c4684ff",
- "metadata": {},
+ "id": "941d5dbb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n",
@@ -513,8 +604,10 @@
},
{
"cell_type": "markdown",
- "id": "899a6755",
- "metadata": {},
+ "id": "df1f9ea0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## From OLS to Ridge and Lasso\n",
"\n",
@@ -526,8 +619,10 @@
},
{
"cell_type": "markdown",
- "id": "35858a3a",
- "metadata": {},
+ "id": "c396361d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
@@ -537,8 +632,10 @@
},
{
"cell_type": "markdown",
- "id": "fcbba7d8",
- "metadata": {},
+ "id": "ac773467",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which leads to the Ridge regression minimization problem where we\n",
"require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
@@ -549,8 +646,10 @@
},
{
"cell_type": "markdown",
- "id": "26cb747b",
- "metadata": {},
+ "id": "e2d61c36",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n",
@@ -559,16 +658,20 @@
},
{
"cell_type": "markdown",
- "id": "ae7c22be",
- "metadata": {},
+ "id": "75611c24",
+ "metadata": {
+ "editable": true
+ },
"source": [
"we have a new optimization equation"
]
},
{
"cell_type": "markdown",
- "id": "0ef7ea7a",
- "metadata": {},
+ "id": "f0892133",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
@@ -578,8 +681,10 @@
},
{
"cell_type": "markdown",
- "id": "285f0e32",
- "metadata": {},
+ "id": "7fe31d0b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n",
"\n",
@@ -588,8 +693,10 @@
},
{
"cell_type": "markdown",
- "id": "48dc1686",
- "metadata": {},
+ "id": "56fb6bd8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n",
@@ -598,8 +705,10 @@
},
{
"cell_type": "markdown",
- "id": "94bea820",
- "metadata": {},
+ "id": "2371926e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Deriving the Ridge Regression Equations\n",
"\n",
@@ -608,8 +717,10 @@
},
{
"cell_type": "markdown",
- "id": "79dbe6d8",
- "metadata": {},
+ "id": "05d8fe56",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n",
@@ -618,8 +729,10 @@
},
{
"cell_type": "markdown",
- "id": "299d4456",
- "metadata": {},
+ "id": "ff200404",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and \n",
"taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
@@ -630,8 +743,10 @@
},
{
"cell_type": "markdown",
- "id": "28f1259b",
- "metadata": {},
+ "id": "6cb12da3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
@@ -640,16 +755,20 @@
},
{
"cell_type": "markdown",
- "id": "ef07fce1",
- "metadata": {},
+ "id": "6dbbd5b9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that"
]
},
{
"cell_type": "markdown",
- "id": "567e07ca",
- "metadata": {},
+ "id": "54d40d41",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n",
@@ -658,16 +777,20 @@
},
{
"cell_type": "markdown",
- "id": "ab200db0",
- "metadata": {},
+ "id": "fdf68a3a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $t$ a finite positive number."
]
},
{
"cell_type": "markdown",
- "id": "e2d8ca37",
- "metadata": {},
+ "id": "5eca2f87",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Note on Scikit-Learn\n",
"\n",
@@ -676,8 +799,10 @@
},
{
"cell_type": "markdown",
- "id": "604b3117",
- "metadata": {},
+ "id": "30981df5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
@@ -686,16 +811,20 @@
},
{
"cell_type": "markdown",
- "id": "da04983b",
- "metadata": {},
+ "id": "ecf8c726",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In our codes where we compare our own codes with **Scikit-Learn**, we do thus not include the $1/n$ factor in the cost function."
]
},
{
"cell_type": "markdown",
- "id": "2bebf877",
- "metadata": {},
+ "id": "9f2bcfc9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Comparison with OLS\n",
"When we compare this with the ordinary least squares result we have"
@@ -703,8 +832,10 @@
},
{
"cell_type": "markdown",
- "id": "6152d137",
- "metadata": {},
+ "id": "4c486a65",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
@@ -713,8 +844,10 @@
},
{
"cell_type": "markdown",
- "id": "eccd4760",
- "metadata": {},
+ "id": "c7def0db",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n",
"\n",
@@ -727,8 +860,10 @@
},
{
"cell_type": "markdown",
- "id": "73aef7e2",
- "metadata": {},
+ "id": "f4c99b49",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## SVD analysis\n",
"\n",
@@ -738,8 +873,10 @@
},
{
"cell_type": "markdown",
- "id": "37b86951",
- "metadata": {},
+ "id": "cf237b5c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
@@ -748,16 +885,20 @@
},
{
"cell_type": "markdown",
- "id": "9c638256",
- "metadata": {},
+ "id": "78759bd0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"For Ridge regression this becomes"
]
},
{
"cell_type": "markdown",
- "id": "a67ccade",
- "metadata": {},
+ "id": "c54f19d1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
@@ -766,16 +907,20 @@
},
{
"cell_type": "markdown",
- "id": "d60a0acb",
- "metadata": {},
+ "id": "0c7e35dd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$."
]
},
{
"cell_type": "markdown",
- "id": "7d16dcf3",
- "metadata": {},
+ "id": "196ef495",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Interpreting the Ridge results\n",
"\n",
@@ -784,8 +929,10 @@
},
{
"cell_type": "markdown",
- "id": "3b162883",
- "metadata": {},
+ "id": "e4054c70",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n",
@@ -794,8 +941,10 @@
},
{
"cell_type": "markdown",
- "id": "fa81d3cf",
- "metadata": {},
+ "id": "5082ae92",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n",
"orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n",
@@ -808,8 +957,10 @@
},
{
"cell_type": "markdown",
- "id": "c35848c1",
- "metadata": {},
+ "id": "a39c69f3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More interpretations\n",
"\n",
@@ -818,8 +969,10 @@
},
{
"cell_type": "markdown",
- "id": "f75f3dea",
- "metadata": {},
+ "id": "74b9d096",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n",
@@ -828,16 +981,20 @@
},
{
"cell_type": "markdown",
- "id": "2b407fea",
- "metadata": {},
+ "id": "a364c430",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In this case the standard OLS results in"
]
},
{
"cell_type": "markdown",
- "id": "5fad7e1b",
- "metadata": {},
+ "id": "ff92d951",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n",
@@ -846,16 +1003,20 @@
},
{
"cell_type": "markdown",
- "id": "de011fe9",
- "metadata": {},
+ "id": "90a08bf8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "ad61d5a1",
- "metadata": {},
+ "id": "e45ab4f7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n",
@@ -864,8 +1025,10 @@
},
{
"cell_type": "markdown",
- "id": "eff9b775",
- "metadata": {},
+ "id": "7cebde68",
+ "metadata": {
+ "editable": true
+ },
"source": [
"that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n",
"the Ridge estimator converges to zero when the hyperparameter goes to\n",
@@ -879,8 +1042,10 @@
},
{
"cell_type": "markdown",
- "id": "6c4f7a24",
- "metadata": {},
+ "id": "77911e86",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Deriving the Lasso Regression Equations\n",
"\n",
@@ -889,8 +1054,10 @@
},
{
"cell_type": "markdown",
- "id": "f4deb050",
- "metadata": {},
+ "id": "1d35c24b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
@@ -899,16 +1066,20 @@
},
{
"cell_type": "markdown",
- "id": "9d6cfa3e",
- "metadata": {},
+ "id": "fb96fe4b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicity)"
]
},
{
"cell_type": "markdown",
- "id": "c2303440",
- "metadata": {},
+ "id": "3aa95bb4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{d \\vert \\beta\\vert}{d \\beta}=\\mathrm{sgn}(\\beta)=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\-1 & \\beta < 0, \\end{array}\\right.\n",
@@ -917,16 +1088,20 @@
},
{
"cell_type": "markdown",
- "id": "7bf0b1fc",
- "metadata": {},
+ "id": "1e09dbbb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"we have that the derivative of the cost function is"
]
},
{
"cell_type": "markdown",
- "id": "25fe5980",
- "metadata": {},
+ "id": "df29cffc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-\\frac{2}{n}\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n",
@@ -935,16 +1110,20 @@
},
{
"cell_type": "markdown",
- "id": "38a46583",
- "metadata": {},
+ "id": "6362d699",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and reordering we have"
]
},
{
"cell_type": "markdown",
- "id": "45666dad",
- "metadata": {},
+ "id": "19cd5978",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=\\boldsymbol{X}^T\\boldsymbol{y}.\n",
@@ -953,16 +1132,20 @@
},
{
"cell_type": "markdown",
- "id": "d72cd292",
- "metadata": {},
+ "id": "37581fab",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. We have absorbed the factor $2/n$ in a redefinition of the parameter $\\lambda$. We will solve this type of problems using libraries like **scikit-learn**."
]
},
{
"cell_type": "markdown",
- "id": "1cdd45ed",
- "metadata": {},
+ "id": "ae4d698d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression\n",
"\n",
@@ -974,8 +1157,10 @@
},
{
"cell_type": "markdown",
- "id": "bf327b4f",
- "metadata": {},
+ "id": "40241013",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n",
@@ -984,16 +1169,20 @@
},
{
"cell_type": "markdown",
- "id": "235e1172",
- "metadata": {},
+ "id": "0c960aa9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and minimizing we have that"
]
},
{
"cell_type": "markdown",
- "id": "c0df9f8a",
- "metadata": {},
+ "id": "f14d2d19",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n",
@@ -1002,8 +1191,10 @@
},
{
"cell_type": "markdown",
- "id": "88642f9a",
- "metadata": {},
+ "id": "18d8a1fb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Ridge Regression\n",
"\n",
@@ -1012,8 +1203,10 @@
},
{
"cell_type": "markdown",
- "id": "c48f87e2",
- "metadata": {},
+ "id": "a9f7ec7d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n",
@@ -1022,16 +1215,20 @@
},
{
"cell_type": "markdown",
- "id": "ae22178d",
- "metadata": {},
+ "id": "fe0598cd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and minimizing we have that"
]
},
{
"cell_type": "markdown",
- "id": "3c8c1324",
- "metadata": {},
+ "id": "4be76e31",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n",
@@ -1040,8 +1237,10 @@
},
{
"cell_type": "markdown",
- "id": "f9f58a15",
- "metadata": {},
+ "id": "7fde9c2a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Lasso Regression\n",
"\n",
@@ -1050,8 +1249,10 @@
},
{
"cell_type": "markdown",
- "id": "865aa159",
- "metadata": {},
+ "id": "e17bf745",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n",
@@ -1060,16 +1261,20 @@
},
{
"cell_type": "markdown",
- "id": "9e6565ec",
- "metadata": {},
+ "id": "4f4a82c9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and minimizing we have that"
]
},
{
"cell_type": "markdown",
- "id": "8b2950b4",
- "metadata": {},
+ "id": "feb65231",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n",
@@ -1078,16 +1283,20 @@
},
{
"cell_type": "markdown",
- "id": "23a7c1a6",
- "metadata": {},
+ "id": "5719acaf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which leads to"
]
},
{
"cell_type": "markdown",
- "id": "4a733817",
- "metadata": {},
+ "id": "66e5bce0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
@@ -1098,16 +1307,20 @@
},
{
"cell_type": "markdown",
- "id": "dc517d14",
- "metadata": {},
+ "id": "a3342dd5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Plotting these results shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$."
]
},
{
"cell_type": "markdown",
- "id": "ed79ab9a",
- "metadata": {},
+ "id": "a14a8b71",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Yet another Example\n",
"\n",
@@ -1116,8 +1329,10 @@
},
{
"cell_type": "markdown",
- "id": "116b5106",
- "metadata": {},
+ "id": "b51043ea",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{y}=\\begin{bmatrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n",
@@ -1126,16 +1341,20 @@
},
{
"cell_type": "markdown",
- "id": "aaa68ab6",
- "metadata": {},
+ "id": "c67d0c89",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and our inputs as a $3\\times 2$ design matrix"
]
},
{
"cell_type": "markdown",
- "id": "17fa6c43",
- "metadata": {},
+ "id": "00d3fa3c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n",
@@ -1144,16 +1363,20 @@
},
{
"cell_type": "markdown",
- "id": "01264686",
- "metadata": {},
+ "id": "e2e2a844",
+ "metadata": {
+ "editable": true
+ },
"source": [
"meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression."
]
},
{
"cell_type": "markdown",
- "id": "1b53fa9e",
- "metadata": {},
+ "id": "ecae41e2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The OLS case\n",
"\n",
@@ -1162,8 +1385,10 @@
},
{
"cell_type": "markdown",
- "id": "0a7cdc65",
- "metadata": {},
+ "id": "7562b941",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
@@ -1172,16 +1397,20 @@
},
{
"cell_type": "markdown",
- "id": "6fac325b",
- "metadata": {},
+ "id": "4a7abbed",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Inserting the above values we obtain that"
]
},
{
"cell_type": "markdown",
- "id": "2befb748",
- "metadata": {},
+ "id": "2e82ec97",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n",
@@ -1190,16 +1419,20 @@
},
{
"cell_type": "markdown",
- "id": "c67fea47",
- "metadata": {},
+ "id": "35f8c263",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The code which implements this simpler case is presented after the discussion of Ridge and Lasso."
]
},
{
"cell_type": "markdown",
- "id": "65e47e22",
- "metadata": {},
+ "id": "31a412f2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Ridge case\n",
"\n",
@@ -1208,8 +1441,10 @@
},
{
"cell_type": "markdown",
- "id": "221a5457",
- "metadata": {},
+ "id": "934eafd3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
@@ -1218,16 +1453,20 @@
},
{
"cell_type": "markdown",
- "id": "5517307a",
- "metadata": {},
+ "id": "fe950f5d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Inserting the above values we obtain that"
]
},
{
"cell_type": "markdown",
- "id": "b20940ce",
- "metadata": {},
+ "id": "fb1d10a3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
@@ -1236,8 +1475,10 @@
},
{
"cell_type": "markdown",
- "id": "4110bef8",
- "metadata": {},
+ "id": "5898fc4a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n",
"Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n",
@@ -1247,8 +1488,10 @@
},
{
"cell_type": "markdown",
- "id": "0dad3074",
- "metadata": {},
+ "id": "4c8e959c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Writing the Cost Function\n",
"\n",
@@ -1257,8 +1500,10 @@
},
{
"cell_type": "markdown",
- "id": "350a0faa",
- "metadata": {},
+ "id": "be640c1f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n",
@@ -1267,8 +1512,10 @@
},
{
"cell_type": "markdown",
- "id": "24cd86b7",
- "metadata": {},
+ "id": "013b3b49",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n",
@@ -1277,16 +1524,20 @@
},
{
"cell_type": "markdown",
- "id": "17c643f0",
- "metadata": {},
+ "id": "b484d29b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and taking the derivative with respect to $\\beta_0$ we get"
]
},
{
"cell_type": "markdown",
- "id": "4cd2f888",
- "metadata": {},
+ "id": "8cbfa1d8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_0=\\frac{8}{4+\\lambda},\n",
@@ -1295,16 +1546,20 @@
},
{
"cell_type": "markdown",
- "id": "13ce9a3c",
- "metadata": {},
+ "id": "db22265b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and for $\\beta_1$ we obtain"
]
},
{
"cell_type": "markdown",
- "id": "9c52a280",
- "metadata": {},
+ "id": "c9cd2884",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_1=\\frac{2}{1+\\lambda},\n",
@@ -1313,16 +1568,20 @@
},
{
"cell_type": "markdown",
- "id": "4aa82b09",
- "metadata": {},
+ "id": "a104bdb3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving"
]
},
{
"cell_type": "markdown",
- "id": "7369b16e",
- "metadata": {},
+ "id": "23ba52f8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n",
@@ -1331,16 +1590,20 @@
},
{
"cell_type": "markdown",
- "id": "1fc52972",
- "metadata": {},
+ "id": "fca5ec07",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$."
]
},
{
"cell_type": "markdown",
- "id": "40baac3a",
- "metadata": {},
+ "id": "3dd9c2aa",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Lasso case\n",
"\n",
@@ -1350,8 +1613,10 @@
},
{
"cell_type": "markdown",
- "id": "c038e0d3",
- "metadata": {},
+ "id": "37a7d2fc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n",
@@ -1360,8 +1625,10 @@
},
{
"cell_type": "markdown",
- "id": "4ce575fa",
- "metadata": {},
+ "id": "c5e17b22",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n",
@@ -1370,16 +1637,20 @@
},
{
"cell_type": "markdown",
- "id": "1e5a661f",
- "metadata": {},
+ "id": "d6921e92",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "0a170322",
- "metadata": {},
+ "id": "0b50b699",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n",
@@ -1388,8 +1659,10 @@
},
{
"cell_type": "markdown",
- "id": "2443ec27",
- "metadata": {},
+ "id": "4f564b2e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n",
"1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n",
@@ -1403,8 +1676,10 @@
},
{
"cell_type": "markdown",
- "id": "64723ae2",
- "metadata": {},
+ "id": "807c604c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The first Case\n",
"\n",
@@ -1413,8 +1688,10 @@
},
{
"cell_type": "markdown",
- "id": "d5723b13",
- "metadata": {},
+ "id": "cfcca8c4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"-4(4-2\\beta_0)+\\lambda=0,\n",
@@ -1423,16 +1700,20 @@
},
{
"cell_type": "markdown",
- "id": "b64de210",
- "metadata": {},
+ "id": "77cf6660",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "207377d9",
- "metadata": {},
+ "id": "6bdad19e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"-2(2-\\beta_1)+\\lambda=0.\n",
@@ -1441,16 +1722,20 @@
},
{
"cell_type": "markdown",
- "id": "38658c17",
- "metadata": {},
+ "id": "c0014adf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which yields"
]
},
{
"cell_type": "markdown",
- "id": "038520f6",
- "metadata": {},
+ "id": "50019bc0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_0=\\frac{16+\\lambda}{8},\n",
@@ -1459,16 +1744,20 @@
},
{
"cell_type": "markdown",
- "id": "342f1a76",
- "metadata": {},
+ "id": "14ec4f6e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "6addac68",
- "metadata": {},
+ "id": "416aeff2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_1=\\frac{4+\\lambda}{2}.\n",
@@ -1477,16 +1766,20 @@
},
{
"cell_type": "markdown",
- "id": "78cc3bce",
- "metadata": {},
+ "id": "8da8b221",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you."
]
},
{
"cell_type": "markdown",
- "id": "420ca9dd",
- "metadata": {},
+ "id": "e7fe0c53",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple code for solving the above problem\n",
"\n",
@@ -1498,229 +1791,12 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "585576f1",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[2. 2.]\n",
- "Training MSE for OLS\n",
- "3.0\n",
- "[1.99995 1.99980002]\n",
- "3.0000000166638334\n",
- "[1.99993978 1.99975913]\n",
- "3.000000024175529\n",
- "[1.99992746 1.99970988]\n",
- "3.0000000350730853\n",
- "[1.99991263 1.99965056]\n",
- "3.000000050882475\n",
- "[1.99989476 1.99957911]\n",
- "3.0000000738172754\n",
- "[1.99987324 1.99949306]\n",
- "3.000000107088393\n",
- "[1.99984732 1.99938942]\n",
- "3.000000155353232\n",
- "[1.9998161 1.99926459]\n",
- "3.000000225367021\n",
- "[1.99977849 1.99911427]\n",
- "3.0000003269271693\n",
- "[1.9997332 1.99893323]\n",
- "3.0000004742423325\n",
- "[1.99967865 1.99871521]\n",
- "3.0000006879168684\n",
- "[1.99961294 1.99845267]\n",
- "3.0000009978267586\n",
- "[1.99953381 1.99813653]\n",
- "3.0000014472865364\n",
- "[1.9994385 1.99775587]\n",
- "3.0000020990850875\n",
- "[1.9993237 1.99729756]\n",
- "3.0000030442256755\n",
- "[1.99918546 1.9967458 ]\n",
- "3.000004414576747\n",
- "[1.99901896 1.99608161]\n",
- "3.0000064011746996\n",
- "[1.99881845 1.99528218]\n",
- "3.0000092806885594\n",
- "[1.998577 1.9943201]\n",
- "3.00001345365979\n",
- "[1.99828624 1.99316252]\n",
- "3.000019499710462\n",
- "[1.99793613 1.99176998]\n",
- "3.0000282571692165\n",
- "[1.99751458 1.99009525]\n",
- "3.0000409377758195\n",
- "[1.99700706 1.98808176]\n",
- "3.0000592916770077\n",
- "[1.9963961 1.98566191]\n",
- "3.0000858443464193\n",
- "[1.99566069 1.98275501]\n",
- "3.0001242360606217\n",
- "[1.9947756 1.97926491]\n",
- "3.0001797070821445\n",
- "[1.99371056 1.97507735]\n",
- "3.000259788951496\n",
- "[1.99242921 1.97005689]\n",
- "3.0003752857813737\n",
- "[1.99088801 1.9640435 ]\n",
- "3.0005416609565345\n",
- "[1.9890348 1.95684892]\n",
- "3.0007809860966828\n",
- "[1.98680716 1.9482527 ]\n",
- "3.001124662299061\n",
- "[1.98413059 1.93799826]\n",
- "3.001617189522961\n",
- "[1.98091621 1.92578916]\n",
- "3.0023213376102738\n",
- "[1.97705827 1.91128596]\n",
- "3.0033251572929984\n",
- "[1.97243128 1.89410423]\n",
- "3.0047513507691748\n",
- "[1.96688672 1.87381451]\n",
- "3.0067695784394064\n",
- "[1.96024953 1.84994524]\n",
- "3.00961227707716\n",
- "[1.95231424 1.82198978]\n",
- "3.013594454782845\n",
- "[1.94284104 1.78941903]\n",
- "3.0191376445333487\n",
- "[1.93155188 1.75170092]\n",
- "3.0267976715211304\n",
- "[1.91812702 1.70832814]\n",
- "3.037295070894927\n",
- "[1.90220243 1.65885453]\n",
- "3.0515458973819936\n",
- "[1.88336879 1.60293962]\n",
- "3.0706894339417037\n",
- "[1.86117291 1.54039921]\n",
- "3.096108240564299\n",
- "[1.83512277 1.47125748]\n",
- "3.129435553875627\n",
- "[1.80469739 1.39579407]\n",
- "3.1725457461446864\n",
- "[1.76936315 1.31457796]\n",
- "3.2275256008513113\n",
- "[1.72859758 1.22847924]\n",
- "3.2966271243130905\n",
- "[1.68192193 1.13865173]\n",
- "3.3822051574630474\n",
- "[1.62894215 1.04648335]\n",
- "3.486643242957762\n",
- "[1.56939714 0.95351665]\n",
- "3.6122675597483926\n",
- "[1.50321091 0.86134827]\n",
- "3.761241783673139\n",
- "[1.43054282 0.77152076]\n",
- "3.9354290505857747\n",
- "[1.35182854 0.68542204]\n",
- "4.136206724367547\n",
- "[1.26780278 0.60420593]\n",
- "4.364230711407981\n",
- "[1.17949575 0.52874252]\n",
- "4.619169150472792\n",
- "[1.0881981 0.45960079]\n",
- "4.899453531129684\n",
- "[0.99539415 0.39706038]\n",
- "5.202115678338607\n",
- "[0.90266948 0.34114547]\n",
- "5.5227784587157664\n",
- "[0.81160425 0.29167186]\n",
- "5.855840955026829\n",
- "[0.7236674 0.24829908]\n",
- "6.194851926696971\n",
- "[0.64012627 0.21058097]\n",
- "6.533015585626998\n",
- "[0.56198284 0.17801022]\n",
- "6.863740078371417\n",
- "[0.48994188 0.15005476]\n",
- "7.181133165628348\n",
- "[0.42441033 0.12618549]\n",
- "7.480370686014802\n",
- "[0.3655222 0.10589577]\n",
- "7.757900495568644\n",
- "[0.31318084 0.08871404]\n",
- "8.011483187026515\n",
- "[0.26710969 0.07421084]\n",
- "8.240099719092596\n",
- "[0.22690428 0.06200174]\n",
- "8.44377034840297\n",
- "[0.19207979 0.0517473 ]\n",
- "8.62333016722461\n",
- "[0.16211139 0.04315108]\n",
- "8.780198640335543\n",
- "[0.13646574 0.0359565 ]\n",
- "8.91616887792369\n",
- "[0.11462415 0.02994311]\n",
- "9.033230859368514\n",
- "[0.09609807 0.02492265]\n",
- "9.133433591261653\n",
- "[0.08043851 0.02073509]\n",
- "9.218784924511205\n",
- "[0.06724062 0.01724499]\n",
- "9.29118422676508\n",
- "[0.05614483 0.01433809]\n",
- "9.352381653444837\n",
- "[0.04683565 0.01191824]\n",
- "9.403957666258659\n",
- "[0.039039 0.00990475]\n",
- "9.447317109798865\n",
- "[0.03251863 0.00823002]\n",
- "9.483693143475648\n",
- "[0.02707227 0.00683748]\n",
- "9.514157367059978\n",
- "[0.02252765 0.0056799 ]\n",
- "9.539633426806517\n",
- "[0.01873869 0.00471782]\n",
- "9.560912182395493\n",
- "[0.01558197 0.00391839]\n",
- "9.578667139901263\n",
- "[0.01295356 0.0032542 ]\n",
- "9.593469326199116\n",
- "[0.01076611 0.00270244]\n",
- "9.605801120347516\n",
- "[0.00894639 0.00224413]\n",
- "9.61606879456562\n",
- "[0.0074331 0.00186347]\n",
- "9.624613676371458\n",
- "[0.00617499 0.00154733]\n",
- "9.631721945196448\n",
- "[0.00512927 0.00128479]\n",
- "9.637633137959385\n",
- "[0.00426027 0.00106677]\n",
- "9.642547471308156\n",
- "[0.00353823 0.00088573]\n",
- "9.64663210283169\n",
- "[0.00293838 0.00073541]\n",
- "9.650026456182994\n",
- "[0.0024401 0.00061058]\n",
- "9.652846730402574\n",
- "[0.00202624 0.00050694]\n",
- "9.655189704973857\n",
- "[0.00168251 0.00042089]\n",
- "9.657135941391186\n",
- "[0.00139705 0.00034944]\n",
- "9.65875247063582\n",
- "[0.00115999 0.00029012]\n",
- "9.660095044772136\n",
- "[0.00096314 0.00024087]\n",
- "9.661210020378476\n",
- "[0.00079968 0.00019998]\n",
- "9.66213593196335\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGwCAYAAACHJU4LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABB20lEQVR4nO3deZzN9eLH8feZMftmGWsGQ91EWUO2K9m3pFJXspYSiaSkbpKrO3FLXXWLUihl0qLSL2shRbYQspTIFCLLzNhmzMz398e3mXHMYmacmc9ZXs/H4/s4y3zOOe8zxpz3fJfP12FZliUAAAA35Gc6AAAAQF4oKgAAwG1RVAAAgNuiqAAAALdFUQEAAG6LogIAANwWRQUAALitUqYDXI6MjAwdPHhQERERcjgcpuMAAIACsCxLycnJqlKlivz88l9n4tFF5eDBg4qJiTEdAwAAFEFCQoKqVq2a7xiPLioRERGS7DcaGRlpOA0AACiIpKQkxcTEZH2O58eji0rm5p7IyEiKCgAAHqYgu22wMy0AAHBbFBUAAOC2KCoAAMBtefQ+KgWVnp6u8+fPm44BHxMQECB/f3/TMQDAo3l1UbEsS4cPH9bJkydNR4GPKl26tCpVqsQ8PwBQRF5dVDJLSoUKFRQaGsqHBUqMZVk6c+aMjhw5IkmqXLmy4UQA4Jm8tqikp6dnlZRy5cqZjgMfFBISIkk6cuSIKlSowGYgACgCr92ZNnOflNDQUMNJ4Msyf/7YRwoAisZri0omNvfAJH7+AODyeH1RAQAAnouiAgAA3BZFBZdl//79cjgc2rJlS55jVq5cKYfD4ZGHiQ8cOFC33HKL6RgA4LMoKm5o4MCBcjgcGjp0aI6vDRs2TA6HQwMHDsy678iRI7r//vtVrVo1BQUFqVKlSurUqZPWrl2bNaZGjRpyOBw5lueeey7PHDfeeGPWuMDAQNWqVUvjxo1TSkpK1piYmBgdOnRI1157rWvefBHl9f4ylxtvvLFIz/vf//5Xs2fPdmlWADDOsqTz56Vz56TTp6XEROn4cenoUSk11XmcYV57eLKni4mJUXx8vF588cWsw1zPnTunefPmqVq1ak5jb7vtNp0/f15z5sxRzZo19ccff+jLL7/U8ePHncZNnDhRQ4YMcbrvUqfYHjJkiCZOnKjU1FRt2LBBgwYNkiTFxcVJkvz9/VWpUqXLeq+usGHDBqWnp0uS1qxZo9tuu027d+/OOqt2YGCg0/jz588rICDgks8bFRXl+rAAfNv581JSkr2cOmUXhVOn7KVTJ+mv3/lavlxavdouExcvKSnSK69IVavaY2fMkKZPt0vG+fM5ly+/lBo2tMdOmSKNHZt3vhUrpMw/7tLSpAL8rixOvllUTp/O+2v+/lJwcMHG+vll/0DlNTYsrPD5JDVq1Ei//PKLPv74Y/Xt21eS9PHHHysmJkY1a9bMGnfy5El98803Wrlypdq0aSNJql69upo2bZrjOSMiIgpdKkJDQ7MeU61aNb333ntaunRpVlHZv3+/YmNjtXnzZjVo0ECS9MUXX2jUqFFKSEjQDTfcoAEDBuR43jfeeEMTJ07UsWPH1KlTJ7Vu3VoTJ0502jy0cOFCTZgwQTt27FCVKlU0YMAAPfnkkypVKuePbfny5bOuly1bVpJUoUIFlS5dWpJ99M1rr72mRYsWafny5RozZozGjx+v++67T1999ZUOHz6satWqadiwYRo5cmTWcw0cOFAnT57UJ598Isley1SvXj0FBwdr5syZCgwM1NChQzVhwoRCfV8BeJG9e+3l6FHpzz+zlxMn7OWDD6S//mjSiBF2wcjLL79IsbH29eXLpcmT8x47cWJ2UTlyRMpnE7zOncu+fqk5nf76o0+S5AZHLvpmUQkPz/trXbtK//d/2bcrVJDOnMl9bJs20sqV2bdr1LB/OC90GavNBg0apFmzZmUVlbfeekuDBw/WygteMzw8XOHh4frkk090ww03KCgoqMivdylbt27Vt99+qxo1auQ5JiEhQbfeequGDh2qBx54QBs3btQjjzziNObbb7/V0KFDNXnyZN18881avny5nnrqKacxS5Ys0d13361p06apdevW2rt3r+677z5J0tNPP12k/E8//bTi4uL04osvyt/fXxkZGapatarmz5+v6OhorVmzRvfdd58qV66sO+64I8/nmTNnjkaPHq1169Zp7dq1GjhwoFq2bKkOHToUKRcAN5SRIR06ZBeQX3+VEhKyl99+k775JvuzJC5OevPNvJ/r2LHsonLh3F4hIVJEhP0HbXh4zj9smzeXHnjAHhccnL0EBdlLlSrZY/v0kZo2lQID7TUgFy6lStmfT5mGDpUGDLALS26L3wV7heTyh2GJszxYYmKiJclKTEzM8bWzZ89aP/74o3X27NmcD7TrQ+5L167OY0ND8x7bpo3z2OjonGOKYMCAAVbPnj2to0ePWkFBQda+ffus/fv3W8HBwdbRo0etnj17WgMGDMga/+GHH1plypSxgoODrRYtWljjxo2ztm7d6vSc1atXtwIDA62wsDCnZcWKFXnmaNOmjRUQEGCFhYVZgYGBliTLz8/P+vDDD7PG7Nu3z5Jkbd682bIsyxo3bpx1zTXXWBkZGVljxo4da0myTpw4YVmWZd15551Wt27dnF6rb9++VlRUVNbt1q1bW//+97+dxrzzzjtW5cqVL/n9W7FihdPrWZZlSbJGjRp1yccOGzbMuu2227JuZ/5bZGrTpo3VqlUrp8c0adLEGjt2bK7Pl+/PIQDzEhMt67vvLOuttyzr/Pns+wcPzv+z4uefs8dOmWJZ9epZVrt2lnXnnZY1fLhljR9vWS+9ZFmzZ1vWyZPOr3f8uPNr+aD8Pr8v5gZVyYBTp/L+2sWrxP46V0uu/C7aF3n//iJHyk10dLS6deumOXPmyLIsdevWTdHR0TnG3XbbberWrZtWr16ttWvXavHixZoyZYpmzpzptNPto48+6nRbkq644op8M/Tt21dPPvmkkpKSNHnyZEVGRuq2227Lc/zOnTt1ww03OE101rx5c6cxu3fvVq9evZzua9q0qT7//POs25s2bdKGDRv07LPPZt2Xnp6uc+fO6cyZM0Wacfj666/Pcd/06dM1c+ZM/frrrzp79qxSU1OzNmHlpV69ek63K1eunHVOHwBu7MgRaf16adMm6fvvpc2b7TUkmVq1kq66yr5evbr9eVC9ur02olo1KSbGXqpWlSpWzH7co4/aS0FkrllBgflmUSnMfiPFNbaABg8erAcffFCS9L///S/PccHBwerQoYM6dOig8ePH695779XTTz/tVEyio6N15ZVXFur1o6Kish4zd+5c1a1bV2+++abuueeeXMdbBdjUZVlWjhlbL35cRkaGnnnmGd166605Hh984T5EhRB20b/P/Pnz9fDDD+uFF15Q8+bNFRERof/85z9at25dvs9z8U64DodDGRkZRcoEoJhkZEjbt0u1amX/bp42Tbrgj58sVapIdeo478cxZoz0xBPusenDx/Ev4OY6d+6s1L8OFevUqVOBH1enTp2sHUBdJSAgQE888YTGjRunPn365LpWI7fX/e6775xu165dW+vXr3e6b+PGjU63GzVqpN27dxe6WBXG6tWr1aJFCw0bNizrvr179xbb6wEoZr//Li1ZIi1ebO+IeuKEtGiR1Lmz/fWmTaW6daVGjaTGje3L666T/trp3gnniXMbFBU35+/vr507d2Zdv9ixY8fUu3dvDR48WPXq1VNERIQ2btyoKVOmqGfPnk5jk5OTdfjwYaf7QkNDsw7hLYi77rpLTzzxhF599VWNGTMmx9eHDh2qF154QaNHj9b999+vTZs25ZiHZMSIEfr73/+uqVOnqkePHvrqq6+0aNEip7Us48ePV/fu3RUTE6PevXvLz89PP/zwg7Zt26ZJkyYVOG9+rrzySr399ttasmSJYmNj9c4772jDhg2KzdzjHoD7S0iQXntN+vxzads256+FhUkHD2bfvvlme4FHYcI3DxAZGZlnmQgPD1ezZs304osv6u9//7uuvfZaPfXUUxoyZIheuegQuPHjx6ty5cpOy2OPPVaoLIGBgXrwwQc1ZcoUncplX59q1arpo48+0sKFC1W/fn1Nnz5d//73v53GtGzZUtOnT9fUqVNVv359LV68WA8//LDTJp1OnTrp888/17Jly9SkSRPdcMMNmjp1qqpXr16ovPkZOnSobr31Vt15551q1qyZjh075rR2BYCbOns2+/qxY/ZRN9u22YfSNmsmTZggrV0rnTwpDR5sKiVcxGEVZKcCN5WUlKSoqCglJibm+CA/d+6c9u3bp9jY2CLv04CSM2TIEO3atUurV682HcWl+DkEXOTUKSk+3p7UrE4d6e237fstS3rwQallS3uytHLlzOZEgeT3+X0xNv3AiOeff14dOnRQWFiYFi1apDlz5ujVV181HQuAu9m2zd60M3eulJxs33fggD0pmb+/vRYlnwMN4PkoKjBi/fr1mjJlipKTk1WzZk1NmzZN9957r+lYANzFunXSv/7lPAHnVVdJ990nDRx46dlV4TUoKjBi/vz5piMAcGcLF9olxc9PuvVWe4bWtm3dYkp3lCyvLyoevAsOvAA/f0ABbd1qryXJPBP7qFH2uXPGjMmehA0+yWuP+smclOtMXufpAUpA5s9fQc7UDPikI0fszTkNG9rnoMks99HR9hmBKSk+z2vXqPj7+6t06dJZU5uHhobmmA0VKC6WZenMmTM6cuSISpcunescOIBPS0uTXnrJPgNw5k6yV1xhn4U+vxPHwud4bVGRpEqVKkkS52GBMaVLl876OQTwl5077bP3bthg327c2C4trVoZjQX35NVFxeFwqHLlyqpQoYLOnz9vOg58TEBAAGtSgIutWye1aSOlpEhRUdLUqfZRPBef5BX4i1cXlUz+/v58YACAO2jc2D6/TnS0NHOmvbkHyIdPFBUAgEFr10pNmthnIi5Vyj5xYJkyHGqMAmFdGwCgeFiW9Nxz9vT2jz+efX/ZspQUFBhrVAAArnf6tH1CwMzJHRMTpYwM9kVBoVFUAACu9dtvUrdu0g8/SAEB0rRp0v33sxYFRUJRAQC4zs8/S+3bS7/+KlWsKH30kb3pBygiigoAwDVSUqQOHeySctVV0rJlUvXqplPBw7GxEADgGkFB9sRtjRpJq1dTUuASFBUAwOW58OSbPXtK69fbm30AF6CoAACK7vvv7Unc9u7Nvo8JNuFCFBUAQNHs3St16SJt3iw9+aTpNPBSFBUAQOEdOSJ17mxf1q8vvf666UTwUhQVAEDhJCdLXbvahyLXqCEtWiRFRppOBS9ltKgkJydr1KhRql69ukJCQtSiRQttyDztNwDA/aSmSrffLm3aZJ9YcMkSqXJl06ngxYwWlXvvvVfLli3TO++8o23btqljx45q3769fv/9d5OxAAB5mTBBWrpUCg2VvvhC+tvfTCeCl3NY1oXHlZWcs2fPKiIiQp9++qm6deuWdX+DBg3UvXt3TZo0KcdjUlJSlJKSknU7KSlJMTExSkxMVCSrHQGg+B0+LN11l/Tww1KPHqbTwEMlJSUpKiqqQJ/fxtaopKWlKT09XcHBwU73h4SE6Jtvvsn1MXFxcYqKispaYmJiSiIqACBTpUrSl19SUlBijBWViIgINW/eXP/617908OBBpaena+7cuVq3bp0OHTqU62PGjRunxMTErCUhIaGEUwOAD0pOlj75JPs2JxdECTK6j8o777wjy7J0xRVXKCgoSNOmTdNdd90l/zwmCwoKClJkZKTTAgAoRpYl3XOP1KuXNHGi6TTwQUaLSq1atbRq1SqdOnVKCQkJWr9+vc6fP6/Y2FiTsQAAmaZNkz74QCpVyj7hIFDC3GIelbCwMFWuXFknTpzQkiVL1LNnT9ORAADbt0uPPmpff+EFqXlzs3ngk0qZfPElS5bIsixdffXV+vnnn/Xoo4/q6quv1qBBg0zGAgCkpUkDB0rnz9s7zo4YYToRfJTRNSqJiYkaPny4ateurf79+6tVq1ZaunSpAgICTMYCAPznP/akbqVLSzNmsAMtjDE2j4orFOY4bABAAf3+u1Szpj0L7dtvS/36mU4EL1OYz2+jm34AAG7oiiukDz+UPvtMuvtu02ng4ygqAICcevRgUje4Bbc46gcA4AZ++UXKY8JNwBSKCgDAnthtyBCpTh17inzATbDpBwAgffqp9NVXUlCQVKuW6TRAFtaoAICvS0mRHnnEvj5mjFSjhtE4wIUoKgDg6156yd4/pXJl6fHHTacBnFBUAMCXHTokTZpkX3/uOSk83Gwe4CIUFQDwZU8+KZ06JTVtypwpcEsUFQDwVZYlhYZK/v725h8/PhLgfvipBABf5XBIr7wi7d/PmZHhtigqAODrqlY1nQDIE0UFAHyNZUljx0rbtplOAlwSRQUAfM3nn0tTpkgtW0rJyabTAPmiqACAL7EsacIE+/qwYVJEhNE4wKVQVADAlyxcKH3/vRQWZs9CC7g5igoA+IoL16aMGCFFRxuNAxQERQUAfMWnn0qbN9uzz7I2BR6CogIAvsCypGeesa8/9JBUrpzZPEABlTIdAABQAtLS7CnyT53KPlMy4AFYowIAviAgwC4ou3dLZcuaTgMUGEUFAHwJ5/OBh+EnFgC83fjx0gcf2Jt/AA/DPioA4M327ZOefVbKyJB27JDq1DGdCCgU1qgAgDebNs0uKZ06UVLgkSgqAOCtEhOlmTPt6w8/bDYLUEQUFQDwVm++aR+OXKeO1LGj6TRAkVBUAMAbpaVJ//2vff3hhyWHw2weoIgoKgDgjRYskA4ckMqXl/r2NZ0GKDKO+gEAbxQdLbVoIbVvL4WEmE4DFBlFBQC8Udu20rffMncKPB6bfgDAm5Xi71F4NooKAHiTU6ekqVOlo0dNJwFcgqICAN4kPt4++eBNN5lOArgERQUAvMkbb9iX/fubzQG4CEUFALzF1q3S+vVSQIA0YIDpNIBLUFQAwFtkrk255RapQgWjUQBXoagAgDc4c0aaO9e+PmSI2SyAC1FUAMAbfPihfRLC2FipXTvTaQCXMVpU0tLS9M9//lOxsbEKCQlRzZo1NXHiRGVkZJiMBQCe59df7X1T7rlH8uNvUHgPozMBTZ48WdOnT9ecOXNUt25dbdy4UYMGDVJUVJRGjhxpMhoAeJannpIeeEDy9zedBHApo0Vl7dq16tmzp7p16yZJqlGjhubNm6eNGzeajAUAnik62nQCwOWMrh9s1aqVvvzyS+3Zs0eStHXrVn3zzTfq2rVrruNTUlKUlJTktACAT0tLk37+2XQKoNgYLSpjx45Vnz59VLt2bQUEBKhhw4YaNWqU+vTpk+v4uLg4RUVFZS0xMTElnBgA3MyXX0pXXSX16GE6CVAsjBaV999/X3PnztV7772n77//XnPmzNHzzz+vOXPm5Dp+3LhxSkxMzFoSEhJKODEAuJl337Uvq1UzmwMoJg7LsixTLx4TE6PHH39cw4cPz7pv0qRJmjt3rnbt2nXJxyclJSkqKkqJiYmKjIwszqgA4H5On5YqVbJPRPjtt1KLFqYTAQVSmM9vo2tUzpw5I7+LDqPz9/fn8GQAKIjPPrNLSmys1Ly56TRAsTB61E+PHj307LPPqlq1aqpbt642b96sqVOnavDgwSZjAYBnyNzs07ev5HCYzQIUE6ObfpKTk/XUU09pwYIFOnLkiKpUqaI+ffpo/PjxCgwMvOTj2fQDwGf9+adUubJ91M/OnVLt2qYTAQVWmM9vo0XlclFUAPis116Thg2TGjWSNm0ynQYolMJ8fhvd9AMAKKKBA6Vy5aTgYNNJgGJFUQEATxQSIt1xh+kUQLHjzFUAAMBtUVQAwJNYltSzpzRpknTypOk0QLGjqACAJ9m82Z4/5dlnOSQZPoGiAgCe5IMP7Mtu3aSoKLNZgBJAUQEAT2FZ0kcf2ddvv91sFqCEUFQAwFNs3y799JMUFGSvUQF8AEUFADzFhx/al506SRERZrMAJYSiAgCeInOzz223mc0BlCCKCgB4gnPnpIYNpehoqUcP02mAEsPMtADgCYKDpXfekdLTJX9/02mAEsMaFQDwJJQU+BiKCgC4uyNHpK1b7cOTAR9DUQEAd/f221KDBlL//qaTACWOogIA7i7zsOTmzc3mAAygqACAO/vtN2ndOvu8Pr16mU4DlDiKCgC4s48/ti9btpQqVzabBTCAogIA7oxJ3uDjKCoA4K6OHZO++ca+zmYf+CiKCgC4q0WLpIwMqV49qXp102kAI5iZFgDcVe/eUoUKUmqq6SSAMRQVAHBXQUFSx46mUwBGsekHAAC4LYoKALijyZOlMWOknTtNJwGMYtMPALgby5KmT5f275dat5auucZ0IsAY1qgAgLvZscMuKUFBUvv2ptMARlFUAMDdfPaZfdmunRQWZjYLYBhFBQDczcKF9uXNN5vNAbgBigoAuJMjR+yTEEpS9+5mswBugKICAO7k//7P3pm2USPpiitMpwGMo6gAgDtJSbHPktyjh+kkgFtwWJZlmQ5RVElJSYqKilJiYqIiIyNNxwEA18jIsAtLSIjpJECxKMznN2tUAMDd+PlRUoC/UFQAwF389pu9NgVAFooKALiLG2+UKlaUNm0ynQRwG0yhDwDu4KefpL17pYAA6W9/M50GcBusUQEAd7BokX3ZurUUEWE2C+BGKCoA4A4yi0rnzmZzAG6GogIApp09K61caV/v0sVoFMDdGC0qNWrUkMPhyLEMHz7cZCwAKFmrVknnzklVq0p165pOA7gVozvTbtiwQenp6Vm3t2/frg4dOqh3794GUwFACcvc7NOli+RwmM0CuBmjRaV8+fJOt5977jnVqlVLbdq0yXV8SkqKUlJSsm4nJSUVaz4AKBH9+klhYVKHDqaTAG7HbQ5PTk1N1dy5czV69Gg58viLIi4uTs8880wJJwOAYnb99fYCIAe3OdfP/Pnzddddd+nAgQOqUqVKrmNyW6MSExPDuX4AAPAghTnXj9usUXnzzTfVpUuXPEuKJAUFBSkoKKgEUwFAMXvtNSkmRmrXjvP7ALlwi6Ly66+/avny5fr4449NRwGAknPunPTII/bhyVu2SPXrm04EuB23mEdl1qxZqlChgrp162Y6CgCUnK+/tktKlSpSvXqm0wBuyXhRycjI0KxZszRgwACVKuUWK3gAoGRcOBsthyUDuTJeVJYvX64DBw5o8ODBpqMAQMm6cP4UALkyvgqjY8eOcpMDjwCg5Pz6q7R7t+TvL7VvbzoN4LaMr1EBAJ+0bJl92ayZVLq00SiAO6OoAIAJGzfalx07ms0BuDnjm34AwCe99pp9aHJYmOkkgFujqACACQ6HdNVVplMAbo9NPwAAwG1RVACgpPXsKd1+u/Tjj6aTAG6PogIAJSk5WfriC+mjj6TgYNNpALdHUQGAkrRypZSWJtWqJdWsaToN4PYoKgBQkpYutS85LBkoEIoKAJQkigpQKBQVACgp+/dLe/bY0+a3bWs6DeARKCoAUFIyp82/4QYpKspsFsBDUFQAoKSULi01b87ZkoFCYGZaACgpvXvbC2eMBwqMNSoAUNIcDtMJAI9BUQGAkrBnj3TypOkUgMcpVFGZMmWKzp49m3X766+/VkpKStbt5ORkDRs2zHXpAMBbDBkilStnz0gLoMAcllXwjaX+/v46dOiQKlSoIEmKjIzUli1bVPOv2RX/+OMPValSRenp6cWT9iJJSUmKiopSYmKiIiMjS+Q1AaDQTp2SypaVzp+Xfv7ZnpUW8GGF+fwu1BqViztNIToOAPiu1avtklKjBtPmA4XEPioAUNwy50/p0IEdaYFCoqgAQHFbvty+bN/ebA7AAxV6HpWZM2cqPDxckpSWlqbZs2crOjpakr0zLQDgAocPS9u22WtSbrrJdBrA4xSqqFSrVk1vvPFG1u1KlSrpnXfeyTEGAPCXL7+0Lxs2lP76ow5AwRWqqOzfv7+YYgCAl2rbVpo+XfprTTSAwmEKfQAoTlWqSPffbzoF4LEKtTPtunXrtGjRIqf73n77bcXGxqpChQq67777nCaAAwAAuByFKioTJkzQDz/8kHV727Ztuueee9S+fXs9/vjjWrhwoeLi4lweEgA80hdfSP/7n8Rmc6DIClVUtmzZonbt2mXdjo+PV7NmzfTGG29o9OjRmjZtmubPn+/ykADgkWbMkB58UIqPN50E8FiFKionTpxQxYoVs26vWrVKnTt3zrrdpEkTJSQkuC4dAHiq8+elFSvs6x06mM0CeLBCFZWKFStq3759kqTU1FR9//33at68edbXk5OTFRAQ4NqEAOCJNmyQkpPtc/w0aGA6DeCxClVUOnfurMcff1yrV6/WuHHjFBoaqtatW2d9/YcfflAtTrYFANmz0d50k+TvbzYL4MEKdXjypEmTdOutt6pNmzYKDw/X7NmzFRgYmPX1t956Sx07dnR5SADwOBee3wdAkTmsIpwCOTExUeHh4fK/6K+E48ePKyIiosQ2/xTmNNEAUGIyN/mkpUl793LGZOAihfn8LtQalcGDBxdo3FtvvVWYpwUA75I5jUNsLCUFuEyFKiqzZ89W9erV1bBhQxVhRQwA+IaWLaXjx5k/BXCBQhWVoUOHKj4+Xr/88osGDx6su+++W2XLli2ubADguSIipOuuM50C8HiFOurn1Vdf1aFDhzR27FgtXLhQMTExuuOOO7RkyRLWsAAAAJcrVFGRpKCgIPXp00fLli3Tjz/+qLp162rYsGGqXr26Tp06VRwZAcBzzJsnNW1qnzEZwGUrdFG5kMPhkMPhkGVZysjIcFUmAPBcS5bYk72xfwrgEoUuKikpKZo3b546dOigq6++Wtu2bdMrr7yiAwcOKDw8vNABfv/9d919990qV66cQkND1aBBA23atKnQzwMAxllW9kRv7dubzQJ4iULtTDts2DDFx8erWrVqGjRokOLj41WuXLkiv/iJEyfUsmVLtW3bVosWLVKFChW0d+9elS5dusjPCQDG7N4t/f67FBRkH/kD4LIVqqhMnz5d1apVU2xsrFatWqVVq1blOu7jjz8u0PNNnjxZMTExmjVrVtZ9NWrUKEwkAHAfmWtTWrWSQkLMZgG8RKGKSv/+/eVwOFz24p999pk6deqk3r17a9WqVbriiis0bNgwDRkyJNfxKSkpSklJybqdlJTksiwAcNkyiwrT5gMuU6Qp9F0lODhYkjR69Gj17t1b69ev16hRozRjxgz1798/x/gJEybomWeeyXE/U+gDMC4tTSpXTkpKkjZulBo3Np0IcFuFmULfaFEJDAzU9ddfrzVr1mTd99BDD2nDhg1au3ZtjvG5rVGJiYmhqAAw788/pQcflLZskXbs4IzJQD6K7Vw/rla5cmXVqVPH6b5rrrlGH330Ua7jg4KCFBQUVBLRAKBwoqOl+Hj7yB8XbiIHfN1lzaNyuVq2bKndu3c73bdnzx5Vr17dUCIAuEyUFMCljBaVhx9+WN99953+/e9/6+eff9Z7772n119/XcOHDzcZCwAK5+xZ6ccf7bUpAFzKaFFp0qSJFixYoHnz5unaa6/Vv/71L7300kvq27evyVgAUDgrVkh16zJ3ClAMjO6jIkndu3dX9+7dTccAgKJbtsy+rFvXbA7ACxldowIAXiGzqDBtPuByFBUAuBwHD9qHIzscFBWgGFBUAOByZK5NadzYnvANgEtRVADgcmQWFabNB4oFRQUAiiojI/v8Ph07ms0CeCnjR/0AgMeyLGn2bLusNG9uOg3glSgqAFBU/v5S5872AqBYsOkHAAC4LYoKABTFuXPS44/bO9NmZJhOA3gtigoAFMU330iTJ0sDB3IiQqAYUVQAoCguPCyZogIUG4oKABTF0qX2JYclA8WKogIAhXXkiLRli32dafOBYkVRAYDCypzkrUEDqUIFo1EAb0dRAYDCYtp8oMRQVACgsLZtsy/ZPwUodsxMCwCFtWGDXVb+9jfTSQCvR1EBgMJyOKR69UynAHwCm34AoDAsy3QCwKdQVACgoE6flqpXl/r1s68DKHYUFQAoqJUrpYQEafVqKTTUdBrAJ1BUAKCgFi+2Lzt3Ztp8oIRQVACgoC4sKgBKBEUFAAri55/tpVQp6aabTKcBfAZFBQAKYskS+7JlSyky0mwWwIdQVACgIDI3+3TpYjYH4GOY8A0ACqJpU/uIH/ZPAUqUw7I8d/aipKQkRUVFKTExUZGsigUAwCMU5vObTT8AAMBtUVQA4FK++oqZaAFDKCoAkJ+EBKldO6lCBenUKdNpAJ9DUQGA/GQellyvnhQebjYL4IMoKgCQH2ajBYyiqABAXlJTpaVL7evMnwIYQVEBgLx8/bWUnCxVqiRdf73pNIBPoqgAQF4+/9y+7NZN8uPXJWAC//MAIDeWJS1caF/v3t1sFsCHMYU+AOTG4ZAWLbLXqrRvbzoN4LMoKgCQl7/9TRo92nQKwKcZ3fQzYcIEORwOp6VSpUomIwEAADdifB+VunXr6tChQ1nLtm3bTEcC4OuOH5duv12aPdveVwWAMcY3/ZQqVYq1KADcy6JF0kcfSXv2SAMHmk4D+DTja1R++uknValSRbGxsfrHP/6hX375Jc+xKSkpSkpKcloAwOUyD0vmaB/AOKNFpVmzZnr77be1ZMkSvfHGGzp8+LBatGihY8eO5To+Li5OUVFRWUtMTEwJJwbg9c6fz542v0cPs1kAyGFZ7rMB9vTp06pVq5Yee+wxjc5lT/uUlBSlpKRk3U5KSlJMTIwSExMVGRlZklEBeKuVK6W2baXoaOnwYcnf33QiwOskJSUpKiqqQJ/fxvdRuVBYWJiuu+46/fTTT7l+PSgoSEFBQSWcCoBPydzs07UrJQVwA8b3UblQSkqKdu7cqcqVK5uOAsBXZRYVNvsAbsFoURkzZoxWrVqlffv2ad26dbr99tuVlJSkAQMGmIwFwFclJ0tly0pBQVLHjqbTAJDhTT+//fab+vTpoz///FPly5fXDTfcoO+++07Vq1c3GQuAr4qIkNaskZKSJPZ7A9yC0aISHx9v8uUBIHeUFMBtuNU+KgBgTFKSlMfUCADMoagAgCTNmSNVrMhJCAE3Q1EBAMmeMj89XWIiScCtUFQA4MgRafVq+3qvXmazAHBCUQGATz+VMjKkxo2lGjVMpwFwAYoKAHz8sX15661mcwDIgaICwLedPCl9+aV9/bbbjEYBkBNFBYBv+/xz+4zJdepIV19tOg2Ai7jVSQkBoMR16iTNmCGFhppOAiAXFBUAvq18eem++0ynAJAHNv0AAAC3xRoVAL4rLs4+EeE//iFFR5tOAyAXFBUAvun0aenZZ+3Lxo0pKoCbYtMPAN+0cKFdUmJjpRtuMJ0GQB4oKgB807vv2pd33SU5HGazAMgTRQWA7zl2TFq82L7et6/ZLADyRVEB4Hs++EBKS5MaNJCuucZ0GgD5oKgA8D2Zm31YmwK4PYoKAN9y/rxUpowUGGgflgzArXF4MgDfEhAgffaZlJQkRUaaTgPgElijAsA3UVIAj0BRAeA7Dh6U9u0znQJAIVBUAPiOadOkmjWlceNMJwFQQBQVAL4hI0N67z37euPGZrMAKDCKCgDf8M03UkKCvW9K9+6m0wAoIIoKAN/w1lv25e23S8HBZrMAKDCKCgDvd/KkNH++ff3ee41GAVA4FBUA3u/dd6WzZ6Vrr+VMyYCHoagA8H6Za1OGDOFMyYCHYWZaAN5v0SLpww/ZiRbwQBQVAN4vNFTq3990CgBFwKYfAN4rLU2yLNMpAFwGigoA7/XWW1KdOtI775hOAqCIKCoAvNcbb0i7dklHjphOAqCIKCoAvNP330sbN0oBAeyfAngwigoA7/TGG/blrbdK5cubzQKgyCgqALxPYqI9yZtkz50CwGNRVAB4n5kzpeRk6ZprpLZtTacBcBkoKgC8S1qa9N//2tdHj5b8+DUHeDL+BwPwLv7+9maffv2ku+82nQbAZXKbohIXFyeHw6FRo0aZjgLAkzkcUuvW0ttvS8HBptMAuExuUVQ2bNig119/XfXq1TMdBQAAuBHjReXUqVPq27ev3njjDZUpUybfsSkpKUpKSnJaACBL377Sww9Lhw6ZTgLARYwXleHDh6tbt25q3779JcfGxcUpKioqa4mJiSmBhAA8wq5d0nvv2TvS8kcM4DWMFpX4+Hh9//33iouLK9D4cePGKTExMWtJSEgo5oQAPMaLL9qXPXpIV19tNgsAlyll6oUTEhI0cuRILV26VMEF3OEtKChIQUFBxZwMgMc5etTeeVaSHnnEbBYALmWsqGzatElHjhxR48aNs+5LT0/X119/rVdeeUUpKSny9/c3FQ+AJ5k2TTp3TmrSxD7iB4DXMFZU2rVrp23btjndN2jQINWuXVtjx46lpAAomKNHpZdesq8//rh9eDIAr2GsqEREROjaa691ui8sLEzlypXLcT8A5On556VTp6TGjaVevUynAeBixooKALjEww/bm326dWNtCuCFHJZlWaZDFFVSUpKioqKUmJioyMhI03EAAEABFObz2/g8KgBQJKmpphMAKAEUFQCe6f77pc6dpe3bTScBUIwoKgA8z65d9rwpS5ZIZ86YTgOgGFFUAHiep5+WMjKkm2+WmjY1nQZAMaKoAPAsa9ZI8+fb1ydONJsFQLGjqADwHGlp0gMP2NcHDZLq1zebB0Cxo6gA8Bz/+5/0ww9SmTLS5Mmm0wAoARQVAJ7BsqT337evx8VJ5cubzQOgRDAzLQDP4HBIK1dKc+dKAwaYTgOghFBUAHiOwEBp8GDTKQCUIDb9AHBvqanSa68xEy3goygqANzb889Lw4ZJHTva+6kA8CkUFQDua8MGe3I3SbrnHs6ODPggigoA95ScLN11lz13Su/e0t13m04EwACKCgD39NBD0s8/SzEx0owZrE0BfBRFBYD7iY+XZs+W/Pykd9+1J3gD4JMoKgDcS2qq9Nhj9vUnn5RatzabB4BRFBUA7iUwUFqxwj6nz/jxptMAMIwJ3wC4n1q1pFdfNZ0CgBtgjQoA9/DCC9Lnn5tOAcDNsEYFgHnz5kljxtg7z27dKl17relEANwEa1QAmPXNN9LAgfb1UaMoKQCcUFQAmPPTT1LPnvaRPr16Sf/5j+lEANwMRQWAGbt2SW3bSsePS02bSnPn2pt+AOAC7KMCoOT9/rv0979LR49K11wjffaZFBpqOhUAN8SfLwBKXpUq0u23S40aSV9/LVWsaDoRADfFGhUAJc/hkF55RTpzRgoPN50GgBtjjQqAkvHWW/ZZkM+ft2/7+VFSAFwSa1QAFK8zZ6Thw+2TDEr2uXseeshoJACeg6ICoPjs2WPvi7Jtm70GZdIk6cEHTacC4EEoKgBcz7Kk996zTyyYnCxVqCDFx9uHIwNAIbCPCgDXe/JJ6e677ZLSurW0eTMlBUCRUFQAuF7//lJkpDRxovTVV/bhyABQBGz6AXB5UlPtWWUPHpT++U/7vtq1pYQEu6wAwGWgqAAomjNnpJkz7fPz/PabPTdKly5S48b21ykpAFyAogKgcH75RZozR3r1VenPP+37KleWxoyx16QAgAtRVAAU3P/9n9S9e/bt2Fhp7FhpwAApONhcLgBei6ICICfLsudAWbxYKlPG3jlWktq0kSIipBtukAYNsmeaLcWvEQDFh98wAKS0NGnHDmn9emndOunLL6X9++2vNWqUXVTCw+39Udj/BEAJMXp48muvvaZ69eopMjJSkZGRat68uRYtWmQyEuDdLMs+Gue775zvr19fatBAuu8+6c037ZISGCi1by/17Ws/LhMlBUAJMrpGpWrVqnruued05ZVXSpLmzJmjnj17avPmzapbt67JaIBnSUuzJ1crUyb7vk8/tSdaO3DAXhIS7Mtz5+zNN4mJ9pE6knTttfaakiZNpKZNpZYtpRtvlMLCjLwdAMjksKwL/1Qyr2zZsvrPf/6je+6555Jjk5KSFBUVpcTEREW6+q+8M2eko0fz/nrZsvYve0k6e1Y6ciTvsWXKZP8VmpIiHT6c99ioKKl0aft6aqp06FDeYyMjsz+Yzp+357HIS0SEnVmS0tPtD6W8hIVJ0dH29YwM+wMuL6GhUvny9nXLsj8I8xISYk+lnunXX/MeGxwsVayYfTshwc6dm6Ag+6iTC8empWVnulBAgBQTk337wAH7+5w5NnO8Zdlja9bMHvvLL/a/dea4zCUjw95P47rrssdu3y6dPGl/LT09+zI93T7nTadO2WOXL8/OkZpq/1umpNiFIj1devbZ7LETJkjffCOdPm0Xk5Mn7cJx6pSdNyUlu3zcequ0YEHO71epUtKVV0pr1mT//Jw8af88+TEHJIDiV6jPb8tNpKWlWfPmzbMCAwOtHTt25Drm3LlzVmJiYtaSkJBgSbISExNdH+izzy7+OHJeXnste+zy5fmPff757LFr1+Y/duLE7LE//JD/2LFjs8f+/HP+Yx98MHvsoUP5jx08OHtsUlL+Y++8M3tsWlr+Y7t3d/4eBwXlPfamm5zHli2b99hmzZzHVq2a99hrr3Uee/XVeY+tWdN5bMOGeY+tWNF5bMuWeY+NjHQe27Fj3mP9/Z3H9uqV//f45Mnssa+9Zln33WdZkyZZ1ttvW9aKFZa1d69lpaZaAGBSYmJigT+/je9Mu23bNjVv3lznzp1TeHi4FixYoDp16uQ6Ni4uTs8880zJBPPzy/9wS3//go+98KgIh8PM2IAA59smxgYGOt8OCcn7L/igoJxjQ0JyjsvtfYeGOm+yyFzDkPm1C4WF2WubMsdcOPbill+mjL2myeHIXvz87MvMtUqZYmKkq66yv+bv77xcvDmlaVP7exMQYF9mLsHB9pKRkf19GjFCuu02+znCw+1MmWvhoqKc/z2GDs35/QIAD2N8009qaqoOHDigkydP6qOPPtLMmTO1atWqXMtKSkqKUlJSsm4nJSUpJiameDb9AACAYlGYTT/Gi8rF2rdvr1q1amnGjBmXHFus+6gAAIBiUZjPb7fbc86yLKe1JgAAwHcZ3UfliSeeUJcuXRQTE6Pk5GTFx8dr5cqVWrx4sclYAADATRgtKn/88Yf69eunQ4cOKSoqSvXq1dPixYvVoUMHk7EAAICbMFpU3nzzTZMvDwAA3Jzb7aMCAACQiaICAADcFkUFAAC4LYoKAABwWxQVAADgtigqAADAbVFUAACA26KoAAAAt0VRAQAAbsvozLSXK/PEz0lJSYaTAACAgsr83M78HM+PRxeV5ORkSVJMTIzhJAAAoLCSk5MVFRWV7xiHVZA646YyMjJ08OBBRUREyOFwuPS5k5KSFBMTo4SEBEVGRrr0ud0B78/zeft79Pb3J3n/e+T9eb7ieo+WZSk5OVlVqlSRn1/+e6F49BoVPz8/Va1atVhfIzIy0mt/ACXenzfw9vfo7e9P8v73yPvzfMXxHi+1JiUTO9MCAAC3RVEBAABui6KSh6CgID399NMKCgoyHaVY8P48n7e/R29/f5L3v0fen+dzh/fo0TvTAgAA78YaFQAA4LYoKgAAwG1RVAAAgNuiqAAAALdFUSmElJQUNWjQQA6HQ1u2bDEdx2VuvvlmVatWTcHBwapcubL69eungwcPmo7lMvv379c999yj2NhYhYSEqFatWnr66aeVmppqOprLPPvss2rRooVCQ0NVunRp03Fc4tVXX1VsbKyCg4PVuHFjrV692nQkl/n666/Vo0cPValSRQ6HQ5988onpSC4TFxenJk2aKCIiQhUqVNAtt9yi3bt3m47lUq+99prq1auXNQla8+bNtWjRItOxik1cXJwcDodGjRpl5PUpKoXw2GOPqUqVKqZjuFzbtm01f/587d69Wx999JH27t2r22+/3XQsl9m1a5cyMjI0Y8YM7dixQy+++KKmT5+uJ554wnQ0l0lNTVXv3r31wAMPmI7iEu+//75GjRqlJ598Ups3b1br1q3VpUsXHThwwHQ0lzh9+rTq16+vV155xXQUl1u1apWGDx+u7777TsuWLVNaWpo6duyo06dPm47mMlWrVtVzzz2njRs3auPGjbrpppvUs2dP7dixw3Q0l9uwYYNef/111atXz1wICwXyxRdfWLVr17Z27NhhSbI2b95sOlKx+fTTTy2Hw2GlpqaajlJspkyZYsXGxpqO4XKzZs2yoqKiTMe4bE2bNrWGDh3qdF/t2rWtxx9/3FCi4iPJWrBggekYxebIkSOWJGvVqlWmoxSrMmXKWDNnzjQdw6WSk5Otq666ylq2bJnVpk0ba+TIkUZysEalAP744w8NGTJE77zzjkJDQ03HKVbHjx/Xu+++qxYtWiggIMB0nGKTmJiosmXLmo6BXKSmpmrTpk3q2LGj0/0dO3bUmjVrDKVCUSUmJkqS1/5/S09PV3x8vE6fPq3mzZubjuNSw4cPV7du3dS+fXujOSgql2BZlgYOHKihQ4fq+uuvNx2n2IwdO1ZhYWEqV66cDhw4oE8//dR0pGKzd+9evfzyyxo6dKjpKMjFn3/+qfT0dFWsWNHp/ooVK+rw4cOGUqEoLMvS6NGj1apVK1177bWm47jUtm3bFB4erqCgIA0dOlQLFixQnTp1TMdymfj4eH3//feKi4szHcV3i8qECRPkcDjyXTZu3KiXX35ZSUlJGjdunOnIhVLQ95fp0Ucf1ebNm7V06VL5+/urf//+stx80uLCvkdJOnjwoDp37qzevXvr3nvvNZS8YIry/ryJw+Fwum1ZVo774N4efPBB/fDDD5o3b57pKC539dVXa8uWLfruu+/0wAMPaMCAAfrxxx9Nx3KJhIQEjRw5UnPnzlVwcLDpOL47hf6ff/6pP//8M98xNWrU0D/+8Q8tXLjQ6Rdkenq6/P391bdvX82ZM6e4oxZJQd9fbj+Ev/32m2JiYrRmzRq3XpVZ2Pd48OBBtW3bVs2aNdPs2bPl5+fePb0o/4azZ8/WqFGjdPLkyWJOV3xSU1MVGhqqDz74QL169cq6f+TIkdqyZYtWrVplMJ3rORwOLViwQLfccovpKC41YsQIffLJJ/r6668VGxtrOk6xa9++vWrVqqUZM2aYjnLZPvnkE/Xq1Uv+/v5Z96Wnp8vhcMjPz08pKSlOXytupUrsldxMdHS0oqOjLzlu2rRpmjRpUtbtgwcPqlOnTnr//ffVrFmz4ox4WQr6/nKT2V1TUlJcGcnlCvMef//9d7Vt21aNGzfWrFmz3L6kSJf3b+jJAgMD1bhxYy1btsypqCxbtkw9e/Y0mAwFYVmWRowYoQULFmjlypU+UVIk+327++/MgmrXrp22bdvmdN+gQYNUu3ZtjR07tkRLiuTDRaWgqlWr5nQ7PDxcklSrVi1VrVrVRCSXWr9+vdavX69WrVqpTJky+uWXXzR+/HjVqlXLrdemFMbBgwd14403qlq1anr++ed19OjRrK9VqlTJYDLXOXDggI4fP64DBw4oPT09a56fK6+8Mutn1pOMHj1a/fr10/XXX6/mzZvr9ddf14EDB7xmv6JTp07p559/zrq9b98+bdmyRWXLls3xO8fTDB8+XO+9954+/fRTRUREZO1XFBUVpZCQEMPpXOOJJ55Qly5dFBMTo+TkZMXHx2vlypVavHix6WguERERkWOfosx9GI3sa2TkWCMPtm/fPq86PPmHH36w2rZta5UtW9YKCgqyatSoYQ0dOtT67bffTEdzmVmzZlmScl28xYABA3J9fytWrDAdrcj+97//WdWrV7cCAwOtRo0aedXhrStWrMj132vAgAGmo122vP6vzZo1y3Q0lxk8eHDWz2b58uWtdu3aWUuXLjUdq1iZPDzZZ/dRAQAA7s/9N9QDAACfRVEBAABui6ICAADcFkUFAAC4LYoKAABwWxQVAADgtigqAADAbVFUAACA26KoAF7oxhtv1KhRo0zHyNWxY8dUoUIF7d+/X5K0cuVKORyOYj+RYlFfZ/bs2SpdunShHtOkSRN9/PHHhXoMgNxRVABc0qFDh3TXXXfp6quvlp+fX54l6KOPPlKdOnUUFBSkOnXqaMGCBTnGxMXFqUePHqpRo0bxhjboqaee0uOPP66MjAzTUQCPR1EBcEkpKSkqX768nnzySdWvXz/XMWvXrtWdd96pfv36aevWrerXr5/uuOMOrVu3LmvM2bNn9eabb+ree+8tqehGdOvWTYmJiVqyZInpKIDHo6gAXu7EiRPq37+/ypQpo9DQUHXp0kU//fST05g33nhDMTExCg0NVa9evTR16lSnzR01atTQf//7X/Xv319RUVG5vs5LL72kDh06aNy4capdu7bGjRundu3a6aWXXsoas2jRIpUqVSrfM3MfO3ZMffr0UdWqVRUaGqrrrrtO8+bNcxpz4403asSIERo1apTKlCmjihUr6vXXX9fp06c1aNAgRUREqFatWlq0aFGO5//2229Vv359BQcHq1mzZjlOZz979mxVq1Yt63tx7Ngxp6/v3btXPXv2VMWKFRUeHq4mTZpo+fLlTmP8/f3VtWvXHLkBFB5FBfByAwcO1MaNG/XZZ59p7dq1sixLXbt21fnz5yXZH9xDhw7VyJEjtWXLFnXo0EHPPvtsoV9n7dq16tixo9N9nTp10po1a7Juf/3117r++uvzfZ5z586pcePG+vzzz7V9+3bdd9996tevn9OaGUmaM2eOoqOjtX79eo0YMUIPPPCAevfurRYtWuj7779Xp06d1K9fP505c8bpcY8++qief/55bdiwQRUqVNDNN9+c9b1Yt26dBg8erGHDhmnLli1q27atJk2a5PT4U6dOqWvXrlq+fLk2b96sTp06qUePHjpw4IDTuKZNm2r16tUF++YByJuRczYDKFaZp2Tfs2ePJcn69ttvs772559/WiEhIdb8+fMty7KsO++80+rWrZvT4/v27WtFRUXl+9wXCwgIsN59912n+959910rMDAw63bPnj2twYMHO41ZsWKFJck6ceJEnu+na9eu1iOPPOKUoVWrVlm309LSrLCwMKtfv35Z9x06dMiSZK1du9bpdeLj47PGHDt2zAoJCbHef/99y7Isq0+fPlbnzp2dXvvOO+/M83uRqU6dOtbLL7/sdN+nn35q+fn5Wenp6fk+FkD+WKMCeLGdO3eqVKlSatasWdZ95cqV09VXX62dO3dKknbv3q2mTZs6Pe7i2wXlcDicbluW5XTf2bNnFRwcnO9zpKen69lnn1W9evVUrlw5hYeHa+nSpTnWWNSrVy/rur+/v8qVK6frrrsu676KFStKko4cOeL0uAs3O5UtW9bpe7Fz584cm6Uuvn369Gk99thjqlOnjkqXLq3w8HDt2rUrR76QkBBlZGQoJSUl3/cLIH+lTAcAUHwsy8rz/swCcXGZyO9x+alUqZIOHz7sdN+RI0eyCoMkRUdH68SJE/k+zwsvvKAXX3xRL730kq677jqFhYVp1KhRSk1NdRoXEBDgdNvhcDjdl/meCnLkzYXfi0t59NFHtWTJEj3//PO68sorFRISottvvz1HvuPHjys0NFQhISGXfE4AeWONCuDF6tSpo7S0NKf9O44dO6Y9e/bommuukSTVrl1b69evd3rcxo0bC/1azZs317Jly5zuW7p0qVq0aJF1u2HDhvrxxx/zfZ7Vq1erZ8+euvvuu1W/fn3VrFkzx86/l+O7777Lun7ixAnt2bNHtWvXlmR/vy78+sXjM/MNHDhQvXr10nXXXadKlSplzQlzoe3bt6tRo0Yuyw34KooK4MWuuuoq9ezZU0OGDNE333yjrVu36u6779YVV1yhnj17SpJGjBihL774QlOnTtVPP/2kGTNmaNGiRTnWsmzZskVbtmzRqVOndPToUW3ZssWpdIwcOVJLly7V5MmTtWvXLk2ePFnLly93mnOlU6dO2rFjR75rVa688kotW7ZMa9as0c6dO3X//ffnWFNzOSZOnKgvv/xS27dv18CBAxUdHa1bbrlFkvTQQw9p8eLFmjJlivbs2aNXXnlFixcvzpHv448/1pYtW7R161bdddddua61Wb16dY6diwEUHkUF8HKzZs1S48aN1b17dzVv3lyWZemLL77I2kzSsmVLTZ8+XVOnTlX9+vW1ePFiPfzwwzn2JWnYsKEaNmyoTZs26b333lPDhg3VtWvXrK+3aNFC8fHxmjVrlurVq6fZs2fr/fffd9o/5rrrrtP111+v+fPn55n3qaeeUqNGjdSpUyfdeOONqlSpUlaRcIXnnntOI0eOVOPGjXXo0CF99tlnCgwMlCTdcMMNmjlzpl5++WU1aNBAS5cu1T//+U+nx7/44osqU6aMWrRooR49eqhTp0451pz8/vvvWrNmjQYNGuSy3ICvclhF2RgNwKsNGTJEu3btKpbDa7/44guNGTNG27dvl5+fd/6t9OijjyoxMVGvv/666SiAx2NnWgB6/vnn1aFDB4WFhWnRokWaM2eOXn311WJ5ra5du+qnn37S77//rpiYmGJ5DdMqVKigMWPGmI4BeAXWqADQHXfcoZUrVyo5OVk1a9bUiBEjNHToUNOxAICiAgAA3Jd3biAGAABegaICAADcFkUFAAC4LYoKAABwWxQVAADgtigqAADAbVFUAACA26KoAAAAt/X/J+Yjt6Z/fRcAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "id": "88a7263d",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
@@ -1761,11 +1837,11 @@
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" Ridgebeta = np.linalg.inv(X.T @ X+lmb*I) @ X.T @ y\n",
- " print(Ridgebeta)\n",
+ "# print(Ridgebeta)\n",
" # and then make the prediction\n",
" ypredictRidge = X @ Ridgebeta\n",
" MSEPredict[i] = MSE(y,ypredictRidge)\n",
- " print(MSEPredict[i])\n",
+ "# print(MSEPredict[i])\n",
" # Now plot the results\n",
"plt.figure()\n",
"plt.plot(np.log10(lambdas), MSEPredict, 'r--', label = 'MSE Ridge Train')\n",
@@ -1777,16 +1853,20 @@
},
{
"cell_type": "markdown",
- "id": "9d887ba8",
- "metadata": {},
+ "id": "60ab2d3b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We see here that we reach a plateau. What is actually happening?"
]
},
{
"cell_type": "markdown",
- "id": "8c6667cd",
- "metadata": {},
+ "id": "1a49fe21",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## With Lasso Regression"
]
@@ -1794,8 +1874,11 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "469fcfa1",
- "metadata": {},
+ "id": "30bff24b",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import os\n",
@@ -1858,40 +1941,23 @@
},
{
"cell_type": "markdown",
- "id": "62e61bf5",
- "metadata": {},
+ "id": "cb4e882b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Another Example, now with a polynomial fit"
]
},
{
"cell_type": "code",
- "execution_count": 2,
- "id": "b0302344",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[ 2.03099776 -0.17917768 5.18029127]\n",
- "Training MSE for OLS\n",
- "0.009163470508352218\n",
- "Test MSE OLS\n",
- "0.008675369724975977\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAGwCAYAAABcnuQpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB22klEQVR4nO3dd3xTZdvA8V+Stulu6W6hk1X2RgFlKLsiKOIEGYIgS0QRqjIfFUEFHkXECagPw4GICAivsofMsinDMmSV2b1z3j9OG4ilpYU2p22u7+dzODknd5IraUiu3FOnKIqCEEIIIUQ5pdc6ACGEEEKIeyHJjBBCCCHKNUlmhBBCCFGuSTIjhBBCiHJNkhkhhBBClGuSzAghhBCiXJNkRgghhBDlmp3WAZQ2k8nE+fPncXNzQ6fTaR2OEEIIIYpAURSSkpIICgpCry+87qXCJzPnz58nODhY6zCEEEIIcRfOnj1LlSpVCi1T4ZMZNzc3QH0x3N3dNY5GCCGEEEWRmJhIcHCw+Xu8MBU+mclrWnJ3d5dkRgghhChnitJFRDoACyGEEKJc0zSZ2bhxI926dSMoKAidTseyZcsKLDt48GB0Oh2zZs2yWnxCCCGEKPs0TWZSUlJo0KABs2fPLrTcsmXL+OuvvwgKCrJSZEIIIYQoLzTtM9OlSxe6dOlSaJlz584xfPhwfv/9d6KioqwUmRBCCCHKizLdAdhkMtGnTx/GjBlDnTp1inSbjIwMMjIyzMeJiYmlFZ4QQgghyoAy3QF42rRp2NnZMXLkyCLfZurUqXh4eJg3mWNGCCGEqNjKbDKze/du/vvf/zJ//vxizdwbHR1NQkKCeTt79mwpRimEEEIIrZXZZGbTpk3Ex8cTEhKCnZ0ddnZ2nD59mldffZWwsLACb2c0Gs1zysjcMkIIIUTFV2b7zPTp04f27dtbnOvUqRN9+vShf//+GkUlhBBCiLJG02QmOTmZEydOmI/j4uKIiYnBy8uLkJAQvL29Lcrb29sTEBBAzZo1rR2qEEIIIcooTZOZXbt20a5dO/Px6NGjAejbty/z58/XKCohhBBClCeaJjNt27ZFUZQilz916lTpBSOEEEKIcqnM9pkRwuZduQIpKQVfX6UKGAzq5atXITm54LKVK4Nd7n/369ehsPmXAgPBwUG9fOMGJCQUXDYgAIxG9XJCglq+IP7+4OioXk5KgmvXCi7r5wdOTurl5GT1+RXExwdcXNTLKSnq61YQb29wdVUvp6bC5csFl/XygrzVetPT4dKlgst6eoKHh3o5IwMuXiy4rIeHWh4gMxMuXCi4rLs7VKqkXs7OhnPnCi7r6qo+P4CcHPjnn4LLuriorxuAyQSFjfp0dlbL5uSoZePiQFFubiaT+n4xGtW/mZeX+l5QFDUGk0m9DOreyUmN1WhU7zc+Xr3u3Lmb95nHyUl9vRwc1PdE3mt1/rwaz7/j9PQEe3v1fZn3/C9dgqwsy7J5cRoM6v+NvOd/u7KOjurrqtdDcDCcOaOej49X/363cnAAX1/Q6SAkRL1fRVHfZ+np+cv6+amXQ0PVeHNy1Pd6WpplWXt79f8PqDFcuKC+H65dU9/Ht7KzU58/qJ8Rec/p+vX8nycGg/r/HdTX4coV9f2bkKD+H72VTqeWAfU216+rzykhQX3dQ0LQlFLBJSQkKICSkJCgdShCFE/fvrd+ZeTf4uNvln3ppcLLxsXdLPvaa4WXPXToZtmJEwsvu2PHzbLTphVedt26m2Vnzy687G+/3Sw7b17hZb///mbZJUsKLztv3s2yK1YUXvaTT26WXbeu8LLTp98s+9dfhZedOPFm2YMHCy87ZoxaLjNTUXbuLLzs0KGKsnu3ovTvryg9ehRetm9fRVm/XlGCghTFz6/wsr16KcqmTYWXydu6dlWUvXuLVrZNG0U5dqxoZZs1U5Rz54pWtnZtRblxo2hlQ0MVJSuraGV9fdW/hb39ncs6O6tlPT2Ldt+KoiiVKxetbEqKolSvXrSy8fGK0rBh0crGxSlKq1ZFK3vokKJ06HDzuH17pTQU5/tbamaEKKvs7W/WZNxr2VvnarKz06as/paZIAwGbcrm1WTl3a6kytrd8lGq0xW/7K1fFSaTWkanU/cLFkC/fgXfn729Gqu9vfrrft68opXNzlZrOAqL085OLau/wyweeWUdHCzfE3cqCzdf53/XtORdZ29/s/Yv7zb/rhG5Xdm8fWam+roWVjbv75WRkb+sXq8+bl4ZR0f19pmZ6t/qTmUdHQsvm6ewsjrdzVjznpujo1rj8u/X7W7L6nQ3Y79d2Vtfp1vLZmdbvqc1olOUf//lKpbExEQ8PDxISEiQOWdE2ZaUBC++CMOGQatWd/5SEOXXpk0wbRrs33/7Jp5vvoE+fdTLK1dC3rp0np5q80ilSmoTlJsbjBgBedNYnDwJ33+vNqM4OalfNkbjzWagevVuNgckJcGJE+oXkcFwc5+3eXrebGbLylKb+/R6ddPp1C3vcl6CAjcTsrwyIO9lcVeK8/2tfTolhFB98QUsXgwxMXDokHwBlHeKov4d16+HdevgpZduJh2pqfDbbzfLurlB1aoQEQHh4RAZefO6hx5S+zJ4elrWFt1O1aoQHV20+NzcoFGjopW1t7/Zd+dOdLo7xylECZNkRoiyICsLZs1SL7/66p2r9UXZpCiwbRssWgQ//mjZETgs7GYy06wZzJ4NDRqoiYu3d8HJa17zgxCiQJLMCFEWfP+92tzg7w+tQmDnV1pHJIrrxg344IObI68qA+Euam1JjRpQz8Py79rcEYiFuFiIK52QFCDbpJBjMpFjguwcBZNiwqRAjkkhR1EwmRQUBXIUdW+6dQ8ouceKAgq5l3Pv35TbS0EBc1+TW6+/5XRuOYsDbtfHoSgdH4rWN6Ls96CoSJ08PCOaULPpQ5o9viQzQmhNUeD999XLfR+HJb20jUfcvRYATv86eQpyTkHMGoixbjg6wD53E6I0bbvRFySZEcKG/fEH7NunzpMReR1OA76R4F1N68hEQa5dg6NH1Tk2OnW62SyYmKjO4VKEPiMmBVIys0nJyCEtK5vUzBzSMnNIy8ohI8tEtqnkf7brdGqCo9fpcvvn6tDnnc9t5sor8+9j0N1yOff+bvnHopFMZ7HjNlfla1azRg8x6YZWeuwDamn6+JLMCKG1vFqZ3r3gzC/q5Z5fQUBd7WISt3flCrz8MixcrR7r9fBcf+jYsdCbZeWY2P/PDfacvsGRC4kcvpDIycvJZOUUnrAY7fT4uRvxcjHi6WSPp7M9nk72eDjZ4+poh4vRDhcHde/sYMDR3oCjvT53b8Bop8fBTo+DQd30evk2FxWTJDNCaElRoHt3OHUK7jfCaRNUfUgSmbJGUeCHH2D4cHU2V70eBgyAsWOhWv4aNEVROHAugS0nrrLt76vsOnWN1Mz883a4Ge2I8HMl1MuZkNytSiUn/Nwd8XM34ma0M9eQCCEKJsmMEFrS6WDoUOj7FMyqp55rOULbmISljAx45hn4+Wf1uE4ddWK6Zs0siimKwtGLSSzfd55f953nn+uWU9JXcranaZgXdYM8qBXoRq1Ad6pUcpJkRYgSIMmMEGXBnvmQlQL+dSGi3R2LCysyGtWaGDs7eOMNdbtlhtWUjGwW7TjDkp1nOR5/c30sZwcDrar50CLCmxZVvanp7ybNPEKUEklmhNDSwYOQkQYbP1WPW46QXopl0SefwPjx6rwwuW6kZrJg62nmbY3jRqq6OKGDQU+7SF+6NQji4Uh/nBxk8jghrEGSGSG01Lu3OpKptzM0DIE6j2sdkQB1PaTNm+Hzz9Xk0t/fvGpxQloWc9ad4Lvtp0nJ7QcT7uPCoAcjeKRBIO6OMhBaCGuTZEYILaXl9quwA+4fAnYOhRYXpUxRYOpUePNN9bhjR+h1c96ftYcv8dayA1xKzAAgMsCNYe2q0bVeIAZpQhJCM5LMCKGlpOvq3skZGvfVNhYBkyerG8C4cfDEEwBcTc5g0q+H+XWfusp0uI8Lb0XV4qFIP+nAK0QZIMmMEFpKTlT39buDk6emodi8JUtuJjIzZ8KoUQCsOnCBN5cd5FpKJnodDGodwSvta+BoL/1hhCgrJJkRQkuZ2eo+tLG2cdi6XbugXz/18quvwqhRKIrCnPUnef/3WEBtUpr+RH3qV/HULEwhxO1JMiOElrJM6t7FXds4bFlaGjz2GKSnQ9euMG0aOSaFicsP8t32MwC88EA4YztH4mAnq5kLURbJ/0whtJKVpS7QA+AqyYxmnJzUodf33QeLFpGWA0O+281328+g08HEbrUZ/0htSWSEKMOkZkYIrSgKdA6ApGvg5ql1NLbt0UehWzeup2Yx4Mvt7D1zAwc7Pf99qiFd6gVqHZ0Q4g4kmRFCKw4O0KESJKaCi5vW0diebdsgPBwCAgDIzFEY/O1u9p65gYeTPV8835Tm4V4aBymEKAqpNxVCS1l588w4aRuHrUlMhJ49ITISduxAURTe/PkAO05dw81ox5LB90siI0Q5IsmMEFpJS4NzSXDDBPaOWkdjW956Cy5cAD8/qF+fLzfF8cPuf9Dr4ONnGxEZIH2YhChPJJkRQiuHD8PHV+HrFKmZsaZdu9QOvwBz5vDnqQTeXXUEgLeiatO2pp+GwQkh7oYkM0JoJW/CPDvAzlhoUVFCcnJgyBAwmeDZZzlW7z5GLopBUeCZ5sH0bxWmdYRCiLsgyYwQWknJTWbsdWAvNTNWMWcO7N4NHh6kTJ3OwAW7SM7I5r5wLyY/WleWJhCinJJkRgitmGtmdGCQBSZL3blzNxeQfO89Zu5P4My1VCp7OjG3dxOZR0aIckz+9wqhlbyaGQc9SI1A6XN3hwEDoFUrDj3yFPO2ngLg7cfqUslFkkkhyjOZZ0YIraQkqXsHWbDQKtzcYNYscjIyeePLneSYFKLqBdJOOvwKUe5JzYwQWslLZuzlN4U1Ldxznn1nb+BmtGNCt9pahyOEKAGSzAihlRph0MIB6lfSOpKK7epV6NULtm4lPjGd6avVVbBf61QTf3eZ30eIikB+EgqhlfrVoKMj+FXROpKKbfZs+PFHOHGCKa99SVJGNvWreND7/lCtIxNClBCpmRFCK9np6l5m/y09KSnw0UcAHO47lBUHLqLXwbuP1cOgl07XQlQUkswIoZVLF9WlDLKkgrTUfPklXLuGUrUq46gGQL+W4dSt7KFxYEKIkiTJjBBambMY/psMv5/ROpKKKSsLPvwQgBPPD2b/xVSc7A0Mf6iaxoEJIUqaJDNCaCU9d8VsR1nKoFQsWgRnz4K/P2/7NAfg6ebBeMmcMkJUOJomMxs3bqRbt24EBQWh0+lYtmyZ+bqsrCzGjh1LvXr1cHFxISgoiOeff57z589rF7AQJSktt8+Mo/SZKXEmE0ybBsD5foPZcCYZO72OgQ9GaByYEKI0aJrMpKSk0KBBA2bPnp3vutTUVPbs2cP48ePZs2cPS5cu5dixYzz66KMaRCpEKUjPTWacJJkpcZmZ8MIL8MADvB/WDoDuDStT2VPWwBKiItK052GXLl3o0qXLba/z8PBg7dq1Fuc+/vhjmjdvzpkzZwgJCbFGiEKUnvQMde8kX7AlztERRo/mRO8XWTZzAwBD2kitjBAVVbkaRpGQkIBOp8PT07PAMhkZGWRkZJiPExMTrRCZEHchL5mRZqZS8/nGkygKdKjtT3V/N63DEUKUknLTATg9PZ1x48bx7LPP4u7uXmC5qVOn4uHhYd6Cg4OtGKUQxZCRqe6dXbSNo6LZuBEWLODiP/H8vPccAC+1rapxUEKI0lQukpmsrCyefvppTCYTc+bMKbRsdHQ0CQkJ5u3s2bNWilKIYmoRDk3soUqA1pFULDNmQL9+nBz9Flk5CveFe9E4RJaMEKIiK/PNTFlZWTz55JPExcXx559/FlorA2A0GjEaZairKAceqQWhByFS+nKUmGvXYOVKAKZ5NQGkVkYIW1Cmk5m8ROb48eOsW7cOb29vrUMSouRk5c4zYyfJd4n54QfIyuJajdrs96xCDX9X2tTw1ToqIUQp0zSZSU5O5sSJE+bjuLg4YmJi8PLyIigoiCeeeII9e/awYsUKcnJyuHjxIgBeXl44OMjEV6Kcu3wDUk2gl2SmxHz3HQAr6qrDsZ9oUgWdTtZgEqKi0zSZ2bVrF+3atTMfjx49GoC+ffsyadIkli9fDkDDhg0tbrdu3Tratm1rrTCFKB1jfoeMHGiXpHUkFUNcHGzejKLT8UlAM3Q6eLRBZa2jEkJYgabJTNu2bVEUpcDrC7tOiHJNUSAzR73sUng/MFFECxcC8E+j+7nk5kOrqt4EeMiwdyFsQbkYzSREhZOVBXm5uqvMf1IiTp1C0elYVL01AD0aSq2MELZCkhkhtJCWdvOy1MyUjC++IHb7AeYFNcNop6dzXRnyLoStkGRGCC1IMlMqvr+kI83BkQ61/XFztNc6HCGElUgyI4QW8pIZO8DBWdNQyj1FgaQksnNMLN93HoDHGkkTkxC2RJIZIbSQt2K2vQ7spJPqPTlxAry8SGrdjqtJaVRytqe1zC0jhE0p05PmCVFhubpCYwcwKJLM3KvVqyE7mysJaSg6Pd0aBGFvkN9pQtgSSWaE0EKgP3TLTWLsJZm5J6tXA/CLfz0AekgTkxA2R36+CKGF7PSbl+2ctIujvEtPh/XrAfi/0EaEejvTKNhT05CEENYnyYwQWkhJgHQFcpC1me7F5s2QmsoNTx+O+obRo2FlWb5ACBskyYwQWvhlOUxLgu/SQL58715uE9OfIQ1Bp6NLPZlbRghbJMmMEFpIyV2Pyd6gbRzlXV4yE9aYAHdHavrLbMpC2CJJZoTQQmqyujdKH/y7pigwbBix97Vjc1hD2tTwlSYmIWyUfJIKoYWU3GTGQf4L3jWdDl56ieFptbkRn0ybmjK3jBC2SmpmhNBCWoq6N8qU+/fi3I00jscnY9DraFXNR+twhBAakZ+FQmghVZKZe5KdDV9/zS7fOgA0CvbEw0leSyFsldTMCKGF1FR1b3TQNo7yascOGDyYh/tEoTfl0EaWLxDCpkkyI4QWqleGOnYQ7q11JOVT7iimTWENMekN0l9GCBsnzUxCaKFjU8j8DmrW0DqS8ik3mfkjtBFeLg7UDfLQOCAhhJakZkYILeQtZyDrMhXflSuwaxcAG8Ib07q6D3q9DMkWwpZJMiOEFlKSwCQrZt+VdetAUYgLjOCyq5c0MQkhJJkRQhNvfAn/SYINJ7WOpPzZuhWAjYG1AHiwuiQzQtg6SWaE0EJ6prp3ctY2jvIoN5nZXbkW9Sp74OMqC3UKYeukA7AQWsiQZOaurVzJ3Gn/Y3O6D8/KkGwhBJLMCKGNvGTGWZKZ4sqp5MVnrpFc12dJfxkhBCDNTEJoIyNL3Tu7aBtHOXTwXALXU7Nwc7SjUbCn1uEIIcoAqZkRQgsZ2epekpnimTAB06mrhDg1okbLhtgZ5PeYEEJqZoTQRmZuMuPipm0c5YmiwNy5NPp2Dj6pN2gaVknriIQQZYQkM0JooZ4vVLeDSl5aR1J+nDwJly+TabDnoH81moRKMiOEUEkzkxBaGBAJ/yRB5cpaR1J+5A7JPuBfFcXoQL3KsoSBEEIlNTNCaCErdzkDO5kjpchumV+mTpAHjvYGjQMSQpQVkswIYW2KAlmp6mU7J21jKU+2bQPUZEaamIQQt5JmJiGs7dIleHkv2AMDpWamSBIT4cABAPZUjqS7JDNCiFtIMiOEtaWl3bxsL5PmFcmJEygeHvyjGLns6iU1M0IIC9LMJIS15SUzdjrpM1NUjRuz7a9YHu/9PpU9nfB3l9XGhRA3STIjhLWZkxnAXvrMFNXuswlSKyOEuC1JZoSwtuQkdW+vAzupYSiq3WeuA0gyI4TIR5IZIawtOUHdS81M0Rw6hBIWxiOzJwGSzAgh8pMOwEJYW0pezQxSM1MUW7eiO32aQDxwsjcQGSBLQAghLEkyI4S1uTtBVQP4OoBOp3U0ZV/eZHlBtWgY7CmLSwoh8tH0U2Hjxo1069aNoKAgdDody5Yts7heURQmTZpEUFAQTk5OtG3blkOHDmkTrBAlpUk96O0C3X20jqR82L4dUOeXkSYmIcTtaJrMpKSk0KBBA2bPnn3b66dPn86MGTOYPXs2O3fuJCAggA4dOpCUlGTlSIUoQdl5SxlIf5k7SkmB2FgADgbI4pJCiNvTtJmpS5cudOnS5bbXKYrCrFmzePPNN3n88ccBWLBgAf7+/ixcuJDBgwff9nYZGRlkZGSYjxMTE0s+cCHuRV4yYy/9Ze5o/35QFC65enHFpRKNQjy1jkgIUQaV2cbnuLg4Ll68SMeOHc3njEYjbdq0YWtuG/rtTJ06FQ8PD/MWHBxsjXCFKLrZX8G7ibAsXutIyr6YGAAO+4VTzc8VT2cHbeMRQpRJZTaZuXjxIgD+/v4W5/39/c3X3U50dDQJCQnm7ezZs6UapxDFlpQEWVCG//uVHS4uXKxeh32BNWgSIk1MQojbK/OjmXT/Gu2hKEq+c7cyGo0YjTJFvCjDUnNXzDZKLcMdPf88L6dV56+4a0yX/jJCiAKU2Z+GAQEBAPlqYeLj4/PV1ghRruQtZ+AoycydmEwKB8+pkww2lP4yQogClNlkJjw8nICAANauXWs+l5mZyYYNG2jZsqWGkQlxj9LzkhnpAFyo9HTOXLxBSmYORjs9ET4uWkckhCijNE1mkpOTiYmJISa3k19cXBwxMTGcOXMGnU7HqFGjePfdd/n55585ePAg/fr1w9nZmWeffVbLsIW4N+aaGWkOLdTPPxMS6s+Hv80gMsBNJssTQhRI0z4zu3btol27dubj0aNHA9C3b1/mz5/P66+/TlpaGkOHDuX69evcd999rFmzBjc3mc5clGPpuUOzHWWemULFxKDPziLF3onaQe5aRyOEKMM0TWbatm2LoigFXq/T6Zg0aRKTJk2yXlBClLZgbwg2gJ+n1pGUbbcMy64T5KFtLEKIMk3qbYWwtn5tYIALtKyjdSRll6LA3r0AHPaPoHag1MwIIQomyYwQ1iYzAN/ZhQtw+TI5Oj3HfENlpWwhRKEkmRHC2rJyOwDbSTJToNwmppNeVQgK9MLFWOanxBJCaEg+IYSwttGL4FISBMdDG62DKaPy+sv4h0sTkxDijiSZEcLarqdCsiI1M4WpXZvd93dks1dtGckkhLgjSWaEsLbMbHXv4qptHGVZjx6MOenJ35dTWCAjmYQQdyB9ZoSwtswcde8inVoLkpqZTdyVFABpZhJC3JHUzAhhTYoCWZLMFOrKFeIOnwaTCV93J3zdZKZkIUThpGZGCGvKzIS8eSJdpfnkthYvpk6bpsxZ9h51pL+MEKIIJJkRwpry1mUCqZkpSO5IpuPewdLEJIQoEklmhLAmkwlCncBfD87yRX1b5mHZETKSSQhRJNJnRghr8vKCYUGQchkcZKHJfLKyUA4eRAcc9otgrIxkEkIUgdTMCGFtWbnLGcg8M/kdPYouI4MkByeu+gUR6uWsdURCiHJAkhkhrC1bljMoUG4T0xG/cCKDPNHrddrGI4QoFySZEcKatm6BD67DtylgL81M+Rw+DECsb5iMZBJCFJn0mRHCmq5fhSQFnGU5g9vq0oVlBy7xh1MonWUkkxCiiCSZEcKaUhLVvZ1OkpnbUB58kCkbMriWksloqZkRQhSRNDMJYU3JCereXg96+e/3bxcT07mWkolBr6OGv8zDI4QoGqmZEcKaUpLVvYNB2zjKokuXuPjrevyT0vGsFo6jvbxGQoiikZ+GQlhTapK6l2Qmvz//pNGgp/l4+XRqBUqtjBCi6CSZEcKaUtWVoHGQStF8ckcynfAOobo0MQkhikGSGSGsycUIvnrwlsng8jEnM8FU93PVOBghRHkiPw+FsKYeD0HqpxBYTetIyhzl8GF0wHGfYPpIzYwQohikZkYIazLP/isT5lnIzIQTJwA45R9GcCV5fYQQRSfJjBDWlJ2h7u1ljhkLJ06gy84mycEJl/AQ7Azy0SSEKDr5xBDCmj5cAJ8kw5YLWkdSttza+TdAJssTQhSP9JkRwpouxMMVE2RoHUgZc//9LBo6hU3nUomUzr9CiGKSZEYIa0pLV/eO0sxkoUoVFtdqxz63BB71l2RGCFE80swkhDWl51bJOEkH11spisLxeHV25Gp+MpJJCFE8kswIYU0ZucmMs8wzY5adTcKH/6X+ib0YdSZCZQ4eIUQxSTIjhDVlZKl7qZm56e+/8RzzCl//OJkwH1fsZSSTEKKY5FNDCGvKyFT3Ti7axlGW3DLzb7UAD42DEUKUR5LMCGFNlZzAQwceMvzYLDeZOe4dTDUZySSEuAuSzAhhTdEPwSg3qFdT60jKjryaGZ8QqstIJiHEXZBkRghrys4dmi0zAJsp5pqZEKrLSCYhxF2QZEYIa8rKW5tJkhkAcnLgyBEA/vYNIdxH+hIJIYpPJs0TwloUBd5cB0oqdMzSOpqy4dQpdOnpZBjsMVQNx8FOfl8JIYpPkhkhrCUjA86pE8PhKDUQAFSuzIpPlrDq9z1UDfDUOhohRDlVpn8GZWdn89ZbbxEeHo6TkxMRERFMmTIFk8mkdWhCFF9a2s3LLjKaCQBHR7YERPJbrQel868Q4q6V6ZqZadOmMXfuXBYsWECdOnXYtWsX/fv3x8PDg5dfflnr8IQonrxkRgc4S0fXPMcv5S1jIMmMEOLulOlkZtu2bXTv3p2oqCgAwsLCWLRoEbt27dI4MiHuQl4yY4eMZsqlzJhB3Q3/cDK0qYxkEkLctTLdzPTAAw/wxx9/cOzYMQD27dvH5s2b6dq1a4G3ycjIIDEx0WITokzIS2bsdWAnyxmgKCgTJzJp+Ux80xKI8JV+REKIu1Oma2bGjh1LQkICkZGRGAwGcnJyeOedd3jmmWcKvM3UqVOZPHmyFaMUooikZsbS+fPok5PJ1ulRqlbD0d6gdURCiHKqTNfMLFmyhO+++46FCxeyZ88eFixYwAcffMCCBQsKvE10dDQJCQnm7ezZs1aMWIhCmHLATQeuUjMDQG6N61lPf8KCKmkcjBCiPCvTNTNjxoxh3LhxPP300wDUq1eP06dPM3XqVPr27Xvb2xiNRoxGozXDFKJoGtWD0bn9QqRmBmJjAfjbqwrVpfOvEOIelOmamdTUVPR6yxANBoMMzRblU1b6zcsyA7A5mYmrFCTDsoUQ96RM18x069aNd955h5CQEOrUqcPevXuZMWMGAwYM0Do0IYovO7fPjN4e9NI/JK+Z6W/vKtznKyOZhBB3r0wnMx9//DHjx49n6NChxMfHExQUxODBg5kwYYLWoQlRfF8vgC+Tob58cQPkHD2KAYirVJmqfjKSSQhx98p0MuPm5sasWbOYNWuW1qEIce9OnIRzJojQOpCy4dCiX5k68xfia9TB2aFMfxQJIcq4Mt1nRogK5fo1de8iHdQBYvVubAutT0Cwn9ahCCHKOUlmhLCW6zfUvat0/gX4+0oKAOE+0sQkhLg3kswIYS03EtS9m8wxwy+/UP+T97jvzAEifGQkkxDi3kgyI4S13Lih7j09NA2jTFi+nC4rFtDizH5ZxkAIcc8kmRHCWhKS1L2Xl7ZxlAFK3rBsrypU9ZWaGSHEvZFkRghrcbIHB8DLW+tINGc6ehSAs75VCPKUZjchxL2RZEYIa/mkP0S7Q7VwrSPR1vXrGK5cAcBUrToGvU7jgIQQ5Z0kM0JYS/oNde/kqWUU2sttYrro6kVgFRmWLYS4d5LMCGEtaTfUvaOnllFoL29NJq/K0vlXCFEiJJkRwhoOHoQ3V8LPaVIzc/w4AH97VZY5ZoQQJULmEBfCGi5cgLgE8NNLzczkyXTPqMXl5Aw+lpFMQogSIMmMENZw/bq6d9TZfM1MSpaJfXoPcIeq0swkhCgBxWpmmj59OmlpaebjjRs3kpGRYT5OSkpi6NChJRedEBVFXjLjpLP5mpm43GUMvFwc8HR20DgaIURFUKxkJjo6mqSkJPPxI488wrlz58zHqampfPbZZyUXnRAVxVV1KDKONp7MnD9Ppf69GbllERHSX0YIUUKKlcwoilLosRCiAFcuqXtHHTja8HIGhw5Ree0Kuh/eIJ1/hRAlRkYzCWENV+LVvasRDDbcVc28jEFlIqTzrxCihEgyI4Q1KNnqUgbuNl4bkTvHzN8yx4wQogQV+yfil19+iaur+osqOzub+fPn4+PjA2DRn0YIcYuJQ6DqWvCL0DoSTSmxsehQk5mHJZkRQpSQYiUzISEhfPHFF+bjgIAAvv3223xlhBD/kjf7r3MlTcPQmuloLAbglHdlgr2ctQ5HCFFBFCuZOXXqVCmFIUQFl7cuky13/k1LQ3/2DAAZEdUw2hk0DkgIUVFInxkhrOGlafC/FEi11zoS7Zw9i8nOnkSjC5XCqmgdjRCiAilWMvPXX3+xatUqi3PffPMN4eHh+Pn58eKLL1pMoieEAEwm2HMCTuSAsw3XzNSowdQfdtLhhU+I8HPTOhohRAVSrGRm0qRJ7N+/33x84MABXnjhBdq3b8+4ceP49ddfmTp1aokHKUS5lpAAeVMy+QRoGorWTl5L55Kbj8wxI4QoUcVKZmJiYnj44YfNx4sXL+a+++7jiy++YPTo0Xz00Ud8//33JR6kEOVa3lIG9oC7j6ahaO3v3KUMZFi2EKIkFSuZuX79Ov7+/ubjDRs20LlzZ/Nxs2bNOHv2bMlFJ0RFcOsikza8lEHOgBcYPX8y1S+fpqpMmCeEKEHFSmb8/f2Ji4sDIDMzkz179tCiRQvz9UlJSdjb23AHRyFuR1bMVi3/he6HN+CuN+HnZtQ6GiFEBVKsZKZz586MGzeOTZs2ER0djbOzMw8++KD5+v3791O1atUSD1KIck1WzIbr1zFcvQqArmZ1dDqdxgEJISqSYs0z8/bbb/P444/Tpk0bXF1dmT9/Pg4ODubrv/76azp27FjiQQpRrmVmglGXm8zY6Gim48cBuOTqRVBlP42DEUJUNMVKZnx9fdm0aRMJCQm4urpiMFhOevXDDz/g5iZDLoWw8MzTcHwoKIrtNjPlLjAZVylIRjIJIUpcsZKZAQMGFKnc119/fVfBCFEhpSeoe50NNzNZrJYtyYwQomQVK5mZP38+oaGhNGrUCEVR7nwDIcTNZMbeGewcCi9bUeU2M8VVqsyjPjKSSQhRsoqVzAwZMoTFixfz999/M2DAAHr37o2Xl1dpxSZExTD5HfgtBR5y1zoSzWSlpILeQJxXZcJ8ZIFJIUTJKtZopjlz5nDhwgXGjh3Lr7/+SnBwME8++SS///671NQIUZDdMXAyBzJtdzjykTkLqDX6Jw7Vb4Gbo0zfIIQoWcVeaNJoNPLMM8+wdu1aDh8+TJ06dRg6dCihoaEkJyeXRoxClG/Xb6h7DxsdyQTEXUkh22BHcICn1qEIISqge1o1W6fTodPpUBQFk8lUUjEJUbEkJKr7SpW0jUNDf19WlzEI95bOv0KIklfsZCYjI4NFixbRoUMHatasyYEDB5g9ezZnzpzB1VU69gmRT2JujaW3t7ZxaGXhQh4b2pNBfy0lXEYyCSFKQbE6AA8dOpTFixcTEhJC//79Wbx4Md62+gEtRFGYTJCcpl728tU2Fq3ExBB24iBBHuEEyRwzQohSUKxkZu7cuYSEhBAeHs6GDRvYsGHDbcstXbq0RIITotxLTIS8vvG+AZqGohXl+HF0qBPmPSDJjBCiFBQrmXn++eetvqbKuXPnGDt2LKtWrSItLY0aNWrw1Vdf0aRJE6vGIcRdSUoCJ3vIzgJ3H62j0UTO0VjsgNNeQYR4y7BsIUTJK/akedZ0/fp1WrVqRbt27Vi1ahV+fn6cPHkST09Pq8YhxF0LDoa5XeHEn7a5lEFODvq/TwKQFl4Vo53hDjcQQojiK1YyY23Tpk0jODiYefPmmc+FhYVpF5AQdyP9BuhtdCmDM2fQZ2aSYbDDqVqE1tEIISqoexqaXdqWL19O06ZN6dWrF35+fjRq1Igvvvii0NtkZGSQmJhosQmhqbQb6t4Wa2ZylzE47RlEuJ/tzoAshChdZTqZ+fvvv/n000+pXr06v//+O0OGDGHkyJF88803Bd5m6tSpeHh4mLfg4GArRizEv/zyC8yJha0Z4GiDk+alpXHZtzInvavIatlCiFKjU8rwOgQODg40bdqUrVu3ms+NHDmSnTt3sm3bttveJiMjg4yMDPNxYmIiwcHBJCQk4O4uvwyFlb33HkRHQwN72Hga3AO1jsjqOszYwPFLSXzzwn20rmGjw9OFEMWWmJiIh4dHkb6/y3TNTGBgILVr17Y4V6tWLc6cOVPgbYxGI+7u7habEJq5ckndO+psspkpx6Rw+moq6HRSMyOEKDVlOplp1aoVsbGxFueOHTtGaGioRhEJUUxX4tW9sx3YO2kbiwbO30gjM8eEg52eIE/be/5CCOso08nMK6+8wvbt23n33Xc5ceIECxcu5PPPP2fYsGFahyZE0Vy7ou7dbbBWIjMTv8hwfvjudWq5gEFv3TmqhBC2o0wnM82aNePnn39m0aJF1K1bl//85z/MmjWL5557TuvQhCiaa1fVvbsNrlv2998YL8dT63IcAZVtc8JAIYR1lOl5ZgAeeeQRHnnkEa3DEOLu3Lih7j1tcCTTsWOAuoxBmK8NJnNCCKsp0zUzQpR7mZnqvpKnpmFoIi+Z8apMhHT+FUKUIklmhChN816D8W5QxwZnv82dMC+uUhDhPlIzI4QoPZLMCFGa0hPUpQxcvbSOxOpyckcixnlVlmHZQohSJcmMEKUp/Ya6t8F1mZRYtZnpkn8IPq4OGkcjhKjIynwHYCHKrfPn4c0loEuDTp5aR2Nd2dnciKhBZkoW1KiOTifDsoUQpUdqZoQoLZcuwb5zcCLb9mpm7Oz48d2vaDl0Pn4hAVpHI4So4CSZEaK0XL+u7p1scymDuMspANJfRghR6iSZEaK05CUzjjrbWzE7O5u4K5LMCCGsQ5IZIUrLrTUzttbM9NxzzBnzCI8c2UiEDMsWQpQySWaEKC3Xrql7G1wxO+foUXySr5Fq70iYj7PW4QghKjhJZoQoLXkrZjvaWM2MyYQud1j2jZAI3BztNQ5ICFHRSTIjRGlJzG1mctaDgw31GzlzBn1GOhkGO5yqV9U6GiGEDZB5ZoQoLZNHg98P4OgDtjTPSu7Mv6c9gwgPtLGOz0IITUjNjBClJe2GupSBWyWtI7Guo0cBOOldRTr/CiGsQpIZIUqLrS5lkFsz87dXZar6STIjhCh9kswIUVqGTYaf0yDVttYlyqleg+0h9TgQUI2qvjbUV0gIoRnpMyNEacjIgHW7QFHAxVPraKzqTJ9BPH2pOo72euZ4OGkdjhDCBkjNjBCl4exZNZGxB/xta22ik/HJAET4uKLX21DHZyGEZiSZEaI0xMWpe089ONtQB+DUVM6cVefXkf4yQghrkWRGiNJwazJjSx2Af/qJAZ3rM3vZe9JfRghhNZLMCFEazMmMjS1lkDuSKcHJlQhfqZkRQliHJDNClAZbrZnJnWPmb68qUjMjhLAaSWaEKA2pqaADKultqmYm+0juhHleMmGeEMJ6ZGi2EKXhl19gsj/kpINHsNbRWEdODvoTJwBICa+Kk4NB44CEELZCamaEKA3J8UAG2BnAvbLW0VjH6dPoMzPIMNjjXEMWmBRCWI8kM0KUhhtn1L1bENjZyAzAuf1l4ioFEeHvrnEwQghbIs1MQpS0devgtWHgmg79QrWOxnr8/Nj44KPszXGWkUxCCKuSZEaIknboEOw5ApF24BmidTTW07QpE7qN4tTVVBbKSCYhhBVJM5MQJe3WYdk2lMxkZOdw5loqANWkZkYIYUWSzAhR0iySGdtpZrqwdQ/2WZm4Ge3wdTNqHY4QwoZIMiNESctLZirpbKdm5sYNwto05/CMJ6jtoUenkwUmhRDWI8mMECXNFpuZcpcxuOziSeUqvhoHI4SwNZLMCFGSbtyAhAT1ciV725ljJjeZOeldharSX0YIYWWSzAhRki5fhkA/cNWBbxUw2MiAQVmTSQihIUlmhChJ1avD6o/hFVeb6vyrHD4MqGsySc2MEMLaJJkRoqTdOA16G+r8C+Ts2w/Acf8wQrydNY5GCGFrJJkRoqRdP63ubaVmJiUFu1Nqp+eU6rUw2skCk0II67KRBn0hrGTgQNj8M9yfbTs1M9nZ7Bk6liNb9+ETbiMdnoUQZYokM0KUpG3bIPYaNHO2nWTGw4Plnfow3+1BXpT+MkIIDZSrZqapU6ei0+kYNWqU1qEIkZ+iwKlT6mVPHVSykWYm4NilJACq+UkyI4SwvnKTzOzcuZPPP/+c+vXrax2KELd3+TKkqmsTUckB3AK1jcdaNm4k/eBh9KYcIgPctI5GCGGDykUyk5yczHPPPccXX3xBpUqVCi2bkZFBYmKixSaEVeTN/OumA+9g0NtGR1jTE71Y+tEL1Is/SXU/SWaEENZXLpKZYcOGERUVRfv27e9YdurUqXh4eJi34OBgK0QoBDebmCrZ0AKT8fHoL8djQkdmjUicHGwjgRNClC1lPplZvHgxe/bsYerUqUUqHx0dTUJCgnk7e/ZsKUcoRC5bXJPp4EEAzngGEBrsp3EwQghbVaZHM509e5aXX36ZNWvW4OjoWKTbGI1GjEZjKUcmxG3o9eDtApWybKdmJjeZifUNpab0lxFCaKRMJzO7d+8mPj6eJk2amM/l5OSwceNGZs+eTUZGBgaDVGuLMuL118FvI8RttJ2RTAcOAHDUN0w6/wohNFOmk5mHH36YA7kflnn69+9PZGQkY8eOlURGlD03ToPOdpYyUA4cQAcc8wmlmyQzQgiNlOlkxs3Njbp161qcc3FxwdvbO995ITSXkw0J59TLtpDMmEwoBw+hA+KCwgn1ltWyhRDaKPMdgIUoF86dg7Aw+CYR9PbgGqB1RKXPZGLff2Yyq9Uz2NesiUGv0zoiIYSNKtM1M7ezfv16rUMQIr+//4Z/zkGl3CYmvQ38TrCzY0O91sy6FMgTlQuf/0kIIUqTDXziCmEFJ06oe1salg3EXlSXMZDOv0IILZW7mhkhyqQtW9R9oMF2RjItX47nukN4uIbLsGwhhKakZkaIkrBhg7oPNdhMzYzpjTeZ+vUbND5/VJIZIYSmJJkR4l6dP682M+mAEDvbmDAvMxNd7FEALoVUx9dVJqoUQmhHkhkh7lVerUxlR3DU2UYyc+wYuuxsEh2c8agRgU4nI5mEENqRZEaIe+XuDm3bQriiHttCM1PuZJbHfEOpGeiucTBCCFsnyYwQ9yoqCn76Gh4ygp0juNrAgou5yUysb6iMZBJCaE6SGSFKwo3T6t4zRF3OoKLLXWDyqG+YdP4VQmhOhmYLcS/OnQOjEa7mzTNjA01MQM7+/RhQ12Sq4S/JjBBCW1IzI8S9ePdd8PWFmXPV48pNtY3HSg7MXsCoR14lIbIuLkb5TSSE0JYkM0Lci7yRTA4X1X1oS+1isaI9niEsq9OO4DAbWINKCFHmSTIjxN26cgUOHVIv+yWA3g6qNNM2JiuRZQyEEGWJ1A8Lcbc2blT3VauAcyIENQYHZ21jsobZs6m24QSBvg2pGdBI62hECcnJySErK0vrMIQNsbe3x2AwlMh9STIjxN3KW8G9pieQaDNNTMqHHzLo1CnWPfW21MxUAIqicPHiRW7cuKF1KMIGeXp6EhAQcM8Tb0oyI8Tdyusv45es7kNbaReLtcTHozt1CoCjVWoS5u2ibTzinuUlMn5+fjg7O8tszsIqFEUhNTWV+Ph4AAIDA+/p/iSZEeJuXLtmnjgO7yuAAULu0zQkq9i5E4ATXlWoEhaAnUG63ZVnOTk55kTG29tb63CEjXFycgIgPj4ePz+/e2pykk8iIe6GgwN8/TUM6AGuegioB44eWkdV+nbsAGBfUA0aBntqG4u4Z3l9ZJydbaCvlyiT8t5799pfS5IZIe6Gqyv06wePR6jHttDEBOZkZm9gTUlmKhBpWhJaKan3niQzQtyL01vVvS10/lUUlLyamUCpmRFClB2SzAhRXLt3w+uvw8nDEH9YPWcLycw//8CNG2QY7DgXUk06/wpRRKdOnUKn0xETE1NgmfXr16PT6crdqLK2bdsyatQorcOQZEaIYnvnHXj/fRjzsnrsUxNcfLSNyRqCg1m89gCP9/6AOmG+6PXSNCG0069fP3Q6HUOGDMl33dChQ9HpdPTr1898Lj4+nsGDBxMSEoLRaCQgIIBOnTqxbds2c5mwsDB0Ol2+7b333iswjrZt25rLOTg4ULVqVaKjo8nIyDCXCQ4O5sKFC9StW7dknvw96NevHz169Cix+1u6dCn/+c9/Suz+7paMZhKiOA4fhp9/VlfG7hIB53bYRq1Mrl1XsjgUUI2R0sQkyoDg4GAWL17MzJkzzSNj0tPTWbRoESEhlou+9uzZk6ysLBYsWEBERASXLl3ijz/+4Nq1axblpkyZwqBBgyzOubkVPp/SoEGDmDJlCpmZmezcuZP+/fsDMHXqVAAMBgMBAeVr6Y+srCzs7e3vWM7Ly8sK0dyZ1MwIURx5v9AeewyUWPWyrXT+BWLOXgegYYintoEIATRu3JiQkBCWLl1qPrd06VKCg4Np1Ojm7NQ3btxg8+bNTJs2jXbt2hEaGkrz5s2Jjo4mKirK4j7d3NwICAiw2FxcCm9SdXZ2JiAggJCQEHr27EmHDh1Ys2aN+frbNTOtXLmSGjVq4OTkRLt27TiVO3/Trb744guCg4NxdnbmscceY8aMGXh6elqU+fXXX2nSpAmOjo5EREQwefJksrOzbxvnpEmTWLBgAb/88ou5Nmn9+vXm+L7//nvatm2Lo6Mj3333HVevXuWZZ56hSpUqODs7U69ePRYtWmRxn/9uZgoLC+Pdd99lwIABuLm5ERISwueff17o61cSJJkRoqji4mDhQvXyqyPhwj71cmgL7WKylqwssh94kN6LZuCUmU6DKp5aRyRKiaIopGZma7IpilLsePv378+8efPMx19//TUDBgywKOPq6oqrqyvLli2zaP4pDfv27WPLli2F1mqcPXuWxx9/nK5duxITE8PAgQMZN26cRZktW7YwZMgQXn75ZWJiYujQoQPvvPOORZnff/+d3r17M3LkSA4fPsxnn33G/Pnz85XL89prr/Hkk0/SuXNnLly4wIULF2jZ8mbN8tixYxk5ciRHjhyhU6dOpKen06RJE1asWMHBgwd58cUX6dOnD3/99Vehr8GHH35I06ZN2bt3L0OHDuWll17i6NGjd3rp7ok0MwlRVO+/Dzk50LEj+GWBYgLPUPCoonVkpe/AAey2bOZxowvznhiJt6tR64hEKUnLyqH2hN81eezDUzrh7FC8r6U+ffoQHR1trl3YsmULixcvZn3eciOAnZ0d8+fPZ9CgQcydO5fGjRvTpk0bnn76aerXr29xf2PHjuWtt96yOLdixQratm1bYAxz5szhyy+/JCsri8zMTPR6PZ988kmB5T/99FMiIiKYOXMmOp2OmjVrcuDAAaZNm2Yu8/HHH9OlSxdee+01AGrUqMHWrVtZsWKFucw777zDuHHj6Nu3LwARERH85z//4fXXX2fixIn5HtfV1RUnJycyMjJu2+w1atQoHn/8cYtzeY8PMGLECFavXs0PP/zAffcVPElo165dGTp0KKC+njNnzmT9+vVERkYWeJt7JcmMEEVx4YI6SR7AG2/A6dylDGyliemWIdkNQmWmWFF2+Pj4EBUVxYIFC1AUhaioKHx88nfI79mzJ1FRUWzatIlt27axevVqpk+fzpdffmnRUXjMmDEWxwCVK1cuNIbnnnuON998k8TERKZNm4a7uzs9e/YssPyRI0e4//77LeZYadHCsoY3NjaWxx57zOJc8+bNLZKZ3bt3s3PnTouamJycHNLT00lNTS32ZIhNmza1OM7JyeG9995jyZIlnDt3joyMDDIyMu7Y7HZrgqjT6QgICDAvW1BaJJkRoigMBnjxRbUDcOvW8PW76nlb6fwr88vYDCd7A4endNLsse/GgAEDGD58OEChNSKOjo506NCBDh06MGHCBAYOHMjEiRMtkhcfHx+qVatWrMf38PAw3+a7776jTp06fPXVV7zwwgu3LV+U5jRFUfJNKPfv25lMJiZPnpyvNgXU51pc/05SPvzwQ2bOnMmsWbOoV68eLi4ujBo1iszMzELv599NbDqdDpPJVOx4ikOSGSGKws8PPvoIFAWun4J/1C93wmyjZkbZsQMdEBNUg6GSzFRoOp2u2E09WuvcubP5C7ZTp6InYrVr12bZsmUlGou9vT1vvPEG0dHRPPPMM7etHbnd427fvt3iODIykh25PyLy7Nq1y+K4cePGxMbGFiv5cnBwICcnp0hlN23aRPfu3enduzegJk/Hjx+nVq1aRX48a5EOwEIUh04H2z5R+8tUfQi8IrSOqPQlJak1UsChoBrUCXLXOCAhLBkMBo4cOcKRI0duu1jh1atXeeihh/juu+/Yv38/cXFx/PDDD0yfPp3u3btblE1KSuLixYsWW2JiYrHiefbZZ9HpdMyZM+e21w8ZMoSTJ08yevRoYmNjWbhwIfPnz7coM2LECFauXMmMGTM4fvw4n332GatWrbKorZkwYQLffPMNkyZN4tChQxw5coQlS5bk6/Nzq7CwMPbv309sbCxXrlwpdE2katWqsXbtWrZu3cqRI0cYPHgwFy9eLNZrYS2SzAhxJ7/+CitXwvXrkHIF9n6nnm81StOwrGbXLnSKwj/uvvhWD8PxLpsChChN7u7uuLvfPtF2dXXlvvvuY+bMmbRu3Zq6desyfvx4Bg0axOzZsy3KTpgwgcDAQIvt9ddfL1YsDg4ODB8+nOnTp5OcnJzv+pCQEH766Sd+/fVXGjRowNy5c3n33XctyrRq1Yq5c+cyY8YMGjRowOrVq3nllVcsmo86derEihUrWLt2Lc2aNeP+++9nxowZhIaGFhjboEGDqFmzJk2bNsXX15ctW7YUWHb8+PE0btyYTp060bZtWwICAkp0wr2SpFPuZixcOZKYmIiHhwcJCQkFvtGFKFS9enDwoJrUOO+DjdMhqBEMWqfW1FR0S5eSOHgYm7yrsv29ufynh/azmIqSkZ6eTlxcHOHh4XfVx0JY16BBgzh69CibNm3SOpQSU9h7sDjf3+WrYVQIa8vMhLz5EWpWhZ+GqZdbjbKNRAbg8ccZcCmAAycv8a70lxHCaj744AM6dOiAi4sLq1atYsGCBQU2Xdk6SWaEKExsLGRng4cHXP4T0m+AV1Wo1U3ryKwmK8fEwfMJZNg50ECSGSGsZseOHUyfPp2kpCQiIiL46KOPGDhwoNZhlUmSzAhRmAMH1H2dOrA99xdRyxGgt5F+I6mpxF5JJz3LhJujHRE+slK2ENby/fffax1CuSEdgIUoTF4yE+QEif+Aix80eEbbmKxpxgyq16vKCzuX0TDYU1bKFkKUSZLMCFGY/fvVvf6kur//JbC3oY6Sa9ZgTLxBqr2jTJYnhCizJJkRojB5NTPO8eDgBk0HFF6+IklMRNm2DYBNYQ1pHFJJ44CEEOL2JJkRojDLl8PoLhBggAZPg5On1hFZz/r16LKz+btSEPHeQdwX4aV1REIIcVtlOpmZOnUqzZo1w83NDT8/P3r06EFsbKzWYQlb0rAh1M4Gow6Cm2sdjXWtWQPApvBGNAuvVO6muBdC2I4yncxs2LCBYcOGsX37dtauXUt2djYdO3YkJSVF69CErTCZ4OJB9XJA/cLLVjR5yUxYY1pX99U4GCGEKFiZ/qm1evVqi+N58+bh5+fH7t27ad26tUZRCZuxcCGcPgqXksDPGbyLt5JuuRYXB8ePk6U3sD2kHq/WkGRGiLt16tQpwsPD2bt3Lw0bNrxtmfXr19OuXTuuX7+Op6enVeOrCMp0zcy/JSQkAODlVXDbfUZGBomJiRabEHfl00/hjf/APzngXwcMZTr3L1kODpwZOpol9Tvi7FOJyAA3rSMSwkK/fv3Q6XQMGTIk33VDhw5Fp9PRr18/87n4+HgGDx5MSEgIRqORgIAAOnXqxLbcTu6gLsKo0+nybe+9916BcbRt29ZczsHBgapVqxIdHU1GRoa5THBwMBcuXKBuXW2XAino+eVtbdu2vev7btu2LaNGjSqxWIur3Hw6K4rC6NGjeeCBBwp9Q0ydOpXJkydbMTJRISnKzZFM/noIqKdtPNZWuTLfRQ3kc7e/eaKGr8VKvUKUFcHBwSxevJiZM2fi5OQEqGv9LFq0iJCQEIuyPXv2JCsriwULFhAREcGlS5f4448/uHbtmkW5KVOmMGjQIItzbm6FJ/ODBg1iypQpZGZmsnPnTvr37w+o30egruodEBBwT8+1JOzcuZOcnBwAtm7dSs+ePYmNjTWve+Tg4KBlePek3NTMDB8+nP3797No0aJCy0VHR5OQkGDezp49a6UIRYVy9iwkJIBBBz42mMwAG49dBqC1NDHZppSUgrf09KKXTUsrWtm70LhxY0JCQli6dKn53NKlSwkODqZRo0bmczdu3GDz5s1MmzaNdu3aERoaSvPmzYmOjiYqKsriPt3c3AgICLDYXFwKn/na2dmZgIAAQkJC6NmzJx06dGBNbp8zUJuZdDodMTEx5nMrV66kRo0aODk50a5dO06dOpXvfr/44guCg4NxdnbmscceY8aMGfmaoH799VeaNGmCo6MjERERTJ48mezs7NvG6evra35OeS0cfn5+5nNHjx6ldevWODk5ERwczMiRIy36qM6ZM4fq1avj6OiIv78/TzzxBKDWkm3YsIH//ve/5lqe2z2f0lQukpkRI0awfPly1q1bR5UqVQotazQazUvBF7YkvBCFyquV8bVXE5qABtrGY00nTnBj8Y/8c/oSOh08WM1H64iEFlxdC9569rQs6+dXcNkuXSzLhoXdvtxd6t+/P/PmzTMff/311wwYYDkflKurK66urixbtsyi+ac07Nu3jy1btmBvb19gmbNnz/L444/TtWtXYmJiGDhwIOPGjbMos2XLFoYMGcLLL79MTEwMHTp04J133rEo8/vvv9O7d29GjhzJ4cOH+eyzz5g/f36+ckVx4MABOnXqxOOPP87+/ftZsmQJmzdvZvjw4QDs2rWLkSNHMmXKFGJjY1m9erW57+p///tfWrRowaBBg7hw4QIXLlwgODi42DHcE6UMM5lMyrBhw5SgoCDl2LFjd3UfCQkJCqAkJCSUcHSiQps6VVFAUeraKcpED0XJSNY6IuuZOFFRQPm15gPKo7M3ax2NKEVpaWnK4cOHlbS0tPxXqo2tt9+6drUs6+xccNk2bSzL+vjcvlwx9e3bV+nevbty+fJlxWg0KnFxccqpU6cUR0dH5fLly0r37t2Vvn37msv/+OOPSqVKlRRHR0elZcuWSnR0tLJv3z6L+wwNDVUcHBwUFxcXi23dunUFxtGmTRvF3t5ecXFxURwcHBRA0ev1yo8//mguExcXpwDK3r17FUVRlOjoaKVWrVqKyWQylxk7dqwCKNevX1cURVGeeuopJSoqyuKxnnvuOcXDw8N8/OCDDyrvvvuuRZlvv/1WCQwMvOPrt27dOovH69Onj/Liiy9alNm0aZOi1+uVtLQ05aefflLc3d2VxMTEAl+Hl19++Y6P+2+FvQeL8/1dpvvMDBs2jIULF/LLL7/g5ubGxYsXAfDw8DC3jwpRKvJqZvwM6igmBxtaYPGW+WXaVJdaGZuVnFzwdYZ/LbQaH19wWf2/GgBKuPnBx8eHqKgoFixYgKIoREVF4eOT/33bs2dPoqKi2LRpE9u2bWP16tVMnz6dL7/80qKj8JgxYyyOASpXrlxoDM899xxvvvkmiYmJTJs2DXd3d3r+u/bqFkeOHOH++++36IvWokULizKxsbE89thjFueaN2/OihUrzMe7d+9m586dFjUxOTk5pKenk5qairOzc6Fx32r37t2cOHGC//3vf+ZziqJgMpmIi4ujQ4cOhIaGEhERQefOnencuTOPPfZYsR6jNJXpZObTTz8FyNfDet68efnebEKUqFs7/wba0PwyN26g/PUXOtRk5iPpL2O77tBPxCpli2jAgAHm5pBPPvmkwHKOjo506NCBDh06MGHCBAYOHMjEiRMtvk98fHyoVq140zB4eHiYb/Pdd99Rp04dvvrqK1544YXbllcU5Y73qShKvo73/76dyWRi8uTJPP744/lu7+hYvDXkTCYTgwcPZuTIkfmuCwkJwcHBgT179rB+/XrWrFnDhAkTmDRpEjt37iwTQ8nLdDJTlD+4EKVi82b46BlI3mRbnX9XrkRnMnHCqwpJfkGyuKQoFzp37kxmZiYAnTp1KvLtateuzbJly0o0Fnt7e9544w2io6N55plnbltzcbvH3b59u8VxZGQkO3bssDi3a9cui+PGjRsTGxtb7OTrdho3bsyhQ4cKvS87Ozvat29P+/btmThxIp6envz55588/vjjODg4mEdKaaFcdAAWwurc3cHjIjjqbGvm3wULAPgt8kEeqOaDnUE+IkTZZzAYOHLkCEeOHMHw7yYw4OrVqzz00EN899137N+/n7i4OH744QemT59O9+7dLcomJSVx8eJFi62485U9++yz6HQ65syZc9vrhwwZwsmTJxk9ejSxsbEsXLiQ+fPnW5QZMWIEK1euZMaMGRw/fpzPPvuMVatWWdTWTJgwgW+++YZJkyZx6NAhjhw5wpIlS3jrrbeKFS/A2LFj2bZtG8OGDSMmJobjx4+zfPlyRowYAcCKFSv46KOPiImJ4fTp03zzzTeYTCZq1qwJqHPY/PXXX5w6dYorV65gMpmKHcO9kE8qIW4nIxmunlQv20rNzD//wNq1APxU9yEZki3KlcJGr7q6unLfffcxc+ZMWrduTd26dRk/fjyDBg1i9uzZFmUnTJhAYGCgxfb6668XKxYHBweGDx/O9OnTSb5N36OQkBB++uknfv31Vxo0aMDcuXN59913Lcq0atWKuXPnMmPGDBo0aMDq1at55ZVXLJqPOnXqxIoVK1i7di3NmjXj/vvvZ8aMGYSGhhYrXoD69euzYcMGjh8/zoMPPkijRo0YP348gYGBAHh6erJ06VIeeughatWqxdy5c1m0aBF16tQB4LXXXsNgMFC7dm18fX05c+ZMsWO4FzqlgrflJCYm4uHhQUJCggzTFkXz1VeweQ3oVkDtKvCajSxu+s030LcvfwXX5aln32PLuIeo7Ckd7Suy9PR04uLiCA8PL3YfC2F9gwYN4ujRo2zatEnrUEpMYe/B4nx/l+k+M0Jo4uef4bffoKsjPGxDTUzPP88azwj+u3Q3VX1dJJERQmMffPABHTp0wMXFhVWrVrFgwYICm65snSQzQvybeVi27c38O++iHYcCqjG6YeFDUYUQpW/Hjh1Mnz6dpKQkIiIi+Oijjxg4cKDWYZVJkswIcauLFyGvrdfPYDvJTHo6Z1JMbPv7Kjod9GxS+EzbQojS9/3332sdQrkhHYCFuNUvv6j7KnbgZCMjmdLTISSEzG6P4pmWyAPVfKSJSQhRrkgyI8Stfv5Z3dc0gIMbVArXNh5rWL4cLl/G9fB+EhxdebKplddUEUKIeyTJjBB5btyAP/5QL9eyg4C6+adir4hy57f4ofZDuDsb6VDbX9t4hBCimKTPjBB5Tp2C8HDIuA7embbRX+bCBfj9dwCW1n2IHg2DcLTPP+mYEEKUZTbws1OIImrYEGJj4fUm6rEtJDPffQcmE7ur1CLOqzK9pIlJCFEOSTIjxL8l506SV9E7/2Znw+efA/BDnYepHehO3coeGgclhBDFJ8mMEADx8ZCRARf2Qdo1sHME30itoypdCxfCiRMkuHjwa63WPNlUhmMLIYpu0qRJNGzYUOswAElmhFC99hr4+sLMCepxjc5gX8Gnd3/ySS5Mmso7D/Yly9mV7jJRnihH+vXrh06nY8iQIfmuGzp0KDqdjn79+pnPxcfHM3jwYEJCQjAajQQEBNCpUye2bdtmLhMWFoZOp8u3vffeewXG0bZtW0aNGlWST63UzJ8/H09PzxK7v9dee40/8gZNaEw6AAuRmQm//gpJSZAWAy5AvV5aR1X6HB35rFE3vk87RVQdfyq5OGgdkRDFEhwczOLFi5k5cyZOTurcSOnp6SxatIiQkBCLsj179iQrK4sFCxYQERHBpUuX+OOPP7h27ZpFuSlTpjBo0CCLc25ubqX7RMqYzMxMHBzu/Hng6uqKq6urFSK6M6mZEWL9enVYto8XeN0ARw+o3kHjoEpRejpkZxOflM73u84CyNwy4iZFgcwUbbZirnvcuHFjQkJCWLp0qfnc0qVLCQ4OplGjRuZzN27cYPPmzUybNo127doRGhpK8+bNiY6OJioqyuI+3dzcCAgIsNhcXFzu+uUcO3YsNWrUwNnZmYiICMaPH09WVpb5+n379tGuXTvc3Nxwd3enSZMm7Nq1C4DTp0/TrVs3KlWqhIuLC3Xq1GHlypXm227YsIHmzZtjNBoJDAxk3LhxZGdn3zaO9evX079/fxISEsw1TpMmTQLUGqm3336bfv364eHhYU7m7hT7v5uZ+vXrR48ePfjggw8IDAzE29ubYcOGWdymtEjNjBB5E+U1CQT9WajdHeyM2sZUmqZPh4UL+e3JUaRmBtMg2JPW1X20jkqUFVmp8G6QNo/9xnlwKF7i0L9/f+bNm8dzzz0HwNdff82AAQNYv369uUxeDcKyZcu4//77MRqt9//bzc2N+fPnExQUxIEDBxg0aBBubm68/vrrADz33HM0atSITz/9FIPBQExMDPb29gAMGzaMzMxMNm7ciIuLC4cPHzbXhJw7d46uXbvSr18/vvnmG44ePcqgQYNwdHQ0Jym3atmyJbNmzWLChAnExqqDHG6tVXn//fcZP348b731VpFjv51169YRGBjIunXrOHHiBE899RQNGzbMV9tV0qRmRti2nJybyUzQJXVfkZuYrlyBDz6A2Fj27j8FwJtda6HT6bSNS4i71KdPHzZv3sypU6c4ffo0W7ZsoXfv3hZl7OzsmD9/PgsWLMDT05NWrVrxxhtvsH///nz3N3bsWHPyk7fdmhgV11tvvUXLli0JCwujW7duvPrqqxZrLp05c4b27dsTGRlJ9erV6dWrFw0aNDBf16pVK+rVq0dERASPPPIIrVu3BmDOnDkEBwcze/ZsIiMj6dGjB5MnT+bDDz/EZDLli8PBwQEPDw90Op25xunWZOahhx7itddeo1q1alSrVq1Isd9OpUqVzDE98sgjREVFWaVfjdTMCNu2fTtcugRuLlA5A9wqQ2grraMqPdOmQVISZ0Jq8GvNVnSo7U/zcC+toxJlib2zWkOi1WMXk4+PD1FRUSxYsABFUYiKisLHJ39NY8+ePYmKimLTpk1s27aN1atXM336dL788kuLjsJjxoyxOAaoXPnuO8f/+OOPzJo1ixMnTpCcnEx2djbu7u7m60ePHs3AgQP59ttvad++Pb169aJq1aoAjBw5kpdeeok1a9bQvn17evbsSf366pQRR44coUWLFhY/RFq1akVycjL//PNPvj5Dd9K0adNix347derUwWC4OfFmYGAgBw4cKFYsd0NqZoRt+/FHdd/ADww6qPs46CvoDLj//AOzZwMwofmz6A0Gxnau4MPPRfHpdGpTjxbbXdYQDhgwwFzzMmDAgALLOTo60qFDByZMmMDWrVvp168fEydOtCjj4+Njrp3I2/I6FxfX9u3befrpp+nSpQsrVqxg7969vPnmm2RmZprLTJo0iUOHDhEVFcWff/5J7dq1+Tm3tnjgwIH8/fff9OnThwMHDtC0aVM+/vhjABRFyVejquT2ObqbmtZ/9wsqSuy3k9dElken0922pqikSTIjbNvEiTDsJahxXT2uqE1MOTnw/POQns6hqg1YH9GEp5oFU82vbIxEEOJedO7cmczMTDIzM+nUqVORb1e7dm1SUlJKLa4tW7YQGhrKm2++SdOmTalevTqnT5/OV65GjRq88sorrFmzhscff5x58+aZrwsODmbIkCEsXbqUV199lS+++MIc+9atW80JDMDWrVtxc3MrsCbJwcGBnJycEo29rJBmJmHbPD1h4AOw7H/gUwMCG2gdUen4z39g3TqynZwZ2W4IzkY7RrWvrnVUQpQIg8HAkSNHzJf/7erVq/Tq1YsBAwZQv3593Nzc2LVrF9OnT6d79+4WZZOSkrh48aLFOWdn50KbVy5fvkxMTIzFuYCAAKpVq8aZM2dYvHgxzZo147fffjPXugCkpaUxZswYnnjiCcLDw/nnn3/YuXMnPXv2BGDUqFF06dKFGjVqcP36df78809q1aoFqHPpzJo1ixEjRjB8+HBiY2OZOHEio0ePRl/AArlhYWEkJyfzxx9/0KBBA5ydnXF2vn3T3p1iL2ukZkbYpnPnbg4DPfCDuq/X666rucs0RYGTJwGY2n0UJ72DebF1BH5uFXxSQGFT3N3dC0w4XF1due+++5g5cyatW7embt26jB8/nkGDBjE7t+k1z4QJEwgMDLTYChu9A7Bw4UIaNWpksc2dO5fu3bvzyiuvMHz4cBo2bMjWrVsZP368+XYGg4GrV6/y/PPPU6NGDZ588km6dOnC5MmTAcjJyWHYsGHUqlWLzp07U7NmTebMmQOo/XhWrlzJjh07aNCgAUOGDOGFF16wGI30by1btmTIkCE89dRT+Pr6Mn369ALL3in2skanKMUc2F/OJCYm4uHhQUJCwh07LgkbkZoKkZFQtSrMnQmL24FighF7wLuq1tGVDkVh0QffEn3VGx9XIxvGtMXFKBWzti49PZ24uDjCw8NxdJTkVlhfYe/B4nx/y6eZsD3TpsHZs2otzLk/1USmctOKl8jk5IBeDzodqw5eJPqqNwATutWWREYIUaFIM5OwLXFxajID8PZb8Ncs9XLDZzQLqdRMmgQ9enAi9gyv/rAPgBceCOfRBhpNiCaEEKVEfp4J2/Laa+rq2A89BKyCjAQIagyN+2kdWclRFLXD79tvA/CdeyNSKzejRYQ30V1kKLYQouKRmhlhGxQF3ngDli4FgwFefAhO/gEGIzw2FwwVJK9XFHj1VXXIObC0x4vMr9yMyp5OzH62EXYG+S8vhKh4KsgnuBB3MH06TJ2qXp7yBpz8VL388HjwraldXCUpJwcGD4avvgLg/4a8wWiPlhjt9HzWpwnerhV4vSkhhE2Tn2nCNvTtCzVqwJw5UHkvZCZB8P1w/1CtIysZGRnwzDPw1Vcoej0/jPgPAz1aAjD18XrUreyhcYBCCFF6JJkRFdetU2gHBMC+fdDEDuI2qmvA9JhTcZYuSEqCDRtQ7O35aPA7jHFuhE4H4x+pzeONq2gdnRBClCpJZkTFoyjqmkt168I339w8f2IlrMmd9KnDlPI/FDsr6+bEfz4+XJ41h+gBU5npXg8newOf92nKCw+EaxujEEJYgSQzouJQFFizBpo1g1694MgReP11uHwJVo2FH/tDdhrU6AJNX9A62nuzdy80bQpLlgDw59FLdDnpzmLPSPzdjfwwpAUdavtrHKQQQliHdAAW5V9GBvzyC3zyCWzcqJ5zdYXRo2Hg0/DLc/DPTvX8A6Oh3ZvqZHLl0e7d8M47kLtGSub4CbyUGsYfx64CUCvQna/7NSXQ4+5W+RVCiPKonH6iC3GL6Gh46ik1kXFwgFGjIPYQ9KgOi6LURMbRA55ZDO0nls9h2Js3Q5cuam1MbiJzuE1XWnedyB/HrmJv0DG4dQQ/DmkhiYywCf369UOn0zFkyJB81w0dOhSdTke/fv3M5+Lj4xk8eDAhISEYjUYCAgLo1KkT27ZtM5cJCwtDp9Pl2957770C42jbti2jRo0qyadWKvJer8K2uzVp0iQaNmxYcsHehXL4qS5s3vHj4OamduoF6N0bvv8e+vWDng/DxdXwzQOQnqBeH9gAei0Ar3Laf2TiRJgyBQDFYOBg665Mqt2N3a7qTL5tavgyoVttqvq6ahmlEFYXHBzM4sWLmTlzJk5OahKfnp7OokWLCAkJsSjbs2dPsrKyWLBgAREREVy6dIk//viDa9euWZSbMmUKgwYNsjjn5uZWuk/ECv773/9aJGWBgYHMmzePzp07axhVyZGaGVF+pKerU/TXras2IQFkZ4J3OiwYCmHb4Zce8NdcNZHxDIGHJ8KANeUjkcnKgr/+gvfeg5gY8+m0Vq3JsXfg/x7sTuuBn9Gt+WB2uwYR5u3MV32bMr9/M0lkRMlLSSl4S08vetm0tKKVvQuNGzcmJCSEpUuXms8tXbqU4OBgGjVqZD5348YNNm/ezLRp02jXrh2hoaE0b96c6OhooqKiLO7Tzc2NgIAAi83FxeWu4gMYO3YsNWrUwNnZmYiICMaPH09WVpb5+n379tGuXTvc3Nxwd3enSZMm7Nq1C4DTp0/TrVs3KlWqhIuLC3Xq1GHlypXm227YsIHmzZtjNBoJDAxk3LhxZGdn3zYODw8Pi+cE4OnpaT7OycnhqaeeolKlSnh7e9O9e3dOnTplvv369etp3rw5Li4ueHp60qpVK06fPs38+fOZPHky+/btM9fwzJ8//65fr7slNTOi7Mvr2Dt8OJw4oZ6L2wdfPwoXdkJW6s2yOgPU7AJN+0PEQ2W3b4yiwPnzcOCA2g9mwwbYutX8oX7lWBxL+oxh47HLxJxKxTj0GxIdXdHroH2kH880D6FtTT8M+ruvGhaiUK6FJMhdu8Jvv9089vNTV6O/nTZtYP36m8dhYXDlSv5yeSPziql///7MmzeP5557DoCvv/6aAQMGsP6Wx3R1dcXV1ZVly5Zx//33YzRabwJJNzc35s+fT1BQEAcOHGDQoEG4ubnx+uuvA/Dcc8/RqFEjPv30UwwGAzExMdjb2wMwbNgwMjMz2bhxIy4uLhw+fBjX3L/LuXPn6Nq1K/369eObb77h6NGjDBo0CEdHRyZNmlSsGFNTU2nXrh0PPvggGzduxM7OjrfffpvOnTuzf/9+9Ho9PXr0YNCgQSxatIjMzEx27NiBTqfjqaee4uDBg6xevZr/+7//A9TEydokmRFl15EjMG8e/PQT/P23es7TETraQeRZOPOPes7FF8IehPDWUKMzuAdqF/OtsrLg4kU4fVrdfHygUyf1ukuXoEr++V+SnN3YEVyX5Zfd+OX32NyzevyCfOnfqApPNQsmyFP6xAiRp0+fPkRHR3Pq1Cl0Oh1btmxh8eLFFsmMnZ0d8+fPZ9CgQcydO5fGjRvTpk0bnn76aerXr29xf2PHjuWtt96yOLdixQratm17V/Hdel9hYWG8+uqrLFmyxJzMnDlzhjFjxhAZqa6bVr16dXP5M2fO0LNnT+rVqwdARESE+bo5c+YQHBzM7Nmz0el0REZGcv78ecaOHcuECRPQF+OH3OLFi9Hr9Xz55ZfmvjPz5s3D09OT9evX07RpUxISEnjkkUeoWlWd0qJWrVrm27u6umJnZ2eu8dFCuUhm5syZw/vvv8+FCxeoU6cOs2bN4sEHH9Q6LFFSFAXOnoVDh6BqVQirAldPwI9z4P3P1DJ2QBMHaGcPRh0ENYK6T0C1h8E3Eu6h89odY4uPV2tMkpIgMVHdkpLgxg0ICoJHHyU7x0R6WgYOnTtCfDz6y5exu3Hd4q6ON2/LZ4kBXExI5/z1VJa4VOKGoyuxvmHsqFKbHcF1ifUNRdHpcTPa0amaNw9U96V1dR9Cve++mluIu5KcXPB1hn9NNhkfX3DZf3+p3tJ0URJ8fHyIiopiwYIFKIpCVFQUPj4++cr17NmTqKgoNm3axLZt21i9ejXTp0/nyy+/tOgoPGbMGItjgMqVK991fD/++COzZs3ixIkTJCcnk52djbu7u/n60aNHM3DgQL799lvat29Pr169zAnDyJEjeemll1izZg3t27enZ8+e5uTryJEjtGjRwqLjbqtWrUhOTuaff/7J12eoMLt37+bEiRP5+galp6dz8uRJOnbsSL9+/ejUqRMdOnSgffv2PPnkkwQGlpEfjpSDZGbJkiWMGjWKOXPm0KpVKz777DO6dOnC4cOHi/XHKmkJ16+QevQAhvMX/nXNzarS7PBw8PQEQHf5MoZ//inw/nKCQ1B8vNWyV65iOHum4LJVqqD4+qplr13HcPpUwWWDKqP4+6kHN25g//ffljPj3lq2chCm3MxaSUzA4WhsbtWvSd2bFFAUdIqJnKBAckKCQTGhS7iBcfceyMpGl50NWZno8rbMDLKrBZNVpyq6nEwMFy7i8sUP6K8noE9IRp+Qgi4hFV1GDgCmh93QP5D7nzNNgbp2EGkP1e3IcvEjwfMBEl1bkKH3Iz3Jn5QEX3JuXEFJz8D79xWQlYmSmQmZmeqQ7cxMdBkZXAurzt9tOpOdo5CTnka798ZiyMhAn5GOISMdu/Q07DLSsU9PI7be/Xz3wptkZZvIyspi3sCWBb6+G6s144W/7MjKUf/uB3fuxjXzZh+BTL0dF9x9OOfux067QH7cffM9cN/Q+Zj0BgLcHakZ4EabQDdeCnQnMsCdqr4usiik0FZx+omUVtkiGjBgAMOHDwfgk08+KbCco6MjHTp0oEOHDkyYMIGBAwcyceJEi+TFx8eHatWqlUhc27dv5+mnn2by5Ml06tQJDw8PFi9ezIcffmguM2nSJJ599ll+++03Vq1axcSJE1m8eDGPPfYYAwcOpFOnTvz222+sWbOGqVOn8uGHHzJixAgURck3AknJbaor7sgkk8lEkyZN+N///pfvOt/c75p58+YxcuRIVq9ezZIlS3jrrbdYu3Yt999/f3FfllJR5pOZGTNm8MILLzBw4EAAZs2axe+//86nn37K1LyFA2+RkZFBRkaG+TgxMbFU4jq8fCYtFs+A1ekFF3rOGarlvsR7MuHXQsr2coLaajspB7Pgp7SCy/ZwhAYO6uXYLFhcSNkoR2iaW/bvbPi2gHZtgI5GaJHblnw2G74upGxbI7TJLXsxBz4rpAPfAw7wsKN6+WoOrL1NWT3grUdvlwkYSVScOWaoTNOggxCbDRsysL98Eh9O4sMCABbX78i4LiMBcMtI4cCslwoMYXmt1kxOVJNfgymHIet+K7Bs6umz/Lb/ZpKaZmdE0UGygzPJDk4kG51JcXAiyejCvoDq5kQGYFS31zC5uJJeyZt0Lx/w8sLT1ZFKzg5UcrZnjIsDAe6OBHk6UdnTCX8PI0a7CrKkghAa6dy5M5mZmQB0ymvKLYLatWuzbNmyUooKtmzZQmhoKG+++ab53OnTp/OVq1GjBjVq1OCVV17hmWeeYd68eTz22GOAOmJryJAhDBkyhOjoaL744gtGjBhB7dq1+emnnyySmq1bt+Lm5lbsmqTGjRuzZMkS/Pz8LGqN/q1Ro0Y0atSI6OhoWrRowcKFC7n//vtxcHAgJyenWI9Z0sp0MpOZmcnu3bsZN26cxfmOHTuydevW295m6tSpTJ48udRj09k5kIUeu0JewSzFQJbiiIIOIzmFls1U7MhS1L4QDooJ+8LuFzsyc8vaAw52BScz2RjIMJdNwyHvfm+TuGdjIE1xVsvqMnB0SLUsa97ryNLbkaR4kIMee30Gnu6p6vV6HRhQkxO9epxkdOU41cnEDnv7TBrXjkGx02My6FB06mbK0ZGTpmet0ppZhoGk4IJXVgLfbX8Jk04HOh05XgZMegMmvZ4cvYFsP39q+ruh1+twyjayr0ZjcuzszZvJPveyvZEbVWvzSP1A7PQ67Ax6fu3/OoqjEZPRCcXRiOLkguLsjOLsRLaXD5OCQ3GwM2Bv0LHmyePY29thtNPjYKfHwaDHyd6Ah0FPdQcDve31ONoZcLQ34Gjf9Z7maxBCFJ/BYODIkSPmy/929epVevXqxYABA6hfvz5ubm7s2rWL6dOn0717d4uySUlJXLx40eKcs7NzoV/yly9fJuaWEYgAAQEBVKtWjTNnzrB48WKaNWvGb7/9xs+580QBpKWlMWbMGJ544gnCw8P5559/2LlzJz179gRg1KhRdOnShRo1anD9+nX+/PNPc1+VoUOHMmvWLEaMGMHw4cOJjY1l4sSJjB49ulj9ZUDthPz+++/TvXt3pkyZQpUqVThz5gxLly5lzJgxZGVl8fnnn/Poo48SFBREbGwsx44d4/nnnwfUvkBxcXHExMRQpUoV3NzcrNrJGgClDDt37pwCKFu2bLE4/8477yg1atS47W3S09OVhIQE83b27FkFUBISEqwRshBClBtpaWnK4cOHlbS0NK1DKba+ffsq3bt3L/D67t27K3379lUURf1eGDdunNK4cWPFw8NDcXZ2VmrWrKm89dZbSmpqqvk2oaGhCmpfAYtt8ODBBT5OmzZtbnubiRMnKoqiKGPGjFG8vb0VV1dX5amnnlJmzpypeHh4KIqiKBkZGcrTTz+tBAcHKw4ODkpQUJAyfPhw899j+PDhStWqVRWj0aj4+voqffr0Ua5cuWJ+7PXr1yvNmjVTHBwclICAAGXs2LFKVlZWkV4/QPn555/NxxcuXFCef/55xcfHRzEajUpERIQyaNAgJSEhQbl48aLSo0cPJTAwUHFwcFBCQ0OVCRMmKDk5OebXt2fPnoqnp6cCKPPmzStSDIpS+HswISGhyN/futwnVSadP3+eypUrs3XrVlq0aGE+/8477/Dtt99y9OjRO95HYmIiHh4eJCQkFJpZCyGErUlPTycuLo7w8HAcHR21DkfYoMLeg8X5/i7TPQx9fHwwGAz5qvzi4+Px95dF9IQQQghRxpMZBwcHmjRpwtq1ay3Or127lpYtCx5hIoQQQgjbUaY7AIM6Br9Pnz40bdqUFi1a8Pnnn3PmzJnbLi4mhBBCCNtT5pOZp556iqtXrzJlyhQuXLhA3bp1WblyJaGhoVqHJoQQQogyoMwnM6AOQRs6dKjWYQghRIVUhseBiAqupN57ZbrPjBBCiNKTt6BhakGLRApRyvLee3nvxbtVLmpmhBBClDyDwYCnpyfxuWsrOTs7y6SPwioURSE1NZX4+Hg8PT1vO9lhcUgyI4QQNixvpeP4whaLFKKUeHp6lshq25LMCCGEDdPpdAQGBuLn50dWVpbW4QgbYm9vf881MnkkmRFCCIHBYCixLxYhrE06AAshhBCiXJNkRgghhBDlmiQzQgghhCjXKnyfmbwJeRITEzWORAghhBBFlfe9XZSJ9Sp8MpOUlARAcHCwxpEIIYQQoriSkpLw8PAotIxOqeDzWJtMJs6fP4+bm1uJTwaVmJhIcHAwZ8+exd3dvUTvuzyQ52/bzx/kNbD15w/yGsjzL73nrygKSUlJBAUFodcX3iumwtfM6PV6qlSpUqqP4e7ubpNv4jzy/G37+YO8Brb+/EFeA3n+pfP871Qjk0c6AAshhBCiXJNkRgghhBDlmiQz98BoNDJx4kSMRqPWoWhCnr9tP3+Q18DWnz/IayDPv2w8/wrfAVgIIYQQFZvUzAghhBCiXJNkRgghhBDlmiQzQgghhCjXJJkRQgghRLkmyUwJy8jIoGHDhuh0OmJiYrQOxypOnTrFCy+8QHh4OE5OTlStWpWJEyeSmZmpdWilas6cOYSHh+Po6EiTJk3YtGmT1iFZxdSpU2nWrBlubm74+fnRo0cPYmNjtQ5LM1OnTkWn0zFq1CitQ7Gqc+fO0bt3b7y9vXF2dqZhw4bs3r1b67CsJjs7m7feesv8uRcREcGUKVMwmUxah1YqNm7cSLdu3QgKCkKn07Fs2TKL6xVFYdKkSQQFBeHk5ETbtm05dOiQ1eKTZKaEvf766wQFBWkdhlUdPXoUk8nEZ599xqFDh5g5cyZz587ljTfe0Dq0UrNkyRJGjRrFm2++yd69e3nwwQfp0qULZ86c0Tq0UrdhwwaGDRvG9u3bWbt2LdnZ2XTs2JGUlBStQ7O6nTt38vnnn1O/fn2tQ7Gq69ev06pVK+zt7Vm1ahWHDx/mww8/xNPTU+vQrGbatGnMnTuX2bNnc+TIEaZPn87777/Pxx9/rHVopSIlJYUGDRowe/bs214/ffp0ZsyYwezZs9m5cycBAQF06NDBvD5iqVNEiVm5cqUSGRmpHDp0SAGUvXv3ah2SZqZPn66Eh4drHUapad68uTJkyBCLc5GRkcq4ceM0ikg78fHxCqBs2LBB61CsKikpSalevbqydu1apU2bNsrLL7+sdUhWM3bsWOWBBx7QOgxNRUVFKQMGDLA49/jjjyu9e/fWKCLrAZSff/7ZfGwymZSAgADlvffeM59LT09XPDw8lLlz51olJqmZKSGXLl1i0KBBfPvttzg7O2sdjuYSEhLw8vLSOoxSkZmZye7du+nYsaPF+Y4dO7J161aNotJOQkICQIX9exdk2LBhREVF0b59e61Dsbrly5fTtGlTevXqhZ+fH40aNeKLL77QOiyreuCBB/jjjz84duwYAPv27WPz5s107dpV48isLy4ujosXL1p8JhqNRtq0aWO1z8QKv9CkNSiKQr9+/RgyZAhNmzbl1KlTWoekqZMnT/Lxxx/z4Ycfah1Kqbhy5Qo5OTn4+/tbnPf39+fixYsaRaUNRVEYPXo0DzzwAHXr1tU6HKtZvHgxe/bsYefOnVqHoom///6bTz/9lNGjR/PGG2+wY8cORo4cidFo5Pnnn9c6PKsYO3YsCQkJREZGYjAYyMnJ4Z133uGZZ57ROjSry/vcu91n4unTp60Sg9TMFGLSpEnodLpCt127dvHxxx+TmJhIdHS01iGXqKI+/1udP3+ezp0706tXLwYOHKhR5Nah0+ksjhVFyXeuohs+fDj79+9n0aJFWodiNWfPnuXll1/mu+++w9HRUetwNGEymWjcuDHvvvsujRo1YvDgwQwaNIhPP/1U69CsZsmSJXz33XcsXLiQPXv2sGDBAj744AMWLFigdWia0fIzUWpmCjF8+HCefvrpQsuEhYXx9ttvs3379nxrUzRt2pTnnnuu3L65i/r885w/f5527drRokULPv/881KOTjs+Pj4YDIZ8tTDx8fH5fplUZCNGjGD58uVs3LiRKlWqaB2O1ezevZv4+HiaNGliPpeTk8PGjRuZPXs2GRkZGAwGDSMsfYGBgdSuXdviXK1atfjpp580isj6xowZw7hx48yfkfXq1eP06dNMnTqVvn37ahyddQUEBABqDU1gYKD5vDU/EyWZKYSPjw8+Pj53LPfRRx/x9ttvm4/Pnz9Pp06dWLJkCffdd19phliqivr8QR2m2a5dO5o0acK8efPQ6ytupZ+DgwNNmjRh7dq1PPbYY+bza9eupXv37hpGZh2KojBixAh+/vln1q9fT3h4uNYhWdXDDz/MgQMHLM7179+fyMhIxo4dW+ETGYBWrVrlG45/7NgxQkNDNYrI+lJTU/N9zhkMhgo7NLsw4eHhBAQEsHbtWho1agSofQs3bNjAtGnTrBKDJDMlICQkxOLY1dUVgKpVq9rEL9bz58/Ttm1bQkJC+OCDD7h8+bL5uryMvaIZPXo0ffr0oWnTpuaaqDNnzjBkyBCtQyt1w4YNY+HChfzyyy+4ubmZa6g8PDxwcnLSOLrS5+bmlq9/kIuLC97e3jbTb+iVV16hZcuWvPvuuzz55JPs2LGDzz//vELXyP5bt27deOeddwgJCaFOnTrs3buXGTNmMGDAAK1DKxXJycmcOHHCfBwXF0dMTAxeXl6EhIQwatQo3n33XapXr0716tV59913cXZ25tlnn7VOgFYZM2Vj4uLibGpo9rx58xTgtltF9sknnyihoaGKg4OD0rhxY5sZmlzQ33revHlah6YZWxuarSiK8uuvvyp169ZVjEajEhkZqXz++edah2RViYmJyssvv6yEhIQojo6OSkREhPLmm28qGRkZWodWKtatW3fb//d9+/ZVFEUdnj1x4kQlICBAMRqNSuvWrZUDBw5YLT6doiiKddImIYQQQoiSV3E7NgghhBDCJkgyI4QQQohyTZIZIYQQQpRrkswIIYQQolyTZEYIIYQQ5ZokM0IIIYQo1ySZEUIIIUS5JsmMEEIIIco1SWaEsFFt27Zl1KhRWodxW1evXsXPz49Tp04BsH79enQ6HTdu3CjVx73bx5k/fz6enp7Fuk2zZs1YunRpsW4jhLg9SWaEECXiwoULPPvss9SsWRO9Xl9govTTTz9Ru3ZtjEYjtWvX5ueff85XZurUqXTr1s1iVfaKZvz48YwbN84mFyYUoqRJMiOEKBEZGRn4+vry5ptv0qBBg9uW2bZtG0899RR9+vRh37599OnThyeffJK//vrLXCYtLY2vvvqKgQMHWit0TURFRZGQkMDvv/+udShClHuSzAghuH79Os8//zyVKlXC2dmZLl26cPz4cYsyX3zxBcHBwTg7O/PYY48xY8YMi6aVsLAw/vvf//L888/j4eFx28eZNWsWHTp0IDo6msjISKKjo3n44YeZNWuWucyqVauws7OjRYsWBcZ79epVnnnmGapUqYKzszP16tVj0aJFFmXatm3LiBEjGDVqFJUqVcLf35/PP/+clJQU+vfvj5ubG1WrVmXVqlX57n/Lli00aNAAR0dH7rvvPg4cOGBx/fz58wkJCTG/FlevXrW4/uTJk3Tv3h1/f39cXV1p1qwZ//d//2dRxmAw0LVr13xxCyGKT5IZIQT9+vVj165dLF++nG3btqEoCl27diUrKwtQv9yHDBnCyy+/TExMDB06dOCdd94p9uNs27aNjh07Wpzr1KkTW7duNR9v3LiRpk2bFno/6enpNGnShBUrVnDw4EFefPFF+vTpY1HDA7BgwQJ8fHzYsWMHI0aM4KWXXqJXr160bNmSPXv20KlTJ/r06UNqaqrF7caMGcMHH3zAzp078fPz49FHHzW/Fn/99RcDBgxg6NChxMTE0K5dO95++22L2ycnJ9O1a1f+7//+j71799KpUye6devGmTNnLMo1b96cTZs2Fe3FE0IUzGrrcwshypQ2bdooL7/8snLs2DEFULZs2WK+7sqVK4qTk5Py/fffK4qiKE899ZQSFRVlcfvnnntO8fDwKPS+/83e3l753//+Z3Huf//7n+Lg4GA+7t69uzJgwACLMuvWrVMA5fr16wU+n65duyqvvvqqRQwPPPCA+Tg7O1txcXFR+vTpYz534cIFBVC2bdtm8TiLFy82l7l69ari5OSkLFmyRFEURXnmmWeUzp07Wzz2U089VeBrkad27drKxx9/bHHul19+UfR6vZKTk1PobYUQhZOaGSFs3JEjR7Czs+O+++4zn/P29qZmzZocOXIEgNjYWJo3b25xu38fF5VOp7M4VhTF4lxaWhqOjo6F3kdOTg7vvPMO9evXx9vbG1dXV9asWZOv5qN+/frmywaDAW9vb+rVq2c+5+/vD0B8fLzF7W5t4vLy8rJ4LY4cOZKvCezfxykpKbz++uvUrl0bT09PXF1dOXr0aL74nJycMJlMZGRkFPp8hRCFs9M6ACGEthRFKfB8XpLx74SjsNsVJiAggIsXL1qci4+PNycVAD4+Ply/fr3Q+/nwww+ZOXMms2bNol69eri4uDBq1CgyMzMtytnb21sc63Q6i3N5z6koI4pufS3uZMyYMfz+++988MEHVKtWDScnJ5544ol88V27dg1nZ2ecnJzueJ9CiIJJzYwQNq527dpkZ2db9De5evUqx44do1atWgBERkayY8cOi9vt2rWr2I/VokUL1q5da3FuzZo1tGzZ0nzcqFEjDh8+XOj9bNq0ie7du9O7d28aNGhAREREvg7L92L79u3my9evX+fYsWNERkYC6ut16/X/Lp8XX79+/XjssceoV68eAQEB5jlzbnXw4EEaN25cYnELYaskmRHCxlWvXp3u3bszaNAgNm/ezL59++jduzeVK1eme/fuAIwYMYKVK1cyY8YMjh8/zmeffcaqVavy1dbExMQQExNDcnIyly9fJiYmxiIxefnll1mzZg3Tpk3j6NGjTJs2jf/7v/+zmJOmU6dOHDp0qNDamWrVqrF27Vq2bt3KkSNHGDx4cL4an3sxZcoU/vjjDw4ePEi/fv3w8fGhR48eAIwcOZLVq1czffp0jh07xuzZs1m9enW++JYuXUpMTAz79u3j2WefvW3tz6ZNm/J1iBZCFJ8kM0II5s2bR5MmTXjkkUdo0aIFiqKwcuVKc5NMq1atmDt3LjNmzKBBgwasXr2aV155JV/flkaNGtGoUSN2797NwoULadSoEV27djVf37JlSxYvXsy8efOoX78+8+fPZ8mSJRb9derVq0fTpk35/vvvC4x3/PjxNG7cmE6dOtG2bVsCAgLMyUZJeO+993j55Zdp0qQJFy5cYPny5Tg4OABw//338+WXX/Lxxx/TsGFD1qxZw1tvvWVx+5kzZ1KpUiVatmxJt27d6NSpU74amHPnzrF161b69+9fYnELYat0yt00fAshbN6gQYM4evRoqQwtXrlyJa+99hoHDx5Er6+Yv7nGjBlDQkICn3/+udahCFHuSQdgIUSRfPDBB3To0AEXFxdWrVrFggULmDNnTqk8VteuXTl+/Djnzp0jODi4VB5Da35+frz22mtahyFEhSA1M0KIInnyySdZv349SUlJREREMGLECIYMGaJ1WEIIIcmMEEIIIcq3itkYLYQQQgibIcmMEEIIIco1SWaEEEIIUa5JMiOEEEKIck2SGSGEEEKUa5LMCCGEEKJck2RGCCGEEOWaJDNCCCGEKNf+H6/moB+6A0ZyAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 6,
+ "id": "b324a833",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"import os\n",
"import numpy as np\n",
@@ -1943,7 +2009,7 @@
"MSETrain = np.zeros(nlambdas)\n",
"MSELassoPredict = np.zeros(nlambdas)\n",
"MSELassoTrain = np.zeros(nlambdas)\n",
- "lambdas = np.logspace(-4, 10, nlambdas)\n",
+ "lambdas = np.logspace(-4, 4, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" Ridgebeta = np.linalg.inv(X_train.T @ X_train+lmb*I) @ X_train.T @ y_train\n",
@@ -1975,16 +2041,20 @@
},
{
"cell_type": "markdown",
- "id": "6e699b2f",
- "metadata": {},
+ "id": "c3f2883f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Material for lecture Thursday September 7"
]
},
{
"cell_type": "markdown",
- "id": "d39bb025",
- "metadata": {},
+ "id": "cc652ee0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Important technicalities: More on Rescaling data\n",
"\n",
@@ -2038,8 +2108,11 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "6f772122",
- "metadata": {},
+ "id": "55057c60",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"\"\"\"\n",
@@ -2062,8 +2135,10 @@
},
{
"cell_type": "markdown",
- "id": "38ff7f71",
- "metadata": {},
+ "id": "4a2940af",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Let us try to understand what this may imply mathematically when we\n",
"subtract the mean values, also known as *zero centering*. For\n",
@@ -2074,8 +2149,10 @@
},
{
"cell_type": "markdown",
- "id": "7dba297c",
- "metadata": {},
+ "id": "9091f979",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n",
@@ -2084,8 +2161,10 @@
},
{
"cell_type": "markdown",
- "id": "772f2dc0",
- "metadata": {},
+ "id": "47d249f7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Recall also that we use the squared value. This expression can lead to an\n",
"increased penalty for higher differences between predicted and\n",
@@ -2099,8 +2178,10 @@
},
{
"cell_type": "markdown",
- "id": "1908cc29",
- "metadata": {},
+ "id": "396b54f3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial \\beta_j} = 0,\n",
@@ -2109,16 +2190,20 @@
},
{
"cell_type": "markdown",
- "id": "fb31721c",
- "metadata": {},
+ "id": "c7845b32",
+ "metadata": {
+ "editable": true
+ },
"source": [
"for all $j$. For $\\beta_0$ we have"
]
},
{
"cell_type": "markdown",
- "id": "01d332e2",
- "metadata": {},
+ "id": "1dd0b8a3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n",
@@ -2127,16 +2212,20 @@
},
{
"cell_type": "markdown",
- "id": "fd2cc608",
- "metadata": {},
+ "id": "e67fd252",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Multiplying away the constant $2/n$, we obtain"
]
},
{
"cell_type": "markdown",
- "id": "abe97943",
- "metadata": {},
+ "id": "35f00341",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n",
@@ -2145,8 +2234,10 @@
},
{
"cell_type": "markdown",
- "id": "4b0c9c12",
- "metadata": {},
+ "id": "a12571e1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Let us specialize first to the case where we have only two parameters $\\beta_0$ and $\\beta_1$.\n",
"Our result for $\\beta_0$ simplifies then to"
@@ -2154,8 +2245,10 @@
},
{
"cell_type": "markdown",
- "id": "addd3d97",
- "metadata": {},
+ "id": "c77e3dcb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n",
@@ -2164,16 +2257,20 @@
},
{
"cell_type": "markdown",
- "id": "a96063de",
- "metadata": {},
+ "id": "d7d8f4a5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We obtain then"
]
},
{
"cell_type": "markdown",
- "id": "61d186e9",
- "metadata": {},
+ "id": "5f1c4e41",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n",
@@ -2182,16 +2279,20 @@
},
{
"cell_type": "markdown",
- "id": "9b976247",
- "metadata": {},
+ "id": "c927193b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"If we define"
]
},
{
"cell_type": "markdown",
- "id": "c935d0d7",
- "metadata": {},
+ "id": "ef7bff5c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mu_{\\boldsymbol{x}_1}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1},\n",
@@ -2200,16 +2301,20 @@
},
{
"cell_type": "markdown",
- "id": "9f4e09e3",
- "metadata": {},
+ "id": "00249d84",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and the mean value of the outputs as"
]
},
{
"cell_type": "markdown",
- "id": "9438c6ec",
- "metadata": {},
+ "id": "146ca7e1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n",
@@ -2218,16 +2323,20 @@
},
{
"cell_type": "markdown",
- "id": "f773450a",
- "metadata": {},
+ "id": "20a05643",
+ "metadata": {
+ "editable": true
+ },
"source": [
"we have"
]
},
{
"cell_type": "markdown",
- "id": "1b530b85",
- "metadata": {},
+ "id": "a4c8df62",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_0 = \\mu_y - \\beta_1\\mu_{\\boldsymbol{x}_1}.\n",
@@ -2236,16 +2345,20 @@
},
{
"cell_type": "markdown",
- "id": "4fa771d8",
- "metadata": {},
+ "id": "34ebe859",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In the general case with more parameters than $\\beta_0$ and $\\beta_1$, we have"
]
},
{
"cell_type": "markdown",
- "id": "09116fa4",
- "metadata": {},
+ "id": "56bbfec4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n",
@@ -2254,16 +2367,20 @@
},
{
"cell_type": "markdown",
- "id": "5011c21f",
- "metadata": {},
+ "id": "bf7c2d55",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can rewrite the latter equation as"
]
},
{
"cell_type": "markdown",
- "id": "b1d51013",
- "metadata": {},
+ "id": "f2a7e47e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\sum_{j=1}^{p-1} \\mu_{\\boldsymbol{x}_j}\\beta_j,\n",
@@ -2272,16 +2389,20 @@
},
{
"cell_type": "markdown",
- "id": "f12877c0",
- "metadata": {},
+ "id": "6100af59",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have defined"
]
},
{
"cell_type": "markdown",
- "id": "0b27f009",
- "metadata": {},
+ "id": "16765ca9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mu_{\\boldsymbol{x}_j}=\\frac{1}{n}\\sum_{i=0}^{n-1} X_{ij},\n",
@@ -2290,8 +2411,10 @@
},
{
"cell_type": "markdown",
- "id": "dc8e6f9a",
- "metadata": {},
+ "id": "d7be14d8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"the mean value for all elements of the column vector $\\boldsymbol{x}_j$.\n",
"\n",
@@ -2300,8 +2423,10 @@
},
{
"cell_type": "markdown",
- "id": "c76d277c",
- "metadata": {},
+ "id": "39dd30ab",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n",
@@ -2310,16 +2435,20 @@
},
{
"cell_type": "markdown",
- "id": "3986fa93",
- "metadata": {},
+ "id": "ed86069f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"If we minimize with respect to $\\boldsymbol{\\beta}$ we have then"
]
},
{
"cell_type": "markdown",
- "id": "07298cf7",
- "metadata": {},
+ "id": "45e51c93",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n",
@@ -2328,8 +2457,10 @@
},
{
"cell_type": "markdown",
- "id": "84f96f80",
- "metadata": {},
+ "id": "1edf9e19",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n",
"and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n",
@@ -2339,8 +2470,10 @@
},
{
"cell_type": "markdown",
- "id": "54ba61b3",
- "metadata": {},
+ "id": "de5717d0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n",
@@ -2349,8 +2482,10 @@
},
{
"cell_type": "markdown",
- "id": "e23b4f5e",
- "metadata": {},
+ "id": "095eac95",
+ "metadata": {
+ "editable": true
+ },
"source": [
"What does this mean? And why do we insist on all this? Let us look at some examples.\n",
"\n",
@@ -2361,8 +2496,11 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "3a6e4b8d",
- "metadata": {},
+ "id": "175c8669",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -2455,8 +2593,10 @@
},
{
"cell_type": "markdown",
- "id": "d938305a",
- "metadata": {},
+ "id": "210e4409",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The intercept is the value of our output/target variable\n",
"when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n",
@@ -2474,8 +2614,10 @@
},
{
"cell_type": "markdown",
- "id": "9797b358",
- "metadata": {},
+ "id": "2215519e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n",
@@ -2484,16 +2626,20 @@
},
{
"cell_type": "markdown",
- "id": "4962007b",
- "metadata": {},
+ "id": "0ddfe3ef",
+ "metadata": {
+ "editable": true
+ },
"source": [
"but when we take out the intercept, this equation becomes"
]
},
{
"cell_type": "markdown",
- "id": "84738217",
- "metadata": {},
+ "id": "8065d63a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n",
@@ -2502,16 +2648,20 @@
},
{
"cell_type": "markdown",
- "id": "340cdb6e",
- "metadata": {},
+ "id": "17c2c714",
+ "metadata": {
+ "editable": true
+ },
"source": [
"For Lasso regression we have"
]
},
{
"cell_type": "markdown",
- "id": "ea0d39ea",
- "metadata": {},
+ "id": "476aaaa4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n",
@@ -2520,8 +2670,10 @@
},
{
"cell_type": "markdown",
- "id": "e18a34e9",
- "metadata": {},
+ "id": "2fd7dcab",
+ "metadata": {
+ "editable": true
+ },
"source": [
"It means that, when scaling the design matrix and the outputs/targets,\n",
"by subtracting the mean values, we have an optimization problem which\n",
@@ -2535,77 +2687,13 @@
},
{
"cell_type": "code",
- "execution_count": 5,
- "id": "c7c1b283",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Beta values for own Ridge implementation\n",
- "[ 1.02910448 0.09882561 -0.872222 0.56647004 0.10230911]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[ 1.02910448 0.09882561 -0.872222 0.56647004 0.10230911]\n",
- "MSE values for own Ridge implementation\n",
- "6.378961675519713e-07\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "6.378961676716729e-07\n",
- "Beta values for own Ridge implementation\n",
- "[ 1.036455 -0.00786759 -0.47643988 0.02948138 0.34461101]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[ 1.036455 -0.00786759 -0.47643988 0.02948138 0.34461101]\n",
- "MSE values for own Ridge implementation\n",
- "7.172986768060827e-06\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "7.172986768080781e-06\n",
- "Beta values for own Ridge implementation\n",
- "[ 1.04531042 -0.11654524 -0.24566996 -0.01019061 0.23403408]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[ 1.04531042 -0.11654524 -0.24566996 -0.01019061 0.23403408]\n",
- "MSE values for own Ridge implementation\n",
- "4.3009536610604754e-05\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "4.300953661055037e-05\n",
- "Beta values for own Ridge implementation\n",
- "[ 1.01467706 -0.08613571 -0.10175655 -0.01725877 0.05286597]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[ 1.01467706 -0.08613571 -0.10175655 -0.01725877 0.05286597]\n",
- "MSE values for own Ridge implementation\n",
- "0.0004127999371307263\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "0.00041279993713071416\n",
- "Beta values for own Ridge implementation\n",
- "[ 8.40379934e-01 1.26650691e-01 8.33755702e-04 -2.69288364e-02\n",
- " -3.14230505e-02]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[ 8.40379934e-01 1.26650691e-01 8.33755702e-04 -2.69288364e-02\n",
- " -3.14230505e-02]\n",
- "MSE values for own Ridge implementation\n",
- "0.01495259804986317\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "0.014952598049863024\n",
- "Beta values for own Ridge implementation\n",
- "[0.37683289 0.1456364 0.0803835 0.05259326 0.03788163]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[0.37683289 0.1456364 0.0803835 0.05259326 0.03788163]\n",
- "MSE values for own Ridge implementation\n",
- "0.26325852059157523\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "0.2632585205915754\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAGwCAYAAABB4NqyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABT/0lEQVR4nO3deVxVdeL/8dcFWUXAFTSB1Mw1TQEVza1iuWpjZuVMZVlmOZVL/tocW/1O4+Q0ZlpmTaU2lWKjlRkXpcWdFEm0FM0xDVPIHXADgfP7g7jjlUVA4AD3/Xw87mO4537uue9zonjPuedzjsUwDAMRERERJ+JidgARERGRmqYCJCIiIk5HBUhEREScjgqQiIiIOB0VIBEREXE6KkAiIiLidFSARERExOk0MDtAbVRQUMDhw4dp1KgRFovF7DgiIiJSDoZhkJ2dTatWrXBxKfsYjwpQCQ4fPkxQUJDZMURERKQSDh48SOvWrcscowJUgkaNGgGFO9DX19fkNCIiIlIeWVlZBAUF2f+Ol0UFqARFX3v5+vqqAImIiNQx5Tl9RSdBi4iIiNNRARIRERGnowIkIiIiTkfnAF2B/Px8Lly4YHYMkWrj5uaGq6ur2TFERKqcClAlGIZBRkYGp06dMjuKSLXz9/cnMDBQ18QSkXpFBagSispPixYt8Pb21h8GqZcMw+Ds2bMcOXIEgJYtW5qcSESk6qgAVVB+fr69/DRt2tTsOCLVysvLC4AjR47QokULfR0mIvWGToKuoKJzfry9vU1OIlIzin7Xdb6biNQnKkCVpK+9xFnod11E6iMVIBEREXE6KkAiIiLidFSAREpx4MABLBYLKSkppY5Zs2YNFotFl0QQEaljVICcyJgxY7BYLIwfP77Ya4888ggWi4UxY8bYlx05coSHH36Y4OBgPDw8CAwMJDo6msTERPuYq6++GovFUuzx97//vSY2qdIGDRpkz+ru7k67du2YOnUqOTk59jFBQUGkp6fTtWtXE5OWvo+LHoMGDar0ugcNGsTkyZOrLKuIyOXk5Z5nU9zb5OWeNzWHpsE7maCgIJYsWcJrr71mn+J8/vx5Fi9eTHBwsMPYkSNHcuHCBRYtWkTbtm357bff+Prrrzlx4oTDuOnTpzNu3DiHZY0aNareDakC48aNY/r06eTm5pKUlMT9998PwIwZMwBwdXUlMDDQzIgAJCUlkZ+fD8CmTZsYOXIke/bswdfXFwB3d3cz44mIVMh3q9+nf/KjXPP1BH76x3ksLuYci9ERoKpgGHDmjDkPw6hQ1J49exIcHMzy5cvty5YvX05QUBA9evSwLzt16hQbNmzglVdeYfDgwYSEhNCrVy+mTp3K0KFDHdbZqFEjAgMDHR4NGzYsNcPJkye59957ady4Md7e3litVvbu3fv7rjRo3rw5y5Yts4+//vrradGihf15YmIibm5unD59GiicpfTuu+8yYsQIvL29ad++PStWrLjsvvD29iYwMJDg4GBGjhxJZGQkq1evtr9e0ldgcXFxXHvttXh5eTF48GAOHDhQbL3/+te/CAoKwtvbmxEjRjBr1iz8/f0dxnzxxReEhobi6elJ27Zteemll8jLyysxZ/Pmze37tUmTJgC0aNHCvmz37t0MGDAALy8vgoKCmDhxImfOnLG/f968ebRv3x5PT08CAgK4/fbbgcIjgmvXruX111+3H00qaXtERKqSbctHAITR0rTyAypAVePsWfDxMedx9myF495///0sWLDA/vz999/ngQcecBjj4+ODj48Pn332mcPXQlVhzJgxbN26lRUrVpCYmIhhGAwZMoQLFy5gsVgYMGAAa9asAQrL0q5du7hw4QK7du0CCs+7CQ0NxcfHx77Ol156iTvvvJMdO3YwZMgQ7r777mJHqsqyfft2Nm7ciJubW6ljDh48yG233caQIUNISUnhwQcf5JlnnnEYs3HjRsaPH8+kSZNISUkhMjKSl19+2WHMqlWruOeee5g4cSK7du3i7bffZuHChcXGlccPP/xAdHQ0t912Gzt27CA2NpYNGzbw2GOPAbB161YmTpzI9OnT2bNnD/Hx8QwYMACA119/nYiICMaNG0d6ejrp6ekEBQVVOIOISEXYsrcBYG0bbW4QQ4rJzMw0ACMzM7PYa+fOnTN27dplnDt37n8LT582jMJjMTX/OH263Nt13333GcOHDzeOHj1qeHh4GPv37zcOHDhgeHp6GkePHjWGDx9u3Hffffbx//nPf4zGjRsbnp6eRt++fY2pU6ca27dvd1hnSEiI4e7ubjRs2NDh8e2335aY4aeffjIAY+PGjfZlx44dM7y8vIylS5cahmEYc+bMMbp27WoYhmF89tlnRlhYmHHbbbcZb775pmEYhhEVFWU8/fTT9vcDxrPPPnvRP47ThsViMWw2W6n7YuDAgYabm5vRsGFDw93d3QAMFxcX4z//+Y99zP79+w3A2LZtm2EYhjF16lSjU6dORkFBgX3M008/bQDGyZMnDcMwjFGjRhlDhw51+Ky7777b8PPzsz/v37+/8be//c1hzL///W+jZcuWpeYt8u233zp83ujRo42HHnrIYcz69esNFxcX49y5c8ayZcsMX19fIysrq9T9MGnSpDI/s8TfeRGRSkjft93gRQxexMj4+YcqX39Zf78vpXOAqoK3N/z+dYwpn11BzZo1Y+jQoSxatAjDMBg6dCjNmjUrNm7kyJEMHTqU9evXk5iYSHx8PDNnzuTdd991OFn6ySefdHgOcNVVV5X42ampqTRo0IDevXvblzVt2pQOHTqQmpoKFJ6YO2nSJI4dO8batWsZNGgQwcHBrF27loceeohNmzYVO3G3W7du9p8bNmxIo0aN7PewKs3dd9/NtGnTyMrK4pVXXsHX15eRI0eWOj41NZU+ffo4XBgwIiLCYcyePXsYMWKEw7JevXqxcuVK+/Pk5GSSkpIcjvjk5+dz/vx5zp49W6GrjCcnJ/Pf//6Xjz76yL7MMAwKCgrYv38/kZGRhISE0LZtW2JiYoiJibF/VSgiUtNW2eYCEHrKm4A25k4wUQGqChYLlHHOS230wAMP2L8mefPNN0sd5+npSWRkJJGRkTz//PM8+OCDvPDCCw6Fp1mzZlxzzTXl+lyjlHOWDMOwF4uuXbvStGlT1q5dy9q1a5k+fTpBQUG8/PLLJCUlce7cOW644QaH91/61ZXFYqGgoKDMLH5+fvbcH374IV26dOG9995j7NixFcpe2naU9r6CggJeeuklbrvttmLv9/T0vOxnXLquhx9+mIkTJxZ7LTg4GHd3d77//nvWrFnD6tWref7553nxxRdJSkoqdl6SiEh1s+1bBX5g9e1pdhSdA+SsYmJiyM3NJTc3l+jo8n8P27lzZ4cTbCuqc+fO5OXlsXnzZvuy48eP89NPP9GpUycA+3lAn3/+OT/++CP9+/fnuuuu48KFC8yfP5+ePXtW+SwzNzc3/vKXv/Dss89ytpTzqjp37sx3333nsOzS5x07dmTLli0Oy7Zu3erwvGfPnuzZs4drrrmm2MOlgicE9uzZk507d5a4rqLZYQ0aNODmm29m5syZ7NixgwMHDvDNN98AhTPIimaYiYhUp7zc86z2+BUAa+97TE6jAuS0XF1dSU1NJTU1tcQ7fB8/fpwbb7yRDz/8kB07drB//34++eQTZs6cyfDhwx3GZmdnk5GR4fDIysoq8XPbt2/P8OHDGTduHBs2bGD79u3cc889XHXVVQ7rHTRoEB9//DHdunXD19fXXoo++uijK7ruTVnuuusuLBYL8+bNK/H18ePHs2/fPqZMmcKePXv4+OOPWbhwocOYCRMmEBcXx6xZs9i7dy9vv/02NpvN4ajQ888/zwcffMCLL77Izp07SU1NJTY2lmeffbbCmZ9++mkSExN59NFHSUlJYe/evaxYsYIJEyYAsHLlSubMmUNKSgq//PILH3zwAQUFBXTo0AEovMbQ5s2bOXDgAMeOHbvsUTMRkcra8tUiTnoa+J+30Ovm+8yOowLkzHx9fe3XkrmUj48PvXv35rXXXmPAgAF07dqV5557jnHjxvHGG284jH3++edp2bKlw+Opp54q9XMXLFhAaGgow4YNIyIiAsMwiIuLc/gaa/DgweTn5zuUnYEDB5Kfn8/AgQOvbMNL4e7uzmOPPcbMmTPtU+wvFhwczLJly/jiiy/o3r078+fP529/+5vDmH79+jF//nxmzZpF9+7diY+P5/HHH3f4ais6OpqVK1eSkJBAeHg4ffr0YdasWYSEhFQ4c7du3Vi7di179+6lf//+9OjRg+eee46WLVsC4O/vz/Lly7nxxhvp1KkT8+fPZ/HixXTp0gWAJ554AldXVzp37kzz5s1JS0urcAYRkfKwbf4QgKic1jRwr9jX/dXBYpTnxAYnk5WVhZ+fH5mZmcUKwvnz59m/fz9t2rSp8Pka4pzGjRvH7t27Wb9+vdlRKkW/8yJSFcIfb8hW/7MsaDqWMY+9Wy2fUdbf70vpJGiRKvbqq68SGRlJw4YNsdlsLFq0qNSv1UREnMGRAzvZ6l94fmXMkOKTNsygAiRSxbZs2cLMmTPJzs6mbdu2zJkzhwcffNDsWCIiplkVVzj9vccpLwLbdrvM6JqhAiRSxZYuXWp2BBGRWsW2Lx58wdqox+UH1xCdBC0iIiLVJv9CLqvcCidYxIT/yeQ0/2N6AZo3b5795MrQ0NAyTxRdvnw5kZGRNG/eHF9fXyIiIli1apXDmIULF9pv7Hjx4/z589W9KSIiInKJpK8/4ISXgd95iIiuPacDmFqAYmNjmTx5MtOmTWPbtm30798fq9Va6lTcdevWERkZSVxcHMnJyQwePJhbbrmFbdu2OYzz9fW139yx6KHZKyIiIjXP9t2/AYjMuapWTH8vYuo5QLNmzWLs2LH2E0Rnz57NqlWreOutt5gxY0ax8bNnz3Z4/re//Y3PP/+cL774gh49/ve9osViITAwsFqzi4iIyOXFZyaDP1ivjjQ7igPTjgDl5uaSnJxMVFSUw/KoqCg2bdpUrnUUFBSQnZ1NkyZNHJafPn2akJAQWrduzbBhw4odIbpUTk4OWVlZDg8RERG5MkfTUknyK7x9Uox1gslpHJlWgI4dO0Z+fj4BAQEOywMCAsjIyCjXOv75z39y5swZ7rzzTvuyjh07snDhQlasWMHixYvx9PSkX79+7N27t9T1zJgxAz8/P/sjKCiochslVebAgQNYLBZSUlJKHbNmzRosFgunTp2qsVwiIlJ+q+PmYlig+ylPWrU3/waoFzP9JOiS7px96bKSLF68mBdffJHY2FhatGhhX96nTx/uueceunfvTv/+/Vm6dCnXXnstc+fOLXVdU6dOJTMz0/44ePBg5TeoFhszZgwWi4Xx48cXe+2RRx7BYrE43OX9yJEjPPzwwwQHB+Ph4UFgYCDR0dEkJibax1x99dUlnnT+97//vdQcgwYNso9zd3enXbt2TJ06lZycHPuYoKAg0tPT6dq1a9VsfCWVtn1Fjyu5L9mgQYOYPHlylWUVEaltbHttAFh9rjc3SAlMOweoWbNmuLq6Fjvac+TIkWJHhS4VGxvL2LFj+eSTT7j55pvLHOvi4kJ4eHiZR4A8PDzw8PAof/g6LCgoiCVLlvDaa6/h5eUFFN7qYPHixQQHBzuMHTlyJBcuXGDRokW0bduW3377ja+//poTJ044jJs+fTrjxo1zWHa5u7WPGzeO6dOnk5ubS1JSEvfffz+A/dwvV1fXWnEeV1JSkv1u6Zs2bWLkyJHs2bPHfon1ojuui4iIo4L8PFa5/QJATNgfTU5TnGlHgNzd3QkNDSUhIcFheUJCAn379i31fYsXL2bMmDF8/PHHDB069LKfYxgGKSkp9ptDOruePXsSHBzM8uXL7cuWL19OUFCQw4nkp06dYsOGDbzyyisMHjyYkJAQevXqxdSpU4vt90aNGhEYGOjwaNiwYZk5vL29CQwMJDg4mJEjRxIZGcnq1avtr5f0FVhcXBzXXnstXl5eDB48mAMHDhRb77/+9S+CgoLw9vZmxIgRzJo1C39/f4cxX3zxBaGhoXh6etK2bVteeukl8vLySszZvHlz+zYVnWvWokUL+7Ldu3czYMAAvLy8CAoKYuLEiZw5c8b+/nnz5tG+fXs8PT0JCAjg9ttvBwqPxq1du5bXX3/dfjSppO0REamrtn79b455GfjmQN/ocZd/Qw0z9SuwKVOm8O677/L++++TmprK448/Tlpamv0rmqlTp3Lvvffaxy9evJh7772Xf/7zn/Tp04eMjAwyMjLIzMy0j3nppZdYtWoVP//8MykpKYwdO5aUlJQSv/apKoZhcCb3jCmPytzL9v7772fBggX25++//z4PPPCAwxgfHx98fHz47LPPHL6aqg7bt29n48aNDneDv9TBgwe57bbbGDJkCCkpKTz44IM888wzDmM2btzI+PHjmTRpEikpKURGRvLyyy87jFm1ahX33HMPEydOZNeuXbz99tssXLiw2Ljy+OGHH4iOjua2225jx44dxMbGsmHDBh577DEAtm7dysSJE5k+fTp79uwhPj6eAQMGAPD6668TERHBuHHj7Jdq0LlnIlKfxCcWTn+/+Xwr3Dy9TU5TnKnT4EeNGsXx48eZPn26/XyPuLg4QkJCAEhPT3e4JtDbb79NXl4ejz76KI8++qh9+X333cfChQuBwiMXDz30EBkZGfj5+dGjRw/WrVtHr169qm07zl44i88Mn2pbf1lOTz1NQ/eyj7ZcavTo0UydOtV+lGXjxo0sWbKENWvW2Mc0aNCAhQsXMm7cOObPn0/Pnj0ZOHAgf/zjH+nWzfE+Lk8//TTPPvusw7KVK1eWeX7MvHnzePfdd7lw4QK5ubm4uLjw5ptvljr+rbfeom3btrz22mtYLBY6dOjADz/8wCuvvGIfM3fuXKxWK0888QQA1157LZs2bWLlypX2MS+//DLPPPMM9913HwBt27bl//7v/3jqqad44YUXLrvvLvaPf/yDu+66y34eT/v27ZkzZw4DBw7krbfeIi0tjYYNGzJs2DAaNWpESEiI/Sibn58f7u7u9iNhIiL1je1UUuH095CyT1Uxi+n3AnvkkUd45JFHSnytqNQUufgPdGlee+01XnvttSpIVn81a9aMoUOHsmjRIgzDYOjQoTRr1qzYuJEjRzJ06FDWr19PYmIi8fHxzJw5k3fffdfhZOknn3zS4TnAVVddVWaGu+++m2nTppGVlcUrr7yCr68vI0eOLHV8amoqffr0cThBPiIiwmHMnj17GDFihMOyXr16ORSg5ORkkpKSHI745Ofnc/78ec6ePYu3d/n/X0pycjL//e9/+eijj+zLDMOgoKCA/fv3ExkZSUhICG3btiUmJoaYmBhGjBhRoc8QEamLjv+6l81+pwGIiXnM5DQlM70A1Qfebt6cnnratM+ujAceeMD+VU1ZR148PT2JjIwkMjKS559/ngcffJAXXnjBofA0a9aMa665pkKf7+fnZ3/Phx9+SJcuXXjvvfcYO3ZsiePL81VfSTMIL31fQUEBL730Erfddlux91f0auEFBQU8/PDDTJw4sdhrwcHBuLu78/3337NmzRpWr17N888/z4svvkhSUlKx85JEROqT1V/OwbDAdZmetO4QbnacEqkAVQGLxVLhr6HMFhMTQ25uLgDR0dHlfl/nzp357LPPqjSLm5sbf/nLX5g6dSp/+tOfSjxCUtLnfvfddw7PO3bsyJYtWxyWbd261eF5z5492bNnT4ULW0l69uzJzp07y1xXgwYNuPnmm7n55pt54YUX8Pf355tvvuG2227D3d3dPsNMRKQ+se2Ng0YQ43Wd2VFKZfp1gMQcrq6upKamkpqaiqura7HXjx8/zo033siHH37Ijh072L9/P5988gkzZ85k+PDhDmOzs7PtJ6QXPSp6Ne277roLi8XCvHnzSnx9/Pjx7Nu3jylTprBnzx4+/vjjYl+RTpgwgbi4OGbNmsXevXt5++23sdlsDkeFnn/+eT744ANefPFFdu7cSWpqKrGxscXOYSqPp59+msTERB599FFSUlLYu3cvK1asYMKEwqudrly5kjlz5pCSksIvv/zCBx98QEFBAR06dAAKrzG0efNmDhw4wLFjxygoKKhwBhGR2qYgP4941/0AWGvh9PciKkBOzNfX1349m0v5+PjQu3dvXnvtNQYMGEDXrl157rnnGDduHG+88YbD2Oeff56WLVs6PJ566qkKZXF3d+exxx5j5syZnD5d/OvE4OBgli1bxhdffEH37t2ZP38+f/vb3xzG9OvXj/nz5zNr1iy6d+9OfHw8jz/+uMNXW9HR0axcuZKEhATCw8Pp06cPs2bNsp94XxHdunVj7dq17N27l/79+9OjRw+ee+45+yUX/P39Wb58OTfeeCOdOnVi/vz5LF68mC5dugDwxBNP4OrqSufOnWnevHmpNwEWEalLvv/2Y456G/jkQr+Yh8yOUyqLUZl51PVcVlYWfn5+ZGZmFisI58+fZ//+/bRp00Z3mK8Dxo0bx+7du1m/fr3ZUeos/c6LSEX8dfrNPGd8za2nWvLpa4dr9LPL+vt9KZ0DJPXKq6++SmRkJA0bNsRms7Fo0aJSv1YTEZGqZzuxBRqDNeRGs6OUSQVI6pUtW7Ywc+ZMsrOzadu2LXPmzOHBBx80O5aIiFM4cXgf3/llA2CNrl13f7+UCpDUK0uXLjU7goiI00r4cg4FLtAl04OgTr3NjlMmnQQtIiIiVcK250sAYry6mpzk8lSAKknnjouz0O+6iJSHw/T3nqNMTnN5KkAVVHTDzrNnz5qcRKRmFP2ul3WzWhGR7es/4TfvAhrmwg0xD5sd57J0DlAFubq64u/vz5EjRwDw9vYudvsFkfrAMAzOnj3LkSNH8Pf3L/GCmSIiRWzrFwBw09lAPBqWPQW9NlABqoSiu3cXlSCR+szf3193rBeRy7Id31w4/T1osNlRykUFqBIsFgstW7akRYsWXLhwwew4ItXGzc1NR35E5LJO/fYLiX6Ft0CyRj9qcpryUQG6Aq6urvrjICIiTi9h5evku0CnTHdCuvQzO0656CRoERERuSK23SsBiPGs/dPfi6gAiYiISKUZBQXEW/YBYO1xh8lpyk8FSERERCptx4b/kN6wAO9cGDDkEbPjlJsKkIiIiFSabd37ANx4tkWdmP5eRAVIREREKs127DsArK3rxvT3IipAIiIiUimZR9LY6JcJQMzN401OUzEqQCIiIlIpX62cS74LXJvlRtvug8yOUyEqQCIiIlIpttQVAFjdu5icpOJUgERERKTCjIIC4vkvANbrbzc5TcWpAImIiEiF/bjpUw75FOB1AQYOrRu3v7iYCpCIiIhUmG3tewAMPtMcTx9/c8NUggqQiIiIVJjtaCIA1lYDTU5SOSpAIiIiUiFZR39lg+8pAKyRfzY3TCWpAImIiEiFfP3lXPJc4ZosN9pdf6PZcSpFBUhEREQqxLbr9+nvbp1MTlJ5KkAiIiJSbkZBAfHGXgCs3UeanKbyVIBERESk3HZ99wUHffLxvACDhj5mdpxKUwESERGRcrOt/RcAg840w8u3iclpKk8FSERERMrN9tsmAKwtB5ic5MqoAImIiEi5ZB8/zHrfkwBYb6pbd3+/lAqQiIiIlMs3X77BBVdom92Aa3rcZHacK6ICJCIiIuUSv/NzAKwNOmFxqdsVom6nFxERkRphFBRgy98DgPW620xOc+VUgEREROSydifF8UujfDzyYPCwujv9vYgKkIiIiFyW7dvC6e8Ds5vi7dfM5DRXTgVIRERELsuWsQEAa8v+JiepGipAIiIiUqbTJzJY1+gEADGDx5mcpmqoAImIiEiZvv3yTXIbwNXZDegQFmN2nCqhAiQiIiJliv/xMwCsrh3q/PT3IvVjK0RERKRaFE5/3w2AtesIk9NUHRUgERERKdVPyavY3ygP9zy4cdgEs+NUGRUgERERKZXtm3cAGJDdhIaNW5icpuqoAImIiEipbOnrAbAG3mBykqqlAiQiIiIlOpt5jLWNjgMQM2isyWmqlgqQiIiIlGjNl2+S0wCCT7vSqdcws+NUKRUgERERKZFtxzIArJZr68309yL1a2tERESkytgupAJg7XqruUGqgQqQiIiIFLM3OYF9vnm45cONQ+v+3d8vZXoBmjdvHm3atMHT05PQ0FDWr19f6tjly5cTGRlJ8+bN8fX1JSIiglWrVhUbt2zZMjp37oyHhwedO3fm008/rc5NEBERqXdsX88HoH9WYxo1bWVymqpnagGKjY1l8uTJTJs2jW3bttG/f3+sVitpaWkljl+3bh2RkZHExcWRnJzM4MGDueWWW9i2bZt9TGJiIqNGjWL06NFs376d0aNHc+edd7J58+aa2iwREZE6z5a+DoCYFhEmJ6keFsMwDLM+vHfv3vTs2ZO33nrLvqxTp07ceuutzJgxo1zr6NKlC6NGjeL5558HYNSoUWRlZWGz2exjYmJiaNy4MYsXLy7XOrOysvDz8yMzMxNfX98KbJGIiEjddy7rBE1mNuW8G/xw83K69qsbt8CoyN9v044A5ebmkpycTFRUlMPyqKgoNm3aVK51FBQUkJ2dTZMmTezLEhMTi60zOjq6zHXm5OSQlZXl8BAREXFWa+Pmcd4NWp92pUvEcLPjVAvTCtCxY8fIz88nICDAYXlAQAAZGRnlWsc///lPzpw5w5133mlflpGRUeF1zpgxAz8/P/sjKCioAlsiIiJSv9i2/wcAq6V9vZv+XsT0rbJYLA7PDcMotqwkixcv5sUXXyQ2NpYWLRzvTVLRdU6dOpXMzEz74+DBgxXYAhERkfrFlrsLAGvnP5icpPo0MOuDmzVrhqura7EjM0eOHCl2BOdSsbGxjB07lk8++YSbb77Z4bXAwMAKr9PDwwMPD48KboGIiEj9sy/lG/b6XqBBPtw0tP7c/f1Sph0Bcnd3JzQ0lISEBIflCQkJ9O3bt9T3LV68mDFjxvDxxx8zdOjQYq9HREQUW+fq1avLXKeIiIgUsiUUTky6Icsf3+atTU5TfUw7AgQwZcoURo8eTVhYGBEREbzzzjukpaUxfvx4oPCrqUOHDvHBBx8AheXn3nvv5fXXX6dPnz72Iz1eXl74+fkBMGnSJAYMGMArr7zC8OHD+fzzz/nqq6/YsGGDORspIiJSh9gOrwV/iGnex+wo1crUc4BGjRrF7NmzmT59Otdffz3r1q0jLi6OkJAQANLT0x2uCfT222+Tl5fHo48+SsuWLe2PSZMm2cf07duXJUuWsGDBArp168bChQuJjY2ld+/eNb59IiIidcn506f4tuFRAKwD6tfd3y9l6nWAaitdB0hERJzR6qUziE79C1edduHgKxfq3AywOnEdIBEREaldbNs+ASCGa+pc+amo+r11IiIiUm623J0AWDvV3+nvRVSAREREhP071rHHN5cG+XDzsPo7/b2ICpCIiIhgS5gHQN8sP/xaBJucpvqpAImIiAi2X78FIKaZc8yaVgESERFxcjlnsvjG+wgA1v4PmJymZqgAiYiIOLn1tvmcdYeWZ1zo3v8Os+PUCBUgERERJ2f7fikAMUa7ej/9vYhzbKWIiIiUynb+BwCsHYeZnKTmqACJiIg4sV92biTVLxfXAogcNunyb6gnVIBEREScmG3VmwD0yWyEf0CIyWlqjgqQiIiIE7MdLJz+bm3qHNPfi6gAiYiIOKncc6f52jsDAGu/MeaGqWEqQCIiIk5qg+1tzrhDwFkXrh84yuw4NUoFSERExEnZkpcAEJPfBhfXBianqVkqQCIiIk7Kdu736e8dhpqcpOapAImIiDihg6mb2emXg0sBRA6daHacGqcCJCIi4oRsq+YC0DvThyat2pmcpuapAImIiDih+F++AcDapJfJScyhAiQiIuJkcs+d5iuvdACsfe8zOY05VIBERESczKZV75LtAS3OWug5+C6z45hCBUhERMTJ2LYuBiDaCae/F1EBEhERcTK2szsAsLYfYnIS86gAiYiIOJFf9yTxg995LAZEOeH09yIqQCIiIk4kPv4NAHplNqRp6/YmpzGPCpCIiIgTif/lawCs/uEmJzGXCpCIiIiTuHD+LAmehwCwRtxrchpzqQCJiIg4icTV75HlAc3OWQi7abTZcUylAiQiIuIkbEkfAxB9IcRpp78XUQESERFxErbTKQBY21vNDVILqACJiIg4gcN7v2e7/+/T34dMMDuO6VSAREREnMCq36e/h2U2pHlwJ5PTmE8FSERExAnY9icAYPULNTlJ7aACJCIiUs/l5Z4nweP36e99nHv2VxEVIBERkXruu9Xvc8rToOk5C+E3Off1f4qoAImIiNRzti0fARB1IRhXN3eT09QOKkAiIiL1nC17GwDWdjEmJ6k9VIBERETqsYyfd7DN/xwAUdbHTE5Te6gAiYiI1GOrbHMBCD3lTUCbrianqT1UgEREROox275VAFh9e5qcpHZRARIREamn8nLPs9rjVwCsve8xOU3togIkIiJST235ahEnPQ0an7fQO/J+s+PUKipAIiIi9ZRt84cAROW01vT3S6gAiYiI1FO2rO8BiGkbZXKS2kcFSEREpB46cmAnyf5nAYgZMtHkNLWPCpCIiEg9tCqucPp7j1NeBLbtZnKa2kcFSEREpB6y7YsHwNqoh8lJaicVIBERkXom/0Iuq9zSALD2utvkNLWTCpCIiEg9k/T1B5zwMvA/b6FP1ANmx6mVVIBERETqGdt3/wYgMucqGrh7mpymdlIBEhERqWdsmckAxFx9s8lJai8VIBERkXrkaFoqW/3OABBjnWBymtpLBUhERKQeWR03F8MC3U950qq9boBaGtML0Lx582jTpg2enp6Ehoayfv36Usemp6dz11130aFDB1xcXJg8eXKxMQsXLsRisRR7nD9/vhq3QkREpHaw7bUBYPW53twgtZypBSg2NpbJkyczbdo0tm3bRv/+/bFaraSlpZU4Picnh+bNmzNt2jS6d+9e6np9fX1JT093eHh66iQwERGp3wry81jl9gsA1vC7TE5Tu5lagGbNmsXYsWN58MEH6dSpE7NnzyYoKIi33nqrxPFXX301r7/+Ovfeey9+fn6lrtdisRAYGOjwKEtOTg5ZWVkODxERkbpm69f/5piXgW8ORESNNTtOrWZaAcrNzSU5OZmoKMcbtEVFRbFp06YrWvfp06cJCQmhdevWDBs2jG3btpU5fsaMGfj5+dkfQUFBV/T5IiIiZrAlfgBA5PmrcPP0NjlN7WZaATp27Bj5+fkEBAQ4LA8ICCAjI6PS6+3YsSMLFy5kxYoVLF68GE9PT/r168fevXtLfc/UqVPJzMy0Pw4ePFjpzxcRETFL/KmtAMSE3GRyktqvgdkBLBaLw3PDMIotq4g+ffrQp08f+/N+/frRs2dP5s6dy5w5c0p8j4eHBx4eHpX+TBEREbMd/3Uvm/1OAxAT85jJaWo/044ANWvWDFdX12JHe44cOVLsqNCVcHFxITw8vMwjQCIiInXd6i/nYFjgukxPWncINztOrWdaAXJ3dyc0NJSEhASH5QkJCfTt27fKPscwDFJSUmjZsmWVrVNERKS2se2NA8Dq3c3kJHWDqV+BTZkyhdGjRxMWFkZERATvvPMOaWlpjB8/Hig8N+fQoUN88MEH9vekpKQAhSc6Hz16lJSUFNzd3encuTMAL730En369KF9+/ZkZWUxZ84cUlJSePPNN2t8+0RERGpCQX4e8a77AbCG/cnkNHWDqQVo1KhRHD9+nOnTp5Oenk7Xrl2Ji4sjJCQEKLzw4aXXBOrRo4f95+TkZD7++GNCQkI4cOAAAKdOneKhhx4iIyMDPz8/evTowbp16+jVq1eNbZeIiEhN+v7bjznqbdAoB/pGP2h2nDrBYhiGYXaI2iYrKws/Pz8yMzPx9fU1O46IiEiZ/m/6TTxvfMOtp1ry6WuHzY5jmor8/Tb9VhgiIiJyZeJPJAFgDbnR5CR1hwqQiIhIHXbi8D6+88sGwBqtu7+XlwqQiIhIHZbw5RwKXKBLpgdBnXqbHafOUAESERGpw2x7vgTA6nWdyUnqFhUgERGROsph+nvoH01OU7eoAImIiNRRKWtj+c27gIa5cIP1YbPj1CkVKkAzZ87k3Llz9ufr1q0jJyfH/jw7O5tHHnmk6tKJiIhIqeI3LgLgprOBuHv5mJymbqlQAZo6dSrZ2dn258OGDePQoUP252fPnuXtt9+uunQiIiJSKtvxzQBYgwabnKTuqVABuvSaibqGooiIiDlO/fYLiX5ZAFijHzU5Td2jc4BERETqoISVr5PvAp0y3Qnp0s/sOHWOCpCIiEgdZNu9EgCrp6a/V0aFb4b67rvv4uNTeKJVXl4eCxcupFmzZgAO5weJiIhI9TAKCoi37APA2vNOk9PUTRW6GerVV1+NxWK57Lj9+/dfUSiz6WaoIiJSm6WsjaXHmj/inQsn/pKJR0P9rYKK/f2u0BGgAwcOXEkuERERqQLx6xcAcOPZFio/laRzgEREROoY27HvALC21vT3yqpQAdq8eTM2m81h2QcffECbNm1o0aIFDz30kMOFEUVERKRqZR5JY6NfJgDWSF18uLIqVIBefPFFduzYYX/+ww8/MHbsWG6++WaeeeYZvvjiC2bMmFHlIUVERKTQVyvnku8CHbLcadNtgNlx6qwKFaCUlBRuuukm+/MlS5bQu3dv/vWvfzFlyhTmzJnD0qVLqzykiIiIFLKlrgDA6t7F5CR1W4UK0MmTJwkICLA/X7t2LTExMfbn4eHhHDx4sOrSiYiIiJ1RUEA8/wXA2uMOk9PUbRUqQAEBAfYp7rm5uXz//fdERETYX8/OzsbNza1qE4qIiAgAP2xcziGfArwuwIAhfzY7Tp1WoQIUExPDM888w/r165k6dSre3t7079/f/vqOHTto165dlYcUERERiF/3PgCDzzTH08ff3DB1XIWuA/TXv/6V2267jYEDB+Lj48PChQtxd3e3v/7+++8TFRVV5SFFREQEbEcToTFYWw00O0qdV6EC1Lx5c9avX09mZiY+Pj64uro6vP7JJ5/QqFGjKg0oIiIikHX0Vzb4ngLAGqmvv65UhQrQAw88UK5x77//fqXCiIiISMm+/nIuea7QPsuNdtffaHacOq9CBWjhwoWEhITQo0cPKnALMREREblCtl0roCFY3TubHaVeqFABGj9+PEuWLOHnn3/mgQce4J577qFJkybVlU1EREQonP5uM/YCYO1+u8lp6ocKzQKbN28e6enpPP3003zxxRcEBQVx5513smrVKh0REhERqSa7vvuCX33y8bwAA4fo9hdVocI3Q/Xw8OBPf/oTCQkJ7Nq1iy5duvDII48QEhLC6dOnqyOjiIiIU7Ot/RcAg840w8tX37xUhSu6G7zFYsFisWAYBgUFBVWVSURERC5i+20TANaWuvdXValwAcrJyWHx4sVERkbSoUMHfvjhB9544w3S0tLw8fGpjowiIiJOK/v4Ydb7ngTAetN4k9PUHxU6CfqRRx5hyZIlBAcHc//997NkyRKaNm1aXdlERESc3jdfvsEFV2iX1YD2oZFmx6k3KlSA5s+fT3BwMG3atGHt2rWsXbu2xHHLly+vknAiIiLOzvbjZ4XT3906mR2lXqlQAbr33nuxWCzVlUVEREQuUjj9/ScAYq67zeQ09UuFL4QoIiIiNWN3UhxpPvl45MHgYY+ZHadeuaJZYCIiIlJ9bN8WTn8fmN0Ub79mJqepX1SAREREailbxgYArC37m5yk/lEBEhERqYVOn8hgXaMTAFhvfMjkNPWPCpCIiEgt9O2Xb5LbANpkN+Da0Giz49Q7KkAiIiK1kO3HTwGwunbE4qI/11VNe1RERKSWMQoKsOXvASCm663mhqmnVIBERERqmZ+SV3GgUR7ueXDjsAlmx6mXVIBERERqGds37wAwILsJDRu3MDlN/aQCJCIiUsvY0tcDYA28weQk9ZcKkIiISC1yNvMYaxsdB8A6eJzJaeovFSAREZFa5NuVb5DTAEKyXekYPsTsOPWWCpCIiEgtYvthOQBW1w6a/l6NtGdFRERqkfgLqQDEdBlucpL6TQVIRESkltibnMA+3zzc8uHGobr7e3VSARIREaklbF/PB6B/VmMaNW1lcpr6TQVIRESklrClrwPAGtDX5CT1nwqQiIhILXAu6wRrGh4DwDpQ09+rmwqQiIhILbDmyzc47wZBp13p3OcWs+PUeypAIiIitUD8jsLp7zGW9pr+XgNM38Pz5s2jTZs2eHp6Ehoayvr160sdm56ezl133UWHDh1wcXFh8uTJJY5btmwZnTt3xsPDg86dO/Ppp59WU3oREZGqYcvdBYC18x9MTuIcTC1AsbGxTJ48mWnTprFt2zb69++P1WolLS2txPE5OTk0b96cadOm0b179xLHJCYmMmrUKEaPHs327dsZPXo0d955J5s3b67OTREREam0fSnfsNf3Ag3y4aahuvt7TbAYhmGY9eG9e/emZ8+evPXWW/ZlnTp14tZbb2XGjBllvnfQoEFcf/31zJ4922H5qFGjyMrKwmaz2ZfFxMTQuHFjFi9eXOK6cnJyyMnJsT/PysoiKCiIzMxMfH19K7FlIiIi5ffGP+5gwtn/MOikP9/OPml2nDorKysLPz+/cv39Nu0IUG5uLsnJyURFRTksj4qKYtOmTZVeb2JiYrF1RkdHl7nOGTNm4OfnZ38EBQVV+vNFREQqynZ4LQDW5hEmJ3EephWgY8eOkZ+fT0BAgMPygIAAMjIyKr3ejIyMCq9z6tSpZGZm2h8HDx6s9OeLiIhUxPnTp/i24VEArAPHmpzGeTQwO4DFYnF4bhhGsWXVvU4PDw88PDyu6DNFREQqY+2Xb3LODa467ULXviPMjuM0TDsC1KxZM1xdXYsdmTly5EixIzgVERgYWOXrFBERqS7xKcsAiOEaTX+vQabtaXd3d0JDQ0lISHBYnpCQQN++lb8EeERERLF1rl69+orWKSIiUl1suTsBsHbS9PeaZOpXYFOmTGH06NGEhYURERHBO++8Q1paGuPHjwcKz805dOgQH3zwgf09KSkpAJw+fZqjR4+SkpKCu7s7nTt3BmDSpEkMGDCAV155heHDh/P555/z1VdfsWHDhhrfPhERkbLs37GOPb65NMiHm4dp+ntNMrUAjRo1iuPHjzN9+nTS09Pp2rUrcXFxhISEAIUXPrz0mkA9evSw/5ycnMzHH39MSEgIBw4cAKBv374sWbKEZ599lueee4527doRGxtL7969a2y7REREysOWMA+Avll++LUINjmNczH1OkC1VUWuIyAiIlJZtzwewEr/I8xwjeaZZ+PNjlPn1YnrAImIiDiz86dP8Y33EQCsAx4wOY3zUQESERExwYb4dzjrDi3PuNDthtvNjuN0VIBERERMYPt+KQAxRjtNfzeB9riIiIgJbOd/AMDacZjJSZyTCpCIiEgN+2XnRlL9cnEtgMhhk8yO45RUgERERGqYbdWbAERk+uIfEGJyGuekAiQiIlLDbAe/BcDaVNeoM4sKkIiISA3KOZPF196F96y09r/f5DTOSwVIRESkBm1c9S/OuEPAWRe697/D7DhOSwVIRESkBtmSlwAQk98GF1dT70jl1FSAREREapDt3O/T3zsMNTmJc1MBEhERqSEHUzez0y8HlwKIHDrR7DhOTQVIRESkhthWzQWgT2YjmrRqZ3Ia56YCJCIiUkNsv3wDgLVJL5OTiAqQiIhIDcg9d5qvvdIBsPYbY24YUQESERGpCZtWvUu2B7Q4a6HHoD+aHcfpqQCJiIjUANvWxQBEa/p7raACJCIiUgNsZ3cAYG0/xOQkAipAIiIi1e7XPUn84HcelwKI0vT3WkEFSEREpJrFx78BQK8sH5q2bm9yGgEVIBERkWpn++UrAKz+4SYnkSIqQCIiItXowvmzfOV5GICYiNEmp5EiKkAiIiLVKHH1e2R5QLNzFsJuUgGqLVSAREREqpEt6WMAoi+EaPp7LaICJCIiUo1sp1MAsLa3mhtEHKgAiYiIVJPDe79nu/95LAZED51kdhy5iAqQiIhINYm3Fd79PTyzIc2COpicRi6mAiQiIlJN4g/8Pv3dL8zkJHIpFSAREZFqkJd7ngSPQwDE9LnH5DRyKRUgERGRavDd6vc55WnQ9JyF8JvuNTuOXEIFSEREpBrYtnwEQNSFYFzd3E1OI5dSARIREakGtuxtAFjbxZicREqiAiQiIlLFMn7ewTb/cwBED5lgchopiQqQiIhIFYuPmwNA2ClvWlzdxeQ0UhIVIBERkSoW//NqAKy+oSYnkdKoAImIiFShvNzzrPb4FYCY3nebnEZKowIkIiJShbZ8tYiTngaNz1voHXm/2XGkFCpAIiIiVci2+UMAonJaa/p7LaYCJCIiUoVsWd8DYG0XbXISKYsKkIiISBX5bf+PJPufBSBmiO7+XpupAImIiFSRVb9Pf+95youANl1NTiNlUQESERGpIkXT32Ma9TA5iVyOCpCIiEgVyL+Qyyq3NACsvTT9vbZTARIREakCSV9/wAkvA//zFvpEPWB2HLkMFSAREZEqYPvu3wBE5lxFA3dPk9PI5agAiYiIVAFbZjIA1jaRJieR8lABEhERuUJH01LZ6ncGgJghE01OI+WhAiQiInKFVsfNxbDA9ae8aNnuerPjSDmoAImIiFwh214bADE+3U1OIuWlAiQiInIFCvLzWOX2CwDW8LtMTiPlpQIkIiJyBbZ+/W+OeRn45kBE1Fiz40g5qQCJiIhcAVviBwBEnr8KN09vk9NIeZlegObNm0ebNm3w9PQkNDSU9evXlzl+7dq1hIaG4unpSdu2bZk/f77D6wsXLsRisRR7nD9/vjo3Q0REnJTtVBIA1qtvNjmJVISpBSg2NpbJkyczbdo0tm3bRv/+/bFaraSlpZU4fv/+/QwZMoT+/fuzbds2/vKXvzBx4kSWLVvmMM7X15f09HSHh6enLkolIiJV69jBPWwpmv4e85jJaaQiGpj54bNmzWLs2LE8+OCDAMyePZtVq1bx1ltvMWPGjGLj58+fT3BwMLNnzwagU6dObN26lVdffZWRI0fax1ksFgIDA2tkG0RExHklxL2BYYHrMj256tows+NIBZh2BCg3N5fk5GSioqIclkdFRbFp06YS35OYmFhsfHR0NFu3buXChQv2ZadPnyYkJITWrVszbNgwtm3bVmaWnJwcsrKyHB4iIiKXY9sbB4DVu5vJSaSiTCtAx44dIz8/n4CAAIflAQEBZGRklPiejIyMEsfn5eVx7NgxADp27MjChQtZsWIFixcvxtPTk379+rF3795Ss8yYMQM/Pz/7Iygo6Aq3TkRE6ruC/DziXfcDYA37k8lppKJMPwnaYrE4PDcMo9iyy42/eHmfPn2455576N69O/3792fp0qVce+21zJ07t9R1Tp06lczMTPvj4MGDld0cERFxEt9/+zFHvQ0a5UC/mIfMjiMVZNo5QM2aNcPV1bXY0Z4jR44UO8pTJDAwsMTxDRo0oGnTpiW+x8XFhfDw8DKPAHl4eODh4VHBLRAREWdm27QIgJvPtdT09zrItCNA7u7uhIaGkpCQ4LA8ISGBvn37lvieiIiIYuNXr15NWFgYbm5uJb7HMAxSUlJo2bJl1QQXEREBbCe2AGAN0fT3usjUr8CmTJnCu+++y/vvv09qaiqPP/44aWlpjB8/Hij8auree++1jx8/fjy//PILU6ZMITU1lffff5/33nuPJ554wj7mpZdeYtWqVfz888+kpKQwduxYUlJS7OsUERG5UicO72Oz32kAYqIfNTmNVIap0+BHjRrF8ePHmT59Ounp6XTt2pW4uDhCQkIASE9Pd7gmUJs2bYiLi+Pxxx/nzTffpFWrVsyZM8dhCvypU6d46KGHyMjIwM/Pjx49erBu3Tp69epV49snIiL1U8KXcyhwgS6ZHgR16m12HKkEi1F0FrHYZWVl4efnR2ZmJr6+vmbHERGRWmbME9ewqNE+nsgN4x8vJ5kdR35Xkb/fps8CExERqUscpr+H/tHkNFJZKkAiIiIVkLI2lt+8C/DJhRusD5sdRypJBUhERKQCbBsXAnDT2UDcvXzMDSOVpgIkIiJSAbbjmwGwBt9ochK5EipAIiIi5XQyfT+JftkAxEQ9YnIauRIqQCIiIuX0VdxcClygU6Y7IV36mR1HroAKkIiISDnZdq8EwOp5nclJ5EqpAImIiJSDUVBAvGUfANaed5qcRq6UCpCIiEg5bF//CekNC2iYC/2tur1SXacCJCIiUg629e8DcOPZADwa6i4BdZ0KkIiISDnEH/t9+nvrwSYnkaqgAiQiInIZmUfS2OiXCUBM5J9NTiNVQQVIRETkMr5aOZd8F+iQ5U6bbgPMjiNVQAVIRETkMmypKwCwuncxOYlUFRUgERGRMhgFBcTzXwCsPe4wOY1UFRUgERGRMvywcTmHfArwzoUBQ3T+T32hAiQiIlIG27r3ABh8tgWePv7mhpEqowIkIiJShvij3wFgvWqgyUmkKqkAiYiIlCLr6K9s8D0FQMzNuvpzfaICJCIiUoqvv5xLniu0z3Kj3fU3mh1HqpAKkIiISClsu4qmv3c2OYlUNRUgERGREhgFBdiMvQBYu99uchqpaipAIiIiJdiZ+Dm/+uTjeQEGDnnE7DhSxVSAREREShBfNP39THO8fJuYnEaqmgqQiIhICWy/bQLA2kr3/qqPVIBEREQukX38MOt9TwIQc+PDJqeR6qACJCIicolvvnyDC67QLqsB7UMjzY4j1UAFSERE5BK2Hz8DwOrWydwgUm1UgERERC5SOP39JwCs3UaanEaqiwqQiIjIRVK3rCTNJx+PPBg09FGz40g1UQESERG5SPyawunvg043xduvmclppLqoAImIiFzElrEBgJjA/iYnkeqkAiQiIvK70ycyWNfoBADWGx8yOY1UJxUgERGR33375ZvkNoA22Q24NjTa7DhSjVSAREREfmf78VMArK4dsbjoT2R9pn+6IiIi/D79PX8PANbrRpicRqqbCpCIiAiwOymOA43ycM+DwUMfMzuOVLMGZgcQERGpaRfOn+XH7z5na4qNpMNJJF34hR8bnQNXGJjdhIaNW5gdUaqZCpCIiNRrBfl57NkaT9LWFSQd/I6tOftJaXia826/D2j4v7EBZ12Y0udxU3JKzVIBEhGResMoKGD/D+vYmvQ5Sfs3kHRmL8kNMznt/vsAr98fgN95CDvXmDCfawlv25/wPrcR1LG3Tn52EipAIiJSZx3e+z1Jif8h6b/r2Jq9m60eJzjuZRS+6P77A/C6AD1PNyLc6xrCQyIIC/sD1/S4CRdX/Rl0VvonLyIidcLxX/eydeNSkvZ8S9LJnWx1O8LhhgWFL7oC/oU/uuVD92xvwt3bEHZVOOE9h9Gp11AauHuaFV1qIRUgERGpdbKPHyZ5/VKSdiWw9dgPJLmks79R3v8G+Bf+j0sBdM72INw1mPDAUMK6xdCt7wg8GvqaklvqDhUgEREx1bmsE6Rs/A9bf1xN0m/bSCr4lT2+uRiW3wf4/W9s+yw3wmlFWPPrCe8aRY9+t2vGllSKCpCIiNSYounnSSlfknRoK1vz0vix0TnyXH8f0Oh/Y4NPuxKWH0B4k+sI73QTPfveTuOWbUzJLfWPCpCIiFSL/Au57EmOZ2vySpIOfkdSzs+k+Jwhp+gvj8//xrY4ayE8tznh/p0Ju2YgYX1vJ6BNV1Nyi3NQARIRkStWNP08acunJB3YyNYz/73s9PPwRh1+n34+ktYdwjX9XGqUCpCIiFTYoZ+2kpS4jK371pOUtZutnic4UcL0c+9c6HnGl3DvawgL7kN4+K20u36wpp+L6fQbKCIiZTp2cA9bN35C0k9F08+Pkn7x9PPGhT+650H30w0Jc7+a8Na9CO95Cx3DrZp+LrWSCpCIiNhlHf2V5A1LSdr1FVuP/0CSSwYHSpl+3iXbk3DXIMJbhhHWLYbrIm7V9HOpM1SAREScVNH086QfVpH02za2GodKnX5+bZYb4VxFWIvfp5/fcAfefs1MyS1SFVSAREScwIXzZ/lh06ckbY9j6+FkkvJ+4Uff8+QXnXd80YGbkGxXwgoCCW96HeGdbqZnv9vxDwgxJbdIdVEBEhGpZ/Iv5LI7ycbWbStJOriZpJyf2V7K9POAsy6E5zYj3L8LYe0Lp5+3uLqLKblFapLpBWjevHn84x//ID09nS5dujB79mz69+9f6vi1a9cyZcoUdu7cSatWrXjqqacYP368w5hly5bx3HPPsW/fPtq1a8fLL7/MiBEjqntTRERqnFFQwM871pC05TOSDmxi69n/8n0p08/9z1sIP9eYsEYdCG/Xn/CI27mqfaimn4tTMrUAxcbGMnnyZObNm0e/fv14++23sVqt7Nq1i+Dg4GLj9+/fz5AhQxg3bhwffvghGzdu5JFHHqF58+aMHDkSgMTEREaNGsX//d//MWLECD799FPuvPNONmzYQO/evWt6E0VEqoxRUMChvcls/W45SfvWFU4/9zrJSc/fp597/P4AGtqnn7cnLCSC8F630q77YJUdkd9ZDMMwzPrw3r1707NnT9566y37sk6dOnHrrbcyY8aMYuOffvppVqxYQWpqqn3Z+PHj2b59O4mJiQCMGjWKrKwsbDabfUxMTAyNGzdm8eLF5cqVlZWFn58fmZmZ+PpW3YyGnDNZpO/fUWXrqylGQYHZESrFMMrOffnXy/5X43L75Uo/v1wZrnQbC65s/UVjCgryMQyj8Of8fAyjAMMwKCjI+9/yi8eU6+eC/63HyP99fRe9blz0ekE+BkbZPxtF7zUc32sUXDTm9+UYhZ9/8XKKXjcq8LPxv3VQwvKi8fC/MaX8nFlwjmS3Y2Q0LP7PxD0Prj/dkDD3NoQH9bZPP3d1cy82VqQ+q8jfb9OOAOXm5pKcnMwzzzzjsDwqKopNmzaV+J7ExESioqIclkVHR/Pee+9x4cIF3NzcSExM5PHHHy82Zvbs2aVmycnJIScnx/48KyurgltTPtvWLyVi87hqWbeI1DDL748a5loAXbI8CW8QTHirMMK6Wbmu7624e/lc/s0iYmdaATp27Bj5+fkEBAQ4LA8ICCAjI6PE92RkZJQ4Pi8vj2PHjtGyZctSx5S2ToAZM2bw0ksvVXJLys/FxRWvC9X+MdXCYtpxwitT1t+ny/3tKmubr+y9Zb+7zMyX+edg1va6GIVb5WIUbl+ZP2PBYvC/nyl8rdiYouW//6/FKFp+8RjLJWMtWCwX/Wwf6/K/MZYSllsuHn/xmFJ+tlgK32v5fT0Wl4vGuGCxcNHrRcsq8bPFgovFFQsWPN286N5pENf3G6np5yJVwPSToC0Wx/+0GoZRbNnlxl+6vKLrnDp1KlOmTLE/z8rKIigo6PLhK6hX1P2cjbq/ytcrIiIiFWNaAWrWrBmurq7FjswcOXKk2BGcIoGBgSWOb9CgAU2bNi1zTGnrBPDw8MDDw6MymyEiIiJ1kGnTAdzd3QkNDSUhIcFheUJCAn379i3xPREREcXGr169mrCwMNzc3MocU9o6RURExPmY+hXYlClTGD16NGFhYURERPDOO++QlpZmv67P1KlTOXToEB988AFQOOPrjTfeYMqUKYwbN47ExETee+89h9ldkyZNYsCAAbzyyisMHz6czz//nK+++ooNGzaYso0iIiJS+5hagEaNGsXx48eZPn066enpdO3albi4OEJCCi+5np6eTlpamn18mzZtiIuL4/HHH+fNN9+kVatWzJkzx34NIIC+ffuyZMkSnn32WZ577jnatWtHbGysrgEkIiIidqZeB6i2qq7rAImIiEj1qcjfb10SVERERJyOCpCIiIg4HRUgERERcToqQCIiIuJ0VIBERETE6agAiYiIiNNRARIRERGnowIkIiIiTkcFSERERJyOqbfCqK2KLo6dlZVlchIREREpr6K/2+W5yYUKUAmys7MBCAoKMjmJiIiIVFR2djZ+fn5ljtG9wEpQUFDA4cOHadSoERaLpUrXnZWVRVBQEAcPHtR9xi5D+6r8tK/KT/uq/LSvKkb7q/yqa18ZhkF2djatWrXCxaXss3x0BKgELi4utG7dulo/w9fXV/+ClJP2VflpX5Wf9lX5aV9VjPZX+VXHvrrckZ8iOglaREREnI4KkIiIiDgdFaAa5uHhwQsvvICHh4fZUWo97avy074qP+2r8tO+qhjtr/KrDftKJ0GLiIiI09ERIBEREXE6KkAiIiLidFSARERExOmoAImIiIjTUQGqBXJycrj++uuxWCykpKSYHadW+sMf/kBwcDCenp60bNmS0aNHc/jwYbNj1ToHDhxg7NixtGnTBi8vL9q1a8cLL7xAbm6u2dFqrZdffpm+ffvi7e2Nv7+/2XFqlXnz5tGmTRs8PT0JDQ1l/fr1ZkeqldatW8ctt9xCq1atsFgsfPbZZ2ZHqpVmzJhBeHg4jRo1okWLFtx6663s2bPHtDwqQLXAU089RatWrcyOUasNHjyYpUuXsmfPHpYtW8a+ffu4/fbbzY5V6+zevZuCggLefvttdu7cyWuvvcb8+fP5y1/+Yna0Wis3N5c77riDP//5z2ZHqVViY2OZPHky06ZNY9u2bfTv3x+r1UpaWprZ0WqdM2fO0L17d9544w2zo9Rqa9eu5dFHH+W7774jISGBvLw8oqKiOHPmjDmBDDFVXFyc0bFjR2Pnzp0GYGzbts3sSHXC559/blgsFiM3N9fsKLXezJkzjTZt2pgdo9ZbsGCB4efnZ3aMWqNXr17G+PHjHZZ17NjReOaZZ0xKVDcAxqeffmp2jDrhyJEjBmCsXbvWlM/XESAT/fbbb4wbN45///vfeHt7mx2nzjhx4gQfffQRffv2xc3Nzew4tV5mZiZNmjQxO4bUIbm5uSQnJxMVFeWwPCoqik2bNpmUSuqbzMxMANP++6QCZBLDMBgzZgzjx48nLCzM7Dh1wtNPP03Dhg1p2rQpaWlpfP7552ZHqvX27dvH3LlzGT9+vNlRpA45duwY+fn5BAQEOCwPCAggIyPDpFRSnxiGwZQpU7jhhhvo2rWrKRlUgKrYiy++iMViKfOxdetW5s6dS1ZWFlOnTjU7smnKu6+KPPnkk2zbto3Vq1fj6urKvffei+EkFzKv6L4COHz4MDExMdxxxx08+OCDJiU3R2X2lxRnsVgcnhuGUWyZSGU89thj7Nixg8WLF5uWQbfCqGLHjh3j2LFjZY65+uqr+eMf/8gXX3zh8B+T/Px8XF1dufvuu1m0aFF1RzVdefeVp6dnseW//vorQUFBbNq0iYiIiOqKWGtUdF8dPnyYwYMH07t3bxYuXIiLi3P9f53K/G4tXLiQyZMnc+rUqWpOV/vl5ubi7e3NJ598wogRI+zLJ02aREpKCmvXrjUxXe1msVj49NNPufXWW82OUmtNmDCBzz77jHXr1tGmTRvTcjQw7ZPrqWbNmtGsWbPLjpszZw5//etf7c8PHz5MdHQ0sbGx9O7duzoj1hrl3VclKertOTk5VRmp1qrIvjp06BCDBw8mNDSUBQsWOF35gSv73RJwd3cnNDSUhIQEhwKUkJDA8OHDTUwmdZlhGEyYMIFPP/2UNWvWmFp+QAXINMHBwQ7PfXx8AGjXrh2tW7c2I1KttWXLFrZs2cINN9xA48aN+fnnn3n++edp166dUxz9qYjDhw8zaNAggoODefXVVzl69Kj9tcDAQBOT1V5paWmcOHGCtLQ08vPz7dfiuuaaa+z/XjqjKVOmMHr0aMLCwoiIiOCdd94hLS1N55OV4PTp0/z3v/+1P9+/fz8pKSk0adKk2H/rndmjjz7Kxx9/zOeff06jRo3s55P5+fnh5eVV84FMmXsmxezfv1/T4EuxY8cOY/DgwUaTJk0MDw8P4+qrrzbGjx9v/Prrr2ZHq3UWLFhgACU+pGT33Xdfifvr22+/NTua6d58800jJCTEcHd3N3r27GnadOXa7ttvvy3xd+i+++4zO1qtUtp/mxYsWGBKHp0DJCIiIk7H+U4OEBEREaenAiQiIiJORwVIREREnI4KkIiIiDgdFSARERFxOipAIiIi4nRUgERERMTpqACJiIiI01EBEpFyGzRoEJMnTzY7RomOHz9OixYtOHDgAABr1qzBYrFU+81NK/s5CxcuxN/fv0LvCQ8PZ/ny5RV6j4iUTAVIREyTnp7OXXfdRYcOHXBxcSm1XC1btozOnTvj4eFB586d+fTTT4uNmTFjBrfccgtXX3119YY20XPPPcczzzxDQUGB2VFE6jwVIBExTU5ODs2bN2fatGl07969xDGJiYmMGjWK0aNHs337dkaPHs2dd97J5s2b7WPOnTvHe++9x4MPPlhT0U0xdOhQMjMzWbVqldlRROo8FSARqZSTJ09y77330rhxY7y9vbFarezdu9dhzL/+9S+CgoLw9vZmxIgRzJo1y+Frn6uvvprXX3+de++9Fz8/vxI/Z/bs2URGRjJ16lQ6duzI1KlTuemmm5g9e7Z9jM1mo0GDBkRERJSa9/jx4/zpT3+idevWeHt7c91117F48WKHMYMGDWLChAlMnjyZxo0bExAQwDvvvMOZM2e4//77adSoEe3atcNmsxVb/8aNG+nevTuenp707t2bH374weH1hQsXEhwcbN8Xx48fd3h93759DB8+nICAAHx8fAgPD+err75yGOPq6sqQIUOK5RaRilMBEpFKGTNmDFu3bmXFihUkJiZiGAZDhgzhwoULQGEhGD9+PJMmTSIlJYXIyEhefvnlCn9OYmIiUVFRDsuio6PZtGmT/fm6desICwsrcz3nz58nNDSUlStX8uOPP/LQQw8xevRohyNJAIsWLaJZs2Zs2bKFCRMm8Oc//5k77riDvn378v333xMdHc3o0aM5e/asw/uefPJJXn31VZKSkmjRogV/+MMf7Pti8+bNPPDAAzzyyCOkpKQwePBg/vrXvzq8//Tp0wwZMoSvvvqKbdu2ER0dzS233EJaWprDuF69erF+/fry7TwRKZ0p96AXkTpp4MCBxqRJk4yffvrJAIyNGzfaXzt27Jjh5eVlLF261DAMwxg1apQxdOhQh/fffffdhp+fX5nrvpSbm5vx0UcfOSz76KOPDHd3d/vz4cOHGw888IDDmG+//dYAjJMnT5a6PUOGDDH+3//7fw4ZbrjhBvvzvLw8o2HDhsbo0aPty9LT0w3ASExMdPicJUuW2MccP37c8PLyMmJjYw3DMIw//elPRkxMjMNnjxo1qtR9UaRz587G3LlzHZZ9/vnnhouLi5Gfn1/me0WkbDoCJCIVlpqaSoMGDejdu7d9WdOmTenQoQOpqakA7Nmzh169ejm879Ln5WWxWByeG4bhsOzcuXN4enqWuY78/HxefvllunXrRtOmTfHx8WH16tXFjrB069bN/rOrqytNmzbluuuusy8LCAgA4MiRIw7vu/jrtyZNmjjsi9TU1GJfz136/MyZMzz11FN07twZf39/fHx82L17d7F8Xl5eFBQUkJOTU+b2ikjZGpgdQETqHsMwSl1eVEwuLSllva8sgYGBZGRkOCw7cuSIvYgANGvWjJMnT5a5nn/+85+89tprzJ49m+uuu46GDRsyefJkcnNzHca5ubk5PLdYLA7LirapPDOxLt4Xl/Pkk0+yatUqXn31Va655hq8vLy4/fbbi+U7ceIE3t7eeHl5XXadIlI6HQESkQrr3LkzeXl5DufPHD9+nJ9++olOnToB0LFjR7Zs2eLwvq1bt1b4syIiIkhISHBYtnr1avr27Wt/3qNHD3bt2lXmetavX8/w4cO555576N69O23bti120vaV+O677+w/nzx5kp9++omOHTsChfvr4tcvHV+Ub8yYMYwYMYLrrruOwMBA+zWNLvbjjz/Ss2fPKsst4qxUgESkwtq3b8/w4cMZN24cGzZsYPv27dxzzz1cddVVDB8+HIAJEyYQFxfHrFmz2Lt3L2+//TY2m63YUaGUlBRSUlI4ffo0R48eJSUlxaHMTJo0idWrV/PKK6+we/duXnnlFb766iuHawZFR0ezc+fOMo8CXXPNNSQkJLBp0yZSU1N5+OGHix1ZuhLTp0/n66+/5scff2TMmDE0a9aMW2+9FYCJEycSHx/PzJkz+emnn3jjjTeIj48vlm/58uWkpKSwfft27rrrrhKPMq1fv77YSeEiUnEqQCJSKQsWLCA0NJRhw4YRERGBYRjExcXZvy7q168f8+fPZ9asWXTv3p34+Hgef/zxYufq9OjRgx49epCcnMzHH39Mjx49GDJkiP31vn37smTJEhYsWEC3bt1YuHAhsbGxDucfXXfddYSFhbF06dJS8z733HP07NmT6OhoBg0aRGBgoL2gVIW///3vTJo0idDQUNLT01mxYgXu7u4A9OnTh3fffZe5c+dy/fXXs3r1ap599lmH97/22ms0btyYvn37cssttxAdHV3sSM+hQ4fYtGkT999/f5XlFnFWFqMyX8qLiFTCuHHj2L17d7VM446Li+OJJ57gxx9/xMWlfv5/uyeffJLMzEzeeecds6OI1Hk6CVpEqs2rr75KZGQkDRs2xGazsWjRIubNm1ctnzVkyBD27t3LoUOHCAoKqpbPMFuLFi144oknzI4hUi/oCJCIVJs777yTNWvWkJ2dTdu2bZkwYQLjx483O5aIiAqQiIiIOJ/6+UW5iIiISBlUgERERMTpqACJiIiI01EBEhEREaejAiQiIiJORwVIREREnI4KkIiIiDgdFSARERFxOv8fYdQDOCkewewAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 9,
+ "id": "e413d7b0",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
@@ -2626,7 +2714,7 @@
"x = np.random.rand(n)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)\n",
"\n",
- "Maxpolydegree = 5\n",
+ "Maxpolydegree = 20\n",
"X = np.zeros((n,Maxpolydegree))\n",
"#We include explicitely the intercept column\n",
"for degree in range(Maxpolydegree):\n",
@@ -2676,8 +2764,10 @@
},
{
"cell_type": "markdown",
- "id": "26406e58",
- "metadata": {},
+ "id": "1e1c1a41",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n",
"We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n",
@@ -2687,137 +2777,13 @@
},
{
"cell_type": "code",
- "execution_count": 6,
- "id": "ce41d0b7",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Beta values for own Ridge implementation\n",
- "[ 3.43579948e-02 -5.43330971e-01 -3.10141413e-03 2.47116868e-01\n",
- " 2.18613217e-01 1.02054837e-01 -4.25617658e-04 -5.90475506e-02\n",
- " -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02\n",
- " 1.11482289e-02 2.88529063e-02 3.67047975e-02 3.38135733e-02\n",
- " 2.02198703e-02 -3.46383925e-03 -3.63025821e-02]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[ 3.43579948e-02 -5.43330971e-01 -3.10141413e-03 2.47116868e-01\n",
- " 2.18613217e-01 1.02054837e-01 -4.25617654e-04 -5.90475506e-02\n",
- " -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02\n",
- " 1.11482289e-02 2.88529063e-02 3.67047975e-02 3.38135733e-02\n",
- " 2.02198702e-02 -3.46383925e-03 -3.63025821e-02]\n",
- "Intercept from own implementation:\n",
- "1.0330308045188872\n",
- "Intercept from Scikit-Learn Ridge implementation\n",
- "1.0330308045183219\n",
- "MSE values for own Ridge implementation\n",
- "3.1392559591206444e-06\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "3.1392559585048734e-06\n",
- "Beta values for own Ridge implementation\n",
- "[-0.05807125 -0.29822833 -0.08551306 0.08156108 0.13679863 0.12333649\n",
- " 0.08251519 0.03815288 0.00111756 -0.02498832 -0.04010697 -0.04566964\n",
- " -0.04355837 -0.03562355 -0.02348765 -0.00848904 0.00831018 0.0260906\n",
- " 0.04423486]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[-0.05807125 -0.29822833 -0.08551306 0.08156108 0.13679863 0.12333649\n",
- " 0.08251519 0.03815288 0.00111756 -0.02498832 -0.04010697 -0.04566964\n",
- " -0.04355837 -0.03562355 -0.02348765 -0.00848904 0.00831018 0.0260906\n",
- " 0.04423486]\n",
- "Intercept from own implementation:\n",
- "1.041148729430595\n",
- "Intercept from Scikit-Learn Ridge implementation\n",
- "1.041148729430523\n",
- "MSE values for own Ridge implementation\n",
- "1.96013048502692e-05\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "1.960130485007504e-05\n",
- "Beta values for own Ridge implementation\n",
- "[-0.1416398 -0.14021063 -0.05383795 0.01367553 0.04784395 0.05796251\n",
- " 0.05447415 0.044613 0.03267527 0.02098261 0.01066519 0.00217499\n",
- " -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081 -0.01416528\n",
- " -0.01290947]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[-0.1416398 -0.14021063 -0.05383795 0.01367553 0.04784395 0.05796251\n",
- " 0.05447415 0.044613 0.03267527 0.02098261 0.01066519 0.00217499\n",
- " -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081 -0.01416528\n",
- " -0.01290947]\n",
- "Intercept from own implementation:\n",
- "1.0495569966278295\n",
- "Intercept from Scikit-Learn Ridge implementation\n",
- "1.0495569966278269\n",
- "MSE values for own Ridge implementation\n",
- "5.4959161509377256e-05\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "5.495916150936645e-05\n",
- "Beta values for own Ridge implementation\n",
- "[-0.13535942 -0.08593216 -0.03568439 -0.0036367 0.01397146 0.02229529\n",
- " 0.02503753 0.0245528 0.02228115 0.01908936 0.01549377 0.01179792\n",
- " 0.00817631 0.00472512 0.00149311 -0.00149956 -0.00424967 -0.00676387\n",
- " -0.00905423]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[-0.13535942 -0.08593216 -0.03568439 -0.0036367 0.01397146 0.02229529\n",
- " 0.02503753 0.0245528 0.02228115 0.01908936 0.01549377 0.01179792\n",
- " 0.00817631 0.00472512 0.00149311 -0.00149956 -0.00424967 -0.00676387\n",
- " -0.00905423]\n",
- "Intercept from own implementation:\n",
- "1.0399676689527966\n",
- "Intercept from Scikit-Learn Ridge implementation\n",
- "1.0399676689527975\n",
- "MSE values for own Ridge implementation\n",
- "7.571105947979352e-05\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "7.571105947979394e-05\n",
- "Beta values for own Ridge implementation\n",
- "[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706 -0.00517114\n",
- " -0.00174276 0.00068734 0.00243186 0.00369758 0.00462287 0.0053018\n",
- " 0.00579953 0.006162 0.00642221 0.00660427 0.00672607 0.0068011\n",
- " 0.00683964]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706 -0.00517114\n",
- " -0.00174276 0.00068734 0.00243186 0.00369758 0.00462287 0.0053018\n",
- " 0.00579953 0.006162 0.00642221 0.00660427 0.00672607 0.0068011\n",
- " 0.00683964]\n",
- "Intercept from own implementation:\n",
- "0.999955585168597\n",
- "Intercept from Scikit-Learn Ridge implementation\n",
- "0.999955585168597\n",
- "MSE values for own Ridge implementation\n",
- "0.0007698473260556344\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "0.0007698473260556325\n",
- "Beta values for own Ridge implementation\n",
- "[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335\n",
- " -0.00323332 -0.00274989 -0.0023548 -0.00202756 -0.00175331 -0.00152117\n",
- " -0.001323 -0.0011526 -0.00100519 -0.00087697 -0.00076495 -0.00066668\n",
- " -0.00058016]\n",
- "Beta values for Scikit-Learn Ridge implementation\n",
- "[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335\n",
- " -0.00323332 -0.00274989 -0.0023548 -0.00202756 -0.00175331 -0.00152117\n",
- " -0.001323 -0.0011526 -0.00100519 -0.00087697 -0.00076495 -0.00066668\n",
- " -0.00058016]\n",
- "Intercept from own implementation:\n",
- "0.9637117593816477\n",
- "Intercept from Scikit-Learn Ridge implementation\n",
- "0.9637117593816477\n",
- "MSE values for own Ridge implementation\n",
- "0.0023813163025848865\n",
- "MSE values for Scikit-Learn Ridge implementation\n",
- "0.002381316302584886\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlIAAAG2CAYAAAC04mh6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABjN0lEQVR4nO3deVhU5fsG8HsYZmEHRRYTEM1UNE3QEMqtElwyt5RKSTNJslS0TNFcc89sc8sylzRX3Co1sQQX0NSv4EZqimEJIS4zrMMy7+8Pfk6NLDIIHJb7c11zxZx5zjn3HMl5fM+Z98iEEAJEREREZDIzqQMQERER1VRspIiIiIjKiY0UERERUTmxkSIiIiIqJzZSREREROXERoqIiIionNhIEREREZUTGykiIiKicmIjRURERFRObKSIiIiIyknyRmr58uXw9PSEWq2Gj48Pjhw5Ump9dHQ0fHx8oFar0aRJE6xcubJITUREBLy8vKBSqeDl5YWdO3cavT5//nx06NABNjY2cHJyQr9+/XDp0iWjmuHDh0Mmkxk9Onbs+OhvmIiIiGoNSRupLVu2ICwsDFOnTsWZM2fQqVMn9OzZE0lJScXWJyYmolevXujUqRPOnDmDKVOmYOzYsYiIiDDUxMbGIigoCMHBwYiPj0dwcDAGDx6MEydOGGqio6Pxzjvv4Pjx44iMjER+fj4CAgKQmZlptL8ePXogOTnZ8Ni7d2/lHAgiIiKqkWRS3rTY19cX3t7eWLFihWFZy5Yt0a9fP8yfP79I/aRJk7Bnzx4kJCQYloWGhiI+Ph6xsbEAgKCgIGi1Wuzbt89Q06NHDzg4OGDTpk3F5rh16xacnJwQHR2Nzp07Aygckbp37x527dpVEW+ViIiIaiFzqXacm5uL06dPY/LkyUbLAwICEBMTU+w6sbGxCAgIMFoWGBiI1atXIy8vDwqFArGxsRg/fnyRms8++6zELBqNBgBQr149o+VRUVFwcnKCvb09unTpgrlz58LJyanE7eh0Ouh0OsNzvV6PO3fuoH79+pDJZCWuR0RERNWHEALp6elo2LAhzMxKP3knWSOVlpaGgoICODs7Gy13dnZGSkpKseukpKQUW5+fn4+0tDS4urqWWFPSNoUQmDBhAp599lm0bt3asLxnz54YNGgQPDw8kJiYiGnTpuG5557D6dOnoVKpit3W/PnzMWvWrIe+dyIiIqr+bty4gUaNGpVaI1kjdd+DIzVCiFJHb4qrf3C5Kdt89913cfbsWRw9etRoeVBQkOHn1q1bo3379vDw8MBPP/2EAQMGFLut8PBwTJgwwfBco9HA3d0dN27cgK2tbYnviYiIiKoPrVYLNzc32NjYPLRWskbK0dERcrm8yEhRampqkRGl+1xcXIqtNzc3R/369UutKW6bY8aMwZ49e3D48OGHdpyurq7w8PDAlStXSqxRqVTFjlbZ2tqykSIiIqphynJZjmTf2lMqlfDx8UFkZKTR8sjISPj7+xe7jp+fX5H6AwcOoH379lAoFKXW/HebQgi8++672LFjB3799Vd4eno+NO/t27dx48YNuLq6lun9ERERUR0gJLR582ahUCjE6tWrxcWLF0VYWJiwsrIS169fF0IIMXnyZBEcHGyov3btmrC0tBTjx48XFy9eFKtXrxYKhUJs377dUHPs2DEhl8vFggULREJCgliwYIEwNzcXx48fN9S8/fbbws7OTkRFRYnk5GTDIysrSwghRHp6unjvvfdETEyMSExMFIcOHRJ+fn7iscceE1qttszvT6PRCABCo9E86qEiIiKiKmLK57ekjZQQQixbtkx4eHgIpVIpvL29RXR0tOG1YcOGiS5duhjVR0VFiXbt2gmlUikaN24sVqxYUWSb27ZtE82bNxcKhUK0aNFCREREGL0OoNjHmjVrhBBCZGVliYCAANGgQQOhUCiEu7u7GDZsmEhKSjLpvbGRIiIiqnlM+fyWdB6p2k6r1cLOzg4ajabUa6QKCgqQl5dXhcmIqpZSqXzoV4iJiKqLsn5+A9XgW3t1mRACKSkpuHfvntRRiCqVmZkZPD09oVQqpY5CRFSh2EhJ6H4T5eTkBEtLS07aSbWSXq/HzZs3kZycDHd3d/6eE1GtwkZKIgUFBYYm6v7UDUS1VYMGDXDz5k3k5+cbvmFLRFQb8KIFidy/JsrS0lLiJESV7/4pvYKCAomTEBFVLDZSEuNpDqoL+HtORLUVGykiIiKicmIjRVSJrl+/DplMhri4uBJroqKiIJPJ+O1NIqIaiI0UmWT48OGQyWQIDQ0t8tro0aMhk8kwfPhww7LU1FSMGjUK7u7uUKlUcHFxQWBgIGJjYw01jRs3hkwmK/JYsGBBVbylcuvatashq1KpRNOmTREeHg6dTmeocXNzQ3JyMlq3bi1h0pKP8f1H165dy73trl27IiwsrMKyEhHVJPzWHpnMzc0NmzdvxqeffgoLCwsAQE5ODjZt2gR3d3ej2oEDByIvLw/r1q1DkyZN8M8//+CXX37BnTt3jOpmz56NkJAQo2Vlueu21EJCQjB79mzk5ubi5MmTeOONNwAA8+fPBwDI5XK4uLhIGREAcPLkScOF3jExMRg4cCAuXbpkmGiO8zsREZUPR6TIZN7e3nB3d8eOHTsMy3bs2AE3Nze0a9fOsOzevXs4evQoFi5ciG7dusHDwwNPP/00wsPD0bt3b6Nt2tjYwMXFxehhZWVVYoa7d+/i9ddfh4ODAywtLdGzZ09cuXIFQOFEpw0aNEBERISh/qmnnoKTk5PheWxsLBQKBTIyMgAUXgz9zTffoH///rC0tESzZs2wZ8+ehx4LS0tLuLi4wN3dHQMHDkT37t1x4MABw+vFndrbu3cvnnjiCVhYWKBbt264fv16ke1+/fXXcHNzg6WlJfr3748lS5bA3t7eqOaHH36Aj48P1Go1mjRpglmzZiE/P7/YnA0aNDAc13r16gEAnJycDMt+//13dO7cGRYWFnBzc8PYsWORmZlpWH/58uVo1qwZ1Go1nJ2d8fLLLwMoHKGMjo7G559/bhjdKu79EBHVVmykqqHMzJIfOTllr83Ofnhteb3xxhtYs2aN4fm3336LESNGGNVYW1vD2toau3btMjrdVRGGDx+OU6dOYc+ePYiNjYUQAr169UJeXh5kMhk6d+6MqKgoAIVN18WLF5GXl4eLFy8CKLwuycfHB9bW1oZtzpo1C4MHD8bZs2fRq1cvDBkypMjIWWni4+Nx7NixUudJunHjBgYMGIBevXohLi4OI0eOxOTJk41qjh07htDQUIwbNw5xcXHo3r075s6da1Tz888/Y+jQoRg7diwuXryIr776CmvXri1SVxbnzp1DYGAgBgwYgLNnz2LLli04evQo3n33XQDAqVOnMHbsWMyePRuXLl3C/v370blzZwDA559/Dj8/P4SEhCA5ORnJyclwc3MzOQMRkSnyC/RoM/ldfBzxC/IL9NKGqdzb/tVtpd30MDs7W1y8eFFkZ2cXeQ0o+dGrl3GtpWXJtQ/c71k4OhatMdWwYcNE3759xa1bt4RKpRKJiYni+vXrQq1Wi1u3bom+ffuKYcOGGeq3b98uHBwchFqtFv7+/iI8PFzEx8cbbfP+TautrKyMHocOHSo2w+XLlwUAcezYMcOytLQ0YWFhIbZu3SqEEOKLL74QrVu3FkIIsWvXLtG+fXsxYMAAsWzZMiGEEAEBAWLSpEmG9QGIDz/80PA8IyNDyGQysW/fvhKPRZcuXYRCoRBWVlZCqVQKAMLMzExs377dUJOYmCgAiDNnzgghhAgPDxctW7YUer3eUDNp0iQBQNy9e1cIIURQUJDo3bu30b6GDBki7OzsDM87deok5s2bZ1Tz3XffCVdX1xLz3nfo0CGj/QUHB4u33nrLqObIkSPCzMxMZGdni4iICGFrayu0Wm2Jx2HcuHGl7rO033ciIlPN33pAYCYEJtuJW/cyK3z7pty0mCNSVC6Ojo7o3bs31q1bhzVr1qB3795wdHQsUjdw4EDcvHkTe/bsQWBgIKKiouDt7Y21a9ca1U2cOBFxcXFGD19f32L3nZCQAHNzc6PX69evj+bNmyMhIQFA4QXQFy5cQFpaGqKjo9G1a1d07doV0dHRyM/PR0xMDLp06WK03TZt2hh+trKygo2NDVJTU0s9DkOGDEFcXBxiY2MxePBgjBgxAgMHDiyxPiEhAR07djSaV8nPz8+o5tKlS3j66aeNlj34/PTp05g9e7Zh1M/a2towKpSVlVVq5gedPn0aa9euNdpWYGAg9Ho9EhMT0b17d3h4eKBJkyYIDg7Gxo0bTd4HEVFFWhb7NQCgtX4IHO2kndiaF5tXQ/9/2U6x5HLj56V9zps90CZX9KUrI0aMMJz+WbZsWYl1arUa3bt3R/fu3TF9+nSMHDkSM2bMMPp2n6OjIx5//PEy7VcIUeLy+w1K69atUb9+fURHRyM6OhqzZ8+Gm5sb5s6di5MnTyI7OxvPPvus0foPnpKTyWTQ60sfMrazszPk3rBhA1q1aoXVq1fjzTffNCl7Se+jpPX0ej1mzZqFAQMGFFlfrVY/dB8PbmvUqFEYO3Zskdfc3d2hVCrxv//9D1FRUThw4ACmT5+OmTNn4uTJk0Wu2yIiqmwXrqfiL+tdAIBpvd+SNgzYSFVLpVxjXWW1ZdGjRw/k5uYCAAIDA8u8npeXF3bt2lXu/Xp5eSE/Px8nTpyAv78/AOD27du4fPkyWrZsCQCG66R2796N8+fPo1OnTrCxsUFeXh5WrlwJb2/vCv9WoEKhwJQpUxAeHo5XX3212Nv/FPfejx8/bvS8RYsW+O2334yWnTp1yui5t7c3Ll26VObmszTe3t64cOFCqdsyNzfHCy+8gBdeeAEzZsyAvb09fv31VwwYMABKpZK3fiGiKvPB9+sAeR6s7nXA4M5tpY7Di82p/ORyORISEpCQkAD5g0NlKGxunnvuOWzYsAFnz55FYmIitm3bhkWLFqFv375Gtenp6UhJSTF6aLXaYvfbrFkz9O3bFyEhITh69Cji4+MxdOhQPPbYY0bb7dq1K77//nu0adMGtra2huZq48aNjzRvUmlee+01yGQyLF++vNjXQ0NDcfXqVUyYMAGXLl3C999/X+Q055gxY7B3714sWbIEV65cwVdffYV9+/YZjVJNnz4d69evx8yZM3HhwgUkJCRgy5Yt+PDDD03OPGnSJMTGxuKdd95BXFwcrly5gj179mDMmDEAgB9//BFffPEF4uLi8Oeff2L9+vXQ6/Vo3rw5gMI5qk6cOIHr168jLS3toaN4RETlpdcLRN7+BgAwwCPkIdVVg40UPRJbW1vDXEQPsra2hq+vLz799FN07twZrVu3xrRp0xASEoKlS5ca1U6fPh2urq5Gjw8++KDE/a5ZswY+Pj548cUX4efnByEE9u7da3R6rlu3bigoKDBqmrp06YKCgoIi10dVFKVSiXfffReLFi0yTK3wX+7u7oiIiMAPP/yAtm3bYuXKlZg3b55RzTPPPIOVK1diyZIlaNu2Lfbv34/x48cbnbILDAzEjz/+iMjISHTo0AEdO3bEkiVL4OHhYXLmNm3aIDo6GleuXEGnTp3Qrl07TJs2Da6urgAAe3t77NixA8899xxatmyJlStXYtOmTWjVqhUA4P3334dcLoeXlxcaNGiApKQkkzMQEZXFlz8cRp7tZSDXGguCX5E6DgBAJspy0QaVi1arhZ2dHTQaTZFmIycnB4mJifD09DT5mhaqe0JCQvD777/jyJEjUkcpF/6+E1FFmL1pH+aenIAm5p2QsGhVpe2ntM/vB/EaKaJqaPHixejevTusrKywb98+rFu3rsTThUREdcX0V3viw6AeSNNUn28Os5EiqoZ+++03LFq0COnp6WjSpAm++OILjBw5UupYRESSMzOTwcmhgr899QjYSBFVQ1u3bpU6AhFRtaHXC0xauwPhL/dCPVsLqeMY4cXmREREVK19vT8Wi2+8jAYfNUNWTp7UcYywkSIiIqJqbfGvhTOZNxHdYaku+X6mUmAjRURERNXWn//cwx/qLQCAic9LP5P5g9hIERERUbU1acP3gCIbKk0rjAzsKHWcIthIERERUbWk1wvs/qtwvqheLiEwM5M9ZI2qx0aKiIiIqqXvfjmFHPt4IF+Fj4cGSx2nWGykiEwQFRUFmUyGe/fulVizdu1a2NvbV1kmIqLaavupwwCAxlkvo2nDehKnKR4bKTLJ8OHDIZPJEBoaWuS10aNHQyaTYfjw4YZlqampGDVqFNzd3aFSqeDi4oLAwEDExsYaaho3bgyZTFbksWDBghJzXLt2Da+++ioaNmwItVqNRo0aoW/fvrh8+bKhRiaTYdeuXWV+b//NYWFhgRYtWuDjjz/Gf++i5O/vj+TkZNjZ2ZV5u5WhuOP138d//wxM1bhxY3z22WcVlpWIqLx+CH8PB1+6glWvzZA6Sok4ISeZzM3NDZs3b8ann34KC4vCidFycnKwadMmuLu7G9UOHDgQeXl5WLduHZo0aYJ//vkHv/zyC+7cuWNUN3v2bISEGN/J28bGptj95+bmonv37mjRogV27NgBV1dX/PXXX9i7dy80Gs0jvbf7OXJycnDw4EG8/fbbsLW1xahRowAU3pTYxcXlkfZREZKTkw0/b9myBdOnT8elS5cMy+7/uRAR1XTPt3tc6gil4ogUmczb2xvu7u7YsWOHYdmOHTvg5uaGdu3aGZbdu3cPR48excKFC9GtWzd4eHjg6aefRnh4OHr37m20TRsbG7i4uBg9rKyKvwXAxYsXce3aNSxfvhwdO3aEh4cHnnnmGcydOxcdOnR4pPd2P0fjxo0xcuRItGnTBgcOHDC8XtypvbVr18Ld3R2Wlpbo378/bt++XWS7c+bMgZOTE2xsbDBy5EhMnjwZTz31lFHNmjVr0LJlS6jVarRo0aLUe+v99zjZ2dlBJpMZLTt8+DB8fHygVqvRpEkTzJo1C/n5+Yb1Z86caRglbNiwIcaOHQsA6Nq1K/7880+MHz/eMLpFRCSFSzfSpI5QJmykqqHM3MwSHzn5OWWuzc7Lfmhteb3xxhtYs2aN4fm3336LESNGGNVYW1vD2toau3btgk6nK/e+HtSgQQOYmZlh+/btKCgoqLDt/pcQAlFRUUhISIBCUfLkbydOnMCIESMwevRoxMXFoVu3bpgzZ45RzcaNGzF37lwsXLgQp0+fhru7O1asWGFU8/XXX2Pq1KmYO3cuEhISMG/ePEybNg3r1q0zOfvPP/+MoUOHYuzYsbh48SK++uorrF27FnPnzgUAbN++HZ9++im++uorXLlyBbt27cKTTz4JoLAhbtSoEWbPno3k5GSjkS8ioqqyJToOLb52hcd7r0KvFw9fQUqCKo1GoxEAhEajKfJadna2uHjxosjOzi7yGmaixEevjb2Mai3nWpZY22VNF6Nax0WORWpMNWzYMNG3b19x69YtoVKpRGJiorh+/bpQq9Xi1q1bom/fvmLYsGGG+u3btwsHBwehVquFv7+/CA8PF/Hx8Ubb9PDwEEqlUlhZWRk9Dh06VGKOpUuXCktLS2FjYyO6desmZs+eLa5evWp8HAGxc+fOMr+3/+ZQKBQCgFCr1eLYsWOGmkOHDgkA4u7du0IIIV599VXRo0cPo+0EBQUJOzs7w3NfX1/xzjvvGNU888wzom3btobnbm5u4vvvvzeq+eijj4Sfn99Dc69Zs8Zof506dRLz5s0zqvnuu++Eq6urEEKITz75RDzxxBMiNze32O15eHiITz/99KH7NUVpv+9ERA9q/cFogZkQbuODJNl/aZ/fD+KIFJWLo6MjevfujXXr1mHNmjXo3bs3HB0di9QNHDgQN2/exJ49exAYGIioqCh4e3tj7dq1RnUTJ05EXFyc0cPX17fE/b/zzjtISUnBhg0b4Ofnh23btqFVq1aIjIx8pPd1P0d0dDS6deuGqVOnwt/fv8T6hIQE+Pn5GS178PmlS5fw9NNPGy377/Nbt27hxo0bePPNNw2jeNbW1pgzZw6uXr1q8ns4ffo0Zs+ebbStkJAQJCcnIysrC4MGDUJ2djaaNGmCkJAQ7Ny50+i0HxGRlFLvZuK8fAMAYMwzIQ+plh4vNq+GMsIzSnxNbiY3ep76fmqJtWYy4z75+rjrj5TrQSNGjMC7774LAFi2bFmJdWq1Gt27d0f37t0xffp0jBw5EjNmzDD6ZpmjoyMef9y0CwptbGzw0ksv4aWXXsKcOXMQGBiIOXPmoHv37uV6P//N8fjjjyMiIgKPP/44OnbsiBdeeKHYeiHKNuT84LVG/11Pr9cDKDy992DzKJcb/3mXhV6vx6xZszBgwIAir6nVari5ueHSpUuIjIzEwYMHMXr0aHz88ceIjo4u9TQmEVFVmLJxG6DSwjy9Ccb36yZ1nIdiI1UNWSmLv8i6KmvLokePHsjNzQUABAYGlnk9Ly8vk6YlKAuZTIYWLVogJiamwrbp4OCAMWPG4P3338eZM2eKvfDay8sLx48fN1r24PPmzZvjt99+Q3Dwv5PJnTp1yvCzs7MzHnvsMVy7dg1Dhgx55Nze3t64dOlSqY2phYWFoQl955130KJFC5w7dw7e3t5QKpWVdu0ZEdHDbPljFeAAPG8fAnN59T9xxkaKyk0ulyMhIcHw84Nu376NQYMGYcSIEWjTpg1sbGxw6tQpLFq0CH379jWqTU9PR0pKitEyS0tL2NraFtluXFwcZsyYgeDgYHh5eUGpVCI6OhrffvstJk2aZFSbmJiIuLg4o2WPP/44rK2ty/Qe33nnHSxcuBARERF4+eWXi7w+duxY+Pv7Y9GiRejXrx8OHDiA/fv3G9WMGTMGISEhaN++Pfz9/bFlyxacPXsWTZo0MdTMnDkTY8eOha2tLXr27AmdTodTp07h7t27mDBhQpmy3jd9+nS8+OKLcHNzw6BBg2BmZoazZ8/i3LlzmDNnDtauXYuCggL4+vrC0tIS3333HSwsLODh4QGgcB6pw4cP45VXXoFKpSr2lC0RUWXYHXMBGQ6xQIE5FgUPlzpO2VT6FVt1WHkvNq/O7l9sXpL/Xmyek5MjJk+eLLy9vYWdnZ2wtLQUzZs3Fx9++KHIysoyrOPh4SEAFHmMGjWq2H3cunVLjB07VrRu3VpYW1sLGxsb8eSTT4rFixeLgoICQ11x2wRQ4kXsJV1kHRISIlq1aiUKCgqKXGwuhBCrV68WjRo1EhYWFqJPnz5i8eLFRhd/CyHE7NmzhaOjo7C2thYjRowQY8eOFR07djSq2bhxo3jqqaeEUqkUDg4OonPnzmLHjh0lHuv7HrzYXAgh9u/fL/z9/YWFhYWwtbUVTz/9tFi1apUQQoidO3cKX19fYWtrK6ysrETHjh3FwYMHDevGxsaKNm3aCJVKJSrqr4ia+vtORFWr3eQwgZkQrmH9Jc1hysXmMiHKeJEHmUyr1cLOzg4ajabIyEpOTg4SExPh6ekJtVotUUKSSvfu3eHi4oLvvvtO6ihVgr/vRFQWicl3MXHDBjzXsh1Gv/isZDlK+/x+EE/tEVWyrKwsrFy5EoGBgZDL5di0aRMOHjz4yN8wJCKqbTxdHbB94hipY5iEjRRRJZPJZNi7dy/mzJkDnU6H5s2bIyIiosRvAhIRUc3BRoqokllYWODgwYNSxyAiqrb2nbyEQd8PwytNQ/HNu8OljmOS6v+9QiIiIqrVpu/6Bpn2J/Dj1R0PL65m2EhJjNf6U13A33MiKok2U4fTBWsBACHe1X8m8wexkZLI/Rmks7KyJE5CVPnuT9xanpnaiah2m7FpN4RFGswyG2JqUE+p45iM10hJRC6Xw97eHqmphbd4sbS0LHbmbKKaTq/X49atW7C0tIS5Of/KISJj684XzmT+jMWbUCtr3t8RNS9xLeLi4gIAhmaKqLYyMzODu7s7/7FAREZ+jbuKuw6/AEKG+YPflDpOubCRkpBMJoOrqyucnJyQl5cndRyiSqNUKmFmxisJiMjYhxGrAXOgviYAz7TykDpOubCRqgbkcjmvHSEiojqnh1cnXP7tf3iz7Sipo5QbbxFTiUyZYp6IiIiqB1M+vznWTkRERFRObKSIiIioSsVeTMJzs2bj5KW/pI7yyNhIERERUZWavHU1DmEGAlcMlzrKI2MjRURERFUmJzcfRzNXAwCGeo2UOM2jYyNFREREVWbe1v3QW/8NWXZ9zBnSX+o4j4yNFBEREVWZVadXAQC8zYbB1kolcZpHx0aKiIiIqsSpy3/jH9ufAACz+9W8GxQXh40UERERVYlJm78FzPSwvdsJvZ5uIXWcCsGZzYmIiKhK5BbkAvmWeK35W1JHqTCc2bwScWZzIiIiY3/d0sLWUlWtr4+qUTObL1++HJ6enlCr1fDx8cGRI0dKrY+OjoaPjw/UajWaNGmClStXFqmJiIiAl5cXVCoVvLy8sHPnTqPX58+fjw4dOsDGxgZOTk7o168fLl26ZFQjhMDMmTPRsGFDWFhYoGvXrrhw4cKjv2EiIqI6rFED22rdRJlK0kZqy5YtCAsLw9SpU3HmzBl06tQJPXv2RFJSUrH1iYmJ6NWrFzp16oQzZ85gypQpGDt2LCIiIgw1sbGxCAoKQnBwMOLj4xEcHIzBgwfjxIkThpro6Gi88847OH78OCIjI5Gfn4+AgABkZmYaahYtWoQlS5Zg6dKlOHnyJFxcXNC9e3ekp6dX3gEhIiKqhS7dSMOGX05LHaNSSHpqz9fXF97e3lixYoVhWcuWLdGvXz/Mnz+/SP2kSZOwZ88eJCQkGJaFhoYiPj4esbGxAICgoCBotVrs27fPUNOjRw84ODhg06ZNxea4desWnJycEB0djc6dO0MIgYYNGyIsLAyTJk0CAOh0Ojg7O2PhwoUYNapsd6nmqT0iIiKg59yF2J8/Gc0y3sDlj7+VOs5D1YhTe7m5uTh9+jQCAgKMlgcEBCAmJqbYdWJjY4vUBwYG4tSpU8jLyyu1pqRtAoBGowEA1KtXD0DhyFdKSorRdlQqFbp06VLqdnQ6HbRardGDiIioLssv0OPgna8BAM+4PStxmoonWSOVlpaGgoICODs7Gy13dnZGSkpKseukpKQUW5+fn4+0tLRSa0raphACEyZMwLPPPovWrVsbtnF/vbJuByi89srOzs7wcHNzK7GWiIioLvh8dxTyba8COhssDA6SOk6Fk/xic5lMZvRcCFFk2cPqH1xuyjbfffddnD17ttjTfqZmCw8Ph0ajMTxu3LhRYi0REVFd8PnRwpnMvQqGwMnBSuI0FU+yeaQcHR0hl8uLjPCkpqYWGQm6z8XFpdh6c3Nz1K9fv9Sa4rY5ZswY7NmzB4cPH0ajRo2M9gMUjky5urqWKRtQePpPpao930QgIiJ6FAlJt3DDqvCb8x/2rD1zR/2XZCNSSqUSPj4+iIyMNFoeGRkJf3//Ytfx8/MrUn/gwAG0b98eCoWi1Jr/blMIgXfffRc7duzAr7/+Ck9PT6N6T09PuLi4GG0nNzcX0dHRJWYjIiIiYx9sXA+Y58Lyng9e7dpO6jiVQtKZzSdMmIDg4GC0b98efn5+WLVqFZKSkhAaGgqg8FTZ33//jfXr1wMo/Ibe0qVLMWHCBISEhCA2NharV682Oi03btw4dO7cGQsXLkTfvn2xe/duHDx4EEePHjXUvPPOO/j++++xe/du2NjYGEaw7OzsYGFhAZlMhrCwMMybNw/NmjVDs2bNMG/ePFhaWuK1116rwiNERERUcx37Zy/gAPT3qJ2jUQAAIbFly5YJDw8PoVQqhbe3t4iOjja8NmzYMNGlSxej+qioKNGuXTuhVCpF48aNxYoVK4psc9u2baJ58+ZCoVCIFi1aiIiICKPXART7WLNmjaFGr9eLGTNmCBcXF6FSqUTnzp3FuXPnTHpvGo1GABAajcak9YiIiGqDbF2e+HD9HvF3mlbqKCYx5fObt4ipRJxHioiIqOapEfNIERERUe10LyMHObn5UseoEmykiIiIqEK9uXwVrD70wNDPVkkdpdKxkSIiIqIKo9cL/JTyNfRWN6HLy5U6TqVjI0VEREQV5tsDJ6CzOw/kqbH49aFSx6l0bKSIiIiowiz6pfB0XpOcwfBwtpc2TBVgI0VEREQVIilVgyvKLQCA97vV4rmj/oONFBEREVWISd99DyizoNS0xKiedeNOIGykiIiIqELs/utrAEAv57dgZiaTOE3VYCNFREREFWLVi1/DKysUi4YESx2lynBm80rEmc2JiIhqHs5sTkRERFQF2EgRERHRI3l7xQY8MXEENv76P6mjVDk2UkRERPRIvrv8Ja5Yr8HG2INSR6lybKSIiIio3LYejkem/W9AgQIfDxkudZwqx0aKiIiIym3O3sIpDx7L6ItWjZ0kTlP12EgRERFRuaRpsnBOtgEA8K5f3ZjJ/EFspIiIiKhcwjdsA9QamKd74v0Bz0sdRxJspIiIiKhctlwpPK3XzW4kzOV1s6Wom++aiIiIHkl+gR4d6gXCXNsUi157Q+o4kuHM5pWIM5sTEVFtp9eLWndfPc5sTkRERFWitjVRpmIjRURERCZZsvMQJn4bgaycPKmjSI6NFBEREZnko+jZWHzjZfRfvFjqKJJjI0VERERl9vOpy7jnEAXozTB74FCp40iOjRQRERGV2bSd3wAAGmh7wLelm8RppMdGioiIiMokIzsXp/LXAgBGtqubM5k/iI0UERERlcn073dDWN6CWaYrpr/SW+o41QIbKSIiIiqTdecKZzL3U4+AWmkucZrqgY0UERERPZQ2U4dcZABChnkvvyl1nGqD7SQRERE9lK2VCumfxSAq/ho6t/GUOk61wREpIiIiKrOubZtIHaFaYSNFREREpfo17ioSk+9KHaNaYiNFREREpXpl3Vg0WdYQY1dtljpKtcNrpIiIiKhEsReTcMtuHyAT6OPjI3WcaocjUkRERFSi8K3fAjIB+7vd0N2nmdRxqh02UkRERFSs3LwCHM1cDQAY2jJE4jTVExspIiIiKta8rftRYP0XZNn18NGQ/lLHqZbYSBEREVGxvjpVOJN5O7NhsLdWS5ymemIjRUREREX8+c89pFj9DACY1Zen9UrCRoqIiIiK8HC2R/yIRIx0XIsXfVtKHafakgkhhNQhaiutVgs7OztoNBrY2tpKHYeIiIjKwJTPb45IERERkZGsnDypI9QYbKSIiIjISPMpQ1Av7HmsP3hK6ijVHmc2JyIiIoPzif/gL+udgDwfliql1HGqPY5IERERkcEH368D5PmwuueLlzu1kTpOtcdGioiIiAAAer3AwTuFc0e97MkpD8qCjRQREREBAD7bHYU82z8AnQ0WDA2SOk6NwEaKiIiIAABfHC0cjWqZ/xpc6llLnKZmYCNFREREuHQjDX9aRgAApgTytF5ZsZEiIiIiODtYY5TrN2ieMRJDn/eROk6NwZnNKxFnNiciIqp5OLM5ERERURVgI0VERFTHdZ4xHX3mf4KrN+9IHaXG4czmREREdVhi8l0cKfgYyM1B37PPoGnDjlJHqlE4IkVERFSHfbBhI6DIgUrTGiMCfKWOU+OwkSIiIqqj9HqBH5NXAQBedH0LZmYyiRPVPGykiIiI6qg1kb8hx+4ckKfGx0OHSh2nRmIjRUREVEctOlg4k3mTnEHwdHWQOE3NxEaKiIioDvrrlhaXlZsAAOO7cCbz8pK8kVq+fDk8PT2hVqvh4+ODI0eOlFofHR0NHx8fqNVqNGnSBCtXrixSExERAS8vL6hUKnh5eWHnzp1Grx8+fBh9+vRBw4YNIZPJsGvXriLbGD58OGQymdGjY0d+k4GIiGqH5DtauGX3gfpeW4zu/azUcWosSRupLVu2ICwsDFOnTsWZM2fQqVMn9OzZE0lJScXWJyYmolevXujUqRPOnDmDKVOmYOzYsYiIiDDUxMbGIigoCMHBwYiPj0dwcDAGDx6MEydOGGoyMzPRtm1bLF26tNR8PXr0QHJysuGxd+/einnjREREEuvQvBGSlmzG3YWneJH5I5D0FjG+vr7w9vbGihUrDMtatmyJfv36Yf78+UXqJ02ahD179iAhIcGwLDQ0FPHx8YiNjQUABAUFQavVYt++fYaaHj16wMHBAZs2bSqyTZlMhp07d6Jfv35Gy4cPH4579+4VO1pVVrxFDBERUc1TI24Rk5ubi9OnTyMgIMBoeUBAAGJiYopdJzY2tkh9YGAgTp06hby8vFJrStpmaaKiouDk5IQnnngCISEhSE1NLbVep9NBq9UaPYiIiKqb91Zvx48nEh5eSA8lWSOVlpaGgoICODs7Gy13dnZGSkpKseukpKQUW5+fn4+0tLRSa0raZkl69uyJjRs34tdff8Unn3yCkydP4rnnnoNOpytxnfnz58POzs7wcHNzM2mfRERElS3lTgaWXB2BPvu9sC7ypNRxajzJbxEjkxmflxVCFFn2sPoHl5u6zeIEBQUZfm7dujXat28PDw8P/PTTTxgwYECx64SHh2PChAmG51qtls0UERFVK+EbtgKqdCi0j2PIcz5Sx6nxJGukHB0dIZfLi4wUpaamFhlRus/FxaXYenNzc9SvX7/UmpK2WVaurq7w8PDAlStXSqxRqVRQqVSPtB8iIqLKtC1xFWAPPF9vJMzlkn95v8aT7AgqlUr4+PggMjLSaHlkZCT8/f2LXcfPz69I/YEDB9C+fXsoFIpSa0raZlndvn0bN27cgKur6yNth4iISCoRR88h0/4EUGCOj18bLnWcWkHSU3sTJkxAcHAw2rdvDz8/P6xatQpJSUkIDQ0FUHiq7O+//8b69esBFH5Db+nSpZgwYQJCQkIQGxuL1atXG30bb9y4cejcuTMWLlyIvn37Yvfu3Th48CCOHj1qqMnIyMAff/xheJ6YmIi4uDjUq1cP7u7uyMjIwMyZMzFw4EC4urri+vXrmDJlChwdHdG/f/8qOjpEREQVa/aPXwMWwGMZfdHa89HO1ND/ExJbtmyZ8PDwEEqlUnh7e4vo6GjDa8OGDRNdunQxqo+KihLt2rUTSqVSNG7cWKxYsaLINrdt2yaaN28uFAqFaNGihYiIiDB6/dChQwJAkcewYcOEEEJkZWWJgIAA0aBBA6FQKIS7u7sYNmyYSEpKMum9aTQaAUBoNBqT1iMiIqpotzVZQjbZXmAmxJzN+6WOU62Z8vkt6TxStR3nkSIioupiwy+n8XpkAMzybZCz8BqvjyqFKZ/fkn9rj4iIiCrf0Od98KLv3zh8nk1UReKRJCIiqiPsrdV4qaOX1DFqFTZSREREtdyvcVeRX6CXOkatxEaKiIioFtNm6vDCZl9YfPAEouKvSR2n1mEjRUREVIt9uHEnhMVtCDMdOrZ0lzpOrcNGioiIqBb77uIqAMAzViOgVvI7ZhWNjRQREVEt9cuZP3DP4RAgZJg3aITUcWolNlJERES11NSIbwAADTQ98EwrD4nT1E5spIiIiGqhjOxc/Ja3BgDwRtsQidPUXmykiIiIaqFFEQcgLFNhlumCGa+8KHWcWotXnREREdVCM1/rjQY/RiPx1j+wVCukjlNrsZEiIiKqhczMZBjzUmepY9R6PLVHRERUy+j1QuoIdQYbKSIiolokJzcflu+3QdvJY5GYfFfqOLUeGykiIqJa5KMte6GzO49zYjMa2FtJHafWYyNFRERUi3zzv68BAD7mw2BtoZQ4Te3HRoqIiKiWOHnpL6Ta7gUAzOnPuaOqAhspIiKiWmLS5m8BMz3s7nZBYPsnpI5TJ7CRIiIiqgVy8wpwOKPwljCvteBoVFVhI0VERFQLLNh+AAXWNyDLccC8oQOljlNncEJOIiKiWuD5Nl746cIHsLawhr21Wuo4dYZMCMFZuyqJVquFnZ0dNBoNbG1tpY5DREREZWDK5zdP7RERERGVExspIiKiGiy/QI9Wk97GnM37kZtXIHWcOoeNFBERUQ22ZOevuGi5EtPiX4E2Syd1nDqHjRQREVENtjSmcCbz1vohcLSzlDhN3WNSI7Vo0SJkZ2cbnh8+fBg63b/db3p6OkaPHl1x6YiIiKhECUm3cMN6JwBgWu+3JE5TN5nUSIWHhyM9Pd3w/MUXX8Tff/9teJ6VlYWvvvqq4tIRERFRiSZuXAfI82B1rwMGd24rdZw6yaRG6sGZEjhzAhERkTT0eoEDaYWn9QZ4cCZzqfAaKSIiohroyx8OI8/2MpBrjQXBr0gdp87izOZEREQ1UJZOB5WmFTzN/dGwvo3Uceoskxupb775BtbW1gCA/Px8rF27Fo6OjgBgdP0UERERVZ7wwQGY9PI53EnPfngxVRqTbhHTuHFjyGSyh9YlJiY+UqjagreIISIiqnlM+fw2aUTq+vXrj5KLiIiIHpFeL/Det9sQPrA3nByspI5T5/FicyIiohrk6/2x+OzvILjOfxxZOXlSx6nzTGqkTpw4gX379hktW79+PTw9PeHk5IS33nrLaIJOIiIiqliLfy2c8qCJvgcs1QqJ05BJjdTMmTNx9uxZw/Nz587hzTffxAsvvIDJkyfjhx9+wPz58ys8JBEREQF//nMPf6i3AAAmPs+ZzKsDkxqpuLg4PP/884bnmzdvhq+vL77++mtMmDABX3zxBbZu3VrhIYmIiAiYtOF7QJENlaYVRgZ2lDoOwcRG6u7du3B2djY8j46ORo8ePQzPO3TogBs3blRcOiIiIgJQeJH57r9WAQB6uYTAzOzh36KnymdSI+Xs7GyY2iA3Nxf/+9//4OfnZ3g9PT0dCgXP1xIREVW07345hRz7eCBfhY+HBksdh/6fSY1Ujx49MHnyZBw5cgTh4eGwtLREp06dDK+fPXsWTZs2rfCQREREdd2u/x0DhAyNs15G04b1pI5D/8+keaTmzJmDAQMGoEuXLrC2tsbatWuhVCoNr3/77bcICAio8JBERER13c5JYTh8ti/yCgqkjkL/YdLM5vdpNBpYW1tDLpcbLb9z5w5sbGx4eu//cWZzIiKimqfSZjYfMWJEmeq+/fZbUzZLREREpbhwPRWtGjtJHYOKYVIjtXbtWnh4eKBdu3Yox0AWERERmWhLdBxe+dUHj6UPQNLirfy2XjVjUiMVGhqKzZs349q1axgxYgSGDh2KevV4wRsREVFlmbP3a8BSDzOYsYmqhkz61t7y5cuRnJyMSZMm4YcffoCbmxsGDx6Mn3/+mSNUREREFSz1bibOyzcAAMY8w5nMqyOTb1qsUqnw6quvIjIyEhcvXkSrVq0wevRoeHh4ICMjozIyEhER1UlTNm4DVFqYpzfB+H7dpI5DxTC5kfovmUwGmUwGIQT0en1FZSIiIiIAW/8ovEHx8/YhMJc/0kc2VRKT/1R0Oh02bdqE7t27o3nz5jh37hyWLl2KpKQkWFtbV0ZGIiKiOmd3zAWkO8QABeZY9NpwqeNQCUy62Hz06NHYvHkz3N3d8cYbb2Dz5s2oX79+ZWUjIiKqs2b/uBpQAa7pfdCmiYvUcagEJk3IaWZmBnd3d7Rr1w4yWcnfHNixY0eFhKvpOCEnERGVV1KqBpO++x6dW7TB272fkTpOnVJpE3K+/vrrpTZQREREVDHcneyw6b23pY5BD2HyhJxEREREVIhfASAiIqpG9p28BKvx7THs82+kjkJlwEaKiIioGpm+6xtk2Z/G/uu7pY5CZcBGioiIqJrQZupwumAtAGCkd4i0YahM2EgRERFVEzM27YawSINZZkNMC+oldRwqAzZSRERE1cT684UzmftbjIBaadL3wUgibKSIiIiqgV/jruKOw0FAyDB/0JtSx6EykryRWr58OTw9PaFWq+Hj44MjR46UWh8dHQ0fHx+o1Wo0adIEK1euLFITEREBLy8vqFQqeHl5YefOnUavHz58GH369EHDhg0hk8mwa9euItsQQmDmzJlo2LAhLCws0LVrV1y4cOGR3isREVFJPoxYDQCorwnAs60bSxuGykzSRmrLli0ICwvD1KlTcebMGXTq1Ak9e/ZEUlJSsfWJiYno1asXOnXqhDNnzmDKlCkYO3YsIiIiDDWxsbEICgpCcHAw4uPjERwcjMGDB+PEiROGmszMTLRt2xZLly4tMduiRYuwZMkSLF26FCdPnoSLiwu6d++O9PT0ijsARERE/69f2+fgdO9FvNl2lNRRyAQm3SKmovn6+sLb2xsrVqwwLGvZsiX69euH+fPnF6mfNGkS9uzZg4SEBMOy0NBQxMfHIzY2FgAQFBQErVaLffv2GWp69OgBBwcHbNq0qcg2ZTIZdu7ciX79+hmWCSHQsGFDhIWFYdKkSQAKb9bs7OyMhQsXYtSosv2S8xYxRERENY8pn9+SjUjl5ubi9OnTCAgIMFoeEBCAmJiYYteJjY0tUh8YGIhTp04hLy+v1JqStlmcxMREpKSkGG1HpVKhS5cupW5Hp9NBq9UaPYiIiKj2kqyRSktLQ0FBAZydnY2WOzs7IyUlpdh1UlJSiq3Pz89HWlpaqTUlbbOk/dxfz5TtzJ8/H3Z2doaHm5tbmfdJRER107ELf6LzjOk4duFPqaNQOUh+sfmDN0EWQpR6Y+Ti6h9cbuo2KypbeHg4NBqN4XHjxg2T90lERHVL+NbVOGL2Efp8zW/q1USSTVLh6OgIuVxeZIQnNTW1yEjQfS4uLsXWm5ubo379+qXWlLTNkvYDFI5Mubq6lnk7KpUKKpWqzPshIqK6LSc3H8eyvgWsgeBWnMm8JpJsREqpVMLHxweRkZFGyyMjI+Hv71/sOn5+fkXqDxw4gPbt20OhUJRaU9I2i+Pp6QkXFxej7eTm5iI6Otqk7RAREZVm3tb90Fv/DVl2fXz0Wj+p41A5SDpt6oQJExAcHIz27dvDz88Pq1atQlJSEkJDQwEUnir7+++/sX79egCF39BbunQpJkyYgJCQEMTGxmL16tVG38YbN24cOnfujIULF6Jv377YvXs3Dh48iKNHjxpqMjIy8McffxieJyYmIi4uDvXq1YO7uztkMhnCwsIwb948NGvWDM2aNcO8efNgaWmJ1157rYqODhER1XarTq8C7AFvs2GwteIZjRpJSGzZsmXCw8NDKJVK4e3tLaKjow2vDRs2THTp0sWoPioqSrRr104olUrRuHFjsWLFiiLb3LZtm2jevLlQKBSiRYsWIiIiwuj1Q4cOCQBFHsOGDTPU6PV6MWPGDOHi4iJUKpXo3LmzOHfunEnvTaPRCABCo9GYtB4REdV+Jy/9JTDdTGAmxA/HL0odh/7DlM9vSeeRqu04jxQREZXkhdlz8IuYBtu7naD57LDUceg/asQ8UkRERHVZgSgAcq3x6hO8yLwm44hUJeKIFBERlSblTgYsVQpeH1XNmPL5LenF5kRERHWZSz1rqSPQI+KpPSIioiqUkHQLq38+Ab2eJ4RqAzZSREREVWj8+tUYebwjmn0wXOooVAHYSBEREVWR/AI9frn7DQCgi0dXacNQhWAjRUREVEU+3x2FfNurgM4WC4YOljoOVQA2UkRERFXk86OrAABeBa/BycFK4jRUEdhIERERVYFLN9Jww2onAGBKIOeOqi3YSBEREVWBDzauB8xzYXnPB0Oe85Y6DlUQNlJERERV4GjKfgBAP3eORtUmnJCTiIioCiR/vA8Lth/AiBeekToKVSA2UkRERFVAqZBj+qs9pY5BFYyn9oiIiCrRvYwcZOXkSR2DKgkbKSIioko0csXXsJnhhlc+WS51FKoEbKSIiIgqiV4v8FPy19Bb/gO94L31aiM2UkRERJXk2wMnkGN3DsizwMfBQ6SOQ5WAjRQREVElWfTL1wCAJjmD4OFsL20YqhRspIiIiCrBX7e0uKLcDAB4v9tbEqehysJGioiIqBJ88N33gDILSk1LjOrpL3UcqiRspIiIiCrBrhuFNyju6RQCMzOZxGmosrCRIiIiqgTrB6zFk9nvYuGQYKmjUCWSCcHvY1YWrVYLOzs7aDQa2NraSh2HiIiIysCUz2+OSBERERGVExspIiKiCvT2ig1o+v7rWBd5UuooVAXYSBEREVWQqPhrWJX4Aa7ZfIctvx2SOg5VATZSREREFeDstRR0Xx8AvVUyVJrWWPVWqNSRqAqwkSIiInpESaka+H7ZA/m2V2Ge7omYt39Gowb8klFdwEaKiIjoEdzRZqPNvJeQYx8Psyxn7B9yAN7NGkodi6oIGykiIqJH8Oyc8dA4HAZ0tvi+13483+5xqSNRFWIjRURE9AhWD58Ci3tP4YuOPyCoy1NSx6EqZi51ACIioprMz8sd2sWnYS7n2ERdxD91IiIiE/WZ/wk+WLPD8JxNVN3FP3kiIiITDPv8G/yY+z4+vj4IEUfPSR2HJMZGioiIqIw+WLMD6++MAgD4FkzEwGeflDgRSY3XSBEREZXBkp2H8PG1VwFzPZ7IeBMxC+dLHYmqATZSRERED7Hhl9N479RLgDIXrvf6I37hSpiZyaSORdUAT+0RERGV4uSlv/D6gZ6AMgP2d7vh4kffQ63kOAQVYiNFRERUinaPN8STsldgcc8b56bugr21WupIVI2wpSYiIiqFudwMZ+Z9jtR7mXCpZy11HKpmOCJFRET0gNS7meg2axYysnMBAGZmMjZRVCw2UkRERP+RkZ0Lr9kDEYWZaPXhMKnjUDXHRoqIiOj/5Rfo0erDYbht/zOQa4lp3cdJHYmqOTZSREREAPR6Ae+pY5FkuxkoUGBOmx0Y2aOj1LGommMjRUREBOD5j2bjnMUyQMgwptF6TA0KlDoS1QBspIiIqM575ZPliMJMAECQzVJ88dYr0gaiGoONFBER1Xkdm3oBOht0wyxsfm+01HGoBuE8UkREVOeF9esK3yfOw7eFm9RRqIbhiBQREdVJ3+w/jp3Hzhue+3m58/55ZDKOSBERUZ2zO+YC3oruBQDYqo/Cy53aSJyIaiqOSBERUZ1y9Px1DNgVAKG+C6uc5ujcuqnUkagGYyNFRER1xoXrqXhuTXforW5CpWmFuIk/wcnBSupYVIOxkSIiojrhr1tadPi8B/Js/4A83QNHQ39G04b1pI5FNRwbKSIiqvXuZeTgybl9kW1/BrKsBtj3aiTaP/GY1LGoFmAjRUREtV6WLq/wB50Nvgvcj+4+zaQNRLUGv7VHRES1XsP6Nkicsw8/n/4dQV2ekjoO1SIckSIiolrr893Rhp/trdVsoqjCsZEiIqJaqfe8jxEW1xVPT/1A6ihUi7GRIiKiWmfEl2uwN6+wgXK0dJQ4DdVmbKSIiKhWCV+3C2vSRgIAOuROxF6OSFElkryRWr58OTw9PaFWq+Hj44MjR46UWh8dHQ0fHx+o1Wo0adIEK1euLFITEREBLy8vqFQqeHl5YefOnSbvd/jw4ZDJZEaPjh07PtqbJSKiSvXZrigs+OMVwEyPZhlv4PhHC6WORLWcpI3Uli1bEBYWhqlTp+LMmTPo1KkTevbsiaSkpGLrExMT0atXL3Tq1AlnzpzBlClTMHbsWERERBhqYmNjERQUhODgYMTHxyM4OBiDBw/GiRMnTN5vjx49kJycbHjs3bu3cg4EERE9sk1RZzD+t5cAcx1c7vXF2bmreBNiqnQyIYSQaue+vr7w9vbGihUrDMtatmyJfv36Yf78+UXqJ02ahD179iAhIcGwLDQ0FPHx8YiNjQUABAUFQavVYt++fYaaHj16wMHBAZs2bSrzfocPH4579+5h165d5X5/Wq0WdnZ20Gg0sLW1Lfd2iIjo4YZ9/g3W3wuB3d0uuDZ7H+rZWkgdiWooUz6/JRuRys3NxenTpxEQEGC0PCAgADExMcWuExsbW6Q+MDAQp06dQl5eXqk197dpyn6joqLg5OSEJ554AiEhIUhNTS31Pel0Omi1WqMHERFVjXXjRuLDJntwdspuNlFUZSRrpNLS0lBQUABnZ2ej5c7OzkhJSSl2nZSUlGLr8/PzkZaWVmrN/W2Wdb89e/bExo0b8euvv+KTTz7ByZMn8dxzz0Gn05X4nubPnw87OzvDw83N7SFHgYiIHkVi8l1cvXnH8Pyj4D5wd7KTMBHVNZLPbC6TGZ+/FkIUWfaw+geXl2WbD6sJCgoy/Ny6dWu0b98eHh4e+OmnnzBgwIBis4WHh2PChAmG51qtls0UEVElSdNkoe3CF5FrpkXM2z/Du1lDqSNRHSRZI+Xo6Ai5XF5k9Ck1NbXIaNF9Li4uxdabm5ujfv36pdbc32Z59gsArq6u8PDwwJUrV0qsUalUUKlUJb5OREQVIysnD14zX0a6QwxkOfb4M/UOGymShGSn9pRKJXx8fBAZGWm0PDIyEv7+/sWu4+fnV6T+wIEDaN++PRQKRak197dZnv0CwO3bt3Hjxg24urqW7Q0SEVGlyC/Qw2vqcNyy3wfkWWBFp5/Q/5nWUseiukpIaPPmzUKhUIjVq1eLixcvirCwMGFlZSWuX78uhBBi8uTJIjg42FB/7do1YWlpKcaPHy8uXrwoVq9eLRQKhdi+fbuh5tixY0Iul4sFCxaIhIQEsWDBAmFubi6OHz9e5v2mp6eL9957T8TExIjExERx6NAh4efnJx577DGh1WrL/P40Go0AIDQazaMeKiIiEkIUFOhFm0ljBGZCYJq5mPX9XqkjUS1kyue3pI2UEEIsW7ZMeHh4CKVSKby9vUV0dLThtWHDhokuXboY1UdFRYl27doJpVIpGjduLFasWFFkm9u2bRPNmzcXCoVCtGjRQkRERJi036ysLBEQECAaNGggFAqFcHd3F8OGDRNJSUkmvTc2UkREFeu5WbMLm6iZEG+v2CB1HKqlTPn8lnQeqdqO80gREVWcv25p0XjRkyiwTsIAi88R8cFYqSNRLVUj5pEiIiIyRaMGtogdeQyDrJayiaJqgyNSlYgjUkREjy4pVcO5oahKcUSKiIhqhdU/n0DjTz0x/putUkchKhYbKSIiqpb2HL+IkEO9INR3seHCGuj1PIFC1Q8bKSIiqnaOXfgT/SMCICzuwOqeL85N2wYzs5LvekEkFTZSRERUrSQk3UK3bwOgt/4bSk1LnHnvJ7jUs5Y6FlGx2EgREVG1cfN2Otp/2hN5tpchz3DH0VEH0KxRfaljEZWIjRQREVUbb3/9NbLsT0OW7YgfBh9Ah+aNpI5EVCo2UkREVG1ETAyDf8EUrH1hH3p2aC51HKKHMpc6ABER1W16vUB+gR5KhRzmcjMcmz1X6khEZcYRKSIiktSzM6bCc9Jg3MvIkToKkcnYSBERkWT6LliCWPP5uGm3Ax/vPCB1HCKTsZEiIiJJvLVsPfbo3gMABMjnYW7wSxInIjIdGykiIqpy0777AV+njgAA+OgmYN+UyRInIiofXmxORERV6ss9hzHn0mBAUYAm6a/j+MKPOWs51VgckSIioipzR5uNsKNBgCIHzvf64Nycb2Au50cR1Vz87SUioipTz9YCX3beBpd7L+HirC2wVCukjkT0SGRCCN5Ou5JotVrY2dlBo9HA1tZW6jhERERUBqZ8fnNEioiIKtWf/9yDY1ggth6OlzoKUYVjI0VERJUmTZOFNvP74LbDAQTvegW5eQVSRyKqUGykiIioUmTl5KHVzMHQOhwFcuywsf8WKBVyqWMRVSg2UkREVOHyC/R48sM3kWr/E5CnxrJnfsTLndpIHYuowrGRIiKiCqXXCzz94Xu4ZvMdoJdjRsvtGP3is1LHIqoUbKSIiKhCvbl0Dc6oPwMAjHJeg5lDeksbiKgSsZEiIqIKNXfIIDjcfR791J9i5ehgqeMQVSreIoaIiCpUw/o2SPn4Z15YTnUCR6SIiOiRLdp+ED3mLIBeXzjHM5soqivYSBER0SNZF3kSk870w88F4Qhd8Z3UcYiqFBspIiIqt72//Y43fukJKDPhcPd5LB4eJHUkoirFRoqIiMrlRMINvLStO4TFbVjd64Dz03bC1koldSyiKsVGioiITHbpRho6fx2AAuu/oNS0wOkJe9Gwvo3UsYiqHBspIiIySVZOHnyW9Eau3e+QZ7jhcMgBNHdzlDoWkSTYSBERkUks1Qr0dR8BWZYT9gw6AN+WblJHIpKMTAghpA5RW2m1WtjZ2UGj0cDW1lbqOEREFeqvW1o0asC/26j2MeXzmyNSRET0UHq9QO95H+PSjTTDMjZRRGykiIioDDrPnIa9eR+g7addkZGdK3UcomqDjRQREZVqwKLPcUw+FwAw0G0MrC2UEiciqj7YSBERUYneXrEBO7PDAAAvmM3BxvGjpA1EVM2wkSIiomLN3PgTVqYMBwA8lTMOP0+dIm0gomrIXOoARERU/Xy1NwazEgYBigJ4aofi5KIlMDOTSR2LqNphI0VEREU0a+gM8xwX1Mv0wvn538JczhMYRMVhI0VEREU891RTnLaNQcP6trBUK6SOQ1Rt8Z8YREQEADif+A8WbIs0PG/TxAWOdpYSJiKq/jgiRURE+POfe+jwRSBybM/jdvoWfDxioNSRiGoEjkgREdVxd7TZaDP/JeTYx8MsxxE9vZ+SOhJRjcFGioioDsvJzUerGa9A63AE0Nni+1778dxTTaWORVRjsJEiIqqj8gv0aD1lJFLs9wB5anzR8QcEdXlK6lhENQobKSKiOkivF+g4/QNctVkH6OWY0nwLxrzUWepYRDUOGykiojpILwQycjUAgJFOqzE3+CWJExHVTPzWHhFRHWQuN8PFhauw/KfX8W6fTlLHIaqxOCJFRFSHrIs8iZzcfACAmZmMTRTRI+KIFBFRDabXC2izdNBk5sDD2d6wPCr+Gi79nYKMnBykZ2cjU5eD67eTsV37Hibv7YGEj7bA3lotXXCiWoKNFBFRBYu7mozkO1pos7KRnp2DjJwcaP+/mdHr9UaTXb69YgPOp1xCTn4OdAXZ0BXkIFefg1yRDb0oQPKnuwy1rSeNxqWCfdCb5UDIcyDMswFzneH17PA8qJWFf60PWz8VSbabi4YzB2SQG+qI6NHw/yQiqlVS72bibkY2tFk5uJeRDW12DjKyCxsZlUKBt3r6GWrHrtqMG3dTkJ2Xg5z8HGTnFzYyuoIcWCtscXLux4baJyaOwN/5Z1Egy0GBLAd6eTbE/zc0ZrkOKPj4L0Nt5y8HId3hWPEBc63xMf5tpLZf3og0+/2AsvhyvV7AzEwGALiXdwv5dtdLfO/aTJ2hQXJUueKmtinkQg25UMNcWMAcajSxegqRH33ERoqogvD/JCKqdHq9QE5uvtHNbz/bFYV/NPdwO0OLu1laZORmIie/sKFxs2uEre+/Y6ht+v7r0OT/g3zkIF+W/W8zY5YNu7wWuP3Zv/eHe2xua+TbXC82h1LTAm/1TDA8X/X7XOjszhc+kQFQ/P8DgDyjEYB/G6nk/ARk2Z8u4f0Z39TXQmaPjBx7yAosYKZXw0yvhlxYwFyooYS1Ue3zjfrg4q3HoZKrYWFuAbW5GhYKNSwVFrBQqqEXAmYobKRWBs1Gyt33YGOhhq2lBWwt1bC3soCdlRr1bCyMmqPT85YAWFJsXiKqOGykiKhEer3AvYwc3LytRfIdLZLvanBLq0Vauha3M7Rwr++ED1/pYahtOjEYWXoNdEILnUyDfLkWBeZaCKUG9bXPIe2znw3bHv9bX0ClLbpTc+D83x0B/NtIXZf9Cr3D38VmzNTYGz030///dT/5SqBADVmBGmYFFpALNeyEp1HtkxY9cEv7JJRmaqjMLKCSq6EyL2xo6jk7GNXO67YQdzIyYK1Ww1pd2MjYWKhha6GGvbWFUe0/n/5Y2mE1svm90WWufdG3ZZlriahqsJEiqoXuN0B/pWmQfEcLawsVnmnlAQDIysnD61+uxL1sDbQ6LTLytMgq0CJbr0WO0KKFtR9OzFkIoHDm6/qfWpa4H8c/ehgaKTMzGa6r9gCq9GJrdTLjpsk+ywf52dlQCVuozWxhYWYNpVwNlZkaTRo2Map9p9li5ObnwVKlhpVSDRsLC0ND42xva1T7z+wzsFQpoFTIH3qc/nvq7mE4WSURFYeNFFE1lZWTh21H4pBqGAHS4G6WFpocLbQ6Ldq7tcXy0CEAgJQ7GfD86Jn/jABpAXm+YVtumiAkLSm88NhcboaIrLGFLyhR5Nqca/esDD8rFXIg1xpQZAI6W8jzbWFeYAuF3hYqmS1a2LY3WneI02KYm8lRz8oW9a1t4WxnByd7W7g62OIxRzuj2ruf/VrmY/HFW6+UuZbfRCOiqsRGiqgC6PUCaZos3LxTeAos9Z4W/2g0SEvXorWbO15/obDhSEy+i16fTkZmvgY5onAEKM+s8KFXaNFavIazC5YCAJLvpGN4zNPF71AJ/HPlFQCFjZS9tRo59meL1gkZoLOFwuzfbkmpkKNJejAUMhWsFbawUdnCTmULe0tb1Leyg1dHN6NNJL+XDEc7S5jLHz7t3Iawt8pwtIiIag/JG6nly5fj448/RnJyMlq1aoXPPvsMnTqVPEFcdHQ0JkyYgAsXLqBhw4b44IMPEBoaalQTERGBadOm4erVq2jatCnmzp2L/v37m7RfIQRmzZqFVatW4e7du/D19cWyZcvQqlWrij0AVGnyC/TQ64XhFE9Gdi5OXf4LGTk6ZObokKXLRaZOh+zcXGTpdPBp6mm4BiUpVYP312+ALl+HdF0GtLlaZOZpkZlf2Py84PYSvp9Q+Ht39Px1dNrWFDDTF5vD6/dQQyOlFwK/W60qMbNWe8fw82OOtpBnuMG8wA7K/x8BspDZwcrcFlYKW/g94WOoVSvNsbBVJOrb2MDZ3hYuDrZo5GhXYgN0dfH6Mh9Hl3rWDy8iIqqjJG2ktmzZgrCwMCxfvhzPPPMMvvrqK/Ts2RMXL16Eu7t7kfrExET06tULISEh2LBhA44dO4bRo0ejQYMGGDiw8OvEsbGxCAoKwkcffYT+/ftj586dGDx4MI4ePQpfX98y73fRokVYsmQJ1q5diyeeeAJz5sxB9+7dcenSJdjY2FTdQarm9HqBLF0etFk6ZGbnIj27sEFJz9bBy93ZMEFgQtItRMScRlauDjl5ucjO1SE7TwddXi5y8nV4pWNXw13nfznzB97b+iXy9LnI0+uQq9chX+QiX+iQL3R4/cmRWPLmIADA1sPxGLp7EPSyXAgzHYRZLoRcB8h1gDwf3TALv86YDgD4+fQlvPxLmxLfS4cLE/Gi7yIAwPWUO9iW+e6/Lz5wCuzsP40NPzvb2/zbROnNIMu1g1le4ekvhd4WHg3+vcD5MUdbPCebDTu1LewtbFHfyhb1bWzRwNYWrg52aOrqaKhVK82R/3FSmf8sPnj5hTLXEhFRxZAJIYRUO/f19YW3tzdWrFhhWNayZUv069cP8+fPL1I/adIk7NmzBwkJ/359OTQ0FPHx8YiNjQUABAUFQavVYt++fYaaHj16wMHBAZs2bSrTfoUQaNiwIcLCwjBp0iQAgE6ng7OzMxYuXIhRo0aV6f1ptVrY2dlBo9HA1tb24SuYIClVg0s3UpHxn5GVrP+MrrzWpSNaujcAAOyOuYD1R3+BriAXunwddPk65BbkIregsEmZ+WIoXu5U2GB8suNXLDjyMQpELvKhg16WiwLooDcr/Hly28/xUXAfAMDEbyOw+MbLJWYc7rAaa8aOAADM3rQPMy73KrF2oOUX2D5xDABg6Q9HMOZ/JV/YGyCfh58/DAcAbPz1fxh6xKfEWv+CKTg2ey4AIPL0FQTseAoyvQqyAhVkQgkzvQpmQgkzoUKgSzB2TgoDAFz56za6LRkFc5kKFnJr2CjsYKuyhZ3aFg6WtujUvBWGde8AoHDk6/z1f9Cwni0c7SwNc/4QEVHNZMrnt2QjUrm5uTh9+jQmT55stDwgIAAxMTHFrhMbG4uAgACjZYGBgVi9ejXy8vKgUCgQGxuL8ePHF6n57LPPyrzfxMREpKSkGO1LpVKhS5cuiImJKbGR0ul00On+nWVYqy3mq90VZOy332K3bkKJr1uqfkZL98L8Eb/FYEf2uH9fNPv/x/9PfXPyandDI3UtNaVwcsAS3Eq/Z/hZqVAULdDLgXwVZHol5Gb/nlJ6rF49WNxrBzlUkAsl5FDBXKaCuUwJhUyFNk88bqj1aeoB/5NToJKroJQroTZXQWmuhIVCBQuFCoFt2xlqn3/qCSzLOgILpRJWahUsVUpYq1WwUithbaGCs/2/p6W6+zSD8Mks8b39V7NG9fHXku1lqjWXm+Gppq5lqiUiotpFskYqLS0NBQUFcHZ2Nlru7OyMlJSUYtdJSUkptj4/Px9paWlwdXUtseb+Nsuy3/v/La7mzz//LPE9zZ8/H7NmzSrx9YpkrbYCtLaFIyv6whGV+6Mrcqhgb/XvN6+e8ngcUUeCoDBTQSFTQilXQWFW+F+VXIVnWjQ31Ab5+yH3yFqozZWwUKqgVihhqVLBSqWChVKJTq3+bXgm9g/Aq7dSYGOpgo2FCtYWyhK/cv5moC/eDPxfmd6bn5e7YRTpYVzqWWP0i8+WqZaIiKiiSX6xuUxmfBpECFFk2cPqH1xelm1WVM1/hYeHY8KEf0eJtFot3NzcSqx/FBvC3sIGlO0bUhP6d8OE/t3KVNu5jSc6t/F8eCEKvynGr5oTEVFdJlkj5ejoCLlcXmT0KTU1tchI0H0uLi7F1pubm6N+/fql1tzfZln26+LiAqBwZMrV1bXYmuKoVCqoVKoSXyciIqLa5eETw1QSpVIJHx8fREZGGi2PjIyEv79/sev4+fkVqT9w4ADat28Pxf9fr1NSzf1tlmW/np6ecHFxMarJzc1FdHR0idmIiIioDhIS2rx5s1AoFGL16tXi4sWLIiwsTFhZWYnr168LIYSYPHmyCA4ONtRfu3ZNWFpaivHjx4uLFy+K1atXC4VCIbZv326oOXbsmJDL5WLBggUiISFBLFiwQJibm4vjx4+Xeb9CCLFgwQJhZ2cnduzYIc6dOydeffVV4erqKrRabZnfn0ajEQCERqN5lMNEREREVciUz29JGykhhFi2bJnw8PAQSqVSeHt7i+joaMNrw4YNE126dDGqj4qKEu3atRNKpVI0btxYrFixosg2t23bJpo3by4UCoVo0aKFiIiIMGm/Qgih1+vFjBkzhIuLi1CpVKJz587i3LlzJr03NlJEREQ1jymf35LOI1XbVeY8UkRERFQ5TPn8luwaKSIiIqKajo0UERERUTmxkSIiIiIqJzZSREREROXERoqIiIionNhIEREREZUTGykiIiKicmIjRURERFRObKSIiIiIyslc6gC12f1J47VarcRJiIiIqKzuf26X5eYvbKQqUXp6OgDAzc1N4iRERERkqvT0dNjZ2ZVaw3vtVSK9Xo+bN2/CxsYGMpmsQret1Wrh5uaGGzdu8D5+D8FjVXY8VmXHY1V2PFZlx2NVdpV5rIQQSE9PR8OGDWFmVvpVUByRqkRmZmZo1KhRpe7D1taW/7OVEY9V2fFYlR2PVdnxWJUdj1XZVdaxethI1H282JyIiIionNhIEREREZUTG6kaSqVSYcaMGVCpVFJHqfZ4rMqOx6rseKzKjseq7Hisyq66HCtebE5ERERUThyRIiIiIionNlJERERE5cRGioiIiKic2EgRERERlRMbqVpEp9PhqaeegkwmQ1xcnNRxqqWXXnoJ7u7uUKvVcHV1RXBwMG7evCl1rGrn+vXrePPNN+Hp6QkLCws0bdoUM2bMQG5urtTRqqW5c+fC398flpaWsLe3lzpOtbN8+XJ4enpCrVbDx8cHR44ckTpStXP48GH06dMHDRs2hEwmw65du6SOVG3Nnz8fHTp0gI2NDZycnNCvXz9cunRJsjxspGqRDz74AA0bNpQ6RrXWrVs3bN26FZcuXUJERASuXr2Kl19+WepY1c7vv/8OvV6Pr776ChcuXMCnn36KlStXYsqUKVJHq5Zyc3MxaNAgvP3221JHqXa2bNmCsLAwTJ06FWfOnEGnTp3Qs2dPJCUlSR2tWsnMzETbtm2xdOlSqaNUe9HR0XjnnXdw/PhxREZGIj8/HwEBAcjMzJQmkKBaYe/evaJFixbiwoULAoA4c+aM1JFqhN27dwuZTCZyc3OljlLtLVq0SHh6ekodo1pbs2aNsLOzkzpGtfL000+L0NBQo2UtWrQQkydPlihR9QdA7Ny5U+oYNUZqaqoAIKKjoyXZP0ekaoF//vkHISEh+O6772BpaSl1nBrjzp072LhxI/z9/aFQKKSOU+1pNBrUq1dP6hhUg+Tm5uL06dMICAgwWh4QEICYmBiJUlFto9FoAECyv5/YSNVwQggMHz4coaGhaN++vdRxaoRJkybBysoK9evXR1JSEnbv3i11pGrv6tWr+PLLLxEaGip1FKpB0tLSUFBQAGdnZ6Plzs7OSElJkSgV1SZCCEyYMAHPPvssWrduLUkGNlLV1MyZMyGTyUp9nDp1Cl9++SW0Wi3Cw8OljiyZsh6r+yZOnIgzZ87gwIEDkMvleP311yHqyAT/ph4rALh58yZ69OiBQYMGYeTIkRIlr3rlOVZUPJlMZvRcCFFkGVF5vPvuuzh79iw2bdokWQbeIqaaSktLQ1paWqk1jRs3xiuvvIIffvjB6C+lgoICyOVyDBkyBOvWravsqJIr67FSq9VFlv/1119wc3NDTEwM/Pz8KititWHqsbp58ya6desGX19frF27FmZmdeffXuX5vVq7di3CwsJw7969Sk5XM+Tm5sLS0hLbtm1D//79DcvHjRuHuLg4REdHS5iu+pLJZNi5cyf69esndZRqbcyYMdi1axcOHz4MT09PyXKYS7ZnKpWjoyMcHR0fWvfFF19gzpw5huc3b95EYGAgtmzZAl9f38qMWG2U9VgV5/6/I3Q6XUVGqrZMOVZ///03unXrBh8fH6xZs6ZONVHAo/1eUSGlUgkfHx9ERkYaNVKRkZHo27evhMmoJhNCYMyYMdi5cyeioqIkbaIANlI1nru7u9Fza2trAEDTpk3RqFEjKSJVW7/99ht+++03PPvss3BwcMC1a9cwffp0NG3atE6MRpni5s2b6Nq1K9zd3bF48WLcunXL8JqLi4uEyaqnpKQk3LlzB0lJSSgoKDDM4/b4448b/p+sqyZMmIDg4GC0b98efn5+WLVqFZKSkni93QMyMjLwxx9/GJ4nJiYiLi4O9erVK/L3fF33zjvv4Pvvv8fu3bthY2NjuN7Ozs4OFhYWVR9Iku8KUqVJTEzk9AclOHv2rOjWrZuoV6+eUKlUonHjxiI0NFT89ddfUkerdtasWSMAFPugooYNG1bssTp06JDU0aqFZcuWCQ8PD6FUKoW3t7dkX1Ovzg4dOlTs79CwYcOkjlbtlPR305o1ayTJw2ukiIiIiMqpbl30QERERFSB2EgRERERlRMbKSIiIqJyYiNFREREVE5spIiIiIjKiY0UERERUTmxkSIiIiIqJzZSRFTlunbtirCwMKljFOv27dtwcnLC9evXAQBRUVGQyWSVfv+88u5n7dq1sLe3N2mdDh06YMeOHSatQ0TFYyNFRDVecnIyXnvtNTRv3hxmZmYlNmkRERHw8vKCSqWCl5cXdu7cWaRm/vz56NOnDxo3bly5oSU0bdo0TJ48GXq9XuooRDUeGykiqvF0Oh0aNGiAqVOnom3btsXWxMbGIigoCMHBwYiPj0dwcDAGDx6MEydOGGqys7OxevVqjBw5sqqiS6J3797QaDT4+eefpY5CVOOxkSIiSd29exevv/46HBwcYGlpiZ49e+LKlStGNV9//TXc3NxgaWmJ/v37Y8mSJUansxo3bozPP/8cr7/+Ouzs7Irdz2effYbu3bsjPDwcLVq0QHh4OJ5//nl89tlnhpp9+/bB3Ny81JtY3759G6+++ioaNWoES0tLPPnkk9i0aZNRTdeuXTFmzBiEhYXBwcEBzs7OWLVqFTIzM/HGG2/AxsYGTZs2xb59+4ps/9ixY2jbti3UajV8fX1x7tw5o9fXrl0Ld3d3w7G4ffu20etXr15F37594ezsDGtra3To0AEHDx40qpHL5ejVq1eR3ERkOjZSRCSp4cOH49SpU9izZw9iY2MhhECvXr2Ql5cHoLCxCA0Nxbhx4xAXF4fu3btj7ty5Ju8nNjYWAQEBRssCAwMRExNjeH748GG0b9++1O3k5OTAx8cHP/74I86fP4+33noLwcHBRiNbALBu3To4Ojrit99+w5gxY/D2229j0KBB8Pf3x//+9z8EBgYiODgYWVlZRutNnDgRixcvxsmTJ+Hk5ISXXnrJcCxOnDiBESNGYPTo0YiLi0O3bt0wZ84co/UzMjLQq1cvHDx4EGfOnEFgYCD69OmDpKQko7qnn34aR44cKdvBI6KSSXKrZCKq07p06SLGjRsnLl++LACIY8eOGV5LS0sTFhYWYuvWrUIIIYKCgkTv3r2N1h8yZIiws7MrddsPUigUYuPGjUbLNm7cKJRKpeF53759xYgRI4xqDh06JACIu3fvlvh+evXqJd577z2jDM8++6zheX5+vrCyshLBwcGGZcnJyQKAiI2NNdrP5s2bDTW3b98WFhYWYsuWLUIIIV599VXRo0cPo30HBQWVeCzu8/LyEl9++aXRst27dwszMzNRUFBQ6rpEVDqOSBGRZBISEmBubg5fX1/Dsvr166N58+ZISEgAAFy6dAlPP/200XoPPi8rmUxm9FwIYbQsOzsbarW61G0UFBRg7ty5aNOmDerXrw9ra2scOHCgyIhPmzZtDD/L5XLUr18fTz75pGGZs7MzACA1NdVovf+eVqxXr57RsUhISChy2vHB55mZmfjggw/g5eUFe3t7WFtb4/fffy+Sz8LCAnq9HjqdrtT3S0SlM5c6ABHVXUKIEpffb3AebHZKW680Li4uSElJMVqWmppqaGgAwNHREXfv3i11O5988gk+/fRTfPbZZ3jyySdhZWWFsLAw5ObmGtUpFAqj5zKZzGjZ/fdUlm/O/fdYPMzEiRPx888/Y/HixXj88cdhYWGBl19+uUi+O3fuwNLSEhYWFg/dJhGVjCNSRCQZLy8v5OfnG11fdPv2bVy+fBktW7YEALRo0QK//fab0XqnTp0yeV9+fn6IjIw0WnbgwAH4+/sbnrdr1w4XL14sdTtHjhxB3759MXToULRt2xZNmjQpcnH8ozh+/Ljh57t37+Ly5cto0aIFgMLj9d/XH6y/n2/48OHo378/nnzySbi4uBjmxPqv8+fPw9vbu8JyE9VVbKSISDLNmjVD3759ERISgqNHjyI+Ph5Dhw7FY489hr59+wIAxowZg71792LJkiW4cuUKvvrqK+zbt6/IKFVcXBzi4uKQkZGBW7duIS4uzqgpGjduHA4cOICFCxfi999/x8KFC3Hw4EGjOacCAwNx4cKFUkelHn/8cURGRiImJgYJCQkYNWpUkZGuRzF79mz88ssvOH/+PIYPHw5HR0f069cPADB27Fjs378fixYtwuXLl7F06VLs37+/SL4dO3YgLi4O8fHxeO2114od9Tpy5EiRi++JyHRspIhIUmvWrIGPjw9efPFF+Pn5QQiBvXv3Gk6DPfPMM1i5ciWWLFmCtm3bYv/+/Rg/fnyRa5natWuHdu3a4fTp0/j+++/Rrl079OrVy/C6v78/Nm/ejDVr1qBNmzZYu3YttmzZYnR91pNPPon27dtj69atJeadNm0avL29ERgYiK5du8LFxcXQ6FSEBQsWYNy4cfDx8UFycjL27NkDpVIJAOjYsSO++eYbfPnll3jqqadw4MABfPjhh0brf/rpp3BwcIC/vz/69OmDwMDAIiNPf//9N2JiYvDGG29UWG6iukomynOxARGRhEJCQvD7779Xytf39+7di/fffx/nz5+HmVnt/LfmxIkTodFosGrVKqmjENV4vNiciKq9xYsXo3v37rCyssK+ffuwbt06LF++vFL21atXL1y5cgV///033NzcKmUfUnNycsL7778vdQyiWoEjUkRU7Q0ePBhRUVFIT09HkyZNMGbMGISGhkodi4iIjRQRERFRedXOCwCIiIiIqgAbKSIiIqJyYiNFREREVE5spIiIiIjKiY0UERERUTmxkSIiIiIqJzZSREREROXERoqIiIionNhIEREREZXT/wEstM74qGAwcwAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 10,
+ "id": "b2891d23",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
@@ -2901,8 +2867,10 @@
},
{
"cell_type": "markdown",
- "id": "3a37bf34",
- "metadata": {},
+ "id": "857b1b0e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We see here, when compared to the code which includes explicitely the\n",
"intercept column, that our MSE value is actually smaller. This is\n",
@@ -2914,8 +2882,10 @@
},
{
"cell_type": "markdown",
- "id": "f33e21aa",
- "metadata": {},
+ "id": "aa5328f0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Test Function for what happens with OLS, Ridge and Lasso\n",
"\n",
@@ -2937,49 +2907,13 @@
},
{
"cell_type": "code",
- "execution_count": 16,
- "id": "94659a7f",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[ 5.73226009e+01 -1.12358261e+03 1.08585442e+04 -5.92310031e+04\n",
- " 1.95886028e+05 -4.06455380e+05 5.31245929e+05 -4.24007434e+05\n",
- " 1.88512287e+05 -3.57414935e+04]\n",
- "Test MSE OLS\n",
- "1.0568260054589842\n",
- "0.0001 [-1.73105987e-02 4.27777349e+00 -1.24072819e+01 -1.57777953e+00\n",
- " 1.58524813e+01 1.44228898e+01 -5.18210622e+00 -2.39296769e+01\n",
- " -1.89984740e+01 2.75776410e+01]\n",
- "0.0001 [ 0.04180978 -0.14790968 -0. -0. -0. -0.\n",
- " -0. -0.07686791 -0. -0. ]\n",
- "0.00046415888336127773 [ 0.90948001 -0.87573584 -5.11160579 3.52244985 7.82028276\n",
- " 3.28968283 -5.38577625 -10.68363991 -6.07362329 12.5913198 ]\n",
- "0.00046415888336127773 [-0. -0.05093789 -0.06174616 -0. -0. -0.\n",
- " -0.06434271 -0. -0. -0. ]\n",
- "0.002154434690031882 [ 1.17265557 -3.17760645 0.08027484 3.21170036 2.69481039 -0.29991171\n",
- " -3.34556923 -4.25014612 -1.55611254 5.43271354]\n",
- "0.002154434690031882 [-0. -0. -0.13930669 -0. -0. -0.\n",
- " -0. -0. -0. -0. ]\n",
- "0.01 [ 0.72733092 -2.07019523 0.57512158 1.62078056 0.89296112 -0.45530968\n",
- " -1.46069988 -1.53490166 -0.40177453 1.99626389]\n",
- "0.01 [-0. -0.03753661 -0. -0. -0. -0.\n",
- " -0. -0. -0. -0. ]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlIAAAGwCAYAAABiu4tnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABvb0lEQVR4nO3deVxUVf8H8M/IjiwqyJaAorllmksqlrk8KaLhRqW5BC4kueujGaa5/Motc8VECUXLxJ40zXLBUtEUC1IUycgMXCFc2ZT9/P44MTqyCCNwZ+Dzfr3uyztnztz7PQzjfDnn3HNVQggBIiIiIiq3WkoHQERERKSvmEgRERERaYmJFBEREZGWmEgRERERaYmJFBEREZGWmEgRERERaYmJFBEREZGWDJUOoDorKCjAjRs3YGlpCZVKpXQ4REREVAZCCKSnp8PJyQm1apXe58REqhLduHEDzs7OSodBREREWrh69SoaNGhQah0mUpXI0tISgHwjrKysFI6GiIiIyiItLQ3Ozs7q7/HSMJGqRIXDeVZWVkykiIiI9ExZpuVwsjkRERGRlphIEREREWmJiRQRERGRljhHSgfk5+cjNzdX6TComjI2Nn7i5btERKQdJlIKEkIgOTkZ9+7dUzoUqsZq1aqFRo0awdjYWOlQiIiqHSZSCipMouzs7GBubs5FO6nCFS4Km5SUBBcXF/6OERFVMCZSCsnPz1cnUTY2NkqHQ9VY/fr1cePGDeTl5cHIyEjpcIiIqhVOnFBI4Zwoc3NzhSOh6q5wSC8/P1/hSIiIqh8mUgrjUAtVNv6OERFVHiZSRERERFpiIkVERESkJSZSVG0lJiZCpVIhJiamxDpHjx6FSqXiEhRERKQVJlJULr6+vlCpVPD39y/y3Pjx46FSqeDr66suS0lJwbhx4+Di4gITExM4ODjAw8MDkZGR6joNGzaESqUqsi1ZsqTEOLp3766uZ2xsjMaNGyMgIADZ2dnqOs7OzkhKSkKrVq0qpvFaKql9hVv37t21Pnb37t0xderUCouViEhv5OYCBw8qHQWXP6Dyc3Z2RlhYGFauXAkzMzMAQFZWFrZv3w4XFxeNut7e3sjNzcWWLVvg5uaGf/75Bz/99BPu3LmjUW/hwoXw8/PTKLO0tCw1Dj8/PyxcuBA5OTmIiorCqFGjAACLFy8GABgYGMDBweGp2loRoqKi1FfMnTx5Et7e3oiPj4eVlRUAcKFMIiJtLFkCfPgh8O67wGefKRYGe6R0iBBAZmbVb0KUL8527drBxcUFu3btUpft2rULzs7OaNu2rbrs3r17+Pnnn7F06VL06NEDrq6u6NixIwICAtCvXz+NY1paWsLBwUFjq127dqlxmJubw8HBAS4uLvD29kavXr0QHh6ufr64ob19+/ahadOmMDMzQ48ePZCYmFjkuMHBwXB2doa5uTkGDRqEFStWoE6dOhp19u7di/bt28PU1BRubm5YsGAB8vLyio2zfv366jbVq1cPAGBnZ6cu++OPP/DKK6/AzMwMzs7OmDx5MjIzM9Wv/+yzz/Dss8/C1NQU9vb2eP311wHI3sGIiAisXr1a3btVXHuIiKqd8+eB//s/ud+1q6KhMJHSIffvAxYWVb/dv1/+WEeNGoXNmzerH2/atAmjR4/WqGNhYQELCwvs3r1bY8itMpw9exYnTpwodcHJq1evYvDgwejbty9iYmIwduxYvP/++xp1Tpw4AX9/f0yZMgUxMTHo1asXPv74Y406Bw8exIgRIzB58mT8/vvv2LBhA0JDQ4vUK4vY2Fh4eHhg8ODBOHfuHHbs2IGff/4ZEydOBABER0dj8uTJWLhwIeLj43HgwAG88sorAIDVq1fD3d0dfn5+SEpKQlJSEpydncsdAxGRXsnLA0aPlkN7/fsDQ4cqG4+gSpOamioAiNTU1CLPPXjwQPz+++/iwYMH6rKMDCFk/1DVbhkZZW+Tj4+PGDBggLh586YwMTERCQkJIjExUZiamoqbN2+KAQMGCB8fH3X9b775RtStW1eYmpqKLl26iICAAHH27FmNY7q6ugpjY2NRu3Ztje3IkSMlxtGtWzdhZGQkateuLYyNjQUAUatWLfHNN9+o6yQkJAgA4syZM0IIIQICAkSLFi1EQUGBus6sWbMEAHH37l0hhBBDhgwR/fr10zjX8OHDhbW1tfpx165dxaJFizTqfPHFF8LR0fGJP78jR45onG/kyJHinXfe0ahz/PhxUatWLfHgwQOxc+dOYWVlJdLS0kr8OUyZMqXUcxb3u0ZEpLeWLZNfXtbWQly/XimnKO37+3GcI6VDzM2BjAxlzltetra26NevH7Zs2QIhBPr16wdbW9si9by9vdGvXz8cP34ckZGROHDgAJYtW4bPP/9cY1L6zJkzNR4DwDPPPFNqDMOHD8cHH3yAtLQ0LF26FFZWVvD29i6x/oULF9C5c2eNBSrd3d016sTHx2PQoEEaZR07dsT333+vfvzbb78hKipKowcqPz8fWVlZuH//frlWq//tt9/w119/Ydu2beoyIQQKCgqQkJCAXr16wdXVFW5ubujTpw/69OmDQYMGcUV8IqqZ4uOBuXPl/sqVgJOTsvGAk811ikoFPGFakE4ZPXq0eghq3bp1JdYzNTVFr1690KtXL3z44YcYO3Ys5s2bp5E42draokmTJuU6v7W1tfo1X375JZ577jmEhIRgzJgxxdYXZZgMJoQoshL4468rKCjAggULMHjw4CKvNzU1LWv46mONGzcOkydPLvKci4sLjI2Ncfr0aRw9ehTh4eH48MMPMX/+fERFRRWZt0VEVK0VFABjxgDZ2UDv3sBjf3wrhYkUaa1Pnz7IyckBAHh4eJT5dS1btsTu3bsrNBYjIyPMnj0bAQEBeOutt4rtsSnuvKdOndJ43Lx5c/z6668aZdHR0RqP27Vrh/j4+HInfsVp164d4uLiSj2WoaEhXn31Vbz66quYN28e6tSpg8OHD2Pw4MEwNjbmPfSIqGZYtw44cUJO7t24UfY+6ABONietGRgY4MKFC7hw4QIMDAyKPH/79m307NkTX375Jc6dO4eEhAT873//w7JlyzBgwACNuunp6UhOTtbY0tLSyhXPsGHDoFKp8FkJl8H6+/vj0qVLmD59OuLj4/HVV18hNDRUo86kSZOwb98+rFixAhcvXsSGDRuwf/9+jV6qDz/8EFu3bsX8+fMRFxeHCxcuYMeOHZgzZ0654gWAWbNmITIyEhMmTEBMTAwuXryI7777DpMmTQIAfP/991izZg1iYmJw+fJlbN26FQUFBWjWrBkAuUbVL7/8gsTERNy6dQsFBQXljoGISOclJACFFwctWwa4uiobzyOYSNFTsbKyUq+H9DgLCwt06tQJK1euxCuvvIJWrVph7ty58PPzQ2BgoEbdDz/8EI6Ojhrbe++9V65YjI2NMXHiRCxbtgwZxUw2c3Fxwc6dO7F37160adMGQUFBWLRokUadl156CUFBQVixYgXatGmDAwcOYNq0aRpDdh4eHvj+++9x6NAhvPjii+jcuTNWrFgBVy0+2K1bt0ZERAQuXryIrl27om3btpg7dy4cHR0BAHXq1MGuXbvQs2dPtGjRAkFBQdi+fTuee+45AMCMGTNgYGCAli1bon79+rhy5Uq5YyAi0mlCAGPHykvMu3UDxo1TOiINKlGWiSOklbS0NFhbWyM1NbVIspGVlYWEhAQ0atSo3PNqqGr5+fnhjz/+wPHjx5UORSv8XSMivbZxo0yezMyAc+eACphW8SSlfX8/jnOkiB6zfPly9OrVC7Vr18b+/fuxZcuWEocLiYioEl29CsyYIfc//rhKkqjyYiJF9Jhff/0Vy5YtQ3p6Otzc3LBmzRqMHTtW6bCIiGoWIWRPVHo60LkzUMzVzbqAiRTRY77++mulQyAioi++APbvB0xMgE2bgGIuatIFnGxOREREuiUpCZgyRe7Pnw+0aKFoOKVhIkVERES6Qwhg/Hjg3j2gffuHc6R0FBMpIiIi0h3/+x+wezdgaCiH9Ax1exYSEykiIiLSDTdvAv/eegwffAC0bq1sPGWgaCJ17NgxeHl5wcnJCSqVqky3DYmIiED79u1hamoKNzc3BAUFlVg3LCwMKpUKAwcO1CifP38+VCqVxubg4KBRRwiB+fPnw8nJCWZmZujevTvi4uK0aSYRERGVxZQpMplq1QqYPVvpaMpE0UQqMzMTbdq0KbLKdUkSEhLQt29fdO3aFWfOnMHs2bMxefJk7Ny5s0jdy5cvY8aMGejatWuxx3ruueeQlJSk3mJjYzWeX7ZsGVasWIHAwEBERUXBwcEBvXr1Qnp6evkbSkRERKXbswfYvh2oVQvYvBkwNlY6ojJRNJHy9PTERx99hMGDB5epflBQEFxcXLBq1Sq0aNECY8eOxejRo7F8+XKNevn5+Rg+fDgWLFgANze3Yo9laGgIBwcH9Va/fn31c0IIrFq1Ch988AEGDx6MVq1aYcuWLbh//z6++uor7RtcDfj6+kKlUsHf37/Ic+PHj4dKpYLvI3fkTklJwbhx4+Di4gITExM4ODjAw8MDkZGR6joNGzYs0kOoUqmwZMmSEuPo3r07pk6dWpFNqxSFP6/SNm3Nnz8fL7zwQsUFS0SklLt3gXfflfszZwIdOigbTzno1RypyMhI9O7dW6PMw8MD0dHRyM3NVZctXLgQ9evXx5gxY0o81sWLF+Hk5IRGjRph6NCh+Pvvv9XPJSQkIDk5WeNcJiYm6NatG06ePFniMbOzs5GWlqaxVUfOzs4ICwvDgwcP1GVZWVnYvn07XFxcNOp6e3vj7Nmz2LJlC/78809899136N69O+7cuaNRb+HChRo9hElJSeob9+qz1atXa7QJADZv3lykjIioRps+XS550KwZMG+e0tGUi14lUsnJybC3t9cos7e3R15eHm7dugUAOHHiBEJCQhAcHFzicTp16oStW7fi4MGDCA4ORnJyMrp06YLbt2+rz1N47MfPVfhccRYvXgxra2v15uzsrFU7dV27du3g4uKCXbt2qct27doFZ2dntG3bVl127949/Pzzz1i6dCl69OgBV1dXdOzYEQEBAejXr5/GMS0tLTV6CB0cHFC7dm2tY5w1axaaNm0Kc3NzuLm5Ye7cuRrJ9tmzZ9GjRw9YWlrCysoK7du3R3R0NAA5LOzl5YW6deuidu3aeO6557Bv3z71ayMiItCxY0eYmJjA0dER77//PvLy8oqNw9raWqNNgLwRceHj/Px8DBkyBHXr1oWNjQ0GDBiAxMRE9euPHj2Kjh07onbt2qhTpw5eeuklXL58GaGhoViwYAHOnj2r7tkKDQ3V+udFRKSYAweA0FBApQJCQuQ99fSIbl9TWIzHh0IK77msUqmQnp6OESNGIDg4GLa2tiUew9PTU73//PPPw93dHY0bN8aWLVswffr0Us9V2lBMQECAxuvT0tLKl0wJIe9uXdXMzeUvcDmMGjUKmzdvxvDhwwEAmzZtwujRo3H06FF1HQsLC1hYWGD37t3o3LkzTExMKjLqUllaWiI0NBROTk6IjY2Fn58fLC0t8d577wEAhg8fjrZt22L9+vUwMDBATEwMjIyMAAATJkxATk4Ojh07htq1a+P333+HhYUFAOD69evo27cvfH19sXXrVvzxxx/w8/ODqakp5s+fX64Y79+/jx49eqBr1644duwYDA0N8dFHH6FPnz44d+4catWqhYEDB8LPzw/bt29HTk4Ofv31V6hUKgwZMgTnz5/HgQMH8OOPPwKQSRsRkV5JSwPeeUfuT54MvPSSsvFoQa8SKQcHhyI9QikpKTA0NISNjQ3i4uKQmJgILy8v9fMFBQUA5Jyo+Ph4NG7cuMhxa9eujeeffx4XL15UnweQPVOOjo4a53q8l+pRJiYmT5cs3L8P/PuFXaUyMoBy9v6MHDkSAQEBSExMhEqlwokTJxAWFqaRSBkaGiI0NBR+fn4ICgpCu3bt0K1bNwwdOhStH7ukddasWZgzZ45G2ffff4/u3btr1aRHj9WwYUP897//xY4dO9SJ1JUrVzBz5kw0b94cAPDss8+q61+5cgXe3t54/vnnAUBjnt1nn30GZ2dnBAYGQqVSoXnz5rhx4wZmzZqFDz/8ELVqlb2TNywsDLVq1cLnn3+uTtA3b96MOnXq4OjRo+jQoQNSU1Px2muvqX9vWzyyuq+FhYV6rh8RkV6aNUvemLhRI3lTYj2kV0N77u7uOHTokEZZeHg4OnToACMjIzRv3hyxsbGIiYlRb/3790ePHj0QExNTYu9QdnY2Lly4oE6aGjVqBAcHB41z5eTkICIiAl26dKm8BuoRW1tb9OvXD1u2bMHmzZvRr1+/YnsBvb29cePGDXz33Xfw8PDA0aNH0a5duyLDUDNnztR432JiYtCpUyet4/vmm2/w8ssvw8HBARYWFpg7dy6uXLmifn769OkYO3YsXn31VSxZsgSXLl1SPzd58mR89NFHeOmllzBv3jycO3dO/dyFCxfg7u6u0TP50ksvISMjA9euXStXjL/99hv++usvWFpaqnvv6tWrh6ysLFy6dAn16tWDr68vPDw84OXlpZ5vRURULRw5AhQuYRQSUu4/6HWFoolURkaG+ksTkJO8Y2Ji1F94AQEBePvtt9X1/f39cfnyZUyfPh0XLlzApk2bEBISghn/Lh9vamqKVq1aaWx16tSBpaUlWrVqBeN/L6WcMWMGIiIikJCQgF9++QWvv/460tLS4OPjA0AO6U2dOhWLFi3Ct99+i/Pnz8PX1xfm5uYYNmxY5f1AzM1l71BVb+bmWoU7evRohIaGYsuWLRg9enSJ9UxNTdGrVy98+OGHOHnyJHx9fTHvscmEtra2aNKkicZmpuU4+alTpzB06FB4enri+++/x5kzZ/DBBx8gJydHXWf+/PmIi4tDv379cPjwYbRs2RLffvstAGDs2LH4+++/MXLkSMTGxqJDhw5Yu3YtgOKHdx8dXi6PgoICtG/fvkgC+eeff6p/zzZv3ozIyEh06dIFO3bsQNOmTXHq1Cmtfi5ERDojMxMYO1bu+/sDPXooG89TUHRoLzo6Gj0e+eEVzi/y8fFBaGgokpKSNHoRGjVqhH379mHatGlYt24dnJycsGbNGnh7e5frvNeuXcNbb72FW7duoX79+ujcuTNOnToFV1dXdZ333nsPDx48wPjx43H37l106tQJ4eHhsLS0fMpWl0Kl0quMvE+fPurkxMPDo8yva9myZZkWX9XWiRMn4Orqig8++EBddvny5SL1mjZtiqZNm2LatGl46623sHnzZgwaNAiAvDLR398f/v7+CAgIQHBwMCZNmoSWLVti586dGgnVyZMnYWlpiWeeeaZccbZr1w47duyAnZ0drKysSqzXtm1btG3bFgEBAXB3d8dXX32Fzp07w9jYGPn5+eU6JxGRTpgzB/j7b8DZGVi6VOlonoqiiVT37t3Vf80Xp7irkLp164bTp0+X+RzFHSMsLOyJr1OpVJg/f365JxDXJAYGBrhw4YJ6/3G3b9/GG2+8gdGjR6N169awtLREdHQ0li1bhgEDBmjUTU9PLzL/zdzcvNQE4+bNm+rezEIODg5o0qQJrly5grCwMLz44ov44Ycf1L1NAPDgwQPMnDkTr7/+Oho1aoRr164hKipKnZBPnToVnp6eaNq0Ke7evYvDhw+r5yaNHz8eq1atwqRJkzBx4kTEx8dj3rx5mD59ernmRwFywvsnn3yCAQMGYOHChWjQoAGuXLmCXbt2YebMmcjNzcXGjRvRv39/ODk5IT4+Hn/++ae6l7Zhw4bqXtwGDRrA0tKySif0ExFp5eRJYPVqub9xI1DK//N6QVClSU1NFQBEampqkecePHggfv/9d/HgwQMFItOej4+PGDBgQInPDxgwQPj4+AghhMjKyhLvv/++aNeunbC2thbm5uaiWbNmYs6cOeL+/fvq17i6ugoARbZx48aVeJ5u3boV+5p58+YJIYSYOXOmsLGxERYWFmLIkCFi5cqVwtraWgghRHZ2thg6dKhwdnYWxsbGwsnJSUycOFH9XkycOFE0btxYmJiYiPr164uRI0eKW7duqc999OhR8eKLLwpjY2Ph4OAgZs2aJXJzc8v08wMgvv32W/XjpKQk8fbbbwtbW1thYmIi3NzchJ+fn0hNTRXJycli4MCBwtHRURgbGwtXV1fx4Ycfivz8fPXP19vbW9SpU0cAEJs3by72nPr6u0ZE1dCDB0I0ayYEIISvr9LRlKi07+/HqYQopUuInkpaWhqsra2RmppapGclKysLCQkJaNSoEUxNTRWKkGoC/q4Rkc4ICACWLAEcHIDffwfq1lU6omKV9v39OL26ao+IiIj0VHQ08Mkncj8oSGeTqPJiIkVERESVKycHGD0ayM8Hhg4FHpsnq8+YSBEREVHlWrQIiI0FbG2BNWuUjqZCMZEiIiKiynPu3MNVywMDgfr1lY2ngjGRUhjn+lNl4+8YESkmLw8YNUr+O3Ag8OabSkdU4ZhIKaTwBrn3lbhJMdUohYumFrfWFxFRpVq+HDh9GqhTB/jsM7nwdDWjVzctrk4MDAxQp04dpKSkAJCLT5b3FiNET1JQUICbN2/C3Nwchob8uBNRFbpwAShc1HrVKuDf+9lWN/yfVUEODg4AoE6miCpDrVq14OLiwkSdiKpOfj4wZgyQnQ14egKP3De3umEipSCVSgVHR0fY2dkhNzdX6XComjI2Ni737WuIiJ7K2rVAZCRgaQls2FAth/QKMZHSAQYGBpy/QkRE1cOlS8Ds2XJ/+XJ5Y+JqjH+mEhERUcUoKADGjgUePAB69gT8/JSOqNIxkSIiIqKKsXEjcPQoYG4OBAdX6yG9QkykiIiI6OlduQLMnCn3Fy8G3NyUjaeKMJEiIiKipyME8M47QEYG8NJLwMSJSkdUZZhIERER0dMJDQUOHgRMTICQEKAGXSlcc1pKREREFe/GDWDaNLm/cCHQrJmy8VQxJlJERESkHSEAf38gNRXo0AGYPl3piKocEykiIiLSTlgYsHcvYGQEbN4M1MBbUTGRIiIiovJLSQEmTZL7c+cCrVopG49CmEgRERFR+U2aBNy+DbRpA7z/vtLRKIaJFBEREZXPrl3A118DBgbApk1yaK+GYiJFREREZXfnDjB+vNyfNQto107ZeBTGRIqIiIjKbto04J9/gBYt5NyoGo6JFBEREZXNvn3A1q3yHnqbNgGmpkpHpDgmUkRERPRkqanAuHFyf9o0oHNnZePREUykiIiI6Mneew+4dg1o3Bj4v/9TOhqdwUSKiIiISvfTT8DGjXI/JAQwN1c2Hh3CRIqIiIhKlpEBjB0r98ePB7p1UzYeHcNEioiIiEo2ezaQmAi4uABLligdjc5hIkVERETFO34cWLtW7gcHA5aWysajg5hIERERUVEPHgBjxsj9MWOA3r2VjUdHMZEiIiKioubNAy5eBJycgOXLlY5GZzGRIiIiIk2//gp8+qnc37ABqFNH0XB0GRMpIiIieig7Gxg9GigoAIYPB157TemIdJqiidSxY8fg5eUFJycnqFQq7N69+4mviYiIQPv27WFqago3NzcEBQWVWDcsLAwqlQoDBw4ssc7ixYuhUqkwdepUjXJfX1+oVCqNrTNXcSUiouru44+BuDjAzg5YvVrpaHSeoolUZmYm2rRpg8DAwDLVT0hIQN++fdG1a1ecOXMGs2fPxuTJk7Fz584idS9fvowZM2aga9euJR4vKioKGzduROvWrYt9vk+fPkhKSlJv+/btK1vDiIiI9FFMDLB4sdxftw6wsVE0HH1gqOTJPT094enpWeb6QUFBcHFxwapVqwAALVq0QHR0NJYvXw5vb291vfz8fAwfPhwLFizA8ePHce/evSLHysjIwPDhwxEcHIyPPvqo2POZmJjAwcGhXG0iIiLSS7m5ckgvLw/w9gZef13piPSCXs2RioyMRO/HLr/08PBAdHQ0cnNz1WULFy5E/fr1Mabwss1iTJgwAf369cOrr75aYp2jR4/Czs4OTZs2hZ+fH1JSUkqNLzs7G2lpaRobERGRXvjkE+DMGaBePaCMI0WkcI9UeSUnJ8Pe3l6jzN7eHnl5ebh16xYcHR1x4sQJhISEICYmpsTjhIWF4fTp04iKiiqxjqenJ9544w24uroiISEBc+fORc+ePfHbb7/BxMSk2NcsXrwYCxYs0KptREREiomLAwq/v1avBjgaU2Z6lUgBgEql0ngshFCXp6enY8SIEQgODoatrW2xr7969SqmTJmC8PBwmJqalnieIUOGqPdbtWqFDh06wNXVFT/88AMGDx5c7GsCAgIwffp09eO0tDQ4OzuXuW1ERERVLj9fDunl5AD9+skr9ajM9CqRcnBwQHJyskZZSkoKDA0NYWNjg7i4OCQmJsLLy0v9fEFBAQDA0NAQ8fHxiI2NRUpKCtq3b6+uk5+fj2PHjiEwMBDZ2dkwMDAocm5HR0e4urri4sWLJcZnYmJSYm8VERGRTlq1Sq4bZWUFBAUBj3VYUOn0KpFyd3fH3r17NcrCw8PRoUMHGBkZoXnz5oiNjdV4fs6cOUhPT8fq1avh7OwMOzu7InVGjRqF5s2bY9asWcUmUQBw+/ZtXL16FY6OjhXbKCIiIqVcvAjMmSP3P/0UaNBA2Xj0kKKJVEZGBv766y/144SEBMTExKBevXpwcXFBQEAArl+/jq1btwIA/P39ERgYiOnTp8PPzw+RkZEICQnB9u3bAQCmpqZo1aqVxjnq/Lsaa2G5sbFxkTq1a9eGjY2NujwjIwPz58+Ht7c3HB0dkZiYiNmzZ8PW1haDBg2qlJ8FERFRlSookPfQy8oCXn314X31qFwUTaSio6PRo0cP9ePC+UU+Pj4IDQ1FUlISrly5on6+UaNG2LdvH6ZNm4Z169bByckJa9as0Vj6oCIYGBggNjYWW7duxb179+Do6IgePXpgx44dsOSdr4mIqDpYvx44fhyoXRsIDuaQnpZUonC2NlW4tLQ0WFtbIzU1FVZWVkqHQ0REJCUmAq1aAZmZcqmDCROUjkinlOf7W6/WkSIiIqKnJATg5yeTqK5dgXffVToivcZEioiIqCbZtAn48UfA1BQICQFqMRV4GvzpERER1RTXrwOF6x1+9BHw7LPKxlMNMJEiIiKqCYQA/P2BtDSgY0dg6lSlI6oWmEgRERHVBF99BXz/PWBsLIf3Slg3kcqHiRQREVF1988/wOTJcv/DD4HnnlM2nmqEiRQREVF1N2ECcOcO8MILwHvvKR1NtcJEioiIqDr75htg507A0BDYvBkwMlI6omqFiRQREVF1devWw8U2339f9khRhWIiRUREVF1NnQqkpMg5UYU3J6YKxUSKiIioOtq7F9i2TS64uWkTYGKidETVEhMpIiKi6ubePblmFAD8979y3SiqFEykiIiIqpsZM4AbN+TK5QsWKB1NtcZEioiIqDo5dEjeQ0+lkkN6ZmZKR1StMZEiIiKqLtLTAT8/uT9xIvDyy8rGUwMwkSIiIqouAgKAy5eBhg2BRYuUjqZGYCJFRERUHRw7BqxbJ/c//xywsFA2nhqCiRQREZG+u38fGDNG7vv5Af/5j7Lx1CBMpIiIiPTd3LnAX38BzzwDfPKJ0tHUKEykiIiI9NmpU8DKlXJ/wwbA2lrZeGoYJlJERET6KisLGD0aEAIYORLo10/piGocJlJERET66v/+D7hwAbC3B1atUjqaGomJFBERkT46fRpYulTur18P1KunbDw1FBMpIiIifZOTA4waBeTnA2++CQwapHRENRYTKSIiIn2zdClw7hxgYwOsXat0NDUaEykiIiJ9cv68nBsFyCTKzk7ZeGo4JlJERET6Ii9PXqWXmwv07w8MHap0RDUeEykiIiJ9sXIlEBUl14pavx5QqZSOqMZjIkVERKQP4uPlCuaATKicnJSNhwAwkSIiItJ9BQXyXnrZ2UDv3oCvr9IR0b+YSBEREem6wEDgxAnAwgLYuJFDejqEiRQREZEu+/tvICBA7i9bBri6KhsPaWAiRUREpKuEAMaOBe7fB7p3B8aNUzoiegwTKSIiIl0VHAwcOQKYmQGffw7U4te2ruE7QkREpIuuXgVmzJD7ixYBjRsrGw8Vi4kUERGRrhFCDuOlpwPu7sCkSUpHRCVgIkVERKRrvvgC2L8fMDEBNm0CDAyUjohKoGgidezYMXh5ecHJyQkqlQq7d+9+4msiIiLQvn17mJqaws3NDUFBQSXWDQsLg0qlwsCBA0uss3jxYqhUKkydOlWjXAiB+fPnw8nJCWZmZujevTvi4uLK2DIiIiItJSUBU6bI/fnzgebNFQ2HSqdoIpWZmYk2bdogMDCwTPUTEhLQt29fdO3aFWfOnMHs2bMxefJk7Ny5s0jdy5cvY8aMGejatWuJx4uKisLGjRvRunXrIs8tW7YMK1asQGBgIKKiouDg4IBevXohPT297A0kIiIqDyGA8eOBe/eA9u0fzpEinWWo5Mk9PT3h6elZ5vpBQUFwcXHBqlWrAAAtWrRAdHQ0li9fDm9vb3W9/Px8DB8+HAsWLMDx48dx7969IsfKyMjA8OHDERwcjI8++kjjOSEEVq1ahQ8++ACDBw8GAGzZsgX29vb46quvMK6Ey0+zs7ORnZ2tfpyWllbmthEREeF//wN27wYMDeWQnqGiX9NUBno1RyoyMhK9e/fWKPPw8EB0dDRyc3PVZQsXLkT9+vUxZsyYEo81YcIE9OvXD6+++mqR5xISEpCcnKxxLhMTE3Tr1g0nT54s8ZiLFy+GtbW1enN2di5P84iIqCa7eROYOFHuf/ABUMxoCekevUp1k5OTYW9vr1Fmb2+PvLw83Lp1C46Ojjhx4gRCQkIQExNT4nHCwsJw+vRpREVFlXiewmM/fq7Lly+XeNyAgABMnz5d/TgtLY3JFBERlc2UKTKZatUKmD1b6WiojPQqkQIA1WP3FxJCqMvT09MxYsQIBAcHw9bWttjXX716FVOmTEF4eDhMTU3Lfa7Hyx5lYmICExOTsjSDiIjooT17gO3b5YKbmzcDxsZKR0RlpFeJlIODg7q3qFBKSgoMDQ1hY2ODuLg4JCYmwsvLS/18QUEBAMDQ0BDx8fGIjY1FSkoK2rdvr66Tn5+PY8eOITAwENnZ2XBwcAAge6YcHR01zvV4LxUREdFTuXsX8PeX+zNnAh06KBsPlYteJVLu7u7Yu3evRll4eDg6dOgAIyMjNG/eHLGxsRrPz5kzB+np6Vi9ejWcnZ1hZ2dXpM6oUaPQvHlzzJo1CwYGBmjUqBEcHBxw6NAhtG3bFgCQk5ODiIgILF26tHIbSURENcv06UByMtCsmVzugPSKoolURkYG/vrrL/XjhIQExMTEoF69enBxcUFAQACuX7+OrVu3AgD8/f0RGBiI6dOnw8/PD5GRkQgJCcH27dsBAKampmjVqpXGOerUqQMA6nJjY+MidWrXrg0bGxt1eeG6UosWLcKzzz6LZ599FosWLYK5uTmGDRtWKT8LIiKqgQ4cAEJDAZVKXqX3hCknpHsUTaSio6PRo0cP9ePCido+Pj4IDQ1FUlISrly5on6+UaNG2LdvH6ZNm4Z169bByckJa9as0Vj6oKK89957ePDgAcaPH4+7d++iU6dOCA8Ph6WlZYWfi4iousvL45X8RaSlAe+8I/enTAG6dFE2HtKKShTO1qYKl5aWBmtra6SmpsLKykrpcIiIqtSDB8C2bcDq1cD583L+tIUFYGkp/9Vm//HHen3nlHffBYKCADc34Nw5oHZtpSOif5Xn+5t/HxARUYVKTgY++wxYvx64detheU4OcOeO3CqKmZn2SVlx++bm8sK5SnfkiEyiAODzz5lE6TEmUkREVCHOngVWrpRX8efkyDIXF2DyZGDoUCA/H8jIkFt6+sP9xx8/aT89XR4LkL1eDx7I5Zcqgkolc5ry9oyVtm9qKo+rlpkJjB0r9/39gUemuJD+YSJFRERaKygAfvhBJlBHjjws79IFmDYNGDiw4udGCQFkZ2uXhJW2L4TcCh9XlFq1NBOsD+/OwbCUv3HT1Bmzbi6F4TvlH97kkoW6g4kUERGVW0YGsGWLnP908aIsMzAA3nhDJlAdO1beuVUq2ctjagqUsPZyuQkhe7YqKilLTwfu35fHLiiQ88rT0gB3nMRQrAYAjMzaiIM7tZs/a2RUsUOaFha8GEBb/LEREVGZXb0KBAYCGzcChfeDr1NHXnw2cSKgr3fFUqnk/Chzc6Ci1l3Oz5fJVGGClXk7C82GjEatqwIJ3Xzx+og+6FPO4c2sLHns3Fy5jufduxUTKyAT04oc0qxdu4rmmymMiRQRET3Rr7/K4bv//e/h/KQmTYCpUwEfH/nlSZoMDGRCoV41J2ABcDUecHBAo29XYGzd8h8zN1dOsaqoIc30dLk0BSCTtKwszQsEnpa5ecX1mFlayosLSrlTmyKYSBERUbHy8oBvvwVWrQJOnnxY3qOHHL7r169m9DhUiOho4JNP5H5QEFBXiywKckivTh25VZScnIod0szIkMOZgOyRu38f+OefiolVpSqaZPn6ApMmVczxtcFEioiINKSmyivy164FLl+WZUZGwLBhsgfqhReUjE4P5eQAo0bJrryhQ4EBA5SOSIOxMWBjI7eKIITs2SpvEvak5wqPnZ4ut6QkWda3b8XErS0mUkREBAC4dAlYs0beqaTwi8vWVq4bOX488O/93Km8Fi2SK5LWry+z02pOpZJDcGZmgJ1dxRyzoED2bBWXbLm5Vcw5tMVEioioBhMCOH5czn/as0c+BoDnnpO9T8OHyy9E0tK5c8DHH8v9wMCKu8ywhilcQsLCQvcSeiZSREQ1UE4O8PXXMoE6ffpheZ8+cv5Tr166N6lX7+TlySG9vDxg0CC5NgRVO0ykiIhqkNu3gQ0bgHXrgBs3ZJmpKfD227IHqkULRcOrXpYvl1lq3bryB87MtFpiIkVEVAP88Ye8+m7rVrnwJAA4OgITJgDjxnHEqcJduADMny/3V62SP2yqlphIERFVU0IAP/4oh+/2739Y3ratHL4bMkResUUVLD8fGDNG3sfG0xMYOVLpiKgSMZEiIqpmsrKAbdtkR8j587JMpQL695cJ1CuvcJSpUq1dC0RGyoWONmzgD7uaYyJFRFRN/PMP8NlnwPr1wM2bsqx2bWD0aGDyZLkSOVWyS5eA2bPl/vLl+nvPHCozJlJERHru7FnZ+/TVV/JqPABwcZHJ05gxFbsKNpWioAAYO1ZOQuvZE/DzUzoiqgJMpIiI9FBBAbBvn5z/dPjww3J3dzl8N2gQYMj/4avWxo3A0aPyBnPBwRzSqyH4MSMi0iOZmcCWLcDq1cCff8oyAwPg9ddlAtWpk7Lx1ViXLwMzZ8r9xYuVX26bqgwTKSIiPXDtmlwYe+NG4O5dWWZtDbzzDjBxohzKI4UIId+IjAzgpZfkG0I1BhMpIiId9uuvcvjuf/+TV9UDctL4lCnyrvcWFoqGRwAQGgqEh8uVTTdtkvczoRqDiRQRkY7JywN275YJ1MmTD8u7d5fDd/36yeE80gE3bsg3BQAWLgSaNlU2HqpyTKSIiHREaioQEgKsWSOn3ACAkRHw1lvy9i1t2yoaHj1OCMDfX75xL774MKGiGoWJFBGRwv7+WyZPmzYB6emyzNZWfkePH8+7i+issDBg716Z7W7axMskayi+60REChAC+PlnOXy3Z49czgAAWraUvU8jRgBmZoqGSKVJSQEmTZL7c+cCrVopGw8phokUEVEVysmRE8dXrgR+++1huYeHHBnq3ZvLD+mFSZOA27eBNm2A999XOhpSEBMpIqIqcOeOvO1aYKCcnwzIi7xGjpQ9UC1bKhoelceuXcDXX8sZ/5s2yaE9qrGYSBERVaL4eHn7li1b5J1DAMDBAZgwQc6BsrVVNDwqrzt35MQ1AJg1C2jXTtl4SHFMpIiIKpgQwE8/yeG7ffselr/wghy+GzIEMDFRLDx6GtOmybtDt2gh50ZRjcdEioiogmRlAdu2yR6o8+dlmUoF9O8vv39feYXzn/Tavn3A1q3yTdy0SY7NUo3HRIqI6Cn98w/w2WfA+vXAzZuyrHZtYPRoYPJkuRI56bnUVGDcOLk/bRrQubOy8ZDOYCJFRKSlc+fk8N1XX8mr8QB5z7tJk4CxY4E6dRQNjyrSzJnyhodNmgD/939KR0M6hIkUEVE5FBTIEZ6VK4HDhx+Wd+4sOyoGD+a6jNXOjz8CwcFyPyQEMDdXNh7SKfy4ExGVQWamvPJu9Wrgzz9lmYEB4O3NkZ5qLSMD8POT+xMmyIluRI9gIkVEVIpr1+TaTxs3AnfvyjJra/ndOmmSHMqjamz2bCAxEXB1BRYvVjoa0kFMpIiIihEVJYfv/vc/IC9PljVuDEyZAvj6ApaWioZHVeH4cWDtWrkfHMw3nYpVqzyVly1bhgeFK8oBOHbsGLKzs9WP09PTMb5wobIyOHbsGLy8vODk5ASVSoXdu3c/8TURERFo3749TE1N4ebmhqCgoBLrhoWFQaVSYeDAgRrl69evR+vWrWFlZQUrKyu4u7tj//79GnV8fX2hUqk0ts7suyeq1vLzgZ07gZdfBjp2BLZvl0lUt27A7t1ycc1Jk/h9WiM8eACMGSP3x4wBevVSNh7SWeVKpAICApBeeGtyAK+99hquX7+ufnz//n1s2LChzMfLzMxEmzZtEBgYWKb6CQkJ6Nu3L7p27YozZ85g9uzZmDx5Mnbu3Fmk7uXLlzFjxgx07dq1yHMNGjTAkiVLEB0djejoaPTs2RMDBgxAXFycRr0+ffogKSlJve17dGU9Iqo20tJk71OTJsDrrwMnTsi7fowcCZw+DRw9CgwYIOdEUQ0xbx5w8SLg5AQsX650NKTDyjW0J4Qo9XF5eXp6wtPTs8z1g4KC4OLiglWrVgEAWrRogejoaCxfvhze3t7qevn5+Rg+fDgWLFiA48eP4969exrH8fLy0nj88ccfY/369Th16hSee+45dbmJiQkcHBzKHF92drZGD11aWlqZX0tEVS8hAVizRl6IVfg3oo2NvHXLhAmAo6Oy8ZFCfv0V+PRTub9hA9exoFKVq0dKaZGRkejdu7dGmYeHB6Kjo5Gbm6suW7hwIerXr48xhd2ypcjPz0dYWBgyMzPh7u6u8dzRo0dhZ2eHpk2bws/PDykpKaUea/HixbC2tlZvzs7O5WgdEVUFIYCff5ZX2zVpIlchT0+Xd/zYsAG4ehX46CMmUTVWdrZcSbWgABg+HHjtNaUjIh2nV5PNk5OTYW9vr1Fmb2+PvLw83Lp1C46Ojjhx4gRCQkIQExNT6rFiY2Ph7u6OrKwsWFhY4Ntvv0XLR26/7unpiTfeeAOurq5ISEjA3Llz0bNnT/z2228wKeEmWQEBAZg+fbr6cVpaGpMpIh2Rmysnjq9cCURHPyz38ACmTpX/8vYthI8/BuLiADs7udYF0ROUO5H6/PPPYWFhAQDIy8tDaGgobP+9ffmj86cqi+qx/+kKhxdVKhXS09MxYsQIBAcHq2MqSbNmzRATE4N79+5h586d8PHxQUREhDqZGjJkiLpuq1at0KFDB7i6uuKHH37A4MGDiz2miYlJiUkWESnjzh25dEFgIFA4pdPUVM5/mjIFeGQ0n2q6mJiHSxysWyfHeYmeoFyJlIuLC4ILV3cF4ODggC+++KJIncri4OCA5ORkjbKUlBQYGhrCxsYGcXFxSExM1JgDVVBQAAAwNDREfHw8GjduDAAwNjZGk39vgNWhQwdERUVh9erVJU6Wd3R0hKurKy5evFgZTSOiChYfLzsUtmwB7t+XZQ4Ocu7TuHFA/frKxkc6JjdXDunl5clx39dfVzoi0hPlSqQSExMrKYyycXd3x969ezXKwsPD0aFDBxgZGaF58+aIjY3VeH7OnDlIT0/H6tWrSx1mE0JoTBR/3O3bt3H16lU4cuIEkc4SQt62ZeVK4IcfHpa/8IJcfXzIEICdxlSsZcuAM2eAevVkbxRRGSk6RyojIwN//fWX+nFCQgJiYmJQr149uLi4ICAgANevX8fWrVsBAP7+/ggMDMT06dPh5+eHyMhIhISEYPv27QAAU1NTtGrVSuMcdf692uLR8tmzZ8PT0xPOzs5IT09HWFgYjh49igMHDqjjmj9/Pry9veHo6IjExETMnj0btra2GDRoUGX+SIhIC1lZ8sbBq1YBhX9LqVSAl5dMoLp14/wnKkVcHLBwodxfswZ4bC4uUalEOZw6dUrs27dPo2zLli2iYcOGon79+sLPz09kZWWV+XhHjhwRAIpsPj4+QgghfHx8RLdu3TRec/ToUdG2bVthbGwsGjZsKNavX1/qOXx8fMSAAQM0ykaPHi1cXV2FsbGxqF+/vvjPf/4jwsPD1c/fv39f9O7dW9SvX18YGRkJFxcX4ePjI65cuVLmtgkhRGpqqgAgUlNTy/U6Iiqb5GQh5s0Tws5OCNkfJUTt2kJMnCjEn38qHR3phbw8ITp2lL88r70mREGB0hGRDijP97dKiLIvBuXp6Ynu3btj1qxZAOSVb+3atYOvry9atGiBTz75BOPGjcP8+fMrPOHTR2lpabC2tkZqaiqsrKyUDoeo2oiNlcN327YBOTmyzNlZrjo+dixQt66y8ZEe+fRTYMYMwMoK+P134JlnlI6IdEB5vr/LNbQXExOD//u//1M/DgsLQ6dOndQT0J2dnTFv3jwmUkRU4QoKgP37ZQL1008Pyzt1ksN3gwfL1ciJyuziRWDOHLm/YgWTKNJKuRKpu3fvaqzjFBERgT59+qgfv/jii7h69WrFRUdENV5mJrB1q7wCLz5eltWqJS+smjYNeGwdXaKyKSiQ99DLygJefVVesUekhXKtbG5vb4+EhAQAQE5ODk6fPq2xGnh6ejqM+CchEVWA69eBgAA5ZDd+vEyirKyA//4X+Ptv4OuvmUTRU1i/Hjh+HKhdGwgO5tUIpLVy9Uj16dMH77//PpYuXYrdu3fD3Nxc46bA586dU6/TRESkjehoOXz39ddySR8AaNxYLp7p6wtYWioaHlUHiYnAv3N9sXQp0LChktGQnitXIvXRRx9h8ODB6NatGywsLBAaGgpjY2P185s2bSpyLzwioifJzwf27JEJ1M8/Pyzv1k0O3732GmBgoFx8VI0IAfj5yTHjrl2Bd99VOiLSc+W6aq9QamoqLCwsYPDY/2x37tyBpaUlh/f+xav2iEqXlgZs2iSX7vl31gCMjIChQ+X979q1UzQ8qo5CQuSlnaamwLlzwLPPKh0R6aBKu2pvdBkn423atKk8hyWiGiYxUSZPn38OFN6i08YG8PeX86GcnBQNj6qr69eBwhvLf/QRkyiqEOVKpEJDQ+Hq6oq2bdtCi44sIqrBhABOnpTDd99+Ky+aAoAWLWTv04gRgLm5oiFSdSaEzNTT0uSaGVOnKh0RVRPlSqT8/f0RFhaGv//+G6NHj8aIESNQr169yoqNiKqB3Fzgm29kAhUV9bC8d285/6l3b7mcAVGl2rYN+P57wNhYjidz0h1VkHL99/XZZ58hKSkJs2bNwt69e+Hs7Iw333wTBw8eZA8VEWm4e1deEOXmBgwbJpMoExM5PeX8eeDgQaBPHyZRVAWSk4HJk+X+vHlAy5bKxkPVilaTzQtdvnwZoaGh2Lp1K3Jzc/H777/DwsKiIuPTa5xsTjXRn3/KxTNDQ4H792WZvT0wYYIcWalfX9HwqCZ6/XVg506gbVvgl1+4BD49UaVNNn+cSqWCSqWCEAIFhRMeiKjGEQI4fBhYtUqOnhRq00YO3w0dKnujiKrcN9/IJMrQUA7pMYmiClbuTvXs7Gxs374dvXr1QrNmzRAbG4vAwEBcuXKFvVFENUx2NrB5M/DCC/IuG99/LxeI9vKSidWZM4CPD5MoUsitW7IrFJDL5L/wgqLhUPVUrh6p8ePHIywsDC4uLhg1ahTCwsJgY2NTWbERkY5KSZF32PjsM7kPyCvuRo2SK5DzqnLSCVOnyl/Q554DPvhA6WiomirXHKlatWrBxcUFbdu2haqU+xLt2rWrQoLTd5wjRdXN+fPy6rtt22RvFAA0aABMmiQXi65bV9n4iNT27gX695dXM0RGAh07Kh0R6ZFKmyP19ttvl5pAEVH1U1AAHDggE6gff3xY3rGjnP/k7c1pJ6Rj7t2TVzYA8i7XTKKoEpV7QU4iqhnu3we2bpVX4P3xhyyrVUsmTtOmAe7uysZHVKIZM4AbN+QY84IFSkdD1dxTXbVHRNXP9evAunXAhg3AnTuyzMpKDt1NmgS4uiobH1GpDh2S99NTqeRVemZmSkdE1RwTKSICIHugJk2SvVB5ebLMzU1OHh81CrC0VDY+oidKT5cZPwBMnAi8/LKy8VCNwESKiJCZKZcsOHJEPn7lFTl85+XFO2mQHgkIAC5fBho1AhYtUjoaqiGYSBHVcOnpQL9+wPHjstdp926gZ0+loyIqp4gIOSYNAMHBANc1pCrCRIqoBktLAzw9gZMn5TyogweBzp2VjoqonO7fB8aMkfvvvAP85z/KxkM1ChMpohrq3j150+BffgHq1JFzdDt0UDoqIi3MnQtcuiQXNVu2TOloqIZhIkVUA925A3h4ANHRQL16Molq107pqIi0cOqUXOQMkJeaWlsrGw/VOEykiGqY27flffFiYgBbW7nIZps2SkdFpIWsLGD0aHnX7LffBvr2VToiqoGYSBHVIDdvyiTq3DnAzg746SegVSuloyLS0v/9H3DhAmBv/7BXiqiKMZEiqiH++UfOwY2LAxwcgMOHgRYtlI6KSEunTwNLl8r99evlGDWRAphIEdUASUlySYM//gCcnGQS1ayZ0lERaSknR64Sm58PvPkmMGiQ0hFRDcZEiqiau35dJlF//ikvajpyBGjSROmoiJ7C0qVyfNrGBli7VuloqIZjIkVUjV29CvToIa8Md3GRSZSbm9JRET2F8+fl3ChAJlF2dsrGQzVeLaUDIKLKkZgIdOsmk6hGjeTCz0yiSK/l5cmr9HJzgf79gaFDlY6IiD1SRNXR33/LnqgrV4DGjWVPlLOz0lERPaWVK4GoKLmC7Pr1gEqldERE7JEiqm7++kv2RF25AjRtKnuimESR3ouPlyuYAzKhcnJSNh6ifzGRIqpG4uNlEnXtGtC8OXD0KPDMM0pHRfSU8vPlkF52tlyS38dH6YiI1JhIEVUTv/8OdO8O3LgBPPecTKIcHZWOiqgCrFsn76xtYQFs3MghPdIpTKSIqoHz5+WcqORkoHVrOSfK3l7pqIgqwN9/AwEBcv+TT+Tlp0Q6RNFE6tixY/Dy8oKTkxNUKhV27979xNdERESgffv2MDU1hZubG4KCgkqsGxYWBpVKhYEDB2qUr1+/Hq1bt4aVlRWsrKzg7u6O/fv3a9QRQmD+/PlwcnKCmZkZunfvjri4OG2aSVSpzp6VSVRKCtC2rVxss359paMiqgBCAGPHAvfvy+7Wd95ROiKiIhRNpDIzM9GmTRsEBgaWqX5CQgL69u2Lrl274syZM5g9ezYmT56MnTt3Fql7+fJlzJgxA127di3yXIMGDbBkyRJER0cjOjoaPXv2xIABAzQSpWXLlmHFihUIDAxEVFQUHBwc0KtXL6Snp2vfYKIKdvq0XGzz1i2gQwd57zwbG6WjIqogwcGye9XMDPj8c6AWB1FIBwkdAUB8++23pdZ57733RPPmzTXKxo0bJzp37qxRlpeXJ1566SXx+eefCx8fHzFgwIAnnr9u3bri888/F0IIUVBQIBwcHMSSJUvUz2dlZQlra2sRFBRU4jGysrJEamqqert69aoAIFJTU594fqLyiooSok4dIQAhOnUS4u5dpSMiqkBXrghhaSl/wVeuVDoaqmFSU1PL/P2tV+l9ZGQkevfurVHm4eGB6Oho5ObmqssWLlyI+vXrY8yYMU88Zn5+PsLCwpCZmQl3d3cAsucrOTlZ41wmJibo1q0bTp48WeKxFi9eDGtra/XmzGvOqZL88gvw6qvAvXtAly5AeLhcWoeoWhACGDcOSE8H3N2BSZOUjoioRHq1IGdycjLsH5tBa29vj7y8PNy6dQuOjo44ceIEQkJCEBMTU+qxYmNj4e7ujqysLFhYWODbb79Fy5Yt1ecpPPbj57p8+XKJxwwICMD06dPVj9PS0ionmQoLk/eaatxYc2vSRN5MzcCg4s9JOuPECcDTU37HdO0K/PADYGmpdFREFeiLL4D9+wETE2DTJv6fRjpNrxIpAFA9dtmrEEJdnp6ejhEjRiA4OBi2tralHqdZs2aIiYnBvXv3sHPnTvj4+CAiIkKdTJV0rsfLHmViYgITE5PyNqn8zp8HYmLk9jgjI3k/kOKSrEaNAFPTyo+PKs2xY0DfvkBmppx7+/33QO3aSkdFVIGSkoApU+T+/PlyQTQiHaZXiZSDg4O6t6hQSkoKDA0NYWNjg7i4OCQmJsLLy0v9fEFBAQDA0NAQ8fHxaNy4MQDA2NgYTZo0AQB06NABUVFRWL16NTZs2AAHBwcAsmfK8ZGFeFJSUor0Uili3Digc2d5E7VHt4QEICcH+PNPuT1OpZKrMxaXZDVuzLEhHXfkCPDaa/ICpldfBfbsAczNlY6KqAIJAYwfL8es27cHZsxQOiKiJ9KrRMrd3R179+7VKAsPD0eHDh1gZGSE5s2bIzY2VuP5OXPmID09HatXry51mE0IgezsbABAo0aN4ODggEOHDqFt27YAgJycHERERGDp0qUV3CotODsXf8+P/Hy5pPXjCdalS/K+Ienp8vlr1+R9Qx5Xr17JSZajIxfBU9CPP8p7tD54APTpA+zaJS9kIqpW/vc/YPdu2bO+aRNgqFdfUVRDKfpbmpGRgb/++kv9OCEhATExMahXrx5cXFwQEBCA69evY+vWrQAAf39/BAYGYvr06fDz80NkZCRCQkKwfft2AICpqSlatWqlcY46//ayPFo+e/ZseHp6wtnZGenp6QgLC8PRo0dx4MABAHJIb+rUqVi0aBGeffZZPPvss1i0aBHMzc0xbNiwyvyRPB0DA8DVVW49e2o+J4S8Rr64JOvSJbmS4507couKKnpsMzPAza1ogtW4sTyfkVHVtLEGOnAAGDhQ3h2jXz/gm284QkvV0M2bwIQJcv+DD+TKskR6QNFEKjo6Gj169FA/Lpyo7ePjg9DQUCQlJeHKlSvq5xs1aoR9+/Zh2rRpWLduHZycnLBmzRp4e3uX67z//PMPRo4ciaSkJFhbW6N169Y4cOAAevXqpa7z3nvv4cGDBxg/fjzu3r2LTp06ITw8HJb6OqtXpZKrNNavL4cFH5eRIVcQLi7JunxZdoXExcntcQYGcrXh4nqy3NzkbR1IKz/8AAweLEdsBwwAduyQ82+Jqp3Jk+Ufe88//3AlcyI9oBKFs7WpwqWlpcHa2hqpqamwsrJSOhzt5ebKZKq44cK//5ZJVmns7YtPsho3BmxtOWRYgj17gDfekD/+wYOB7dsBY2OloyKqBLt3A4MGyT/KfvlFzo8iUlB5vr85AE1PZmQkk59/J+drEEJeZVNcknXpkhwq/OcfuRW3BpelZfEJVuPGNXoph507gaFDgbw84M03gS+/5OgpVVN37wLvviv3Z85kEkV6hz1Slaja9Eg9jXv3Sk6yrl0r/bXGxkDDhsUnWdV4KYevvwaGDZPXDgwbBmzZwjm3VI2NGgWEhgLNmsklXarp55r0S3m+v5lIVSImUk+QlSWXbHg8wSpcyuGR1eqLeHQph8eTLD1eyuGrr4CRI4GCAuDtt7kWIVVzBw7I1WVVKuDnn+Uy/UQ6gImUjmAi9RQeX8rh0STr0iW5lENpCpdyKC7J0tGlHLZulX+cFxQAo0cDGzcyiaJqLC0NaNUKuHoVmDoVWLlS6YiI1JhI6QgmUpXk8aUcHk+y/vmn9NebmRVNrhReymHTJmDsWNm0d94B1q/nje6pmnv3XSAoSF7Ze+4cl+gnncJESkcwkVLIo0s5PJ5kXb4su3xKUriUQ3E9WY0bV8p/9hs3ysXqAbmo89q1TKKomjty5OFad4cPA48sg0OkC5hI6QgmUjro0aUcHk+yyrqUQ0lJlhZLOaxbB0ycKPenTJGjGzo46khUcTIz5WKbf/8N+PvL7lciHcNESkcwkdIzjy7l8HiSVbiUQ2msrEoeMixmKYfVq+XUEAD473+BTz5hEkU1wLRpwKpV8jZX58/Lzw2RjmEipSOYSFUzd++WfIudsizl0KiROrE6fKUxVuxpjEtojNdnNMLCZaZMoqj6O3kSePll+UfLgQOAh4fSEREVi4mUjmAiVYM8eKC5lMOj2xOWchAqFVQNGpTcm6WnSzkQaXjwAGjbFoiPl5enbtqkdEREJWIipSOYSBEAuZTD1avApUv4fs0lxH13CY1xCS85XIJjZhmWcrCxKT7BatIEcHDgeCDph/ffB5YulcuPxMUBdesqHRFRiXiLGCJdYmAA4doQ8zc3xMLv/gMA+Phj4PXZkEMcN2+WPGT4zz/A7dty+/XXosc2N5eXjxeXZLm48L4yVLGEkH8Y5OXJfx/dL+3fxEQ5CRCQSx4wiaJqhD1SlYg9UgTI7545c4BFi+TjZcvkLcXKJD394VIOj99m58qVJy/l4OpafJLl5sZ1ex5XUFD2xKAq/tWFGB7/t7Tft7J46y25fD+RjuPQno5gIkVCALNmPfxjfMUKedFShcjJebiUw+NJ1t9/y1vwlMbBofghQyur6psIlBYj/yt8OrVqyZtCGhgU/2/DhsDu3XKZECIdx0RKRzCRqtmEkMsaFN75Yu3ah2tGVbqCgodLORR3w+i7d6sokGqipOSgOvxbUcfgXD2qRjhHikhhQsgFNteulY/Xr5drD1aZWrXkTZ2feQZ45ZWizxe3lENhT9b9+8p/uetS0lCrFpMEIioREymiClZQAEyYIOfUqlTyFjBjxyod1WPq1gU6dJAbERFpjYkUUQUqKJD3zfv8c5lEbdoE+PoqHRUREVUWJlJEFSQ/X/Y8hYbK0aAtW4ARI5SOioiIKhMTKaIKkJcnF2v+8ks5rebLL4GhQ5WOioiIKhsTKaKnlJcHjBwJhIXJ+cnbtwOvv650VEREVBWYSBE9hdxcYNgw4Jtv5CLiX38NDByodFRERFRVmEgRaSknRw7fffstYGwskykvL6WjIiKiqsREikgL2dnAG28Ae/cCJiYymfL0VDoqIiKqakykiMopKwsYPBjYvx8wNQX27AF691Y6KiIiUgITKaJyePBAzoEKDwfMzGSP1H/+o3RURESkFCZSRGWUmQn07w8cPgzUrg388APQrZvSURERkZKYSBGVQUYG0K8fcOwYYGEhh/VeflnpqIiISGlMpIieID1dTiQ/cQKwsgIOHADc3ZWOioiIdAETKaJSpKYCffoAp04BdeoABw8CHTsqHRUREekKJlJEJbh7F/DwAKKigLp1gR9/BNq1UzoqIiLSJUykiIpx5w7Qqxdw+jRgYyOTqBdeUDoqIiLSNUykiB5z6xbw6qvA2bNA/frATz8Bzz+vdFRERKSLmEgRPSIlRSZRsbGAvb1c6qBlS6WjIiIiXcVEiuhfyclycc3ffwccHWUS1by50lEREZEuYyJFBODGDaBnTyA+HnjmGeDIEeDZZ5WOioiIdF0tJU9+7NgxeHl5wcnJCSqVCrt3737iayIiItC+fXuYmprCzc0NQUFBJdYNCwuDSqXCwIEDNcoXL16MF198EZaWlrCzs8PAgQMRHx+vUcfX1xcqlUpj69y5szbNJB137RrQvbtMolxcgIgIJlFERFQ2iiZSmZmZaNOmDQIDA8tUPyEhAX379kXXrl1x5swZzJ49G5MnT8bOnTuL1L18+TJmzJiBrl27FnkuIiICEyZMwKlTp3Do0CHk5eWhd+/eyMzM1KjXp08fJCUlqbd9+/Zp11DSWZcvy9u8XLwINGwok6jGjZWOioiI9IWiQ3uenp7w9PQsc/2goCC4uLhg1apVAIAWLVogOjoay5cvh7e3t7pefn4+hg8fjgULFuD48eO4d++exnEOHDig8Xjz5s2ws7PDb7/9hldeeUVdbmJiAgcHh/I3jPRCQgLQo4dMptzc5HCei4vSURERkT5RtEeqvCIjI9G7d2+NMg8PD0RHRyM3N1ddtnDhQtSvXx9jxowp03FTU1MBAPXq1dMoP3r0KOzs7NC0aVP4+fkhJSWl1ONkZ2cjLS1NYyPddOmS7Im6fFkO40VEMIkiIqLy06tEKjk5Gfb29hpl9vb2yMvLw61btwAAJ06cQEhICIKDg8t0TCEEpk+fjpdffhmtWrVSl3t6emLbtm04fPgwPv30U0RFRaFnz57Izs4u8ViLFy+GtbW1enN2dtailVTZ/vxTJlFXr8qr8o4eBRo0UDoqIiLSR3p31Z5KpdJ4LIRQl6enp2PEiBEIDg6Gra1tmY43ceJEnDt3Dj///LNG+ZAhQ9T7rVq1QocOHeDq6ooffvgBgwcPLvZYAQEBmD59uvpxWloakykd88cf8uq8pCS5PtThw3K9KCIiIm3oVSLl4OCA5ORkjbKUlBQYGhrCxsYGcXFxSExMhJeXl/r5goICAIChoSHi4+PR+JGZxJMmTcJ3332HY8eOocETuiQcHR3h6uqKixcvlljHxMQEJiYm2jSNqkBcnFwn6p9/5ErlP/4I2NkpHRUREekzvUqk3N3dsXfvXo2y8PBwdOjQAUZGRmjevDliY2M1np8zZw7S09OxevVqde+QEAKTJk3Ct99+i6NHj6JRo0ZPPPft27dx9epVODo6VlyDqMqcOydXLL95U94z79AhoIydlkRERCVSNJHKyMjAX3/9pX6ckJCAmJgY1KtXDy4uLggICMD169exdetWAIC/vz8CAwMxffp0+Pn5ITIyEiEhIdi+fTsAwNTUVGOeEwDUqVMHADTKJ0yYgK+++gp79uyBpaWlupfL2toaZmZmyMjIwPz58+Ht7Q1HR0ckJiZi9uzZsLW1xaBBgyrzR0KVICZGJlG3bwPt2wPh4cBj1xUQERFpRdFEKjo6Gj169FA/Lpxf5OPjg9DQUCQlJeHKlSvq5xs1aoR9+/Zh2rRpWLduHZycnLBmzRqNpQ/KYv369QCA7t27a5Rv3rwZvr6+MDAwQGxsLLZu3Yp79+7B0dERPXr0wI4dO2Bpaalla0kJv/0G9OoF3L0LdOwIHDwI/JtbExERPTWVKJytTRUuLS0N1tbWSE1NhZWVldLh1Di//gr07g2kpgLu7sD+/YC1tdJRERGRrivP97deLX9AVFaRkbInKjUVePll2RPFJIqIiCoaEymqdn7+WfZEpaXJ9aL27wc4IktERJWBiRRVKxERQJ8+QEaGXC/qhx8ACwuloyIiouqKiRRVGz/9BHh6ApmZskfq+++B2rWVjoqIiKozJlJULYSHA6+9Bjx4APTtC+zZA5iZKR0VERFVd0ykSO/t2wf07w9kZQFeXsCuXYCpqdJRERFRTcBEivTa3r3AoEFAdrb895tvAN6lh4iIqgoTKdJb334LeHsDOTnAG28AO3YAxsZKR0VERDUJEynSS998A7z5JpCbCwwdCnz1FWBkpHRURERU0zCRIr0TFiaTp7w8YMQI4IsvAEO9uv02ERFVF0ykSK98+SUwfDiQnw/4+gKhoUyiiIhIOUykSG+EhgJvvw0UFAB+fkBICGBgoHRURERUkzGRIr3w+efA6NGAEMC77wJBQUAt/vYSEZHC+FVEOm/9etkDJQQwaRKwbh2TKCIi0g38OiKdtnYtMH683J8+HVi9GlCplI2JiIioEBMp0lkrVwKTJ8v9994Dli9nEkVERLqFiRTppGXLZA8UAHzwAbBkCZMoIiLSPUykSOcsWgTMmiX3588H/u//mEQREZFu4go8pFMWLgTmzZP7H30ke6OIiIh0FRMp0glCAB9+KJMnQA7lFfZKERER6SomUqQ4IYCAAGDpUvn4008fzo8iIiLSZUykSFFCADNnyuQJkMsbFF6pR0REpOuYSJFihACmTZPJEyAX2ixcM4qIiEgfMJEiRRQUyFXKP/tMPt6wAXjnHWVjIiIiKi8mUlTlCgrk/fI2bpTLGoSEAKNGKR0VERFR+TGRoiqVny97njZtkvfLCw0FRo5UOioiIiLtMJGiKpOfL3uevvhCJlFffgm89ZbSUREREWmPiRRVibw8wMcH+OorwMAA2L4deOMNpaMiIiJ6OkykqNLl5gIjRgBffw0YGgI7dgCDBysdFRER0dNjIkWVKidHDt/t2gUYGQHffAP07690VERERBWDiRRVmuxs4M03ge++A4yNZTLVr5/SUREREVUcJlJUKbKyAG9vYN8+wNQU2L0b8PBQOioiIqKKxUSKKtyDB8CgQcDBg4CZGbB3L/Cf/ygdFRERUcVjIkUV6v59YMAA4McfAXNz4IcfgO7dlY6KiIiocjCRogqTmQm89hpw9ChgYSGH9bp2VToqIiKiysNEiipEerqcSH78OGBpCRw4AHTponRURERElauWkic/duwYvLy84OTkBJVKhd27dz/xNREREWjfvj1MTU3h5uaGoKCgEuuGhYVBpVJh4MCBGuWLFy/Giy++CEtLS9jZ2WHgwIGIj4/XqCOEwPz58+Hk5AQzMzN0794dcXFx2jSz2ktLA/r0kUmUtTVw6BCTKCIiqhkUTaQyMzPRpk0bBAYGlql+QkIC+vbti65du+LMmTOYPXs2Jk+ejJ07dxape/nyZcyYMQNdixlbioiIwIQJE3Dq1CkcOnQIeXl56N27NzIzM9V1li1bhhUrViAwMBBRUVFwcHBAr169kJ6ern2Dq6F794DevYGTJ4G6deXcqE6dlI6KiIioaqiEEELpIABApVLh22+/LdJ79KhZs2bhu+++w4ULF9Rl/v7+OHv2LCIjI9Vl+fn56NatG0aNGoXjx4/j3r17pfZ23bx5E3Z2doiIiMArr7wCIQScnJwwdepUzJo1CwCQnZ0Ne3t7LF26FOPGjStTm9LS0mBtbY3U1FRYWVmV6TX65M4duaRBdDRgYyN7otq2VToqIiKip1Oe729Fe6TKKzIyEr1799Yo8/DwQHR0NHJzc9VlCxcuRP369TFmzJgyHTc1NRUAUK9ePQCy5ys5OVnjXCYmJujWrRtOnjxZ4nGys7ORlpamsVVXt2/LJQ2iowFbW+DwYSZRRERU8+hVIpWcnAx7e3uNMnt7e+Tl5eHWrVsAgBMnTiAkJATBwcFlOqYQAtOnT8fLL7+MVq1aqc9TeOzHz1X4XHEWL14Ma2tr9ebs7FzmtumTmzeBHj2AmBjAzg44cgRo3VrpqIiIiKqeXiVSgBwCfFThyKRKpUJ6ejpGjBiB4OBg2Nralul4EydOxLlz57B9+/YynevxskcFBAQgNTVVvV29erVMMeiTf/6RSVRsLODgIJc6+Df/JCIiqnH0avkDBweHIj1CKSkpMDQ0hI2NDeLi4pCYmAgvLy/18wUFBQAAQ0NDxMfHo3HjxurnJk2ahO+++w7Hjh1DgwYNNM4DyJ4pR0dHjXM93kv1KBMTE5iYmDxdI3VYUhLQsyfwxx/AM8/I4bymTZWOioiISDl61SPl7u6OQ4cOaZSFh4ejQ4cOMDIyQvPmzREbG4uYmBj11r9/f/To0QMxMTHqoTYhBCZOnIhdu3bh8OHDaNSokcYxGzVqBAcHB41z5eTkICIiAl1q6HX916/LFcr/+ANwdgYiIphEERERKdojlZGRgb/++kv9OCEhATExMahXrx5cXFwQEBCA69evY+vWrQDkFXqBgYGYPn06/Pz8EBkZiZCQEPWwnKmpqXqeU6E6deoAgEb5hAkT8NVXX2HPnj2wtLRU93JZW1vDzMwMKpUKU6dOxaJFi/Dss8/i2WefxaJFi2Bubo5hw4ZV5o9EJ129KofzLl0CXF3lnKjHck8iIqIaSdFEKjo6Gj169FA/nj59OgDAx8cHoaGhSEpKwpUrV9TPN2rUCPv27cO0adOwbt06ODk5Yc2aNfD29i7XedevXw8A6P7YTeA2b94MX19fAMB7772HBw8eYPz48bh79y46deqE8PBwWFpaatFS/ZWYKIfzEhIANzc5nOfqqnRUREREukFn1pGqjvR9Ham//5Y9UVeuAE2ayCSqml6ISEREpFZt15GiqvPXX0C3bjKJatpUXp3HJIqIiEgTEykqIj5eJlHXrgEtWsiJ5c88o3RUREREuoeJFGn4/Xd5dd6NG3J9qKNH5XpRREREVBQTKVI7f14mUcnJQJs28uo8OzuloyIiItJdTKQIAHD2rJxYfvOmvGfeTz/Je+gRERFRyZhIEU6flksc3LoFvPiiTKJsbJSOioiISPcxkarhoqKA//wHuHMH6NwZOHQIqFtX6aiIiIj0AxOpGuzUKeDVV4F794CXXgIOHgSsrZWOioiISH8wkaqhTpwAevcG0tKAV14BDhwA9HDNUCIiIkUxkaqBjh0DPDyA9HQ5N2rfPsDCQumoiIiI9A8TqRrmyBHA0xPIzAR69QL27gVq11Y6KiIiIv3ERKoG+fFHoF8/4P59oE8f4LvvAHNzpaMiIiLSX0ykaogDB4DXXgMePJD/7t4NmJoqHRUREZF+YyJVA3z/PTBgAJCdLf/duRMwMVE6KiIiIv3HRKqa27MHGDwYyMkBvL2B//0PMDZWOioiIqLqgYlUNbZzJ/D660BuLjBkCLB9O2BkpHRURERE1QcTqWpqxw6ZPOXlAcOHA19+ySSKiIioojGRqoa++goYNgzIzwd8fIAtWwBDQ6WjIiIiqn6YSFUzW7cCI0cCBQXAmDHApk2AgYHSUREREVVPTKSqkU2bAF9fmUT5+wMbNwK1+A4TERFVGn7NVhMbN8oeKCGAiROBzz5jEkVERFTZ+FVbDaxbB4wbJ/enTgXWrAFUKkVDIiIiqhGYSOm51atlDxQAzJwJrFjBJIqIiKiqMJHSY8uXyx4oAJg9G1i6lEkUERFRVWIipaeWLJE9UADw4YfARx8xiSIiIqpqXF1IDy1aBHzwgdxfuBCYO1fZeIiIiGoqJlJ6qHFjeUXexx8D77+vdDREREQ1FxMpPTRkCNCmDdC8udKREBER1WycI6WnmEQREREpj4kUERERkZaYSBERERFpiYkUERERkZaYSBERERFpiYkUERERkZaYSBERERFpiYkUERERkZYUTaSOHTsGLy8vODk5QaVSYffu3U98TUREBNq3bw9TU1O4ubkhKCioxLphYWFQqVQYOHBguc/r6+sLlUqlsXXu3LmcLSQiIqLqTNFEKjMzE23atEFgYGCZ6ickJKBv377o2rUrzpw5g9mzZ2Py5MnYuXNnkbqXL1/GjBkz0LVrV63P26dPHyQlJam3ffv2la1hREREVCMoeosYT09PeHp6lrl+UFAQXFxcsGrVKgBAixYtEB0djeXLl8Pb21tdLz8/H8OHD8eCBQtw/Phx3Lt3T6vzmpiYwMHBoczxERERUc2iV3OkIiMj0bt3b40yDw8PREdHIzc3V122cOFC1K9fH2PGjHmq8x09ehR2dnZo2rQp/Pz8kJKSUmr97OxspKWlaWxERERUfelVIpWcnAx7e3uNMnt7e+Tl5eHWrVsAgBMnTiAkJATBwcFPdS5PT09s27YNhw8fxqeffoqoqCj07NkT2dnZJb5m8eLFsLa2Vm/Ozs5PFQMRERHpNkWH9rShUqk0Hgsh1OXp6ekYMWIEgoODYWtr+1TnGTJkiHq/VatW6NChA1xdXfHDDz9g8ODBxb4mICAA06dPVz9OS0tjMkVERFSN6VUi5eDggOTkZI2ylJQUGBoawsbGBnFxcUhMTISXl5f6+YKCAgCAoaEh4uPj0bhxY63O7ejoCFdXV1y8eLHEOiYmJjAxMVE/LkzyOMRHRESkPwq/twu/x0ujV4mUu7s79u7dq1EWHh6ODh06wMjICM2bN0dsbKzG83PmzEF6ejpWr179VL1Dt2/fxtWrV+Ho6Fjm16SnpwMAe6WIiIj0UHp6OqytrUuto2gilZGRgb/++kv9OCEhATExMahXrx5cXFwQEBCA69evY+vWrQAAf39/BAYGYvr06fDz80NkZCRCQkKwfft2AICpqSlatWqlcY46deoAgEb5k86bkZGB+fPnw9vbG46OjkhMTMTs2bNha2uLQYMGlbl9Tk5OuHr1KiwtLYsMST6twmHDq1evwsrKqkKPrQvYPv1X3dvI9um/6t5Gtk97Qgikp6fDycnpiXUVTaSio6PRo0cP9ePC+UU+Pj4IDQ1FUlISrly5on6+UaNG2LdvH6ZNm4Z169bByckJa9as0Vj6oCLOa2BggNjYWGzduhX37t2Do6MjevTogR07dsDS0rLM56lVqxYaNGhQrtjKy8rKqlp+QAqxffqvureR7dN/1b2NbJ92ntQTVUjRRKp79+6ljj+GhoYWKevWrRtOnz5d5nMUd4wnndfMzAwHDx4s8zmIiIioZtKr5Q+IiIiIdAkTKT1lYmKCefPmaVwlWJ2wffqvureR7dN/1b2NbF/VUImyXNtHREREREWwR4qIiIhIS0ykiIiIiLTERIqIiIhIS0ykiIiIiLTERErHZWdn44UXXoBKpUJMTEypdYUQmD9/PpycnGBmZobu3bsjLi6uyPEmTZoEW1tb1K5dG/3798e1a9cqsQXF69+/P1xcXGBqagpHR0eMHDkSN27cKPU1KpWq2O2TTz5R1+nevXuR54cOHVrZzSlCm/b5+voWib1z584adXTl/QPK38bc3FzMmjULzz//PGrXrg0nJye8/fbbRV6jz++hPn0GExMTMWbMGDRq1AhmZmZo3Lgx5s2bh5ycnFJfpy+fQ23bpy+fQ23ap2+fQW3fwyr/HArSaZMnTxaenp4CgDhz5kypdZcsWSIsLS3Fzp07RWxsrBgyZIhwdHQUaWlp6jr+/v7imWeeEYcOHRKnT58WPXr0EG3atBF5eXmV3BJNK1asEJGRkSIxMVGcOHFCuLu7C3d391Jfk5SUpLFt2rRJqFQqcenSJXWdbt26CT8/P4169+7dq+zmFKFN+3x8fESfPn00Yr99+7ZGHV15/4Qofxvv3bsnXn31VbFjxw7xxx9/iMjISNGpUyfRvn17jXr6/B7q02dw//79wtfXVxw8eFBcunRJ7NmzR9jZ2Yn//ve/pb5OXz6H2rZPXz6H2rRP3z6D2r6HVf05ZCKlw/bt2yeaN28u4uLinphIFRQUCAcHB7FkyRJ1WVZWlrC2thZBQUFCCPkhMjIyEmFhYeo6169fF7Vq1RIHDhyotHaUxZ49e4RKpRI5OTllfs2AAQNEz549Ncq6desmpkyZUsHRPb2ytM/Hx0cMGDCgxOd1+f0TQrv38NdffxUAxOXLl9Vl+voe6vtnUAghli1bJho1alSu1+jT57As7dPnz6E2758+fQaFeHIblfgccmhPR/3zzz/w8/PDF198AXNz8yfWT0hIQHJyMnr37q0uMzExQbdu3XDy5EkAwG+//Ybc3FyNOk5OTmjVqpW6jhLu3LmDbdu2oUuXLjAyMirTa/755x/88MMPGDNmTJHntm3bBltbWzz33HOYMWMG0tPTKzrkcilP+44ePQo7Ozs0bdoUfn5+SElJUT+nq+8foN17CACpqalQqVTqm4sX0sf3UJ8/g4VSU1NRr169MtfXp88hUPb26evnsLzvX+Fr9OEzWOhJbVTic8hESgcJIeDr6wt/f3906NChTK9JTk4GANjb22uU29vbq59LTk6GsbEx6tatW2KdqjRr1izUrl0bNjY2uHLlCvbs2VPm127ZsgWWlpYYPHiwRvnw4cOxfft2HD16FHPnzsXOnTuL1Kkq5W2fp6cntm3bhsOHD+PTTz9FVFQUevbsiezsbAC69/4BT/ceZmVl4f3338ewYcM0bjiqr++hPn4GH3Xp0iWsXbsW/v7+ZX6NPnwOC5W1ffr4OQS0e//04TP4qLK0UZHPYbn7sEhr8+bNEwBK3aKiosTq1atFly5d1GO1CQkJTxzaO3HihAAgbty4oVE+duxY4eHhIYQQYtu2bcLY2LjIa1999VUxbty4KmtfoZs3b4r4+HgRHh4uXnrpJdG3b19RUFBQpnM1a9ZMTJw48Yn1oqOjBQDx22+/ad2uQlXZPiGEuHHjhjAyMhI7d+4UQlT++ydE1bUxJydHDBgwQLRt21akpqaWWldf3kNd+Axq00Yh5LBGkyZNxJgxY8p1Ln34HAqhffuEqPrPYVW1T6nPoBCV20YlPoeG5U+9SFsTJ0584pUPDRs2xEcffYRTp04VuX9Qhw4dMHz4cGzZsqXI6xwcHADITNvR0VFdnpKSos7MHRwckJOTg7t372pk4ikpKejSpYvW7SpU1vYVsrW1ha2tLZo2bYoWLVrA2dkZp06dgru7e6nHOH78OOLj47Fjx44nxtSuXTsYGRnh4sWLaNeuXZnaUZKqal8hR0dHuLq64uLFiwAq//0DqqaNubm5ePPNN5GQkIDDhw9r/CVcHH15D3XhMwiUv403btxAjx494O7ujo0bN5b5PPryOdS2fYWq+nNYFe1T8jMIVG4bFfkcljv1okp3+fJlERsbq94OHjwoAIhvvvlGXL16tdjXFE6wW7p0qbosOzu72Al2O3bsUNe5ceOGTkySvHLligAgjhw58sS6Pj4+Ra4yKUlsbKwAICIiIp4ywqdTnvYVunXrljAxMRFbtmwRQuj2+ydE2dqYk5MjBg4cKJ577jmRkpJSpuPqy3uoj5/Ba9euiWeffVYMHTq03Fcr6cPn8GnaV0iXP4fatE/fPoPlbaMSn0MmUnqgpKG9Zs2aiV27dqkfL1myRFhbW4tdu3aJ2NhY8dZbbxV7yWeDBg3Ejz/+KE6fPi169uxZ5Zft/vLLL2Lt2rXizJkzIjExURw+fFi8/PLLonHjxiIrK6vE9gkhRGpqqjA3Nxfr168vcty//vpLLFiwQERFRYmEhATxww8/iObNm4u2bdvqfPvS09PFf//7X3Hy5EmRkJAgjhw5Itzd3cUzzzyjc++ftm3Mzc0V/fv3Fw0aNBAxMTEal1ZnZ2cLIfT7PRRCfz6DQjwcKunZs6e4du2axvvxKH39HGrTPn36HGrTPn36DGrbRiGq/nPIREoPlJRIARCbN29WPy4oKBDz5s0TDg4OwsTERLzyyisiNjZW4zUPHjwQEydOFPXq1RNmZmbitddeE1euXKmCVjx07tw50aNHD1GvXj1hYmIiGjZsKPz9/cW1a9c06j3ePiGE2LBhgzAzMyt2TZMrV66IV155RdSrV08YGxuLxo0bi8mTJxdZA6ayadO++/fvi969e4v69esLIyMj4eLiInx8fIq8N7rw/gmhXRsLf4+L2wp7efT5PRRCfz6DQgixefPmEt+PR+nr51Cb9unT51Cb9unTZ1AI7X9Hq/pzqPo3CCIiIiIqJy5/QERERKQlJlJEREREWmIiRURERKQlJlJEREREWmIiRURERKQlJlJEREREWmIiRURERKQlJlJEREREWmIiRURVrnv37pg6darSYRTr9u3bsLOzQ2JiIgDg6NGjUKlUuHfvXqWeV9vzhIaGok6dOuV6zYsvvohdu3aV6zVEVDwmUkSk95KSkjBs2DA0a9YMtWrVKjFJ27lzJ1q2bAkTExO0bNkS3377bZE6ixcvhpeXl8bd56ubuXPn4v3330dBQYHSoRDpPSZSRKT3srOzUb9+fXzwwQdo06ZNsXUiIyMxZMgQjBw5EmfPnsXIkSPx5ptv4pdfflHXefDgAUJCQjB27NiqCl0R/fr1Q2pqKg4ePKh0KER6j4kUESnq7t27ePvtt1G3bl2Ym5vD09MTFy9e1KgTHBwMZ2dnmJubY9CgQVixYoXGcFbDhg2xevVqvP3227C2ti72PKtWrUKvXr0QEBCA5s2bIyAgAP/5z3+watUqdZ39+/fD0NAQ7u7uJcZ7+/ZtvPXWW2jQoAHMzc3x/PPPY/v27Rp1unfvjkmTJmHq1KmoW7cu7O3tsXHjRmRmZmLUqFGwtLRE48aNsX///iLHP3HiBNq0aQNTU1N06tQJsbGxGs+HhobCxcVF/bO4ffu2xvOXLl3CgAEDYG9vDwsLC7z44ov48ccfNeoYGBigb9++ReImovJjIkVEivL19UV0dDS+++47REZGQgiBvn37Ijc3F4BMLPz9/TFlyhTExMSgV69e+Pjjj8t9nsjISPTu3VujzMPDAydPnlQ/PnbsGDp06FDqcbKystC+fXt8//33OH/+PN555x2MHDlSo2cLALZs2QJbW1v8+uuvmDRpEt5991288cYb6NKlC06fPg0PDw+MHDkS9+/f13jdzJkzsXz5ckRFRcHOzg79+/dX/yx++eUXjB49GuPHj0dMTAx69OiBjz76SOP1GRkZ6Nu3L3788UecOXMGHh4e8PLywpUrVzTqdezYEcePHy/bD4+ISiaIiKpYt27dxJQpU8Sff/4pAIgTJ06on7t165YwMzMTX3/9tRBCiCFDhoh+/fppvH748OHC2tq61GM/zsjISGzbtk2jbNu2bcLY2Fj9eMCAAWL06NEadY4cOSIAiLt375bYnr59+4r//ve/GjG8/PLL6sd5eXmidu3aYuTIkeqypKQkAUBERkZqnCcsLExd5/bt28LMzEzs2LFDCCHEW2+9Jfr06aNx7iFDhpT4syjUsmVLsXbtWo2yPXv2iFq1aon8/PxSX0tEpWOPFBEp5sKFCzA0NESnTp3UZTY2NmjWrBkuXLgAAIiPj0fHjh01Xvf447JSqVQaj4UQGmUPHjyAqalpqcfIz8/Hxx9/jNatW8PGxgYWFhYIDw8v0uPTunVr9b6BgQFsbGzw/PPPq8vs7e0BACkpKRqve3RYsV69eho/iwsXLhQZdnz8cWZmJt577z20bNkSderUgYWFBf74448i8ZmZmaGgoADZ2dmltpeISmeodABEVHMJIUosL0xwHk92SntdaRwcHJCcnKxRlpKSok5oAMDW1hZ3794t9TiffvopVq5ciVWrVuH5559H7dq1MXXqVOTk5GjUMzIy0nisUqk0ygrbVJYr5x79WTzJzJkzcfDgQSxfvhxNmjSBmZkZXn/99SLx3blzB+bm5jAzM3viMYmoZOyRIiLFtGzZEnl5eRrzi27fvo0///wTLVq0AAA0b94cv/76q8broqOjy30ud3d3HDp0SKMsPDwcXbp0UT9u27Ytfv/991KPc/z4cQwYMAAjRoxAmzZt4ObmVmRy/NM4deqUev/u3bv4888/0bx5cwDy5/Xo84/XL4zP19cXgwYNwvPPPw8HBwf1mliPOn/+PNq1a1dhcRPVVEykiEgxzz77LAYMGAA/Pz/8/PPPOHv2LEaMGIFnnnkGAwYMAABMmjQJ+/btw4oVK3Dx4kVs2LAB+/fvL9JLFRMTg5iYGGRkZODmzZuIiYnRSIqmTJmC8PBwLF26FH/88QeWLl2KH3/8UWPNKQ8PD8TFxZXaK9WkSRMcOnQIJ0+exIULFzBu3LgiPV1PY+HChfjpp59w/vx5+Pr6wtbWFgMHDgQATJ48GQcOHMCyZcvw559/IjAwEAcOHCgS365duxATE4OzZ89i2LBhxfZ6HT9+vMjkeyIqPyZSRKSozZs3o3379njttdfg7u4OIQT27dunHgZ76aWXEBQUhBUrVqBNmzY4cOAApk2bVmQuU9u2bdG2bVv89ttv+Oqrr9C2bVv07dtX/XyXLl0QFhaGzZs3o3Xr1ggNDcWOHTs05mc9//zz6NChA77++usS4507dy7atWsHDw8PdO/eHQ4ODupEpyIsWbIEU6ZMQfv27ZGUlITvvvsOxsbGAIDOnTvj888/x9q1a/HCCy8gPDwcc+bM0Xj9ypUrUbduXXTp0gVeXl7w8PAo0vN0/fp1nDx5EqNGjaqwuIlqKpXQZrIBEZGC/Pz88Mcff1TK5fv79u3DjBkzcP78edSqVT3/1pw5cyZSU1OxceNGpUMh0nucbE5EOm/58uXo1asXateujf3792PLli347LPPKuVcffv2xcWLF3H9+nU4OztXyjmUZmdnhxkzZigdBlG1wB4pItJ5b775Jo4ePYr09HS4ublh0qRJ8Pf3VzosIiImUkRERETaqp4TAIiIiIiqABMpIiIiIi0xkSIiIiLSEhMpIiIiIi0xkSIiIiLSEhMpIiIiIi0xkSIiIiLSEhMpIiIiIi39P6W/PWhKEV4vAAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 11,
+ "id": "d11ccfa9",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
@@ -2993,17 +2927,17 @@
" return np.sum((y_data-y_model)**2)/n\n",
"\n",
"# Make data set.\n",
- "n = 1000\n",
+ "n = 10000\n",
"x = np.random.rand(n)\n",
"y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)+ np.random.randn(n)\n",
"\n",
- "Maxpolydegree = 10\n",
+ "Maxpolydegree = 5\n",
"X = np.zeros((len(x),Maxpolydegree))\n",
- "#X[:,0] = 1.0\n",
+ "X[:,0] = 1.0\n",
"\n",
"\n",
- "for polydegree in range(0,Maxpolydegree):\n",
- " X[:,polydegree] = x**(polydegree+1)\n",
+ "for polydegree in range(1,Maxpolydegree):\n",
+ " X[:,polydegree] = x**(polydegree)\n",
"\n",
"# We split the data in test and training data\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
@@ -3019,13 +2953,13 @@
"nlambdas = 4\n",
"MSERidgePredict = np.zeros(nlambdas)\n",
"MSELassoPredict = np.zeros(nlambdas)\n",
- "lambdas = np.logspace(-4, -2, nlambdas)\n",
+ "lambdas = np.logspace(-3, 1, nlambdas)\n",
"for i in range(nlambdas):\n",
" lmb = lambdas[i]\n",
" # Make the fit using Ridge and Lasso\n",
- " RegRidge = linear_model.Ridge(lmb,fit_intercept=True)\n",
+ " RegRidge = linear_model.Ridge(lmb,fit_intercept=False)\n",
" RegRidge.fit(X_train,y_train)\n",
- " RegLasso = linear_model.Lasso(lmb,fit_intercept=True)\n",
+ " RegLasso = linear_model.Lasso(lmb,fit_intercept=False)\n",
" RegLasso.fit(X_train,y_train)\n",
" # and then make the prediction\n",
" ypredictRidge = RegRidge.predict(X_test)\n",
@@ -3047,16 +2981,20 @@
},
{
"cell_type": "markdown",
- "id": "2056e242",
- "metadata": {},
+ "id": "5e3262b3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"How can we understand this?"
]
},
{
"cell_type": "markdown",
- "id": "3a93dae3",
- "metadata": {},
+ "id": "74be82d7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Linking the regression analysis with a statistical interpretation\n",
"\n",
@@ -3082,8 +3020,10 @@
},
{
"cell_type": "markdown",
- "id": "265c2329",
- "metadata": {},
+ "id": "7fe2602e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*} \n",
@@ -3096,8 +3036,10 @@
},
{
"cell_type": "markdown",
- "id": "1eb62f34",
- "metadata": {},
+ "id": "93dbe57a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The randomness of $\\varepsilon_i$ implies that\n",
"$\\mathbf{y}_i$ is also a random variable. In particular,\n",
@@ -3113,8 +3055,10 @@
},
{
"cell_type": "markdown",
- "id": "93523fed",
- "metadata": {},
+ "id": "1ee68fd5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Assumptions made\n",
"\n",
@@ -3125,8 +3069,10 @@
},
{
"cell_type": "markdown",
- "id": "5b2fe793",
- "metadata": {},
+ "id": "8d0dd5b5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n",
@@ -3135,8 +3081,10 @@
},
{
"cell_type": "markdown",
- "id": "645cfe7a",
- "metadata": {},
+ "id": "335af34a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We approximate this function with our model from the solution of the linear regression equations, that is our\n",
"function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with"
@@ -3144,8 +3092,10 @@
},
{
"cell_type": "markdown",
- "id": "acd315af",
- "metadata": {},
+ "id": "5c45c5a2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n",
@@ -3154,8 +3104,10 @@
},
{
"cell_type": "markdown",
- "id": "3e5bcdc9",
- "metadata": {},
+ "id": "84cca583",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Expectation value and variance\n",
"\n",
@@ -3164,8 +3116,10 @@
},
{
"cell_type": "markdown",
- "id": "445b0b0d",
- "metadata": {},
+ "id": "d3837bb0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*} \n",
@@ -3178,8 +3132,10 @@
},
{
"cell_type": "markdown",
- "id": "9ba5bb36",
- "metadata": {},
+ "id": "f1c3036d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"while\n",
"its variance is"
@@ -3187,8 +3143,10 @@
},
{
"cell_type": "markdown",
- "id": "cd53bf61",
- "metadata": {},
+ "id": "e14f5ca2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n",
@@ -3208,8 +3166,10 @@
},
{
"cell_type": "markdown",
- "id": "3d7101bc",
- "metadata": {},
+ "id": "dfa9ab36",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n",
"mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)."
@@ -3217,8 +3177,10 @@
},
{
"cell_type": "markdown",
- "id": "3670204e",
- "metadata": {},
+ "id": "58386e3b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Expectation value and variance for $\\boldsymbol{\\beta}$\n",
"\n",
@@ -3227,8 +3189,10 @@
},
{
"cell_type": "markdown",
- "id": "09f3c5ea",
- "metadata": {},
+ "id": "0d3fec7a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbb{E}(\\boldsymbol{\\hat{\\beta}}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n",
@@ -3237,8 +3201,10 @@
},
{
"cell_type": "markdown",
- "id": "814246ef",
- "metadata": {},
+ "id": "1f2c1761",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This means that the estimator of the regression parameters is unbiased.\n",
"\n",
@@ -3249,8 +3215,10 @@
},
{
"cell_type": "markdown",
- "id": "e831728d",
- "metadata": {},
+ "id": "361db6cc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{eqnarray*}\n",
@@ -3278,8 +3246,10 @@
},
{
"cell_type": "markdown",
- "id": "522aaab4",
- "metadata": {},
+ "id": "126d7a69",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n",
"\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n",
@@ -3298,8 +3268,10 @@
},
{
"cell_type": "markdown",
- "id": "0dfec07a",
- "metadata": {},
+ "id": "33f45af8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n",
@@ -3308,8 +3280,10 @@
},
{
"cell_type": "markdown",
- "id": "c0b31a8b",
- "metadata": {},
+ "id": "e4007597",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We see clearly that \n",
"$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n",
@@ -3319,8 +3293,10 @@
},
{
"cell_type": "markdown",
- "id": "287d622e",
- "metadata": {},
+ "id": "345b2c9a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n",
@@ -3329,8 +3305,10 @@
},
{
"cell_type": "markdown",
- "id": "0afd8826",
- "metadata": {},
+ "id": "24dfdffe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n",
"\n",
@@ -3339,8 +3317,10 @@
},
{
"cell_type": "markdown",
- "id": "bb005c68",
- "metadata": {},
+ "id": "22abadec",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n",
@@ -3349,8 +3329,10 @@
},
{
"cell_type": "markdown",
- "id": "2d9cb9a1",
- "metadata": {},
+ "id": "0e3d6108",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The difference is non-negative definite since each component of the\n",
"matrix product is non-negative definite. \n",
@@ -3359,8 +3341,10 @@
},
{
"cell_type": "markdown",
- "id": "4760d2c7",
- "metadata": {},
+ "id": "616e24dc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Deriving OLS from a probability distribution\n",
"\n",
@@ -3380,8 +3364,10 @@
},
{
"cell_type": "markdown",
- "id": "5af957ae",
- "metadata": {},
+ "id": "91727842",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
@@ -3390,8 +3376,10 @@
},
{
"cell_type": "markdown",
- "id": "fd6a18ae",
- "metadata": {},
+ "id": "bf6d769e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Independent and Identically Distrubuted (iid)\n",
"\n",
@@ -3401,8 +3389,10 @@
},
{
"cell_type": "markdown",
- "id": "401d6e0a",
- "metadata": {},
+ "id": "446bf095",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n",
@@ -3411,8 +3401,10 @@
},
{
"cell_type": "markdown",
- "id": "48e126ea",
- "metadata": {},
+ "id": "9ec34b7f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n",
"\n",
@@ -3421,8 +3413,10 @@
},
{
"cell_type": "markdown",
- "id": "34eda977",
- "metadata": {},
+ "id": "e55aa7e4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n",
@@ -3431,8 +3425,10 @@
},
{
"cell_type": "markdown",
- "id": "90714c71",
- "metadata": {},
+ "id": "682bd937",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n",
"in case we have a simple one-dimensional input and output case"
@@ -3440,8 +3436,10 @@
},
{
"cell_type": "markdown",
- "id": "74731195",
- "metadata": {},
+ "id": "1053d0d0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n",
@@ -3450,8 +3448,10 @@
},
{
"cell_type": "markdown",
- "id": "1bc063bc",
- "metadata": {},
+ "id": "685615d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n",
"We can now rewrite the above probability as"
@@ -3459,8 +3459,10 @@
},
{
"cell_type": "markdown",
- "id": "a9de140e",
- "metadata": {},
+ "id": "e9daac1f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
@@ -3469,16 +3471,20 @@
},
{
"cell_type": "markdown",
- "id": "b8c38a5a",
- "metadata": {},
+ "id": "674078d8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$."
]
},
{
"cell_type": "markdown",
- "id": "2b9c970d",
- "metadata": {},
+ "id": "13efc918",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Maximum Likelihood Estimation (MLE)\n",
"\n",
@@ -3506,8 +3512,10 @@
},
{
"cell_type": "markdown",
- "id": "9f18acb3",
- "metadata": {},
+ "id": "6666bcf2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## A new Cost Function\n",
"\n",
@@ -3516,8 +3524,10 @@
},
{
"cell_type": "markdown",
- "id": "db679b3a",
- "metadata": {},
+ "id": "dbdd23c7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n",
@@ -3526,16 +3536,20 @@
},
{
"cell_type": "markdown",
- "id": "6b743abf",
- "metadata": {},
+ "id": "40a2a7ee",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which becomes"
]
},
{
"cell_type": "markdown",
- "id": "43048bc0",
- "metadata": {},
+ "id": "9d2925ab",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n",
@@ -3544,16 +3558,20 @@
},
{
"cell_type": "markdown",
- "id": "53939c7a",
- "metadata": {},
+ "id": "1ee409ee",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely"
]
},
{
"cell_type": "markdown",
- "id": "22af3bc5",
- "metadata": {},
+ "id": "5f4a56ee",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n",
@@ -3562,16 +3580,20 @@
},
{
"cell_type": "markdown",
- "id": "89e9f36e",
- "metadata": {},
+ "id": "a192d3e6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which leads to the well-known OLS equation for the optimal paramters $\\beta$"
]
},
{
"cell_type": "markdown",
- "id": "10eef97b",
- "metadata": {},
+ "id": "7e4befd2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n",
@@ -3580,16 +3602,20 @@
},
{
"cell_type": "markdown",
- "id": "6f491288",
- "metadata": {},
+ "id": "b5af2fdd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics."
]
},
{
"cell_type": "markdown",
- "id": "aa38d555",
- "metadata": {},
+ "id": "5e7a6338",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More basic Statistics and Bayes' theorem\n",
"\n",
@@ -3606,8 +3632,10 @@
},
{
"cell_type": "markdown",
- "id": "e9b2c3a4",
- "metadata": {},
+ "id": "e7abbba0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n",
@@ -3616,16 +3644,20 @@
},
{
"cell_type": "markdown",
- "id": "68cb51e6",
- "metadata": {},
+ "id": "0d906411",
+ "metadata": {
+ "editable": true
+ },
"source": [
"**The product rule (aka joint probability) is given by.**"
]
},
{
"cell_type": "markdown",
- "id": "e0325fa4",
- "metadata": {},
+ "id": "c46b32fe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n",
@@ -3634,8 +3666,10 @@
},
{
"cell_type": "markdown",
- "id": "9e0da087",
- "metadata": {},
+ "id": "8d719e61",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n",
"\n",
@@ -3644,8 +3678,10 @@
},
{
"cell_type": "markdown",
- "id": "6af44809",
- "metadata": {},
+ "id": "69c13c62",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Marginal Probability\n",
"\n",
@@ -3654,8 +3690,10 @@
},
{
"cell_type": "markdown",
- "id": "71162259",
- "metadata": {},
+ "id": "06c92406",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X)=\\sum_{i=0}^{n-1}p(X,Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert Y=y_i)p(Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert y_i)p(y_i).\n",
@@ -3664,8 +3702,10 @@
},
{
"cell_type": "markdown",
- "id": "622c0483",
- "metadata": {},
+ "id": "50314a95",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conditional Probability\n",
"\n",
@@ -3674,8 +3714,10 @@
},
{
"cell_type": "markdown",
- "id": "157eb8eb",
- "metadata": {},
+ "id": "59287aa4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)}=\\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}.\n",
@@ -3684,8 +3726,10 @@
},
{
"cell_type": "markdown",
- "id": "a73c711d",
- "metadata": {},
+ "id": "908a76a6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Bayes' Theorem\n",
"\n",
@@ -3694,8 +3738,10 @@
},
{
"cell_type": "markdown",
- "id": "2d83fce3",
- "metadata": {},
+ "id": "4c91459e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n",
@@ -3704,16 +3750,20 @@
},
{
"cell_type": "markdown",
- "id": "fa09b6e5",
- "metadata": {},
+ "id": "9004d3cc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which we can rewrite as"
]
},
{
"cell_type": "markdown",
- "id": "c21e7b34",
- "metadata": {},
+ "id": "80cbe78c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n",
@@ -3722,16 +3772,20 @@
},
{
"cell_type": "markdown",
- "id": "b0f6f2ec",
- "metadata": {},
+ "id": "c0331acf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$."
]
},
{
"cell_type": "markdown",
- "id": "fdb50d53",
- "metadata": {},
+ "id": "c6fe8797",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Interpretations of Bayes' Theorem\n",
"\n",
@@ -3747,8 +3801,10 @@
},
{
"cell_type": "markdown",
- "id": "80ea6e5e",
- "metadata": {},
+ "id": "27cd0c90",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Example of Usage of Bayes' theorem\n",
"\n",
@@ -3766,8 +3822,10 @@
},
{
"cell_type": "markdown",
- "id": "bc467488",
- "metadata": {},
+ "id": "b4e2c5da",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X=1\\vert Y=1) =0.8.\n",
@@ -3776,8 +3834,10 @@
},
{
"cell_type": "markdown",
- "id": "e2f5ad4f",
- "metadata": {},
+ "id": "f8386bad",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n",
"It is however not correct, as the following Bayesian analysis shows."
@@ -3785,8 +3845,10 @@
},
{
"cell_type": "markdown",
- "id": "488981c2",
- "metadata": {},
+ "id": "61cdeb43",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Doing it correctly\n",
"\n",
@@ -3796,8 +3858,10 @@
},
{
"cell_type": "markdown",
- "id": "82e6140d",
- "metadata": {},
+ "id": "2432bedc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(Y=1) =0.004.\n",
@@ -3806,16 +3870,20 @@
},
{
"cell_type": "markdown",
- "id": "a4453800",
- "metadata": {},
+ "id": "5d6ac5c4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have"
]
},
{
"cell_type": "markdown",
- "id": "346f9f17",
- "metadata": {},
+ "id": "71119552",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(X=1\\vert Y=0) =0.1.\n",
@@ -3824,16 +3892,20 @@
},
{
"cell_type": "markdown",
- "id": "a16b35ad",
- "metadata": {},
+ "id": "d44efffd",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute"
]
},
{
"cell_type": "markdown",
- "id": "4cd7eeb4",
- "metadata": {},
+ "id": "c2aa1834",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n",
@@ -3842,16 +3914,20 @@
},
{
"cell_type": "markdown",
- "id": "0bad6e84",
- "metadata": {},
+ "id": "67c60f91",
+ "metadata": {
+ "editable": true
+ },
"source": [
"That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!"
]
},
{
"cell_type": "markdown",
- "id": "de649672",
- "metadata": {},
+ "id": "27222f1c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Bayes' Theorem and Ridge and Lasso Regression\n",
"\n",
@@ -3862,8 +3938,10 @@
},
{
"cell_type": "markdown",
- "id": "247ef7ea",
- "metadata": {},
+ "id": "9bf4c2da",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n",
@@ -3872,16 +3950,20 @@
},
{
"cell_type": "markdown",
- "id": "e668ae58",
- "metadata": {},
+ "id": "cea2e5e6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"is given by"
]
},
{
"cell_type": "markdown",
- "id": "503efa2d",
- "metadata": {},
+ "id": "837f3932",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n",
@@ -3890,16 +3972,20 @@
},
{
"cell_type": "markdown",
- "id": "b2b875a0",
- "metadata": {},
+ "id": "e0b179d2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability"
]
},
{
"cell_type": "markdown",
- "id": "55a767c5",
- "metadata": {},
+ "id": "70bc7ff5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n",
@@ -3908,16 +3994,20 @@
},
{
"cell_type": "markdown",
- "id": "14bbfa0a",
- "metadata": {},
+ "id": "025bb774",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Bayes' theorem comes to our rescue here since (omitting the normalization constant)"
]
},
{
"cell_type": "markdown",
- "id": "240ed0c3",
- "metadata": {},
+ "id": "4d66e6f6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n",
@@ -3926,16 +4016,20 @@
},
{
"cell_type": "markdown",
- "id": "f458005f",
- "metadata": {},
+ "id": "b90b87a8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!"
]
},
{
"cell_type": "markdown",
- "id": "b8b753eb",
- "metadata": {},
+ "id": "3567513f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Ridge and Bayes\n",
"\n",
@@ -3948,8 +4042,10 @@
},
{
"cell_type": "markdown",
- "id": "5e4534e0",
- "metadata": {},
+ "id": "07c3a73d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n",
@@ -3958,16 +4054,20 @@
},
{
"cell_type": "markdown",
- "id": "c601b314",
- "metadata": {},
+ "id": "50fb3075",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Our posterior probability becomes then (omitting the normalization factor which is just a constant)"
]
},
{
"cell_type": "markdown",
- "id": "fe20511b",
- "metadata": {},
+ "id": "48e0399b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n",
@@ -3976,8 +4076,10 @@
},
{
"cell_type": "markdown",
- "id": "254593d7",
- "metadata": {},
+ "id": "fdb45d4d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n",
"did for OLS, this is most conveniently done by taking the negative\n",
@@ -3987,8 +4089,10 @@
},
{
"cell_type": "markdown",
- "id": "a62f7507",
- "metadata": {},
+ "id": "6fea7113",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
@@ -3997,16 +4101,20 @@
},
{
"cell_type": "markdown",
- "id": "7c1a1880",
- "metadata": {},
+ "id": "4cc8558b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and replacing $1/2\\tau^2$ with $\\lambda$ we have"
]
},
{
"cell_type": "markdown",
- "id": "5719b77c",
- "metadata": {},
+ "id": "3f88a5ae",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n",
@@ -4015,16 +4123,20 @@
},
{
"cell_type": "markdown",
- "id": "976134e0",
- "metadata": {},
+ "id": "f8e5eab1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which is our Ridge cost function! Nice, isn't it?"
]
},
{
"cell_type": "markdown",
- "id": "ddc710a7",
- "metadata": {},
+ "id": "625ffa9a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Lasso and Bayes\n",
"\n",
@@ -4033,8 +4145,10 @@
},
{
"cell_type": "markdown",
- "id": "9eba4148",
- "metadata": {},
+ "id": "c9f42b80",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n",
@@ -4043,16 +4157,20 @@
},
{
"cell_type": "markdown",
- "id": "1e028fb0",
- "metadata": {},
+ "id": "d4a9db87",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Our posterior probability becomes then (omitting the normalization factor which is just a constant)"
]
},
{
"cell_type": "markdown",
- "id": "4ed079fb",
- "metadata": {},
+ "id": "07422075",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n",
@@ -4061,8 +4179,10 @@
},
{
"cell_type": "markdown",
- "id": "879dc4c0",
- "metadata": {},
+ "id": "1d2ae416",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Taking the negative\n",
"logarithm of the posterior probability and leaving out the\n",
@@ -4071,8 +4191,10 @@
},
{
"cell_type": "markdown",
- "id": "570d880d",
- "metadata": {},
+ "id": "eaef554a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
@@ -4081,16 +4203,20 @@
},
{
"cell_type": "markdown",
- "id": "9b9e59f3",
- "metadata": {},
+ "id": "5f691944",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and replacing $1/\\tau$ with $\\lambda$ we have"
]
},
{
"cell_type": "markdown",
- "id": "9bed769e",
- "metadata": {},
+ "id": "b4f8d4c5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
@@ -4099,32 +4225,16 @@
},
{
"cell_type": "markdown",
- "id": "0d2b3bbb",
- "metadata": {},
+ "id": "280a7a70",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which is our Lasso cost function!"
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.10"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index d630ca7c8..ab571a832 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -15,11 +15,11 @@ DATE: September 4-8, 2023
* Linear Regression and links with Statistics
* "Recommended Reading: Goodfellow et al chapter 3 on probability theory":"https://www.deeplearningbook.org/"
* See also Murphy, sections 2.4 (Gaussian distributions) and 3.2 (Bayesian Statistics, basis)
-
+ * "Video of lecture":"https://youtu.be/Kc20CFK0z7Y"
!split
===== Material for the active learning sessions Tuesday and Wednesday =====
-The material here contains a summary from last Week and discussion of SVD, Ridge and Lasso regression with examples
+The material here contains a summary from last week and discussion of SVD, Ridge and Lasso regression with examples
!split
===== Linear Regression and the SVD =====