From fa06cccfd8183b3468f141027a8d06e27e83c004 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Thu, 8 Sep 2022 14:30:23 +0200 Subject: [PATCH] minor typo --- doc/pub/week35/html/._week35-bs000.html | 2 +- doc/pub/week35/html/._week35-bs050.html | 11 +- doc/pub/week35/html/week35-bs.html | 2 +- doc/pub/week35/html/week35-reveal.html | 13 +- doc/pub/week35/html/week35-solarized.html | 13 +- doc/pub/week35/html/week35.html | 13 +- doc/pub/week35/ipynb/ipynb-week35-src.tar.gz | Bin 192 -> 191 bytes doc/pub/week35/ipynb/week35.ipynb | 836 ++++---- doc/pub/week36/ipynb/week36.ipynb | 1915 +++++++++--------- doc/src/week35/week35.do.txt | 10 +- 10 files changed, 1398 insertions(+), 1417 deletions(-) diff --git a/doc/pub/week35/html/._week35-bs000.html b/doc/pub/week35/html/._week35-bs000.html index 878b6dfb3..2ba0f1ed3 100644 --- a/doc/pub/week35/html/._week35-bs000.html +++ b/doc/pub/week35/html/._week35-bs000.html @@ -303,7 +303,7 @@ MathJax.Hub.Config({
-

Sep 7, 2022

+

Sep 8, 2022


diff --git a/doc/pub/week35/html/._week35-bs050.html b/doc/pub/week35/html/._week35-bs050.html index 12c092123..12aec1326 100644 --- a/doc/pub/week35/html/._week35-bs050.html +++ b/doc/pub/week35/html/._week35-bs050.html @@ -316,13 +316,18 @@ $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, $$ -

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),,

+

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, $$ -

It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

+

It means that the ordinary least square model (with the optimal +parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal +transformation of the output (or target) vector \( \boldsymbol{y} \) by the +vectors of the matrix \( \boldsymbol{U} \). Note that the summation ends at \( p-1 \), +that is \( \boldsymbol{\tilde{y}}\ne \boldsymbol{y} \). +

diff --git a/doc/pub/week35/html/week35-bs.html b/doc/pub/week35/html/week35-bs.html index 878b6dfb3..2ba0f1ed3 100644 --- a/doc/pub/week35/html/week35-bs.html +++ b/doc/pub/week35/html/week35-bs.html @@ -303,7 +303,7 @@ MathJax.Hub.Config({

-

Sep 7, 2022

+

Sep 8, 2022


diff --git a/doc/pub/week35/html/week35-reveal.html b/doc/pub/week35/html/week35-reveal.html index e502d9141..c7195b2a3 100644 --- a/doc/pub/week35/html/week35-reveal.html +++ b/doc/pub/week35/html/week35-reveal.html @@ -181,7 +181,7 @@ MathJax.Hub.Config({
-

Sep 7, 2022

+

Sep 8, 2022


@@ -2552,15 +2552,20 @@ $$ $$

 
-

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),,

+

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

 
$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, $$

 
-

It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

+

It means that the ordinary least square model (with the optimal +parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal +transformation of the output (or target) vector \( \boldsymbol{y} \) by the +vectors of the matrix \( \boldsymbol{U} \). Note that the summation ends at \( p-1 \), +that is \( \boldsymbol{\tilde{y}}\ne \boldsymbol{y} \). +

diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 0eede8123..a17d8d38b 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -327,7 +327,7 @@ MathJax.Hub.Config({
-

Sep 7, 2022

+

Sep 8, 2022


@@ -2523,13 +2523,18 @@ $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, $$ -

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),,

+

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, $$ -

It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

+

It means that the ordinary least square model (with the optimal +parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal +transformation of the output (or target) vector \( \boldsymbol{y} \) by the +vectors of the matrix \( \boldsymbol{U} \). Note that the summation ends at \( p-1 \), +that is \( \boldsymbol{\tilde{y}}\ne \boldsymbol{y} \). +











Further properties (important for our analyses later)

diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 2176d58d8..653392d4b 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -404,7 +404,7 @@ MathJax.Hub.Config({
-

Sep 7, 2022

+

Sep 8, 2022


@@ -2600,13 +2600,18 @@ $$ \tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\left(\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^{2}(\boldsymbol{V}^T\right)^{-1}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, $$ -

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),,

+

which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),

$$ -\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{n-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_i\boldsymbol{y}, $$ -

It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).

+

It means that the ordinary least square model (with the optimal +parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal +transformation of the output (or target) vector \( \boldsymbol{y} \) by the +vectors of the matrix \( \boldsymbol{U} \). Note that the summation ends at \( p-1 \), +that is \( \boldsymbol{\tilde{y}}\ne \boldsymbol{y} \). +











Further properties (important for our analyses later)

diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 26f96ced54cfa55c27bb524dd0f51b30240191f5..c4a6ba91b27801e287ec36f5a3f2f52360cb1145 100644 GIT binary patch delta 169 zcmV;a09OCN0lxtyABzY8P~aJ300ZsM%?iRW3hq{ X-q2SQoGXaHonlU_ROgc(k0sxvB!>fPKhR+b<3 z%&)*R|HQG97PkA|Ra$}44s)$*xFOaoiDcWW914wgY=OaRCk=v7J&2-^PHH7CVQcis Yh{i_Yub=Te&-1?a03POG)c^\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "efb06e86", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 36: Statistical interpretation of Linear Regression and Resampling techniques\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "477ad46e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for week 36\n", "\n", @@ -51,9 +45,7 @@ { "cell_type": "markdown", "id": "09fd21b1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Thursday September 8" ] @@ -61,9 +53,7 @@ { "cell_type": "markdown", "id": "a2bbaa2d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Summary from last Week and discussion of SVD, Ridge and Lasso regression with examples" ] @@ -71,9 +61,7 @@ { "cell_type": "markdown", "id": "6c6d0800", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear Regression and the SVD\n", "\n", @@ -83,9 +71,7 @@ { "cell_type": "markdown", "id": "cf8ac89f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n", @@ -95,9 +81,7 @@ { "cell_type": "markdown", "id": "5f91422a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix" ] @@ -105,9 +89,7 @@ { "cell_type": "markdown", "id": "518a8a19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\end{bmatrix},\n", @@ -117,9 +99,7 @@ { "cell_type": "markdown", "id": "bf357b3c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is" ] @@ -127,9 +107,7 @@ { "cell_type": "markdown", "id": "46ecb6d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n", @@ -144,9 +122,7 @@ { "cell_type": "markdown", "id": "8a0f212c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning we can write" ] @@ -154,9 +130,7 @@ { "cell_type": "markdown", "id": "a44c155e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n", @@ -166,9 +140,7 @@ { "cell_type": "markdown", "id": "f18bd3ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get" ] @@ -176,9 +148,7 @@ { "cell_type": "markdown", "id": "54581079", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n", @@ -188,9 +158,7 @@ { "cell_type": "markdown", "id": "e9d26af5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What does it mean?\n", "\n", @@ -202,9 +170,7 @@ { "cell_type": "markdown", "id": "4b726bc7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n", @@ -214,9 +180,7 @@ { "cell_type": "markdown", "id": "2ac9e42f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n", "square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n", @@ -236,9 +200,7 @@ { "cell_type": "markdown", "id": "51045d47", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n", @@ -248,9 +210,7 @@ { "cell_type": "markdown", "id": "10117be4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n", "the number of samples) are the eigenvalues of the covariance\n", @@ -263,9 +223,7 @@ { "cell_type": "markdown", "id": "682f761c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n", "\n", @@ -275,9 +233,7 @@ { "cell_type": "markdown", "id": "266c3d4c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n", @@ -287,9 +243,7 @@ { "cell_type": "markdown", "id": "1f77adcf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Since the matrices here have dimension $n\\times n$, we have" ] @@ -297,9 +251,7 @@ { "cell_type": "markdown", "id": "cf376fdd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n", @@ -309,9 +261,7 @@ { "cell_type": "markdown", "id": "889f5d8c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "leading to" ] @@ -319,9 +269,7 @@ { "cell_type": "markdown", "id": "ffef11b1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n", @@ -331,9 +279,7 @@ { "cell_type": "markdown", "id": "269b2d8d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem" ] @@ -341,9 +287,7 @@ { "cell_type": "markdown", "id": "84be23ce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n", @@ -353,9 +297,7 @@ { "cell_type": "markdown", "id": "573fd926", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n", "the non-zero singular values plus now a series of zeros. The column\n", @@ -370,9 +312,7 @@ { "cell_type": "markdown", "id": "ace48064", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code for SVD and Inversion of Matrices\n", "\n", @@ -384,10 +324,7 @@ "cell_type": "code", "execution_count": 1, "id": "8bef14cd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "Ainv = np.linlag.pinv(A)" @@ -396,22 +333,42 @@ { "cell_type": "markdown", "id": "853fb3cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Let us first look at a matrix which does not causes problems and write our own function where we just use the SVD." ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 5, "id": "1d675928", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[ 1. -1. 2.]\n", + " [ 1. 0. 1.]\n", + " [ 1. 2. -1.]\n", + " [ 1. 1. 0.]]\n", + "test U\n", + "[[-3.33066907e-16 -1.11022302e-16 3.33066907e-16]\n", + " [-1.11022302e-16 4.44089210e-16 -2.49800181e-16]\n", + " [ 3.33066907e-16 -2.49800181e-16 0.00000000e+00]]\n", + "test VT\n", + "[[ 2.22044605e-16 5.55111512e-17 -2.22044605e-16]\n", + " [ 5.55111512e-17 -2.22044605e-16 5.55111512e-17]\n", + " [-2.22044605e-16 5.55111512e-17 2.22044605e-16]]\n", + "[[ 0.11111111 0.05555556 0.05555556]\n", + " [ 0.05555556 0.07777778 -0.02222222]\n", + " [ 0.05555556 -0.02222222 0.07777778]]\n", + "[[1.82969604e+30 1.82969604e+30 1.82969604e+30]\n", + " [1.82969604e+30 1.82969604e+30 1.82969604e+30]\n", + " [1.82969604e+30 1.82969604e+30 1.82969604e+30]]\n" + ] + } + ], "source": [ "import numpy as np\n", "# SVD inversion\n", @@ -433,23 +390,22 @@ " return np.matmul(V,np.matmul(invD,UT))\n", "\n", "\n", - "#X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n", + "X = np.array([ [1.0, -1.0, 2.0], [1.0, 0.0, 1.0], [1.0, 2.0, -1.0], [1.0, 1.0, 0.0] ])\n", "# Non-singular square matrix\n", - "X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n", + "#X = np.array( [ [1,2,3],[2,4,5],[3,5,6]])\n", "print(X)\n", "A = np.transpose(X) @ X\n", "# Brute force inversion\n", - "B = np.linalg.inv(A) # here we could use np.linalg.pinv(A)\n", + "B = np.linalg.pinv(A) # here we could use np.linalg.pinv(A)\n", "C = SVDinv(A)\n", + "print(B)\n", "print(np.abs(B-C))" ] }, { "cell_type": "markdown", "id": "73f83a23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Inverse of Rectangular Matrix\n", "\n", @@ -467,9 +423,7 @@ { "cell_type": "markdown", "id": "880eb333", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}_{\\mathrm{PI}}= \\boldsymbol{V}\\boldsymbol{D}_{\\mathrm{PI}}\\boldsymbol{U}^T,\n", @@ -479,22 +433,32 @@ { "cell_type": "markdown", "id": "149c1834", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{D}_{\\mathrm{PI}}$ can be calculated by creating a diagonal matrix from $\\boldsymbol{\\Sigma}$ where we only keep the singular values (the non-zero values). The following code computes the pseudoinvers of the matrix based on the SVD." ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 6, "id": "48111712", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[0.3 0.4]\n", + " [0.5 0.6]\n", + " [0.7 0.8]\n", + " [0.9 1. ]]\n", + "[[-13. -6. 1. 8. ]\n", + " [ 11.5 5.5 -0.5 -6.5]]\n", + "[[0. 0. 0. 0.]\n", + " [0. 0. 0. 0.]]\n" + ] + } + ], "source": [ "import numpy as np\n", "# SVD inversion\n", @@ -524,9 +488,7 @@ { "cell_type": "markdown", "id": "fc85c135", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "As you can see from this example, our own decomposition based on the SVD agrees the pseudoinverse algorithm provided by **Numpy**." ] @@ -534,9 +496,7 @@ { "cell_type": "markdown", "id": "acc5f27e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and LASSO Regression\n", "\n", @@ -547,9 +507,7 @@ { "cell_type": "markdown", "id": "ef91a46f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -559,9 +517,7 @@ { "cell_type": "markdown", "id": "494f0101", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or we can state it as" ] @@ -569,9 +525,7 @@ { "cell_type": "markdown", "id": "e6a6cb0d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -582,9 +536,7 @@ { "cell_type": "markdown", "id": "7d3cc66b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used the definition of a norm-2 vector, that is" ] @@ -592,9 +544,7 @@ { "cell_type": "markdown", "id": "9c9070d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -604,9 +554,7 @@ { "cell_type": "markdown", "id": "b78801a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## From OLS to Ridge and Lasso\n", "\n", @@ -619,9 +567,7 @@ { "cell_type": "markdown", "id": "87ee54c6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -632,9 +578,7 @@ { "cell_type": "markdown", "id": "38bc193b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -644,9 +588,7 @@ { "cell_type": "markdown", "id": "a45c66ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -656,9 +598,7 @@ { "cell_type": "markdown", "id": "caa621d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have a new optimization equation" ] @@ -666,9 +606,7 @@ { "cell_type": "markdown", "id": "66de1412", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -679,9 +617,7 @@ { "cell_type": "markdown", "id": "1926d209", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -691,9 +627,7 @@ { "cell_type": "markdown", "id": "816037d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -703,9 +637,7 @@ { "cell_type": "markdown", "id": "4e126790", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving the Ridge Regression Equations\n", "\n", @@ -715,9 +647,7 @@ { "cell_type": "markdown", "id": "2fdd9447", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n", @@ -727,9 +657,7 @@ { "cell_type": "markdown", "id": "b372151b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n", @@ -741,9 +669,7 @@ { "cell_type": "markdown", "id": "af777550", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -753,9 +679,7 @@ { "cell_type": "markdown", "id": "5bc74cfb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\boldsymbol{I}$ being a $p\\times p$ identity matrix with the constraint that" ] @@ -763,9 +687,7 @@ { "cell_type": "markdown", "id": "c9f98734", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i=0}^{p-1} \\beta_i^2 \\leq t,\n", @@ -775,9 +697,7 @@ { "cell_type": "markdown", "id": "3ed7d5f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $t$ a finite positive number." ] @@ -785,9 +705,7 @@ { "cell_type": "markdown", "id": "0c76382e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Note on Scikit-Learn\n", "\n", @@ -797,9 +715,7 @@ { "cell_type": "markdown", "id": "f6b38206", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+n\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -809,9 +725,7 @@ { "cell_type": "markdown", "id": "d79b6a4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In our codes where we compare our own codes with **Scikit-Learn**, we do thus not include the $1/n$ factor in the cost function." ] @@ -819,9 +733,7 @@ { "cell_type": "markdown", "id": "098d6225", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Comparison with OLS\n", "When we compare this with the ordinary least squares result we have" @@ -830,9 +742,7 @@ { "cell_type": "markdown", "id": "a7396d95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n", @@ -842,9 +752,7 @@ { "cell_type": "markdown", "id": "e8bf0330", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n", "\n", @@ -858,9 +766,7 @@ { "cell_type": "markdown", "id": "1b2c6733", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SVD analysis\n", "\n", @@ -871,9 +777,7 @@ { "cell_type": "markdown", "id": "1d2e7543", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n", @@ -883,9 +787,7 @@ { "cell_type": "markdown", "id": "0140b01d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For Ridge regression this becomes" ] @@ -893,9 +795,7 @@ { "cell_type": "markdown", "id": "bbef0231", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n", @@ -905,9 +805,7 @@ { "cell_type": "markdown", "id": "ddc14a15", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$." ] @@ -915,9 +813,7 @@ { "cell_type": "markdown", "id": "3d394ec0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpreting the Ridge results\n", "\n", @@ -927,9 +823,7 @@ { "cell_type": "markdown", "id": "feafeed5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda} \\leq 1.\n", @@ -939,9 +833,7 @@ { "cell_type": "markdown", "id": "33ce5825", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Ridge regression finds the coordinates of $\\boldsymbol{y}$ with respect to the\n", "orthonormal basis $\\boldsymbol{U}$, it then shrinks the coordinates by\n", @@ -955,9 +847,7 @@ { "cell_type": "markdown", "id": "274c480d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More interpretations\n", "\n", @@ -967,9 +857,7 @@ { "cell_type": "markdown", "id": "b2e1d024", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}=(\\boldsymbol{X}^T\\boldsymbol{X})^{-1} =\\boldsymbol{I}.\n", @@ -979,9 +867,7 @@ { "cell_type": "markdown", "id": "d483b952", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In this case the standard OLS results in" ] @@ -989,9 +875,7 @@ { "cell_type": "markdown", "id": "e6d5928e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{OLS}} = \\boldsymbol{X}^T\\boldsymbol{y}=\\sum_{i=0}^{n-1}\\boldsymbol{u}_i\\boldsymbol{u}_i^T\\boldsymbol{y},\n", @@ -1001,9 +885,7 @@ { "cell_type": "markdown", "id": "7f742759", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1011,9 +893,7 @@ { "cell_type": "markdown", "id": "178ba1fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{I}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\left(1+\\lambda\\right)^{-1}\\boldsymbol{\\beta}^{\\mathrm{OLS}},\n", @@ -1023,9 +903,7 @@ { "cell_type": "markdown", "id": "cd132b07", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "that is the Ridge estimator scales the OLS estimator by the inverse of a factor $1+\\lambda$, and\n", "the Ridge estimator converges to zero when the hyperparameter goes to\n", @@ -1040,9 +918,7 @@ { "cell_type": "markdown", "id": "94d79072", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving the Lasso Regression Equations\n", "\n", @@ -1052,9 +928,7 @@ { "cell_type": "markdown", "id": "5540bb11", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -1064,9 +938,7 @@ { "cell_type": "markdown", "id": "12370aea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)" ] @@ -1074,9 +946,7 @@ { "cell_type": "markdown", "id": "c230a75c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\ 0 & \\beta =0\\\\-1 & \\beta < 0, \\end{array}\\right.\n", @@ -1086,9 +956,7 @@ { "cell_type": "markdown", "id": "73d88740", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have that the derivative of the cost function is" ] @@ -1096,9 +964,7 @@ { "cell_type": "markdown", "id": "162ec0ee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n", @@ -1108,9 +974,7 @@ { "cell_type": "markdown", "id": "d6f9d09d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and reordering we have" ] @@ -1118,9 +982,7 @@ { "cell_type": "markdown", "id": "65420725", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1130,9 +992,7 @@ { "cell_type": "markdown", "id": "122ad869", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later." ] @@ -1140,9 +1000,7 @@ { "cell_type": "markdown", "id": "e69bf6f3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression\n", "\n", @@ -1155,9 +1013,7 @@ { "cell_type": "markdown", "id": "3d3d54d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2,\n", @@ -1167,9 +1023,7 @@ { "cell_type": "markdown", "id": "e29ddb4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -1177,9 +1031,7 @@ { "cell_type": "markdown", "id": "08653ba9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{OLS}} = y_i.\n", @@ -1189,9 +1041,7 @@ { "cell_type": "markdown", "id": "37ecb550", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge Regression\n", "\n", @@ -1201,9 +1051,7 @@ { "cell_type": "markdown", "id": "f7d27ca7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,\n", @@ -1213,9 +1061,7 @@ { "cell_type": "markdown", "id": "24344af4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -1223,9 +1069,7 @@ { "cell_type": "markdown", "id": "b17ea824", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\beta}_i^{\\mathrm{Ridge}} = \\frac{y_i}{1+\\lambda}.\n", @@ -1235,9 +1079,7 @@ { "cell_type": "markdown", "id": "63a47ac8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso Regression\n", "\n", @@ -1247,9 +1089,7 @@ { "cell_type": "markdown", "id": "6ca7c397", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\vert\\beta_i\\vert=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\sqrt{\\beta_i^2},\n", @@ -1259,9 +1099,7 @@ { "cell_type": "markdown", "id": "0809d498", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and minimizing we have that" ] @@ -1269,9 +1107,7 @@ { "cell_type": "markdown", "id": "f50e056d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-2\\sum_{i=0}^{p-1}(y_i-\\beta_i)+\\lambda \\sum_{i=0}^{p-1}\\frac{(\\beta_i)}{\\vert\\beta_i\\vert}=0,\n", @@ -1281,9 +1117,7 @@ { "cell_type": "markdown", "id": "11bc185e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to" ] @@ -1291,9 +1125,7 @@ { "cell_type": "markdown", "id": "3b3490a3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n", @@ -1305,9 +1137,7 @@ { "cell_type": "markdown", "id": "a9a67944", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Plotting these results ([figure in handwritten notes for week 36](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2021/NotesSeptember9.pdf)) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$." ] @@ -1315,9 +1145,7 @@ { "cell_type": "markdown", "id": "52ed41b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Yet another Example\n", "\n", @@ -1327,9 +1155,7 @@ { "cell_type": "markdown", "id": "5f8eb7f7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}=\\begin{bmatrix}4 \\\\ 2 \\\\3\\end{bmatrix},\n", @@ -1339,9 +1165,7 @@ { "cell_type": "markdown", "id": "34ffb73c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and our inputs as a $3\\times 2$ design matrix" ] @@ -1349,9 +1173,7 @@ { "cell_type": "markdown", "id": "97e0f59e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n", @@ -1361,9 +1183,7 @@ { "cell_type": "markdown", "id": "f0f297a7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "meaning that we have two features and two unknown parameters $\\beta_0$ and $\\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression." ] @@ -1371,9 +1191,7 @@ { "cell_type": "markdown", "id": "8f7b4cac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The OLS case\n", "\n", @@ -1383,9 +1201,7 @@ { "cell_type": "markdown", "id": "920b0eea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1395,9 +1211,7 @@ { "cell_type": "markdown", "id": "9a299a7f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Inserting the above values we obtain that" ] @@ -1405,9 +1219,7 @@ { "cell_type": "markdown", "id": "3b4667c6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n", @@ -1417,9 +1229,7 @@ { "cell_type": "markdown", "id": "00475f1e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The code which implements this simpler case is presented after the discussion of Ridge and Lasso." ] @@ -1427,9 +1237,7 @@ { "cell_type": "markdown", "id": "44cf98fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Ridge case\n", "\n", @@ -1439,9 +1247,7 @@ { "cell_type": "markdown", "id": "54ef4ca0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\left( \\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n", @@ -1451,9 +1257,7 @@ { "cell_type": "markdown", "id": "dd900f61", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Inserting the above values we obtain that" ] @@ -1461,9 +1265,7 @@ { "cell_type": "markdown", "id": "7d17594b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n", @@ -1473,9 +1275,7 @@ { "cell_type": "markdown", "id": "71397dfd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "There is normally a constraint on the value of $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2$ via the parameter $\\lambda$.\n", "Let us for simplicity assume that $\\beta_0^2+\\beta_1^2=1$ as constraint. This will allow us to find an expression for the optimal values of $\\beta$ and $\\lambda$.\n", @@ -1486,9 +1286,7 @@ { "cell_type": "markdown", "id": "fce415df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Writing the Cost Function\n", "\n", @@ -1498,9 +1296,7 @@ { "cell_type": "markdown", "id": "eef55634", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n", @@ -1510,9 +1306,7 @@ { "cell_type": "markdown", "id": "e24cbcf8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n", @@ -1522,9 +1316,7 @@ { "cell_type": "markdown", "id": "b396bfad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and taking the derivative with respect to $\\beta_0$ we get" ] @@ -1532,9 +1324,7 @@ { "cell_type": "markdown", "id": "15a48b85", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0=\\frac{8}{4+\\lambda},\n", @@ -1544,9 +1334,7 @@ { "cell_type": "markdown", "id": "c43dd748", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and for $\\beta_1$ we obtain" ] @@ -1554,9 +1342,7 @@ { "cell_type": "markdown", "id": "d485f754", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_1=\\frac{2}{1+\\lambda},\n", @@ -1566,9 +1352,7 @@ { "cell_type": "markdown", "id": "39381345", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving" ] @@ -1576,9 +1360,7 @@ { "cell_type": "markdown", "id": "7315dc73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n", @@ -1588,9 +1370,7 @@ { "cell_type": "markdown", "id": "6780cca8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$." ] @@ -1598,9 +1378,7 @@ { "cell_type": "markdown", "id": "0ccd5a7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso case\n", "\n", @@ -1611,9 +1389,7 @@ { "cell_type": "markdown", "id": "291adfaf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n", @@ -1623,9 +1399,7 @@ { "cell_type": "markdown", "id": "b18fc94b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-4(4-2\\beta_0)+\\lambda\\mathrm{sgn}(\\beta_0)=0,\n", @@ -1635,9 +1409,7 @@ { "cell_type": "markdown", "id": "a1f7fe6f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1645,9 +1417,7 @@ { "cell_type": "markdown", "id": "10b0cf2a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}(\\beta_1)=0.\n", @@ -1657,9 +1427,7 @@ { "cell_type": "markdown", "id": "b475e47e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n", "1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n", @@ -1674,9 +1442,7 @@ { "cell_type": "markdown", "id": "ee8e912a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The first Case\n", "\n", @@ -1686,9 +1452,7 @@ { "cell_type": "markdown", "id": "c599e318", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-4(4-2\\beta_0)+\\lambda=0,\n", @@ -1698,9 +1462,7 @@ { "cell_type": "markdown", "id": "13b0d0ab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1708,9 +1470,7 @@ { "cell_type": "markdown", "id": "6cfa2770", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "-2(2-\\beta_1)+\\lambda=0.\n", @@ -1720,9 +1480,7 @@ { "cell_type": "markdown", "id": "a602fc93", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which yields" ] @@ -1730,9 +1488,7 @@ { "cell_type": "markdown", "id": "d29afe49", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0=\\frac{16+\\lambda}{8},\n", @@ -1742,9 +1498,7 @@ { "cell_type": "markdown", "id": "b6f0aa6e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1752,9 +1506,7 @@ { "cell_type": "markdown", "id": "0e0478c7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_1=\\frac{4+\\lambda}{2}.\n", @@ -1764,9 +1516,7 @@ { "cell_type": "markdown", "id": "bb1ed229", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using the constraint on $\\beta_0$ and $\\beta_1$ we can then find the optimal value of $\\lambda$ for the different cases. We leave this as an exercise to you." ] @@ -1774,9 +1524,7 @@ { "cell_type": "markdown", "id": "505e3984", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple code for solving the above problem\n", "\n", @@ -1787,13 +1535,32 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 7, "id": "aa938fb6", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[2. 2.]\n", + "Training MSE for OLS\n", + "3.0\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjgUlEQVR4nO3daZhU1bn28f8DIs0kyOTAIBiVIAgILYoomggiwkFFI4hDEBETJzxOUWM0CfFED573KEQxGlSMiEQRRASiHkQxERkEEYWoEAcEsQFBQGae98OqHmi66YHetaur7t911dVVtXbVvqHph9Vrr1rL3B0REUk/VeIOICIi0VCBFxFJUyrwIiJpSgVeRCRNqcCLiKSpg+IOUFDDhg29RYsWcccQEak0FixYsNbdGxXVllIFvkWLFsyfPz/uGCIilYaZfVFcm4ZoRETSlAq8iEiaUoEXEUlTKTUGX5SdO3eycuVKtm3bFncUOUBZWVk0bdqUatWqxR1FJCOkfIFfuXIlderUoUWLFphZ3HGknNyddevWsXLlSlq2bBl3HJGMkPJDNNu2baNBgwYq7pWcmdGgQQP9JiaSRClf4AEV9zSh76NIcqX8EI2ISKx++AG2bIFt22DrVti+HapXh+OOC+3vvAMbNsDOnfm3ww+Hs84K7WPHwsaNsHt3/q1VKzj//ND+xz/C1VdDw4YVn93dU+bWqVMnL+zjjz/e57lkA/zSSy/Ne7xz505v2LCh9+7d293dv/nmG+/du7e3a9fOW7du7b169XJ393//+9+elZXl7du3z7uNHTt2n/c/44wz/LjjjvN27dp5dna2L1y4MK+tV69e/t133+3zmnvvvddHjBhxQH+utWvX5uU67LDD/Mgjj8x7vH379hJfP2/ePL/hhhvKdM5U+H5Khlm/3v3DD91nznSfMMH9kUfcH300v/2ee9y7d3fv3Nm9dWv3Zs3cu3XLb+/c2R32vp16an778cfv296zZ357s2b7tl90UX57vXruB/BzAcz3YmqqevClUKtWLZYsWcLWrVupUaMGr7/+Ok2aNMlrv+eee+jRowfDhg0DYPHixXltP/rRj1i0aFGJ5xg3bhzZ2dk89dRT3Hbbbbz++usATJs2rWL/MAU0aNAgL9tvf/tbateuza233rrXMbt27eKgg4r+Z5KdnU12dnZk+URKZc0a+Ogj+OwzWLECvvgi9Jhzf3aGDoUXX9z7NYcdBr/8Zbi/fn3ooR96KDRvDrVrQ8GJAP/5n7B2LWRl5d8aN85vf+452LEDqlXLv9Wpk9++cGH4WrVq/q3gz9T69RDR8KUKfCmde+65vPrqq1x00UWMHz+eSy65hNmzZwOwevVqzj777Lxj27VrV+7zdOnShREjRuQ9zl2+oWHDhtx3332MHTuWxo0b06xZMzp16gTAvHnzuOqqq6hSpQo9evRg+vTpLFmyhN27d3PHHXcwa9Ystm/fznXXXcc111xTYoZBgwaRlZXFwoUL6dq1KwMGDGDYsGFs27aNGjVq8NRTT9GqVStmzZrFgw8+yNSpU/ntb3/Ll19+yYoVK/jyyy+56aabuPHGG8v99yCyjy1bQrGcPx8WL4bRo8NQyf33w0MPhWMOPhiaNYOjjoJdu0Ihvf56uOgiaNQoFOaGDaF+/fz3HTVq/+cdMGD/7e3b77+9QYP9t0d4baryFfgzz9z3uYsvhmuvDWNl5567b/ugQeG2dm34Rhc0a1apTjtgwAB+//vf06dPHxYvXszgwYPzCvx1111H//79+dOf/kT37t258sorOfLIIwFYvnw5HTp0yHufUaNGcfrppxd7nhkzZnB+7thcAQsWLOD5559n0aJF7Nq1i44dO+YV+CuvvJInnniCLl26cMcdd+S9ZsyYMdStW5d58+axfft2unbtytlnn12qaYorV67kn//8J1WrVuX7779n9uzZHHTQQbzxxhvcddddTJw4cZ/XLFu2jDfffJNNmzbRqlUrfvnLX2rOuxy4adPgzjthyRLYsyc8d9hhsGpV6GkPGQL/8R9w7LHQpAlUKTR35Iwzkp85RVS+Ah+Tdu3a8fnnnzN+/HjOLfSfSM+ePVmxYgUzZsxg+vTpnHjiiSxZsgQo/RDNpZdeyo4dO9i8eXORx8+ePZsLLriAmjVrAtC3b18ANmzYwKZNm+jSpQsAAwcOZOrUqQC89tprLF68mBcTv55u3LiRTz/9tFQF/mc/+xlVq1bNe93Pf/5zPv30U8yMnTt3Fvma3r17U716dapXr07jxo1Zs2YNTZs2LfFcIkAYnV60CF55BV59NVx8/OlPw3BH48Zw991w0kmQnR0uYuZq0ybcZB+Vr8Dvr8dds+b+2xs2LHWPvSh9+/bl1ltvZdasWaxbt26vtvr16zNw4EAGDhxInz59ePvtt/N62KUxbtw4OnXqxG233cYNN9zASy+9VO6cudydUaNG0bNnzzK/tlatWnn3f/Ob3/CTn/yESZMm8fnnn3NmUb9FAdWrV8+7X7VqVXbt2lXm80oG2rwZHnwwjGV/+mkYsujcOQyxAJx+OiSuSUnZVIp58Kli8ODB3HvvvZxwwgl7PT9z5kx++OEHADZt2sTy5ctp3rx5md/fzBg+fDhz5sxh2bJle7V169aNyZMns3XrVjZt2sQrr7wCQL169ahTpw7vvfceAM8//3zea3r27Mno0aPzetyffPIJW7ZsKXOujRs35l1Ufvrpp8v8epF97NoVijmEcfTHHoOmTeHxx+Gbb2DOHChwXUvKRwW+DJo2bVrkhcMFCxaQnZ1Nu3bt6NKlC0OGDOGkk04C8sfgc28jR47c7zlq1KjBLbfcsteFVoCOHTvSv39/2rdvT69evfLeH8JY+9VXX02HDh3YsmULdevWBWDIkCEcf/zxdOzYkbZt23LNNdeUq1d9++23c+edd3LiiSeqVy4HZutWGDkyjJ137x7mhFerBsuXw8yZYT54wRkqckAsTKNMDdnZ2V54w4+lS5fSunXrmBJVDps3b6Z27doA3H///axevZqHH3445lRF0/czQ23fHnrp998feujdusEtt0Dv3mHaoJSbmS1w9yLnK1e+MXjZx6uvvsof//hHdu3axVFHHaVhFEk9r70GN90UZsGNH1/0bDipcCrwaaB///70798/7hgie/viC5g7F372M+jTB959F045Je5UGaVSjMGn0jCSlJ++jxnCHf76V2jXLnzIaNu2MDNGxT3pUr7AZ2VlsW7dOhWHSs4T68FnZWXFHUWi9N134ZOfV1wBJ5wQZsPoex6blB+iadq0KStXriQnJyfuKHKAcnd0kjS1ZQucfDL8+9/wX/8Ft9+uC6gxS/kCX61aNe0AJFIZ1KoVFvY69dRwk9ilfIEXkRQ3enQYjjntNCi0GqnEK+XH4EUkhT34YFjo78kn404iRYi0wJvZMDNbYmYfmdlNUZ5LRJLsD3+A224Lq7n++c9xp5EiRFbgzawtcDXQGWgP9DGzY6I6n4gk0fDh8JvfwOWXw7hxYbkBSTlR9uBbA++5+w/uvgt4C+gX4flEJBn27IEPPghTIZ9+eu/diSSlRPmdWQLcZ2YNgK3AucD8wgeZ2VBgKFCuFRhFJIncw4YaEyaEQl94cw1JKZF9d9x9KfAA8BowA1gE7C7iuMfdPdvdsxs1ahRVHBE5UAsWhOmPK1eG+e0alkl5kf736+5j3L2Tu3cDvgM+ifJ8IhKRtWvh/PNh9Wp9eKkSiXTwzMwau/u3ZtacMP6uxShEKps9e+CyyyAnJywYdsQRcSeSUor66sjExBj8TuA6d98Q8flEpKLddx/8/e9hKuSJJ8adRsog0gLv7qdH+f4iErEdO2DSpNCDv/rquNNIGWl+k4gU7+CD4R//CMM0ZnGnkTLSHCcRKdrYsbBxI9SoERYSk0pHBV5E9jV9OgwaBI88EncSOQAq8CKyt40bw7K/xx8fNsaWSktj8CKyt9tvh1WrYOJEqF497jRyANSDF5F88+fDE0/ATTdB585xp5EDpAIvIvnq1w9TIu+9N+4kUgE0RCMi+Y4+Gp55Ju4UUkHUgxcR+OEHGDIEli+PO4lUIBV4EYH//m8YMyZcXJW0oQIvkunWrAkF/uKL4XStLpJOVOBFMt0DD4Q1Z/7wh7iTSAVTgRfJZKtWwejRYW/VY4+NO41UMM2iEclkWVlw3XVw7bVxJ5EIqMCLZLL69eHBB+NOIRHREI1Ipvrf/4U33og7hURIBV4kE61aBb/6Fbz0UtxJJEIq8CKZaORI2L0bbr017iQSIRV4kUyzaRM89hhceGFYmkDSlgq8SKYZMyas+a613tNepAXezP7TzD4ysyVmNt7MsqI8n4iUQr16cMklcPLJcSeRiEVW4M2sCXAjkO3ubYGqwICozicipTRoEDz3XNwpJAmiHqI5CKhhZgcBNQGtZCQSF3eYNAm2b487iSRJZAXe3b8GHgS+BFYDG939tajOJyIlePdd6NcPnn027iSSJFEO0RwKnAe0BI4EapnZZUUcN9TM5pvZ/JycnKjiiMjo0VCnDvTvH3cSSZIoh2i6A/929xx33wm8BJxa+CB3f9zds909u1GjRhHGEclga9fCCy/AFVdA7dpxp5EkibLAfwmcYmY1zcyAs4ClEZ5PRIozdmwYe//FL+JOIkkU5Rj8e8CLwPvAh4lzPR7V+URkP2bPhtNOg7Zt404iSRTpapLufi+g7dlF4jZpEnz3XdwpJMn0SVaRdLdjB5iFpYElo6jAi6Sz1avh8MNDD14yjgq8SDp79tkwNNOmTdxJJAYq8CLpyh2efhq6dIHjjos7jcRABV4kXS1YAB9/HNaekYykAi+Srp5+OmyqffHFcSeRmGjTbZF0NWQIZGeH5YElI6nAi6SrDh3CTTKWhmhE0tHIkTBnTtwpJGYq8CLpZs0auPlmmDw57iQSMxV4kXTzwguwezdcfnncSSRmKvAi6Wb8eDjhBH24SVTgRdLKF1/AP/8ZNtWWjKcCL5JOPvkEGjfWrk0CaJqkSHrp0QNWrYKqVeNOIilAPXiRdLF1K+zZo+IueVTgRdLFAw/AMceEQi+CCrxIenAPs2datIAaNeJOIylCBV4kHSxaFC6wDhgQdxJJISrwIunghRfC2Hu/fnEnkRSiAi9S2bnDiy/CT34CDRvGnUZSSGTTJM2sFTChwFNHA/e4+0NRnVMkI7nDgw9C7dpxJ5EUE1mBd/d/AR0AzKwq8DWgnX9FKlqVKtC3b9wpJAUla4jmLGC5u3+RpPOJZI6RI+HTT+NOISkoWZ9kHQCML6rBzIYCQwGaN2+epDgiaWLZMhg2DMzg2GPjTiMpJvIevJkdDPQFXiiq3d0fd/dsd89u1KhR1HFE0svEieGrZs9IEZIxRNMLeN/d1yThXCKZZeJEOPVUaNIk7iSSgpJR4C+hmOEZETkAy5fDwoVw4YVxJ5EUFWmBN7NaQA/gpSjPI5KRPvgAsrI0PCPFivQiq7tvARpEeQ6RjNWvH6xfr7VnpFj6JKtIZeQevqq4y36owItURmPGQOfOsG5d3EkkhanAi1RGkybB2rVQv37cSSSFqcCLVDabNsEbb8D554cPOIkUQwVepLKZPh127IALLog7iaQ4FXiRymby5LAs8Kmnxp1EUtx+C7yZXVbgftdCbddHFUpE9qNXL7j7bm2uLSUqqQd/c4H7owq1Da7gLCJSGpdfHhYYEylBSQXeirlf1GMRido778Dq1XGnkEqipALvxdwv6rGIRMk9bKp97bVxJ5FKoqSlCn5sZosJvfUfJe6TeHx0pMlEZG/vvw9ffw1/+EPcSaSSKKnAt05KChEp2ZQpYXu+3r3jTiKVxH4LfOEt9sysAdAN+NLdF0QZTEQKmTIlTI3UxjhSSiVNk5xqZm0T948AlhBmz/zVzG6KPp6IAOHC6qJFcN55cSeRSqSkIZqW7r4kcf9K4HV3v8LM6gD/AB6KMpyIJBxxBHz1VVj/XaSUSirwOwvcPwt4AsDdN5nZnshSici+mjaNO4FUMiVNk/zKzG4wswuAjsAMADOrAVSLOpyIABs3hoXF5s6NO4lUMiUV+KuANsAgoL+7b0g8fwrwVHSxRCTPjBnw8suwc2fJx4oUUNIsmm+BXxTx/JvAm1GFEpECpkwJM2dOOSXuJFLJ7LfAm9mU/bW7e9+KjSMie9m5E6ZNC0sDa3ExKaOSLrJ2Ab4CxgPvofVnRJJr9mzYsAH6qi8lZVfSGPzhwF1AW+BhoAew1t3fcve3SnpzM6tnZi+a2TIzW2pmXQ48skgG2bEDTj4ZevSIO4lUQvst8O6+291nuPvPCRdWPwNmlWEt+IeBGe7+Y6A9sPSA0opkmnPOgTlzoFatuJNIJVTSEA1mVh3oDVwCtABGApNK8bq6hGUNBgG4+w5gR/mjimSY77+HatWgRo24k0glVdJSBc8A7xLmwP/O3U9y9+Hu/nUp3rslkAM8ZWYLzewvZrZPN8TMhprZfDObn5OTU54/g0h6GjUKGjcOhV6kHMy9+GXdE59W3ZJ4WPBAA9zdD9nPa7OBOUBXd3/PzB4Gvnf33xT3muzsbJ8/f35Z8oukr5NPBrMwRCNSDDNb4O7ZRbWVNAZfxd3rJG6HFLjV2V9xT1gJrHT39xKPXyT8JiAiJVm9OnxyVbNn5ACUNIum3Nz9G8JSB60ST50FfBzV+UTSyiuvhK8q8HIASrzIeoBuAMaZ2cHACsKKlCJSkpdfhpYtoU2buJNIJRZpgXf3RUCRY0Mish/Dh8OaNWEMXqScou7Bi0h5dNTlKjlwkY3Bi0g5PfkkzJwZdwpJAyrwIqlk50645RZ45pm4k0gaUIEXSSVvvx0WFzv//LiTSBpQgRdJJZMnh6UJzj477iSSBlTgRVKFe5ge2aMH1KwZdxpJAyrwIqli1SrYtUvDM1JhNE1SJFU0aQIrV8Lu3XEnkTShAi+SKtyhSpVwE6kA+pckkgo++QSaNYO3StwoTaTUVOBFUsHEifD113D00XEnkTSiAi+SCiZOhM6dQy9epIKowIvE7YsvYMECuPDCuJNImlGBF4nbSy+Fr/36xZtD0o4KvEjcTjkF7r4bjjkm7iSSZjRNUiRuXbqEm0gFUw9eJE5z58KiRWEOvEgFUw9eJE533hk+vbpsWdxJJA2pBy8Sl2++gVmzoH9/bc0nkVCBF4nLiy/Cnj2hwItEINIhGjP7HNgE7AZ2ubs24BbJNWECtG0LbdrEnUTSVDLG4H/i7muTcB6RymPDBli4MIzBi0REF1lF4lCvHqxZE9Z/F4lI1GPwDrxmZgvMbGjE5xKpXGrVgrp1404haSzqAn+au3cEegHXmVm3wgeY2VAzm29m83NyciKOI5ICli+HE08Mc+BFIhRpgXf3rxNfvwUmAZ2LOOZxd8929+xGjRpFGUckNUyYED7cdPjhcSeRNBdZgTezWmZWJ/c+cDawJKrziVQK7vDcc3DqqdC8edxpJM1FeZH1MGCShQ9wHAQ85+4zIjyfSOpbtAg++ghGj447iWSAyAq8u68A2kf1/iKV0tixcPDBcPHFcSeRDKBpkiLJ1K0bNGwI9evHnUQygAq8SDL166eNPSRptBaNSLK8/DKsXh13CskgKvAiybB+fRh3f+CBuJNIBlGBF0mGCRNgxw74+c/jTiIZRAVeJBmefhpOOAE6dIg7iWQQFXiRqC1aFJYlGDxYG3tIUqnAi0TtnXegZk244oq4k0iGUYEXidr114d9VzX3XZJMBV4kStu3h6+HHhpvDslIKvAiUXEPi4rdfHPcSSRDqcCLRGXuXHj/fTjuuLiTSIZSgReJymOPQe3acOmlcSeRDKUCLxKFnBx4/vlQ3OvUiTuNZCgVeJEoPPoobNsGw4bFnUQymFaTFInCtdfCMcdA69ZxJ5EMph68SBQaNdLYu8ROBV6kIu3eDZdcArNmxZ1ERAVepEJNmRIurq5dG3cSERV4kQrjDiNGQMuWcMEFcacR0UVWkQrzf/8H774Lf/oTVK0adxqR6HvwZlbVzBaa2dSozyUSG3e4+25o1gyGDIk7jQiQnB78MGApcEgSziUSjz17YNCgsKhY9epxpxEBIi7wZtYU6A3cB2jFJUlfVavCL34RdwqRvUQ9RPMQcDuwp7gDzGyomc03s/k5OTkRxxGJwLRpMHo07NoVdxKRvURW4M2sD/Ctuy/Y33Hu/ri7Z7t7dqNGjaKKIxKN3bvh1lth1ChtxycpJ8ohmq5AXzM7F8gCDjGzZ939sgjPKZJcf/4zLF0KEydq5oyknMh68O5+p7s3dfcWwABgpoq7pJWcHPj1r+GsszTvXVKSPugkUl533QWbN8PIkRqekZSUlA86ufssYFYyziWSNP37Q5s2cPzxcScRKZI+ySpSXt27h5tIitIQjUhZPfQQ3HabpkVKylOBFymLJUvgV7+Czz7TrBlJeSrwIqW1fTtcdhnUqwePP64Lq5LyNAYvUlr33AMffACvvBJ2bBJJcerBi5TG2rXwyCMwdCj06RN3GpFSUQ9epDQaNoR33gkbaYtUEurBi+zPhg0wYUK436ED1K4dZxqRMlGBFynOrl0wYABcfjmsWBF3GpEy0xCNSFF274Yrr4S//x2eeAKOPjruRCJlph68SGG5xf3ZZ+G++7QFn1RaKvAihc2aBX/9KwwfHhYUE6mkNEQjkmvPHqhSJSz/O28eZGfHnUjkgKgHLwLw+edw8skwe3Z4rOIuaUAFXuT116FzZ/j007AcgUiaUIGXzPX993DNNXD22dCgAbz3npb/lbSiAi+Za9w4+MtfwqbZ778PrVrFnUikQukiq2SWt96C9evDHqpDh0KXLuETqiJpSD14SX87dsDf/gZnnAFnnhnmtruH9dxV3CWNqcBLevvb3+Coo8L+qV99Bf/zP2GmjNZylwygIRpJH6tWwdtvh2GYG2+E1q2hVi3o1AmuvRZ69tQuTJJRIivwZpYFvA1UT5znRXe/N6rzSQbZtAl27oT69UNRv+46WLQozGUHqFMnDMe0bg29e4ebSAaKsge/Hfipu282s2rAO2Y23d3nRHhOSWXuYYXGHTvC41q1wtd//Qs2b4YffgjF+/vvoVkz6No1fLr0qqvgm29CMf/6a1i3Dm6/HR54AOrWDa8/+WS44YZQ2Nu3h4P0y6lIZD8F7u7A5sTDaombR3KyDz8My7oWNmIEnHtumN88ePC+7Y8+GgrCzJmhOBQ2dmz4ROPUqWGj5cJefDH0EidMgN//ft/2adPC+O+TT4ax38LeeitsJDFqFDz22L7t8+ZBzZpw//3wzDN7t5nBRx+F+/fcE7IUVLs2zJ0b7t9yC0yfHgpsrsaNw/khzAXPvZ97TMuWMGNGuD9wYHgv9/z2du1g8uRw/5xzwvdgz57Qvns3nHYaTJoU2tu0gU8+CcU913nn5b/+9NMhJ2fv/AMHhgJfpQq8+y4ccgi0aAGnnhq+dusWjqtVCz7+eN+/OxGJdgzezKoCC4BjgEfc/b0ijhkKDAVo3rx5+U5UowYcf/y+z9etG77WrFl0e+7mDXXqFN1es2b++xTVnpUVvh56aNHtBx8cvjZoUHR77nhw48ZFt1dJXAM/4gho23bvtoIXCZs02be9Ro38+82a5bfnvu7QQ/PbW7aEjRv3fu8jj8x//OMfh+dyb7D38rlduoRz5LZXrbr3nPJBg8LGGdWq5d8Kto8Zk5/5kEPC96Nhw/z2ZcsQkbIz92g61XudxKweMAm4wd2XFHdcdna2z58/P/I8IiLpwswWuHuRiyclZZqku28A3gTOScb5REQkwgJvZo0SPXfMrAbQA9Dv2iIiSRLlGPwRwNjEOHwV4G/uPjXC84mISAFRzqJZDJwY1fuLiMj+aakCEZE0pQIvIpKmVOBFRNKUCryISJpKygedSsvMcoAvyvnyhsDaCoxTUZSrbJSrbJSrbNIx11Hu3qiohpQq8AfCzOYX92muOClX2ShX2ShX2WRaLg3RiIikKRV4EZE0lU4F/vG4AxRDucpGucpGucomo3KlzRi8iIjsLZ168CIiUoAKvIhImkrLAm9mt5iZm1nDko+OnpkNN7PFZrbIzF4zsyNLflX0zGyEmS1LZJuUu7xz3MzsZ2b2kZntMbNYp7SZ2Tlm9i8z+8zM7ogzS0Fm9qSZfWtmxW6gEwcza2Zmb5rZx4nv4bC4MwGYWZaZzTWzDxK5fhd3plxmVtXMFppZha+2m3YF3syaAWcDX8adpYAR7t7O3TsAU4F7Ys6T63Wgrbu3Az4B7ow5T64lQD/g7ThDJJa6fgToBRwPXGJmReytGIunSc0NdHYBt7j78cApwHUp8ne2Hfipu7cHOgDnmNkp8UbKMwxYGsUbp12BB/4XuJ2oNvguB3f/vsDDWqRINnd/zd1zd8KeAzSNM08ud1/q7v+KOwfQGfjM3Ve4+w7geeC8mDMB4O5vA+vjzlGYu6929/cT9zcRCleTeFOBB5sTD6slbrH/HJpZU6A38Jco3j+tCryZnQd87e4fxJ2lMDO7z8y+Ai4ldXrwBQ0GpscdIsU0Ab4q8HglKVCsKgsza0HYE+K9mKMAeUMhi4BvgdfdPRVyPUTokO6J4s2j3NEpEmb2BnB4EU2/Bu4iDM8k3f5yufvL7v5r4NdmdidwPXBvKuRKHPNrwq/W45KRqbS5pPIys9rAROCmQr/BxsbddwMdEteaJplZW3eP7RqGmfUBvnX3BWZ2ZhTnqHQF3t27F/W8mZ0AtAQ+MDMIww3vm1lnd/8mrlxFGAdMI0kFvqRcZjYI6AOc5Un8UEQZ/r7i9DXQrMDjponnZD/MrBqhuI9z95fizlOYu28wszcJ1zDivEjdFehrZucCWcAhZvasu19WUSdImyEad//Q3Ru7ewt3b0H4dbpjMop7Sczs2AIPzyNFNh83s3MIvx72dfcf4s6TguYBx5pZSzM7GBgATIk5U0qz0LsaAyx19/8Xd55cZtYod5aYmdUAehDzz6G73+nuTRP1agAwsyKLO6RRgU9x95vZEjNbTBhCSompY8CfgDrA64kpnI/FHQjAzC4ws5VAF+BVM/t7HDkSF6CvB/5OuFj4N3f/KI4shZnZeOBdoJWZrTSzq+LOlNAVuBz4aeLf1KJEDzVuRwBvJn4G5xHG4Ct8WmKq0VIFIiJpSj14EZE0pQIvIpKmVOBFRNKUCryISJpSgRcRSVMq8JKSzGxzyUcV+9rrE6s/7rWiqAUjE22LzaxjgbYjclfzM7MzK2plPzObVZoVMc3s85JWPzWzN8zs0IrIJZlBBV7S0T+A7sAXhZ7vBRybuA0FRhdouxl4Iinpyu+vwLVxh5DKQwVeUlqi1z0i8UGxD82sf+L5Kmb2aGI9+9fNbJqZXQTg7gvd/fMi3u484JnEyoJzgHpmdkSi7UJgRhHn72xm7ybW6/6nmbVKPD/IzCYnzv154reGmxPHzTGz+gXe5vLEB36WmFnnxOsbWNgb4CMz+wtgBc452cwWJNqGFnifKcAl5f27lMyjAi+prh9h/e72hF75iERR7ge0IKzTfjnhU68lKXJ1SDNrCXzn7tuLeM0y4HR3P5GwCuh/FWhrm8hxEnAf8EPiuHeBKwocVzOxF8C1wJOJ5+4F3nH3NsAkoHmB4we7eycgG7jRzBoAuPt3QPXcxyIlqXSLjUnGOQ0Yn1gJcI2ZvUUoqKcBL7j7HuCbxOJR5XUEkFNMW11gbGI9ISesI57rzcSa55vMbCPwSuL5D4F2BY4bD2ENdzM7JLEmSjfCfw64+6tm9l2B4280swsS95sRhpTWJR5/CxxZ4LFIsdSDl0xS3OqQWwmr+RVlOKGQtwX+o9BxBXv8ewo83sPenafC64EUuz5IYtnY7kCXxO5DCwudMyuRV6REKvCS6mYD/RObNTQi9HznEi6kXpgYiz8MOLMU7zUFuCIxrn8KsNHdVxO2K2xRzGvqkr9E8KBy/hlyrxucljjnRsJ2hAMTz/cCcmfH1CUMF/1gZj8mbHtH4jgjrKH/eTlzSIZRgZdUNwlYDHwAzARuTywBPZEwhv4x8CzwPrARwMxuTKxG2RRYnLiICWEd/hXAZ4QZM9cCuPsWYLmZHVPE+f8b+KOZLaT8Q5rbEq9/DMhd9fF3QDcz+4gwVJO7h/AM4CAzWwrcT9hKMVcnYE6BbRZF9kurSUqlZWa13X1z4qLjXKBredf/T4x5d3L3uys0ZAUys4eBKe7+f3FnkcpBF1mlMpuauGB5MDD8QDZ3cfdJlWB2yhIVdykL9eBFRNKUxuBFRNKUCryISJpSgRcRSVMq8CIiaUoFXkQkTf1/NBgygEwx7y4AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "%matplotlib inline\n", "\n", @@ -1851,9 +1618,7 @@ { "cell_type": "markdown", "id": "cd5cdbc0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see here that we reach a plateau. What is actually happening?" ] @@ -1861,22 +1626,239 @@ { "cell_type": "markdown", "id": "f1c9ebe1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## With Lasso Regression" ] }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 9, "id": "df01a2cf", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[2. 2.]\n", + "Training MSE for OLS\n", + "3.0\n", + "[1.99995 1.99980002]\n", + "[ 0.50001525 -0.99953475]\n", + "[1.99993978 1.99975913]\n", + "[ 0.50001525 -0.99944272]\n", + "[1.99992746 1.99970988]\n", + "[ 0.50001525 -0.99933188]\n", + "[1.99991263 1.99965056]\n", + "[ 0.50001525 -0.99919837]\n", + "[1.99989476 1.99957911]\n", + "[ 0.50001524 -0.99903755]\n", + "[1.99987324 1.99949306]\n", + "[ 0.50001524 -0.99884384]\n", + "[1.99984732 1.99938942]\n", + "[ 0.50001524 -0.99861053]\n", + "[1.9998161 1.99926459]\n", + "[ 0.50001523 -0.9983295 ]\n", + "[1.99977849 1.99911427]\n", + "[ 0.50001523 -0.99799099]\n", + "[1.9997332 1.99893323]\n", + "[ 0.50001522 -0.99758326]\n", + "[1.99967865 1.99871521]\n", + "[ 0.50001521 -0.99709215]\n", + "[1.99961294 1.99845267]\n", + "[ 0.50001521 -0.99650061]\n", + "[1.99953381 1.99813653]\n", + "[ 0.50001519 -0.99578809]\n", + "[1.9994385 1.99775587]\n", + "[ 0.50001518 -0.99492986]\n", + "[1.9993237 1.99729756]\n", + "[ 0.50001517 -0.99389612]\n", + "[1.99918546 1.9967458 ]\n", + "[ 0.50001515 -0.99265097]\n", + "[1.99901896 1.99608161]\n", + "[ 0.50001512 -0.99115119]\n", + "[1.99881845 1.99528218]\n", + "[ 0.5000151 -0.9893447]\n", + "[1.998577 1.9943201]\n", + "[ 0.50001506 -0.98716878]\n", + "[1.99828624 1.99316252]\n", + "[ 0.50001502 -0.98454786]\n", + "[1.99793613 1.99176998]\n", + "[ 0.50001498 -0.98139097]\n", + "[1.99751458 1.99009525]\n", + "[ 0.50001492 -0.97758848]\n", + "[1.99700706 1.98808176]\n", + "[ 0.50001485 -0.97300836]\n", + "[1.9963961 1.98566191]\n", + "[ 0.50001476 -0.9674916 ]\n", + "[1.99566069 1.98275501]\n", + "[ 0.50001466 -0.96084663]\n", + "[1.9947756 1.97926491]\n", + "[ 0.50001454 -0.95284275]\n", + "[1.99371056 1.97507735]\n", + "[ 0.50001439 -0.94320205]\n", + "[1.99242921 1.97005689]\n", + "[ 0.50001422 -0.93158979]\n", + "[1.99088801 1.9640435 ]\n", + "[ 0.500014 -0.91760278]\n", + "[1.9890348 1.95684892]\n", + "[ 0.50001374 -0.90075537]\n", + "[1.98680716 1.9482527 ]\n", + "[ 0.50001343 -0.88046261]\n", + "[1.98413059 1.93799826]\n", + "[ 0.50001306 -0.85601992]\n", + "[1.98091621 1.92578916]\n", + "[ 0.50001261 -0.8265786 ]\n", + "[1.97705827 1.91128596]\n", + "[ 0.50001207 -0.79111643]\n", + "[1.97243128 1.89410423]\n", + "[ 0.50001142 -0.74840212]\n", + "[1.96688672 1.87381451]\n", + "[ 0.50001063 -0.69695259]\n", + "[1.96024953 1.84994524]\n", + "[ 0.50000969 -0.63498144]\n", + "[1.95231424 1.82198978]\n", + "[ 0.50000855 -0.56033697]\n", + "[1.94284104 1.78941903]\n", + "[ 0.50000718 -0.47042744]\n", + "[1.93155188 1.75170092]\n", + "[ 0.50000553 -0.3621311 ]\n", + "[1.91812702 1.70832814]\n", + "[ 0.50001414 -0.23167717]\n", + "[1.90220243 1.65885453]\n", + "[ 0.50000455 -0.07456491]\n", + "[1.88336879 1.60293962]\n", + "[ 0.47132891 -0. ]\n", + "[1.86117291 1.54039921]\n", + "[ 0.41433969 -0. ]\n", + "[1.83512277 1.47125748]\n", + "[ 0.34569596 -0. ]\n", + "[1.80469739 1.39579407]\n", + "[ 0.26301436 -0. ]\n", + "[1.76936315 1.31457796]\n", + "[ 0.16342407 -0. ]\n", + "[1.72859758 1.22847924]\n", + "[ 0.04346721 -0. ]\n", + "[1.68192193 1.13865173]\n", + "[ 0. -0.]\n", + "[1.62894215 1.04648335]\n", + "[ 0. -0.]\n", + "[1.56939714 0.95351665]\n", + "[ 0. -0.]\n", + "[1.50321091 0.86134827]\n", + "[ 0. -0.]\n", + "[1.43054282 0.77152076]\n", + "[ 0. -0.]\n", + "[1.35182854 0.68542204]\n", + "[ 0. -0.]\n", + "[1.26780278 0.60420593]\n", + "[ 0. -0.]\n", + "[1.17949575 0.52874252]\n", + "[ 0. -0.]\n", + "[1.0881981 0.45960079]\n", + "[ 0. -0.]\n", + "[0.99539415 0.39706038]\n", + "[ 0. -0.]\n", + "[0.90266948 0.34114547]\n", + "[ 0. -0.]\n", + "[0.81160425 0.29167186]\n", + "[ 0. -0.]\n", + "[0.7236674 0.24829908]\n", + "[ 0. -0.]\n", + "[0.64012627 0.21058097]\n", + "[ 0. -0.]\n", + "[0.56198284 0.17801022]\n", + "[ 0. -0.]\n", + "[0.48994188 0.15005476]\n", + "[ 0. -0.]\n", + "[0.42441033 0.12618549]\n", + "[ 0. -0.]\n", + "[0.3655222 0.10589577]\n", + "[ 0. -0.]\n", + "[0.31318084 0.08871404]\n", + "[ 0. -0.]\n", + "[0.26710969 0.07421084]\n", + "[ 0. -0.]\n", + "[0.22690428 0.06200174]\n", + "[ 0. -0.]\n", + "[0.19207979 0.0517473 ]\n", + "[ 0. -0.]\n", + "[0.16211139 0.04315108]\n", + "[ 0. -0.]\n", + "[0.13646574 0.0359565 ]\n", + "[ 0. -0.]\n", + "[0.11462415 0.02994311]\n", + "[ 0. -0.]\n", + "[0.09609807 0.02492265]\n", + "[ 0. -0.]\n", + "[0.08043851 0.02073509]\n", + "[ 0. -0.]\n", + "[0.06724062 0.01724499]\n", + "[ 0. -0.]\n", + "[0.05614483 0.01433809]\n", + "[ 0. -0.]\n", + "[0.04683565 0.01191824]\n", + "[ 0. -0.]\n", + "[0.039039 0.00990475]\n", + "[ 0. -0.]\n", + "[0.03251863 0.00823002]\n", + "[ 0. -0.]\n", + "[0.02707227 0.00683748]\n", + "[ 0. -0.]\n", + "[0.02252765 0.0056799 ]\n", + "[ 0. -0.]\n", + "[0.01873869 0.00471782]\n", + "[ 0. -0.]\n", + "[0.01558197 0.00391839]\n", + "[ 0. -0.]\n", + "[0.01295356 0.0032542 ]\n", + "[ 0. -0.]\n", + "[0.01076611 0.00270244]\n", + "[ 0. -0.]\n", + "[0.00894639 0.00224413]\n", + "[ 0. -0.]\n", + "[0.0074331 0.00186347]\n", + "[ 0. -0.]\n", + "[0.00617499 0.00154733]\n", + "[ 0. -0.]\n", + "[0.00512927 0.00128479]\n", + "[ 0. -0.]\n", + "[0.00426027 0.00106677]\n", + "[ 0. -0.]\n", + "[0.00353823 0.00088573]\n", + "[ 0. -0.]\n", + "[0.00293838 0.00073541]\n", + "[ 0. -0.]\n", + "[0.0024401 0.00061058]\n", + "[ 0. -0.]\n", + "[0.00202624 0.00050694]\n", + "[ 0. -0.]\n", + "[0.00168251 0.00042089]\n", + "[ 0. -0.]\n", + "[0.00139705 0.00034944]\n", + "[ 0. -0.]\n", + "[0.00115999 0.00029012]\n", + "[ 0. -0.]\n", + "[0.00096314 0.00024087]\n", + "[ 0. -0.]\n", + "[0.00079968 0.00019998]\n", + "[ 0. -0.]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEHCAYAAACp9y31AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsaklEQVR4nO3deXxV1dX/8c9KAoR5SqAyGaSCIDIGhVJBRUCFCtQBVFRAxKoodaxK61BFUXwcap9HilYEtfijWpUCiqhQcUAJ8yjIpKGIzBDGDPv3x74JSW4gDElOcu/3/XrdF/eede49KyEsdvbZZx1zziEiItEjJugERESkZKnwi4hEGRV+EZEoo8IvIhJlVPhFRKKMCr+ISJSJK64PNrPXgN7Az865lqFttYD/ByQBG4CrnXM7C/ushIQEl5SUVFypiohEpPnz529zziXm327FtY7fzLoAacDEXIX/GWCHc260mT0A1HTO/aGwz0pOTnYpKSnFkqeISKQys/nOueT824ttqsc59zmwI9/mPsCE0PMJQN/iOr6IiBSspOf46zrnNoee/wTUPdqOZjbMzFLMLGXr1q0lk52ISBQI7OSu83NMR51ncs6Nc84lO+eSExPDpqhEROQkFdvJ3aPYYmanOec2m9lpwM8n+0Hp6emkpqZy8ODBIkxPghAfH0+DBg0oV65c0KmIRIWSLvxTgBuB0aE/PzjZD0pNTaVq1aokJSVhZkWVn5Qw5xzbt28nNTWVxo0bB52OSFQotqkeM5sEfA00M7NUM7sJX/C7m9ka4OLQ65Ny8OBBateuraJfxpkZtWvX1m9uIiWo2Eb8zrlrjhLqVlTHUNGPDPp7FClZJT3VIyJStmRlwaFDcPAgHDjg/zSD7KnJxYthxw5ITz/yqF4dLrzQxz/4ALZtg8zMI4/69aFfPx//299g505/nOzHWWfB1VcX39fknCv1j/bt27v8VqxYEbatpAHuuuuuy3mdnp7uEhISXK9evZxzzv3000+uV69erlWrVq558+bu0ksvdc45t379ehcfH+9at26d85gwYULY53ft2tU1bdrUtWrVyiUnJ7uFCxfmxC699FK3c+fOsPc88sgjbsyYMaf0dW3bti0nr7p167p69erlvD506FCh7583b5674447TuiYpeHvUyLUgQPOrV/v3LffOjdtmnMTJjj3t78dib/wgnN9+zp34YXOdejgXPPmznXqdCTeo4dzkPfRosWReOfO4fFzzz0Sb9UqPN6t25F448bh8b59i+RLB1JcATVVI/5TULlyZZYtW8aBAweoWLEiM2fOpH79+jnxhx9+mO7duzNixAgAlixZkhNr0qQJixYtKvQYb731FsnJyYwfP5777ruPmTNnAjB9+vSi/WJyqV27dk5ujz76KFWqVOHee+/Ns09GRgZxcQX/+CQnJ5OcHHaxoEjx2L0bVq2C77+H9eth40bYtAmmToWYGLjjDnj11bzvqVABhg3zz3/4AdauhWrVoHZtOP10qFfvyL6DB/vRe8WK/n3x8ZCQcCT+wguQlgblyh15VK16JD59uh/lx8YeeVSocCS+bJn/DSI21udr5h/FSIX/FF122WVMmzaNK6+8kkmTJnHNNdcwZ84cADZv3kyPHj1y9m3VqtVJH6dTp06MGTMm53VSUhIpKSkkJCQwatQoJkyYQJ06dWjYsCHt27cHYN68edx0003ExMTQvXt3PvzwQ5YtW0ZmZiYPPPAAs2fP5tChQ9x+++3ccsstheYwaNAg4uPjWbhwIZ07d2bAgAGMGDGCgwcPUrFiRcaPH0+zZs2YPXs2zz77LFOnTuXRRx/lhx9+YN26dfzwww/8/ve/58477zzp74NEscOHfZGcP99PrzzxBNSo4Qvvo48e2a9uXWjUCPbu9VMu118PHTtCnTqQmOiLdq1afmxtBv/zP8c+7oABx44XNsjJNRgsUKVKx44Xg8gp/BdcEL7t6qvhtttg/3647LLw+KBB/rFtG1x5Zd7Y7NnHddgBAwbw5z//md69e7NkyRKGDBmSU/hvv/12+vfvz1//+lcuvvhiBg8eTL3QSGLt2rW0adMm53Neeuklzj///KMe56OPPqJv375h2+fPn8/bb7/NokWLyMjIoF27djmFf/Dgwbzyyit06tSJBx54IOc9f//736levTrz5s3j0KFDdO7cmR49ehzXcsrU1FS++uorYmNj2bNnD3PmzCEuLo5PPvmEhx56iHfffTfsPatWrWLWrFns3buXZs2aceutt2rNvhy/OXPgvvtg4UJf/MGPqG++2Rf+q6+Gdu3gl7+EpCQ/Ms+tSxf/kByRU/gD0qpVKzZs2MCkSZO4LN9/Lj179mTdunV89NFHfPjhh7Rt25Zly5YBxz/Vc91113H48GHS0tIK3H/OnDn069ePSqFRw+WXXw7Arl272Lt3L506dQLg2muvZerUqQB8/PHHLFmyhHfeeQeA3bt3s2bNmuMq/FdddRWxsbE577vxxhtZs2YNZkZ6enqB7+nVqxcVKlSgQoUK1KlThy1bttCgQYNCjyVRxjk/op82zU+P3HMP9OkDVapAXBzceSd06OBH2I0bH5kOad7cP+S4RU7hP9YIvVKlY8cTEo57hF+Qyy+/nHvvvZfZs2ezffv2PLFatWpx7bXXcu2119K7d28+//zznBH58Xjrrbdo37499913H3fccQf/+te/TjrPbM45XnrpJXr27HnC761cuXLO8z/96U9ceOGFvPfee2zYsIELCvqtC6iQaz4zNjaWjIyMEz6uRLCDB+GZZ+Dtt2HlSr+tbVu/uiX7+RdfBJdfBNKNWIrAkCFDeOSRRzjnnHPybP/ss8/Yv38/AHv37mXt2rU0atTohD/fzHj88ceZO3cuq1atyhPr0qUL77//PgcOHGDv3r38+9//BqBGjRpUrVqVb775BoC333475z09e/bk5Zdfzhmhr169mn379p1wXrt37845mf3666+f8Pslih0+DNmLHSpUgAkT/Pz7//6vPzG7YMGR5Y5S5FT4i0CDBg0KPGE5f/58kpOTadWqFZ06dWLo0KF06NABODLHn/34y1/+csxjVKxYkXvuuSfPCV6Adu3a0b9/f1q3bs2ll16a8/ng5/Jvvvlm2rRpw759+6hevToAQ4cOpUWLFrRr146WLVtyyy23nNQo/P777+fBBx+kbdu2GsXL8dmzB55+2p98veiiI2vily6F//zHn5PLvaJGikWx3YilKBV0I5aVK1fSXPN6x5SWlkaVKlUAGD16NJs3b+bFF18MOKuC6e8zwu3ZA88/Dy++6C9W6t4d7roLevb0SxilWBztRiyRM8cvYaZNm8ZTTz1FRkYGp59+uqZjJDgLF/oll336wMiR/iStBEaFP4L179+f/v37B52GRKtvv4Wvv4YRI6BrV/juO2jaNOisBM3xi0hRO3gQHnwQOnXyUzuHDvntKvqlhgq/iBSdxYuhfXsYPdq3Oli4MG97AikVNNUjIkVj505/hWyVKvDhh3DJJUFnJEehwi8ipya7503NmvD66/CrX/l+OVJqaarnFJgZAwcOzHmdkZFBYmIivXv3BmDLli307t2b1q1b06JFi5yWDhs2bKBixYp51vFPnDgx7PMvuOAC8i9jLQmjRo3KySs2Nva4rzXINnToUFasWFHMWUqpsGcP9OgB2T2a+vVT0S8DNOI/BSXRljkII0eOZOTIkQBUqVIlLM/snt4xR1l//Wr+FrgSmX76CS691PfXufHGoLORE6AR/ynKbssM5LRlzrZ58+Y8zchOpS1ztg0bNnD++efTrl072rVrx1dffZVzrC5dutCmTRtatmzJnDlzyMzMZNCgQbRs2ZJzzjmH559/HoBFixbRsWNHWrVqRb9+/di5c+dxHbdZs2bccMMNtGzZkh9//JFbb72V5ORkzj77bB555JGcfXP/plKlShVGjhxJ69at6dixI1u2bDnl74GUAuvWQefOsHo1/PvfkOs3Xyn9ImbEf8HrF4Rtu/rsq7mtw23sT9/PZW+Ft2Ue1GYQg9oMYtv+bVw5OW9b5tmDZh/XcUuqLXO2OnXqMHPmTOLj41mzZg3XXHMNKSkp/OMf/6Bnz56MHDmSzMxM9u/fz6JFi9i0aVNOR9Bdu3YBcMMNN/DSSy/RtWtXHn74YR577DFeeOGFQo+9Zs0aJkyYQMeOHQE/JVSrVi0yMzPp1q0bS5YsCfvPbd++fXTs2JFRo0Zx//3388orr/DHP/6x0GNJKbZ9u78xyb598NlncN55QWckJyhiCn9Qirstc37p6ekMHz6cRYsWERsby+rVqwHo0KEDQ4YMIT09nb59+9KmTRvOOOMM1q1bxx133EGvXr3o0aMHu3fvZteuXXTt2hWAG2+8kauuuuq4jn366afnFH2AyZMnM27cODIyMti8eTMrVqwIK/zly5fPOefRvn37nDuISRlWqxYMHQq9e/vOmVLmREzhP9YIvVK5SseMJ1RKOO4RfkGKsy1zfs8//zx169Zl8eLFZGVlER8fD/gunZ9//jnTpk1j0KBB3H333dxwww0sXryYGTNmMHbsWCZPnpwz3XMycrdkXr9+Pc8++yzz5s2jZs2aDBo0iIMHD4a9p1y5cliob7paMpdxW7f60f5ZZ8Gf/hR0NnIKNMdfBIq7LXNuu3fv5rTTTiMmJoY33niDzMxMADZu3EjdunW5+eabGTp0KAsWLGDbtm1kZWVxxRVX8MQTT7BgwQKqV69OzZo1c6aj3njjjZzR/4nYs2cPlStXpnr16mzZsoUPP/zwlL4uKeUOHoTLL/dN1bKvxJUyK2JG/EE6Vlvm4cOHExcXR1ZWVk5b5g0bNoTN8Q8ZMqTAz+jVq1fObQo7derEk08+yRVXXMHEiRO55JJLckbhs2fPZsyYMZQrV44qVaowceJENm3axODBg8kK3dDiqaeeAmDChAn87ne/Y//+/ZxxxhmMHz/+hL/m1q1b07ZtW8466ywaNmxI586dT/gzpIxwDm66CebO9cs2dSVumae2zFIq6O+zFBs1Cv74R3jySd+DR8qMo7Vl1lSPiBzd9Om+6A8cCA88EHQ2UkRU+EXk6Lp2hccfh1deOXJzcynzyvQcv3MuZ8WIlF1lYbox6hw+DOnpULmyH/FLRCmzI/74+Hi2b9+uolHGOefYvn17zrJUKSUeeMDfJSstLehMpBiU2RF/gwYNSE1NZevWrUGnIqcoPj4+T2sLCdjUqf7+uMOH+xbLEnHKbOEvV64cjRs3DjoNkciyfbtfutm6NYwZE3Q2UkzKbOEXkWIwYgTs2AEzZ4Km3yJWIHP8ZnaXmS03s2VmNsnM9BMmErR9+3y3zZEjoQg6yUrpVeIjfjOrD9wJtHDOHTCzycAA4PWSzkVEcqlcGb76yl+pKxEtqFU9cUBFM4sDKgH/DSgPEQF47TXYtg3i4iDUIkQiV4kXfufcJuBZ4AdgM7DbOfdx/v3MbJiZpZhZilbuiBSjuXP9Cd0XXww6EykhJV74zawm0AdoDNQDKptZ2O17nHPjnHPJzrnkxMTEkk5TJDpkZvplm/Xqwf33B52NlJAgpnouBtY757Y659KBfwG/CiAPEXntNZg/H559FqpWDTobKSFBFP4fgI5mVsl8v4VuwMoA8hCJbjt2+G6bXbrAgAFBZyMlKIg5/m+Ad4AFwNJQDuNKOg+RqJeeDhdfDC+9pAZsUabM9uMXEZFjUz9+ETli9GhYujToLCQgatkgEm2++srP7WdkQL77REt00IhfJJo455dt/uIXcNddQWcjAdGIXySaTJkCX34JY8f6Fg0SlTTiF4kWmZl+iqdpU3+lrkQtjfhFosXhw9C3L5x3nu/JI1FLf/si0aJiRXjyyaCzkFJAUz0i0WDGDH9LxTJw3Y4UP434RSJdZibceae/o9Zll+kqXVHhF4l4kyb5O2u9+y7E6Jd80VSPSGTLzITHH/c3T+/bN+hspJTQiF8kkr3zjh/tv/OORvuSQz8JIpEsNhZ69IB+/YLOREoRFX6RSHbllX5Fj0b7kot+GkQi1XvvwaFDQWchpZAKv0gk+uIL+O1v/a0VRfJR4ReJRE89BQkJcOONQWcipZAKv0ikWbIEpk+HESOgUqWgs5FSSIVfJNL8z//4lsu33RZ0JlJKqfCLRJLMTL9u/6aboFatoLORUkoXcIlEkthYf2tFreaRY9CIXyRS7NsHO3f6Jmzx8UFnI6WYCr9IpPjb36BhQ/jvf4POREo5FX6RSJCeDi+8AB06QL16QWcjpZzm+EUiwT//CT/+CP/3f0FnImWARvwiZZ1z8Nxz0KyZv9GKSCE04hcp65YuhfnzYexYNWOT46LCL1LWtWrlr9Zt0iToTKSMUOEXKcuc88s3zzkn6EykDNHvhSJl2T33+EZszgWdiZQhKvwiZdXOnTBunH9uFmwuUqao8IuUVa+84q/WveuuoDORMiaQwm9mNczsHTNbZWYrzaxTEHmIlFnp6fDSS3DhhdCmTdDZSBkT1MndF4GPnHNXmll5QE3DRU7Eu+9Caiq8/HLQmUgZVOKF38yqA12AQQDOucPA4ZLOQ6RMO/98ePJJXbAlJyWIqZ7GwFZgvJktNLNXzaxy/p3MbJiZpZhZytatW0s+S5HSrH59ePBBXbAlJyWIn5o4oB3wsnOuLbAPeCD/Ts65cc65ZOdccmJiYknnKFJ6jRoFn34adBZShgVR+FOBVOfcN6HX7+D/IxCRwqxbBw8/rMIvp6TEC79z7ifgRzNrFtrUDVhR0nmIlEkvvOCnd26/PehMpAwLalXPHcBboRU964DBAeUhUnbs2AGvvQbXXuvn+EVOUiCF3zm3CEgO4tgiZdbf/uYv2LrnnqAzkTJOSwJEyorERBg0yHfjFDkFKvwiZcXQoTB+fNBZSARQ4Rcp7ZzzV+oe1nWOUjRU+EVKu5kz4corYfLkoDORCKHCL1LajR4N9erBVVcFnYlECBV+kdLsm29g1iy4+26oUCHobCRCqPCLlGZPPw01asCwYUFnIhFEhV+ktDp40LdoGD4cqlYNOhuJILrZukhpFR8PCxdqNY8UOY34RUqjHTtg925/L13N7UsRU+EXKY0eewzOOAPS0oLORCLQMQu/mQ3M9bxzvtjw4kpKJKpt3gzjxkGfPlClStDZSAQqbMR/d67nL+WLDSniXEQE4Jln/M3UR44MOhOJUIUVfjvK84Jei8ip+uknGDsWBg6EJk2CzkYiVGGF3x3leUGvReRUTZmi0b4Uu8KWc55lZkvwo/smoeeEXp9RrJmJRKNhw6B7d2jcOOhMJIIVVvibl0gWIgI7d0LNmir6UuyOOdXjnNuY+wGk4W+MnhB6LSJF4YcfoGFDeOONoDORKFDYcs6pZtYy9Pw0YBl+Nc8bZvb74k9PJEo88ghkZEDXrkFnIlGgsJO7jZ1zy0LPBwMznXO/Ac5DyzlFisby5TBxou/J06hR0NlIFCis8Kfnet4NmA7gnNsLZBVXUiJR5aGH/IVaDz4YdCYSJQo7ufujmd0BpOLn9j8CMLOKQLlizk0k8m3cCB99BI8+CrVrB52NRInCCv9NwJ+Bi4H+zrldoe0dAd31WeRUnX46fPcdJCYGnYlEkWMWfufcz8DvCtg+C5hVXEmJRIXNm+G00yApKehMJMocs/Cb2ZRjxZ1zlxdtOiJRYvduaNMGhg6FUaOCzkaiTGFTPZ2AH4FJwDeoP49I0Xj0Udi6Fa64IuhMJAoVVvh/AXQHrgGuBaYBk5xzy4s7MZGItXw5vPSSb8/Qrl3Q2UgUKuzK3Uzn3EfOuRvxJ3S/B2arF7/IScrKgttvh2rV4Ikngs5GolSh99w1swpAL/yoPwn4C/Be8aYlEqHWr4cVK2DMGEhICDobiVKFndydCLTEX7j1WK6reEXkZDRp4pdv1qgRdCYSxQq7cncgcCYwAvjKzPaEHnvNbE/xpycSIZyDSZMgM9N34DStk5DgFLaOXzdjFykKr77qT+bGxED//kFnI1EusMJuZrFmttDMpgaVg0iJWLUK7roLLroIrroq6GxEgiv8+OmjlQEeX6T4HTgAV18NFSv6Dpwx+iVaghfIT6GZNcCvFHo1iOOLlJi774alS+HNN6F+/aCzEQGOYzlnMXkBuB+oerQdzGwYMAygkXqUS1k1aBA0bQo9ewadiUiOEh/xm1lv4Gfn3Pxj7eecG+ecS3bOJSeqc6GUNdu2+T/PO8/P74uUIkFM9XQGLjezDcDbwEVm9mYAeYgUjw0boGVLePbZoDMRKVCJF37n3IPOuQbOuSRgAPCZc25gSechUix27YJeveDQIf+nSCkU1By/SOTZvx9++1tYvRo+/hiaNw86I5ECBVr4nXOzgdlB5iBSJJyDfv1g9my/bPPCC4POSOSoNOIXKQpmcP31cN11MFAzl1K6qfCLnIqdO/06/S5dVPClzNBlhCIna80a6NjRn8T9+eegsxE5bir8Iifjs8/8Gv0dO2D6dKhTJ+iMRI6bCr/IiXr0UejeHerVg2+/hfPPDzojkROiwi9yotLS/Incr76Cxo2DzkbkhOnkrkhh0tLg6afhggugWzd45hl12ZQyTYVf5GgyMnxXzYcegs2b/Vr9bt1U9KXM00+wSEEmTfJdNQcPhoYN4csv4Ykngs5KpEio8IuAH80vWAB7QreS3rkT6taFf/0Lvv4afvWrYPMTKUIq/BK9MjL8SH7kSN9Ns317mDzZx373O3/ytl8/Te1IxNEcv0SPXbtg714/dZOWBo0a+ZF9bCx07gxjx8IVV/h9VewlgqnwS+Q4dMhP1WTfuGfcOFi82F9hu3w5/Pe/cPnl8MEHUKUKDBvmR/ndu0ONGoGmLlKSIr/wp6RAVlbebYmJfv21czBvXvh7fvELPxrMyPDzvvnVr+8fhw/DokXh8UaN/GccOOD7uOSXlOSv9ExLgxUrwuNNmkDt2r6IrVoVHm/a1BeqnTt9UcvvrLOgWjV/F6h168LjZ58NlSv7NgMbNuSNOQetW0N8vC+UP/6YNwbQrh2ULw8//ACpqUe2Z+/TqZMfRa9d69+fleW3Zz+6d/f7LloEGzdCZqb/Xmdm+pF2//4+Pm2a//4cOuS/lwcO+Lwff9zH77nHd8PcscM/9uyBNm1g4UIfHz8eVq6EX/4SLr7Yf93nnnsk19Gjw783ItHAOVfqH+3bt3cnrVKl3CXHP265xccyM8Nj4Ny99/r4rl0Fxx97zMdTUwuOP/ecj69cWXB83Dgf//bbguOTJvn4p58WHJ861cfff7/g+H/+4+NvvllwfMECH3/55YLjq1f7+DPPFBzfvNnH//SnguNpaT7++9+Hx8yO/N3cdFN4vGrVI/H+/Y9sj4lxrkoV584++0j8D39wrlcv5wYOdO7OO53785+d+8c/jsT37z/xnxeRCAKkuAJqqrnco7VSKjk52aWkpJzcm2fM8CPJ3Bo18ifznIMPPwx/T+PG/iYa6ekwc2Z4/Mwz/ePgQd+zJb/mzf1npKXB55+Hx1u29Dns2uVPIObXpo1vB7Btm28JkF9ysv+NYcsWmF/ArYs7doRatWDTJj/VkV/nzlC9uh9tL18eHu/a1Y+s162D777LGzPzveYrVIDvvz/yG4XZkX0uvBDi4vxvI5s2+ZiZH82b+eODP/727X7f2Fj/KF8ezjjDx9PS/J8VKvh9ch9DRAplZvOdc8lh2yO+8IuIRKmjFX4tXRARiTIq/CIiUUaFX0Qkyqjwi4hEGRV+EZEoo8IvIhJlVPhFRKKMCr+ISJRR4RcRiTIq/CIiUUaFX0Qkyqjwi4hEGRV+EZEoo8IvIhJlVPhFRKJMiRd+M2toZrPMbIWZLTezESWdg4hINAvinrsZwD3OuQVmVhWYb2YznXMF3HxWRESKWomP+J1zm51zC0LP9wIrgfolnYeISLQKdI7fzJKAtsA3BcSGmVmKmaVs3bq1xHMTEYlUgRV+M6sCvAv83jm3J3/cOTfOOZfsnEtOTEws+QRFRCJUIIXfzMrhi/5bzrl/BZGDiEi0CmJVjwF/B1Y6554r6eOLiES7IEb8nYHrgYvMbFHocVkAeYiIRKUSX87pnPsCsJI+roiIeLpyV0Qkyqjwi4hEGRV+EZEoo8IvIhJlVPhFRKKMCr+ISJRR4RcRiTIq/CIiUUaFX0Qkyqjwi4hEGRV+EZEoo8IvIhJlVPhFRKKMCr+ISJRR4RcRiTIq/CIiUUaFX0Qkyqjwi4hEGRV+EZEoo8IvIhJlSvxm6yJl2bj549i+f3uebWfUPIP+LfsD8Ndv/8reQ3vzxM9KOIt+zfsB8NzXz3Eo41CeeKu6rejVtBcAo78YjXMuTzy5XjLdm3QnPTOdZ796Niynzo060+X0Luw7vI+/fPOXsPgFSRfQqWEndh7YydiUsWHxHk160L5ee7akbeG1ha+FxX/T7De0rNOSH3f/yJtL3gyLX9HiCprWbsraHWuZvHxyWPyac64hqUYSK7eu5P1V74fFb2xzI/Wq1mPxT4uZvmZ6WPzm9jeTUCmBeZvm8cm6T8Lit597O9UqVOOLH75gzsY5YfG7Ot1FfFw8s9bPYm7q3LD4H379B2Ishhnfz2DB5gV5YnExcdzX+T4Apnw3heU/L88Tr1SuEiM6jgDgnRXvsGb7mjzxGvE1uLXDrQD8Y+k/2LhrY554ncp1uKndTQC8vuh1Nu/dnCfe85c9aXdau7CcT5UKv8hRZLks3l72Nst+XsaT3Z4E4IW5L7By28o8+13yy0tyCv/TXz5N6p7UPPGrWlyVU/j//J8/s/vQ7jzxIW2G5BT+kZ+NJMtl5YmPOG8E3Zt0JyMrg4c+eygszz+e/0df+NP3FRgf3W00nRp2YseBHQXGq8dXp3299mxO21xgvEG1BrSs05INuzYUGG+e2JymtZuyevvqAuPn1j+XpBpJLP15aYHxixpfRL2q9Uj5b0qB8T5n9SGhUgJf/vhlgfGBrQZSrUI1Zq2fxcOzHw6L39rhVuLj4vno+4945qtnwuL3d74fDD747gNeTnk5Tyw+Lj6n8P9zxT/D/uNLrJSYU/jfXPImH3z3QZ74GTXPyCn8f1/4dz5b/1meeOu6rXMK/8spL/Ptpm/zxGtWrFkshd/yjy5Ko+TkZJeSkhJ0GhJF0jPTGTJlCG8ueZMza53J4t8tpmK5ihzOPBw2IjczyseWBwgbzQPEWAzlYsuddDw2Jpa4mDiccxzOPFzk8biYOGJjYslyWaRnphd5vFxsOWIshsysTDKyMk46npGVQWZWZli8fGx5zOyU4+mZ6WH/6QJUiKsAcMrxwn52Copnf29PlpnNd84l59+uEb9IPvvT93P1P69m2pppPHHhEzx0/kOYGUDOP9Kjyf5HXhxxMyvWeIzFFGs8Nib2mEWssHhcTBxxMUcvWacaz/7Pt7jihf3sFBYvSir8IvlcMfkKZnw/g7G9xnJL8i1BpyNS5FT4RfDz+Vkui7iYOO7tdC83tb2JK1tcGXRaIsVCyzklqjnnmLNxDl1f78rTXzwNQLczuqnoS0RT4ZeotO/wPiYunkjyK8l0eb0LK7eupFH1RkGnJVIiNNUjUSHLZbF6+2qa1m5KjMVw2/TbmLh4Is0TmjO211iub309lcpVCjpNkRKhwi8RyTnH4i2LmfLdFGZtmMX8/85n7+G9rLp9Fc0SmjHivBEMbjOYrqd3zVmxIxItVPglomS5LGIshinfTaHv/+uLYbQ7rR3Xt7qeDvU7kFg5EaBYLooRKStU+KXMW7tjLR+v/Zi3lr5Fn2Z9uK/zfVzU+CJeu/w1ejXtRZ3KdYJOUaRUCaTwm9klwItALPCqc250EHlI2eKc40DGgZy5+Ltn3M0H333Aup3rAGhWu1lOka9aoSqD2w4OLFeR0qzEC7+ZxQL/C3QHUoF5ZjbFObeipHORkuWcI9Nl5lw9uevgLnYd3MX+9P3sO7yPfen7OJx5mB5NegAwfuF4vk79mi37trBx10bW71pPw2oNWXbbMgBS96TSsk5L7up4Fz2a9ODMWmdqvl7kOAQx4j8X+N45tw7AzN4G+gDFUviTxyVzIONAnm1Xt7iaRy54hCyXxTkvnxP2nsFtBnPvr+5l76G9dPx7x7D48A7DubXDrWxJ28JFEy8Ki9//q/u5sc2NrNu5jt9M+k1Y/NGuj3LV2VexdMtSBrw7ICw+pvsYLjvzMuamzmXIB0PC4i/3epmuSV35dN2n3PHhHWHxif0mklwvmX9/92/+8MkfwuLvXv0uzROb8/ayt3nsP4+F9QeZMXAGp9c4nVcXvMozXz6Dw8ez95s7dC4JlRJ47uvnePGbF3HOkeWycPg/1965lkrlKvHQpw/x12//6nukuEwOZx72PVke9v1S7p5xN+MXjc9z7GoVqrH7Ad/E7NP1n/LJuk+oW6UuDas15PxG59M8sXnOvpOvCu8EKSKFC6Lw1wd+zPU6FTgv/05mNgwYBtCo0cmvrz4r4SwOZeZtfHVa1dNynrdIbBH2nrqV6wK+90hB8YRKCYDv/VFQvHal2gBUiK1QYLxmxZoAVCxXscB49QrVAahSvgot67QMi1etUBXwRbKgePZUSI34GgXG4+PifZ4Va9OqbisAjCMj5ex+K3Ur1yW5nu/vlD2SNixnxJ5UI4kLky7EzIghhhiL8c/NXx5ybv1zGdpuKLHme7CUjy1PuZhyOOcwMwa1GcSvG/2ayuUqU7l8ZSqXq0z1+Oo5ebzR7w2N4EWKQYl35zSzK4FLnHNDQ6+vB85zzg0/2nvUnVNE5MQdrTtnEFfubgIa5nrdILRNRERKQBCFfx5wppk1NrPywABgSgB5iIhEpRKf43fOZZjZcGAGfjnna8655YW8TUREikgg6/idc9OB8JtriohIsVN3ThGRKKPCLyISZVT4RUSijAq/iEiUKfELuE6GmW0FNp7k2xOAbUWYTlFRXidGeZ0Y5XViIjWv051zifk3lonCfyrMLKWgK9eCprxOjPI6McrrxERbXprqERGJMir8IiJRJhoK/7igEzgK5XVilNeJUV4nJqryivg5fhERySsaRvwiIpKLCr+ISJSJqsJvZveYmTOzhKBzATCzx81siZktMrOPzaxe0DkBmNkYM1sVyu09M6sRdE4AZnaVmS03sywzC3zpnZldYmbfmdn3ZvZA0PkAmNlrZvazmS0LOpfczKyhmc0ysxWhv8MRQecEYGbxZvatmS0O5fVY0DnlZmaxZrbQzKYW5edGTeE3s4ZAD+CHoHPJZYxzrpVzrg0wFXg44HyyzQRaOudaAauBBwPOJ9sy4LfA50EnYmaxwP8ClwItgGvMLPw+miXvdeCSoJMoQAZwj3OuBdARuL2UfL8OARc551oDbYBLzCz8RtvBGQGsLOoPjZrCDzwP3A+UmrPZzrk9uV5WppTk5pz72DmXEXo5F3+XtMA551Y6574LOo+Qc4HvnXPrnHOHgbeBPgHnhHPuc2BH0Hnk55zb7JxbEHq+F1/M6gebFTgvLfSyXOhRKv4dmlkDoBfwalF/dlQUfjPrA2xyzi0OOpf8zGyUmf0IXEfpGfHnNgT4MOgkSqH6wI+5XqdSCgpZWWBmSUBb4JuAUwFyplMWAT8DM51zpSIv4AX8YDWrqD84kBuxFAcz+wT4RQGhkcBD+GmeEnesvJxzHzjnRgIjzexBYDjwSGnIK7TPSPyv6G+VRE7Hm5eUXWZWBXgX+H2+33gD45zLBNqEzmW9Z2YtnXOBniMxs97Az865+WZ2QVF/fsQUfufcxQVtN7NzgMbAYjMDP22xwMzOdc79FFReBXgLf1eyEin8heVlZoOA3kA3V4IXe5zA9ytom4CGuV43CG2TozCzcvii/5Zz7l9B55Ofc26Xmc3CnyMJ+uR4Z+ByM7sMiAeqmdmbzrmBRfHhET/V45xb6pyr45xLcs4l4X8lb1cSRb8wZnZmrpd9gFVB5ZKbmV2C/xXzcufc/qDzKaXmAWeaWWMzKw8MAKYEnFOpZX7U9XdgpXPuuaDzyWZmidmr1sysItCdUvDv0Dn3oHOuQahmDQA+K6qiD1FQ+Eu50Wa2zMyW4KeiSsUSN+CvQFVgZmip6digEwIws35mlgp0AqaZ2Yygcgmd/B4OzMCfqJzsnFseVD7ZzGwS8DXQzMxSzeymoHMK6QxcD1wU+plaFBrNBu00YFbo3+A8/Bx/kS6dLI3UskFEJMpoxC8iEmVU+EVEoowKv4hIlFHhFxGJMir8IiJRRoVfyhwzSyt8r6O+d3iom2aeLq3m/SUUW2Jm7XLFTsvujmhmFxRVp0Qzm308XUbNbENhHWXN7BMzq1kUeUnkU+GXaPMlcDGwMd/2S4EzQ49hwMu5YncDr5RIdifvDeC2oJOQskGFX8qs0Ch9TOgiuKVm1j+0PcbM/i90T4GZZjbdzK4EcM4tdM5tKODj+gATQ90a5wI1zOy0UOwK4KMCjn+umX0d6pf+lZk1C20fZGbvh469IfRbxt2h/eaaWa1cH3N96GKmZWZ2buj9tc3fn2G5mb0KWK5jvm9m80OxYbk+Zwpwzcl+LyW6qPBLWfZbfA/11vhR/JhQsf4tkITvk389/krfwhTYbdPMGgM7nXOHCnjPKuB851xbfGfVJ3PFWoby6ACMAvaH9vsauCHXfpVC92O4DXgttO0R4Avn3NnAe0CjXPsPcc61B5KBO82sNoBzbidQIfu1yLFETJM2iUq/BiaFuituMbP/4Avtr4F/OueygJ9CjbdO1mnA1qPEqgMTQj2XHL6Xe7ZZob7ze81sN/Dv0PalQKtc+00C30ffzKqF+sZ0wf+ngXNumpntzLX/nWbWL/S8IX5qanvo9c9AvVyvRQqkEb+Id7Rumwfw3REL8ji+wLcEfpNvv9y/IWTlep1F3gFX/p4pR+2hEmrPezHQKXTHqIX5jhkfylfkmFT4pSybA/QP3UgjET9S/hZ/AveK0Fx/XeCC4/isKcANofMGHYHdzrnN+FtPJh3lPdU50op50El+DdnnJX4dOuZu/K0lrw1tvxTIXq1THT/ttN/MzsLfwpDQfoa/j8GGk8xDoogKv5Rl7wFLgMXAZ8D9oXbb7+Ln6FcAbwILgN0AZnZnqMNnA2BJ6OQp+HshrAO+x6/guQ3AObcPWGtmvyzg+M8AT5nZQk5+2vRg6P1jgexOmo8BXcxsOX7KJ/s+0R8BcWa2EhiNvy1mtvbA3Fy3zBQ5KnXnlIhkZlWcc2mhk53fAp1P9h4MoTn19s65PxZpkkXIzF4EpjjnPg06Fyn9dHJXItXU0InS8sDjp3LjHefce2VgtcwyFX05Xhrxi4hEGc3xi4hEGRV+EZEoo8IvIhJlVPhFRKKMCr+ISJT5/yt5TWbLNwHdAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "import os\n", "import numpy as np\n", @@ -1929,7 +1911,7 @@ "# Now plot the results\n", "plt.figure()\n", "plt.plot(np.log10(lambdas), MSERidgePredict, 'r--', label = 'MSE Ridge Train')\n", - "plt.plot(np.log10(lambdas), MSELassoPredict, 'r--', label = 'MSE Lasso Train')\n", + "plt.plot(np.log10(lambdas), MSELassoPredict, 'g--', label = 'MSE Lasso Train')\n", "plt.xlabel('log10(lambda)')\n", "plt.ylabel('MSE')\n", "plt.legend()\n", @@ -1939,22 +1921,41 @@ { "cell_type": "markdown", "id": "a0b15745", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Another Example, now with a polynomial fit" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 10, "id": "616cf30d", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 2.03099776 -0.17917768 5.18029127]\n", + "Training MSE for OLS\n", + "0.00916347050835222\n", + "Test MSE OLS\n", + "0.008675369724976584\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEKCAYAAAAVaT4rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4aUlEQVR4nO3dd3hUZfbA8e9JIw2SkIRQAoQmJIQQSAIEpIggIBixgh0QWFcprsrqys+ya2PX3jsgLotigaULrkZUQOkISBEIEEQ6IUBIfX9/zBDSIJTM3MzM+TxPnszc95Yzk+Tknfe+91wxxqCUUspzeFkdgFJKKefSxK+UUh5GE79SSnkYTfxKKeVhNPErpZSH0cSvlFIexmGJX0Qmish+EVlfQduDImJEJMJRx1dKKVUxR/b4JwN9yy4UkYbAVcAuBx5bKaXUWTgs8RtjFgOHK2h6GfgroFeOKaWUBXyceTARuRbYY4xZKyLnvV1ERISJiYlxWFxKKeWOVq5cedAYE1l2udMSv4gEAo9iG+Y5n/VHAiMBGjVqxIoVKxwYnVJKuR8R2VnRcmfO6mkGNAHWikgGEA2sEpG6Fa1sjHnPGJNsjEmOjCz3D0sppdRFclqP3xjzC1Dn9HN78k82xhx0VgxKKaUcO51zGrAUaCkimSJyt6OOpZRS6vw5rMdvjLmlkvaYS9l/fn4+mZmZnDp16lJ2oyzm7+9PdHQ0vr6+VoeilMdw6qyeqpSZmUnNmjWJiYnhQmYIqerDGMOhQ4fIzMykSZMmVoejlMdw2ZINp06dIjw8XJO+CxMRwsPD9VObUk7msokf0KTvBvRnqJTzuXTiV0opt5STA19/DevWOWT3mvgvgYhw++23Fz8vKCggMjKSAQMGALBv3z4GDBhA27ZtiYuL4+qrrwYgIyODgIAAEhMTi7+mTJlSbv89evSgZcuWtG3blpSUFNasWVPcdvXVV3P06NFy2zz55JO88MILl/zajh49yltvvXVR254tNqXUebj5ZggNhd694c03HXIIlz25Wx0EBQWxfv16cnJyCAgIYNGiRTRo0KC4/fHHH6d3796MHTsWgHUl/ns3a9asVCI/m6lTp5KcnMykSZMYN24cixYtAmDevHlV+2LKOJ3477333nJtBQUF+Pic/VfH0bEp5TZ27oSPP4YffoD580EE4uOhcWPo2RO6dnXIYbXHf4muvvpq5s6dC8C0adO45ZYzs1j37t1LdHR08fOEhISLPk5qaip79uwpfh4TE8PBg7Zr35555hkuu+wyLr/8cjZv3ly8zvLly0lISCAxMZFx48YRHx8PQGFhIePGjSMlJYWEhATefffdcsd75JFH2LZtW/G26enpdO3albS0NOLi4gAYOHAgSUlJtG7dmvfee69cbBkZGcTGxjJixAhat27NVVddRU5OzkW/B0q5hYIC+OwzW2KPiYHHHrMN7Zz+lPz44/D889CvH3sLvcktKKzyENyix//32RvY+PuxKt1nXP1aPHFN60rXGzx4MP/4xz8YMGAA69atY9iwYXz//fcA3HfffQwaNIg33niDXr16MXToUOrXrw9QnFRPe/311+l6jv/uCxYsYODAgeWWr1y5kk8++YQ1a9ZQUFBA+/btSUpKAmDo0KG8//77pKam8sgjjxRv8+GHHxISEsLy5cvJzc2lS5cuXHXVVaWmVE6YMIH169cXfypJT09n1apVrF+/vni9iRMnUrt2bXJyckhJSeGGG24gPDy8VHxbt25l2rRpvP/++9x888188cUXpYbHlPI4M2fahnOaNIF//APuuMP2DwA4diqf77ccZMm2gyzZdogdB08wdXhHujSv2luXuEXit1JCQgIZGRlMmzateAz/tD59+rB9+3YWLFjA/PnzadeuHevX2+5Lc75DPbfddht5eXkcP368wvW///57rrvuOgIDAwFIS0sDbEM12dnZpKamAnDrrbcyZ84cABYuXMi6dev4/PPPAcjKymLr1q2VzqXv0KFDqXVee+01ZsyYAcDu3bvZunVrucTfpEmT4n9wSUlJZGRkVPqalXIrOTm2sfpatWDkSLj2Wpg9G/r1A29vDmTnMn9pBos27mPptkMUFBmCa/jQsUltbuvYiKaRQVUeklsk/vPpmTtSWloaDz30EOnp6Rw6dKhUW+3atbn11lu59dZbGTBgAIsXLy7ukZ+PqVOnkpSUxLhx4xg9ejRffvnlJcdrjOH111+nT58+F7RdUNCZX8D09HS+/vprli5dSmBgID169KhwPn6NGjWKH3t7e+tQj/IcxsCUKTB+POzZA7feakv8vr7k9u3H1xv288WqTL7bcoDCIkPTyCDu7tqE3rFRJDYMxcfbcSPxOsZfBYYNG8YTTzxBmzZtSi3/5ptvOHnyJADZ2dls27aNRo0aXfD+RYSnnnqKZcuWsWnTplJt3bp1Y+bMmeTk5JCdnc3s2bMBCA0NpWbNmvz0008AfPLJJ8Xb9OnTh7fffpv8/HwAtmzZwokTJ0rtt2bNmmRnZ581pqysLMLCwggMDGTTpk0sW7bsgl+XUm5ryxbbGP6QIRAdDenpMHUqh47n8tr/tnL5P7/lvv+sYuPvxxjRtSkL/9KNbx7swd/6xZIcU9uhSR/cpMdvtejoaMaMGVNu+cqVKxk1ahQ+Pj4UFRUxfPhwUlJSyMjIKDfGP2zYsAr3cVpAQAAPPvggzz//PB9++GHx8vbt2zNo0CDatm1LnTp1SElJKW778MMPGTFiBF5eXnTv3p2QkBAAhg8fTkZGBu3bt8cYQ2RkJDNnzix1vPDwcLp06UJ8fDz9+vWjf//+pdr79u3LO++8Q2xsLC1btqRTp04X8pYp5d5+/x3WroX33oO772b/iTze/O96pi3fTV5BEd0vi2TojTF0bRGJt5fzL2IUY6r/HRCTk5NN2Rux/Prrr8TGxloUkWs4fvw4wcHBgO1k7d69e3n11Vctjqo8/Vkqt7B3r+2iqzvusD3PzibLx593v9vGpB8zyCss4sb20Yzo1oTmdWo6JSQRWWmMSS67XHv8bmzu3Lk899xzFBQU0LhxYyZPnmx1SEq5p6+/to3h5+RA376YiAhmbTvGU3OWcehEHmlt6/OXXpcRE1H1J2ovhiZ+NzZo0CAGDRpkdRhKubf334c//xlatYLp09npFcj4D3/mh98O0rZhKJOHdiC+QYjVUZaiiV8ppS7WY4/B009D374wfTozfzvG+Fe/x0uEp65tza0dG1syhl8ZTfxKKXWxoqJg5EhOvvwqT8zdzGcrM0mJCePVwe2oHxpgdXRnpYlfKaUuRGEhbN4McXEwahS/HznJsPd+ZvO+bEb3bM7YK1s4fDrmpdLEr5RS56uoCP70J5g2DTZsYL1vGMMmLycnr5CPhnag22WRVkd4Xqr3v6Vqzl3LMh86dKg4rrp169KgQYPi53l5eee1j/T0dJYsWXJJcShVrRgDo0fDhx/CAw/wbW4gN7+7FB8v4fM/d3aZpA/a478k7lqWOTw8vDi2J598kuDgYB566KEL2kd6ejrBwcF07tzZAREqZYFXXoG33oKHHuJ/t9zHPVNWcFlUTSYNSaFOLX+ro7sgDuvxi8hEEdkvIutLLHteRDaJyDoRmSEioY46vrO4a1nmiqxcuZLu3buTlJREnz592Lt3L2Ar1hYXF0dCQgKDBw8mIyODd955h5dffpnExMTiaqVKuawlS+Chh+CGG0gf9iB/nrqa2Hq1mDayk8slfXBsj38y8AZQcgxjEfA3Y0yBiPwT+BvwcJUcrUeP8stuvhnuvRdOnoQylTMBWx2NIUPg4EG48cbSbenp53VYdy3LXFZ+fj6jR4/mv//9L5GRkXz66aeMHz+eiRMnMmHCBHbs2EGNGjU4evQooaGh3HPPPRf1SUGpaqlDB5gwgR+vuomRU1fTIiqYj4d1pJa/r9WRXRSHJX5jzGIRiSmzbGGJp8uAMtnW9XhKWebNmzezfv16evfuDdg+NdSrV6/4PbjtttsYOHBghf+clHJZWVmQnw8REWy4bSTD315K04gg/n13R0ICXTPpg7Vj/MOAT8/WKCIjgZHA+VW0PFcPPTDw3O0REefdw6+IJ5RlNsbQunVrli5dWq5t7ty5LF68mNmzZ/PMM8/wyy+/XHKMSlnOGLjnHliyhP0/r2b4R6sJDfRlyrAOhAX5WR3dJbFkVo+IjAcKgKlnW8cY854xJtkYkxwZWb3PlrtjWeayWrZsyYEDB4oTf35+Phs2bKCoqIjdu3dzxRVX8M9//pOsrCyOHz9eaVlnpaq9jz+GTz4h/+7hjPj8V46ezOf9O5Ndcky/LKcnfhEZAgwAbjOuUBr0PJyrLHNycjIJCQmkpqYWl2WGM2P8p79ee+21cx6jZFnmkkqWZe7Xr1+FZZkTExM5ceJEqbLMcXFxtG/fnvj4eP70pz9RUFBwzuP7+fnx+eef8/DDD9O2bVsSExNZsmQJhYWF3H777bRp04Z27doxZswYQkNDueaaa5gxY4ae3FWu6bff4L77MN2781CzfqzLPMorgxOrXc2di+XQssz2Mf45xph4+/O+wEtAd2PMgfPdj5Zlvjhallmpi1BQAF26wNatzJw8j/uXHGJcn5bcd0VzqyO7YE4vyywi04AeQISIZAJPYJvFUwNYJCIAy4wx9zgqBk+nZZmVugjZ2RAVxd677+WR5Ufo2iKCP3dvZnVUVcqRs3puqWDxhxUsUw6iZZmVughhYZz6/EuGvvkjgX55vHhTW7yqYYXNS6ElG5RS6rSnn4Zt2/jnV5vZtO84L9yU4BYnc8vSkg1KKQUwcyY89hi7jhcwiRTuSm1Mz1ZRVkflEJr4lVLqxAkYPZqi+DYMr3050caLh/u1sjoqh9GhHqWUeuklyMzky+GPsuVwLk8NjCfQz337xZr4L4EzyjKXncbqLDNnzmTjxo0XvN2sWbOYMGGCAyJSykH274d//YsT/a/h0QMh9E+oxxUt61gdlUO57780J3BGWWarzJw5kwEDBhAXF1euraCgAB+fin910tLSiusFKeUS/PwwI0bwZO1O1Mj34okB5X/n3Y32+C+Rs8oyn5aRkUHXrl1p37497du3L77Zyd69e+nWrRuJiYnEx8fz/fffU1hYyJAhQ4iPj6dNmza8/PLLAKxZs4ZOnTqRkJDAddddx5EjR0odY8mSJcyaNYtx48aRmJjItm3b6NGjB/fffz/Jycm8+uqrzJ49m44dO9KuXTt69erFvn37AJg8eTKjRo0CYMiQIYwZM4bOnTvTtGnT4qJwSlUroaEsvPuvfHY8iHF9W7rlLJ6y3KPHP/8R+KOKC4PVbQP9Kh+ycFZZ5tPq1KnDokWL8Pf3Z+vWrdxyyy2sWLGC//znP/Tp04fx48dTWFjIyZMnWbNmDXv27CmuCHr6jl133nknr7/+Ot27d+fxxx/n73//O6+88krxMTp37kxaWhoDBgzgxhLlqvPy8oqHno4cOcKyZcsQET744AP+9a9/8eKLL5aLd+/evfzwww9s2rSJtLS0UvtTynKPP05h1278c50PzSKDuLXDhdfSckXukfgt5OiyzGXl5+czatQo1qxZg7e3N1u2bAEgJSWFYcOGkZ+fz8CBA0lMTKRp06Zs376d0aNH079/f6666iqysrI4evQo3bt3B+Cuu+7ipptuOq9jl7wYLDMzk0GDBrF3717y8vLOWtJ54MCBeHl5ERcXV/ypQKlqYfVqeOopNg4bw/bIq3j/zuRqf5P0quIeif88euaO5MiyzGW9/PLLREVFsXbtWoqKivD3t30s7datG4sXL2bu3LkMGTKEBx54gDvvvJO1a9fy1Vdf8c477zB9+vTi4Z6LERQUVPx49OjRPPDAA6SlpZGens6TTz5Z4TY1atQofuwmNfmUu3jmGUxICKPq9SQlOoxese59Qrckz/j35mCOLstcUlZWFvXq1cPLy4uPP/6YwsJCAHbu3ElUVBQjRoxg+PDhrFq1ioMHD1JUVMQNN9zA008/zapVqwgJCSEsLKx4OOrjjz8u7v2XVFlZ5aysrOIT2R999NElvSalnO7XX+HLL1ne/1Z2Fvjwt6tjsdcP8wju0eO32LnKMo8aNQofHx+KioqKyzJnZGSUG+MfNmxYhfvo378/vr62O/2kpqby7LPPcsMNNzBlyhT69u1b3AtPT0/n+eefx9fXl+DgYKZMmcKePXsYOnQoRUVFADz33HOALVHfc889nDx5kqZNmzJp0qRyxx08eDAjRozgtddeq/Ck7JNPPslNN91EWFgYPXv2ZMeOHRf+xilllQkTMAEB3F+vO/3i69K+UZjVETmVQ8syVxUty+ze9GepnO6tt0hfsY2hUT1Z9JfuNK8TbHVEDuH0ssxKKVVdZQ0Zwah933B1y0i3TfrnoolfKeU5/vgD5szh4+hUjucWcF8P17u5SlXQk7tKKc/x9tuYkSP5av7P9IqtQ1z9WlZHZAlN/Eopz5CbC+++y+5OPfgloI5L3kqxqmjiV0p5hs8+g337eP6y3nRpHk47D5vJU5KO8SulPMPrr3OsUVPmRMUz1UPH9k/THv8lcNeyzM8880xxXN7e3sWPX3vttfPex7PPPuvACJW6QEePYgoKmJp8DS3rhZDaLNzqiCzlsB6/iEwEBgD7jTHx9mW1gU+BGCADuNkYc+Rs+6ju3LUs8/jx4xk/fjwAwcHBFxXns88+y6OPPlrFkSl1kUJDWf7ZQp5/50ee7RzjUVfpVsSRPf7JQN8yyx4B/meMaQH8z/7cpbljWeaKFBYWMm7cOFJSUkhISODdd98963EfeeQRcnJySExM5Lbbbrvk16zUJcnKgqNHmbw0g1rB/lyb2KDybdydMcZhX9h69utLPN8M1LM/rgdsPp/9JCUlmbI2btxYekH37uW/3nzT1nbiRMXtkybZ2g8cKN92HoKCgszatWvNDTfcYHJyckzbtm3Nt99+a/r372+MMWbBggUmJCTE9OjRwzz99NNmz549xhhjduzYYfz9/U3btm2LvxYvXlxu/927dzfLly8vtezEiRMmJyfHGGPMli1bzOn35oUXXjBPP/20McaYgoICc+zYMbNixQrTq1ev4m2PHDlijDGmTZs2Jj093RhjzGOPPWbGjh17ztdojDHvvvuueeqpp4wxxpw6dcokJSWZ7du3V3jcktudj3I/S6Wq0oQJpjAoyCSN/Y95dp5n/a4BK0wFOdXZJ3ejjDF77Y//AFz+FvaeUpZ54cKFrFu3rrhuT1ZWFlu3bq3wuEpVG8bAxInsadKKwwG1uKNTY6sjqhYsm9VjjDEictZCQSIyEhgJnF9Fy/T0s7cFBp67PSLi3O2V8ISyzMYYXn/9dfr06VOuraLjKlUt/PgjbNnC+wMfpHdcFNFhgVZHVC04e1bPPhGpB2D/vv9sKxpj3jPGJBtjkiMjI50W4MVwx7LMZfXp04e3336b/Px8ALZs2cKJEycqPC6Ar69v8bpKWWbiRPIDg/i8SSfu6hxjdTTVhrN7/LOAu4AJ9u//dfLxHcIdyzKXNXz4cDIyMmjfvj3GGCIjI5k5c2aFxwUYOXIkCQkJtG/fnqlTp17YG6pUVTh5EqZPZ3G7nkTVjyC1qWdP4SzJYWWZRWQa0AOIAPYBTwAzgelAI2Antumchyvbl5Zldm/6s1SOkvnjCm6bsppBt1zBvR540ZbTyzIbY245S9OVjjqmUkqV9J/sYDLDG3Bj++jKV/YgeuWuUsr9bN5M0eDBLFm0nB6XRVKnlr/VEVUrLp34HTVMpZxHf4bKIaZMgc8/Z0+ucHNKQ6ujqXZcNvH7+/tz6NAhTRwuzBjDoUOHiqekKlUljIFp0/g1NgVTN4qerepYHVG147LVOaOjo8nMzOTAgQNWh6Iugb+/f6myFkpdsp9+gh07mNx/INe3j8bX22X7tw7jsonf19eXJk2aWB2GUqq6+c9/KPDzY36LVGYkaaeiIvqvUCnlXho0YH7qNTRpWo8WUTWtjqZactkev1JKVWT7sPsYfeQ7/i+xvtWhVFua+JVS7mPDBmZnCiIwIEET/9noUI9Syj2cOoXp3JnG/xhPpybh1A3R2WJno4lfKeUe5s1Djh3js4bJXKvDPOekiV8p5R6mT+d4aDgrmiTQL76e1dFUa5r4lVKuLycHM3cuC1t0oltsPUICfa2OqFrTxK+Ucn1ff40cP86XTToxUO+pWymd1aOUcn39+vHB3z9gXW4UH8RqiYbKaOJXSrm8Qi9v3pZGdGsdjr+vt9XhVHs61KOUcm2LF7NvxH3kHzrM1W30pO750MSvlHJtU6ZQe9oUTEAAPVpW7/tzVxea+JVSrqugADNzJuktOtIlrgGBfjp6fT408SulXNfixcihQ8xo2ol+bepaHY3L0MSvlHJdX35Jvp8/S1sk6w1XLoAmfqWUyzI1avBVmx50aB1NTX+9aOt8WTIgJiJ/AYYDBvgFGGqMOWVFLEop17Xu/scY5fsjL2iJhgvi9B6/iDQAxgDJxph4wBsY7Ow4lFIu7uhR5q//Ax8voXdslNXRuBSrToH7AAEikg8EAr9bFIdSylX17EkKYay/91mtzXOBnN7jN8bsAV4AdgF7gSxjzEJnx6GUcmG7dsHq1fwU0pjecdrbv1BWDPWEAdcCTYD6QJCI3F7BeiNFZIWIrDhw4ICzw1RKVWezZwPwdfOO9NLEf8GsmNXTC9hhjDlgjMkHvgQ6l13JGPOeMSbZGJMcGalX4ymlSpg1iz1RjQhoE0eD0ACro3E5ViT+XUAnEQkUEQGuBH61IA6llCvKysJ8+y1zYpJ1mOciOf3krjHmJxH5HFgFFACrgfecHYdSykX5+bHsiZf5bIcXr+hsnotiyaweY8wTwBNWHFsp5eICApjYsCM53sdoXb+W1dG4JL1yVynlOgoKyHvhJX5bsYFesXWwjRarC6WJXynlOpYtw2/cg8Tu2kTvOC3KdrE08SulXMfcuRR6+7C6ZQodmtS2OhqXpYlfKeUyzJw5rGoUT1LbGPx8NH1dLH3nlFKuYedOZP16FsQkcaXeUP2SaOJXSrmGlSsp8vImvXkK3S/TxH8pzpn4S5ZSEJEuZdpGOSoopZQq5/rrueW5OYQmxlM7yM/qaFxaZT3+B0o8fr1M27AqjkUppc5q37FT/HS4kJ560dYlqyzxy1keV/RcKaUcY/585IorqHfsgI7vV4HKEr85y+OKniullGP897/U2rgOv/r1aBlV0+poXF5lJRtaicg6bL37ZvbH2J83dWhkSikFYAxm3jx+aNyWrvH19WrdKlBZ4o91ShRKKXU269cju3ezqO9A+rbS8f2qcM6hHmPMzpJfwHGgPRBhf66UUo41bx4ASy9LIbVZuMXBuIfKpnPOEZF4++N6wHpss3k+FpH7HR+eUsrTmSZNmNWhPy3atcLf19vqcNxCZSd3mxhj1tsfDwUWGWOuATqi0zmVUk6wrUc/xlzxZ65opbN5qkpliT+/xOMrgXkAxphsoMhRQSmlFAC7d7Pk560AmvirUGWJf7eIjBaR67CN7S8AEJEAwNfRwSmlPNzjj3Ptrb1oFRmk99atQpUl/ruB1sAQYJAx5qh9eSdgkuPCUkp5vKIiiubPZ3F0G3po7f0qdc7pnMaY/cA9FSz/FvjWUUEppRSrV+O1bx/fJN/G4JaRVkfjVs6Z+EVk1rnajTFpVRuOUkrZzZtHkQirYjvwfOMwq6NxK5VdwJUK7AamAT+h9XmUUk5i5s5lY4OWxLdrgY+3VpCvSpW9m3WBR4F44FWgN3DQGPOdMea7iz2oiISKyOcisklEfhWR1Ivdl1LKPW19/k0e6zGcni11Nk9Vq+zK3UJjzAJjzF3YTuj+BqRXQS3+V4EFxphWQFvg10vcn1LKzSzIq8Wa6FZ01/H9KlfZUA8iUgPoD9wCxACvATMu9oAiEgJ0wzZTCGNMHpB3sftTSrmhN9/k+LpsEtpcTkRwDaujcTuVndydgm2YZx7w9xJX8V6KJsABYJKItAVWAmONMSfKHHskMBKgUaNGVXBYpZRLKCig6LHHaBGdROBN11sdjVuqbIz/dqAFMBZYIiLH7F/ZInLsIo/pg+1isLeNMe2AE8AjZVcyxrxnjEk2xiRHRupHPaU8xrJleB05wjdNk+mpV+s6RGXz+B1xKj0TyDTG/GR//jkVJH6llIeaN49CL282xncivn6I1dG4JafPkTLG/IGtFERL+6IrgY3OjkMpVT2ZuXNZ1TCODm1j8PLSGeSOUOnJXQcZDUwVET9gO7bKn0opT3fiBKeOn2RRk646zONAliR+Y8waINmKYyulqrGgIF55YxYfLf6N0S0irI7GbenlcEqp6sMYvt20n6SmkdT01wLAjqKJXylVPZw6RUF0QxIXfqnDPA6miV8pVT2kp+Pz+x72B9emh5ZpcChN/Eqp6mH2bHL9/MlM7ESzyCCro3FrmviVUtYzhqI5c/g+JpHL20QjotM4HUkTv1LKer/8gteuXSxqmsKVsTrM42ia+JVS1qtVi8Vpd/FTbCc6Ngm3Ohq3Z9UFXEopVayoUWMe6nA7yTFh+Plof9TRNPErpax15Ag7Fv3A4aNFXNmqldXReAT916qUstasWTQblEbswQyu0Pn7TqGJXyllrdmzORgSQY2UZGoH+VkdjUfQxK+Usk5uLkULF7IwJokr4+paHY3H0MSvlLLO//6HV3Y2C1t0opdO43QaTfxKKessWECOfxC726XSvE6w1dF4DJ3Vo5SyzInn/sVNua3pGq9X6zqT9viVUpb5bvsRNoZF06e1ju87k/b4lVLWePFFfL/fQFj7waTEhFkdjUfRHr9SyvmMwbz9NgEbfqFXbBQ+3pqKnEnfbaWU823YgGzbxrxmHXWYxwKa+JVSzjdjBkUi/BDXmcv13rpOZ1niFxFvEVktInOsikEpZQ0zcybro2OJT2qJv6+31eF4HCtP7o4FfgVqWRiDUsrZ8vM5HNOc6eHhOsxjEUt6/CISDfQHPrDi+EopC/n68u7dT/JpygAtymYRq4Z6XgH+ChRZdHyllEXMli18teEPUptFUMvf1+pwPJLTE7+IDAD2G2NWVrLeSBFZISIrDhw44KTolFIOtWUL0rIlHb6dSb94HeaxihU9/i5AmohkAJ8APUXk32VXMsa8Z4xJNsYkR0ZGOjtGpZQjTJ8OwNKmSfTV8X3LOD3xG2P+ZoyJNsbEAIOBb4wxtzs7DqWU85lPP2VtTDzNk1oRprX3LaPz+JVSzrFxI7J+PV8278KAhPpWR+PRLE38xph0Y8wAK2NQSjnJ9OkYEb6O68pVraOsjsajaZE2pZRTFN7/F/6SUYO4pJY6m8diOtSjlHKK5YcLmFU3gWva6jCP1bTHr5RyvJde4sCmQ/hHXc6VetGW5bTHr5RyrLw8zLPP4r/kB66MjSKohvY3raaJXynlWHPnIocOMTX2CtJ0mKda0MSvlHKsyZPJCglnY+tO9NRhnmpBE79SynH278fMm8f02B4MSGqEr95pq1rQn4JSynEOHWJv2w5Mj+vJTcnRVkej7PQsi1LKcWJjGXHnBGoIxNbTW29UF9rjV0o5xu7dbFm9hQ2/H+OmpIZWR6NK0MSvlHKMxx+nYbcUgkyhzuapZjTxK6Wq3sGDmGnTmB3Xne4JDbQSZzWjiV8pVfU++ADJzeW9Nv24OVmHeaobTfxKqapVUABvv826y5IobBVLtxZ6I6XqRhO/UqpqrViByczkzdZ9uDO1MV5eYnVEqgydzqmUqlqdOvH0a7NZutfwQpLO3a+ONPErpapOfj4Hc4v4+Hfhlg6NqKl196slHepRSlWdIUM4fM315BUUcmfnGKujUWehiV8pVTW2bMF88gnLCoPp1rIOzSKDrY5InYUmfqVU1Xj2WQp9/Xi1bRpDtbdfrWniV0pduu3bMf/+NzM6DCCqeSN6tNQpnNWZ00/uikhDYAoQBRjgPWPMq86OQyl+/x1yc0sv8/KCxo1tj/ftg5MnS7f7+EBD+wVJe/fCqVOl2/38oEGDs++/Rg2oby9fkJkJ+fml2wMCoG5d2+Ndu6CwsHR7YCBERdke79wJRUWl24ODIdKedHfsKP+aa9WC8HDbdjt3lm8PCYHatW3H3bWrfHtYGISG2uLOzDyz/IknKPLy5q2WvXj4yhZIXp7t9ZcVEQE1a9ret717y7fXqQNBQbb3fd++8u1169reoxMnYP/+8u316oG/P2Rnw8GD5dsbNLD9jLKy4PDh8u0NG9p+xkePwpEj5dsbNQJvb9u2WVnl22NiQMR27Ozs0m0itnaAAwfg+PHS7RX97p3+eVQ1Y4xTv4B6QHv745rAFiDuXNskJSUZparUgQPG9OljDJT+iog4s87AgeXbmzQ5096rV/n2hIQz7R07lm/v3PlMe1xc+fY+fc60N2xYvv3GG8+0h4aWbx869Ey7j0/59jFjbG05OeXbwJhHH7W1799fcftzz9nat22rsP2NG+43hYVFxqxZU/H2H31k2/6HHypu//JLW/v8+RW3L1xoa58+veL2pUtt7R9+WHH7+vW29ldeqbh9505b+9NPV9x++LCt/eGHK27Py7O133tv+bYaNc78bO644/x+9x5+2FwKYIWpIKc6vcdvjNkL7LU/zhaRX4EGwEZnx6I8WEQEfPAB/O9/pZfXqHHm8ZgxMHBg6fbgEicsx42D228v3R4WdubxY4+V73VGlhgCeeaZ8r3G+iWKmb30kq1nW9LpHiHAW29BXl7p9ubNzzz+8ENbSikpNtb23dcXJk+mnDZtbN9r1qy4vX37M6/jdHtREat2ZzH1p11ce+dA2wVbDRtWvH2XLrbvLVqce/9t2lTc3rq17XtKSsXtzZrZvnftWnH76U9jvXtX3H66d33NNRBdwTUIgYG27zfddOa9LMnb2/b9jjugQ4fSbV4lRtZHjIArryzdXtHvXnx8+WNUATFlfzGcSERigMVAvDHmWJm2kcBIgEaNGiXtrOhjqVIX48gR28fo00lAXbxlyzAjRzKk/1/5o04j5o/tqlfqViMistIYk1x2uWUnd0UkGPgCuL9s0gcwxrxnjEk2xiRHRuqJIlWF3n/fNla7Z4/Vkbi2oiIYO5ZTe/exPNef0Vc216TvIixJ/CLiiy3pTzXGfGlFDMpDGQMTJ0Jqqvb4L9XUqfDzz/yr2120aFaPq+PrWR2ROk9WzOoR4EPgV2PMS84+vvJwy5bB5s0wagR8NgQK8irdRFWgsAD+9zU5d0eRWm8FYwO24/Xpm1ZH5Z66/xXqJ1bpLq2o1dMFuAP4RUTW2Jc9aoyZZ0EsytNMmmQ7QVf/d/hlFtSJszoi13RgP/jnsS8sgtZBWYTm5kFu5Zupi1BwqvJ1LpAVs3p+AHQgUDlfXh58/jnccB1s+QziroWbJlkdlWsqKODFh99ikl9LvrmvO9T0tzoidQH0yl3lOfz8YP16uLkdnMqCDiOsjsj1HDsGBw+yaMshXvdtxpgrm1NHk77L0cSvPENRke3Ebr16kDnTNsTTKNXqqFzP/fdTmNCWv//nJ1rVrcmQzk2sjkhdBE38yjN88AFccQVsTIc/1kHK3bZL6NX5++ILmDSJ+Ul92I8vrw5uh5+PphBXpD815f5yc+Hpp23fN00Dv5qQMMjqqFzL2rVw110cjGvLX1ql8UjfVrSsW9PqqNRF0sSv3N/778Pu3fDoQ7BxBrQdDDU0aZ23ffsgLY2CkFCu7/1XOrWqxxAtu+zS9NaLyr3l5MCzz9pqt/ivg8I82zCPOn8BAeQlJTO6UR+ya0fw/I1t9QpdF6eJX7m3Dz6wlf9990VYNgYSBkOdCoprqfLy8+HkSXKDgrmjz0Os2XWUqXcmUzdEZ/G4Ok38yr3dfbetfvyJOeDlC72etDoi15CXB4MGYfbsYfzYN/l5x2FeGZRISowDasMrp9MxfuW+CgttV+l2joZNc6Dbg1BL68lUKjcXbrwRZs5kUVJvPv9lP/f3asHAdlrbyF1o4lfuadUqaNUKVq+CBY9AWAx0us/qqKq/gwehXz+YPZs5I8czMqQzt3ZsxNgrW1gdmapCOtSj3I8xMHas7SYn+76CA5tg0FTw1bHpSt15J2bJEqaNeppHgxIZ0bUJj14di+g1D25FE79yP1Onwg8/wD8fhZ9fgNhroFV/q6OqvoyxDe/4+5P93PO8/MUKJuZFcH+vFoy9soUmfTekiV+5l1Wr4E9/gk4dwGsG1GoAaW/oVbpnk5kJ99wDoaGsm/AG9y7Yx77CSJ66No47UmOsjk45iI7xK/fy3nu2++neEQU5B+DmjyAg1Oqoqp/8fHj7bWjdGvPNNyyr3YQb31pCUZHh0z+latJ3c9rjV+7ljTdg+jjYMhGufgHqt7M6oupn1Sq45RbYsoXjqZfzl16jWJQXzBUtInjx5kRqB/lZHaFyME38yvUdOQL/93/w8F9h7cuwZRIkDYWU4VZHVn3k5dlKLzRsCA0akFszhE8feZUnTFMiavjz+g1xDEiop+P5HkITv3JtmzZBWhpkZIDPOghbB10fhJ6P6bg+wM6dMGUKvPMOxMSwa9Yi3vphH59f9SQicFfHxjxw1WXU8ve1OlLlRJr4lWsyBmbMgGHDwMcLxrSC4HXQ5zlIvdfq6Kw3fz7885/w3XcAHEztxuQO1/Hm89/g6+3NrR0bcU/3ZtQPDbA4UGUFTfzKNb3+um2uftMoSMuBurnQfyrEDrA6MufLzYXly+Gbb+DPf4bISPI3byF/ZyY/DL6PV+t1ZINfbeqF+DM2pSGDUxppvR0Pp4lfuQZj4McfwdsbYqOh/i5IC4OEHOh4N/R6AvxDrI7Sefbssf3z+/lnWLoUTp3CiLDQty7T6iay6kAzjt38CsH+vvRsVYcHEuvTo2UdvLWqpsKixC8ifYFXAW/gA2PMBCviUNVcURGsXg0LF8JHk2DzVmhbF67LBQzcdg10HgvRSVZHWvUKC233B962DX77DbZuhQ0bYMgQsm4fys4dB2j94kvsa9yCFalpLAhvyY/148jKqkkzv5OkpTSmZ6s6dGkeQQ0fb6tfjapmnJ74RcQbeBPoDWQCy0VkljFmo7NjUdVIYaHtYqLt26FjWzi8HQYPgx9/sbVHe0OaP6SGQdKNthk7oY2sjfl8FRbC8eO2EhJeXhAdbVs+ZQr88QccOAB//EHR3r3kXN6dPX++n8OHj9OxXTvEGACya4WxIyqGaQu3M237QjAGv7HT8Qn0p3X9WrSNDuWphqF0iKmtwziqUlb0+DsAvxljtgOIyCfAtYAmfldljC255efbxpsD/YFC2P8HZGyH7KNwLAuOHITDB6H/5WByYOZXMPsH2HsQ9mdDYREI8Lea4CsQnQ+DIuDyDtC2J1zWz1ZLv+RsHWNsnwwKC4u/TFCQLaQjRyg6fhzy8inMz6coPx9TBPmXtaSwyCC/boT9+ynKzcPk52Py8ijy8yO7R28KiwxBC+fhvXsXJi8Pyc3F5OWRFxbOnluGUlhkaPjGCwRkbENOncLrVA5ep05xJKYFS//yJPmFhmvuuZ7aO7bid+pkcbi/JHXnjbEvcDKvkFcfeIDaxw6R6+PH/uDa7A8M5euCbbx9ajEAva8bz+81I9gX2YDgOuE0rB1Io9qBPBoeSPM6wbSoU5MGoQF6UxR1waxI/A2A3SWeZwIdHXGgpRP/SodnXsXraEGp5QUNarD3xhhbMB9sxetEUan2vCb+7EtrDED021uQPAPmTHtuywD297P1NqNf24IUmVLbn4oP5kCvBkh+IdFv/lYurpx2NTnYvT7eJ/Kp//72cu0nO4ZwKLUufodPETVlZ7n2413DONo+khr7ThL5SWa59mM9wznWJpyA3ccJ/+L3cu1Z/SI50TKEoG3HCPnv/hItttdxbGAdcprWJHjDEYLmHT7TZP86cVs4BdH+BKzMwm9eNlJU5gD3BkGkNyzNhYW55Y7PgXchxAtW5GH25WNqCaauL0U1hEIfHyYziN+8m9Du2Cqu/m4RMut7vMxivMzjeBcVkvrIlxSKN3+b9ya3rphdatenfPxo9eCXALw050Wu3/BtqfaDgSEkj54KwHtfPs1VW5eVat8ZWpc+f/oAgH9/MoHLd64t1b4+qhl35rQG4P2vvqPFwd3kefuS41uDUz5+bOAo/5ht68McC4+nZnBTTvkHkhsQyKmAYPbXbcTOQycJ8PPmiccnIyGh+NYMJiTQj5AAX+oG+vJ6kB/hQX6EB3ejbi1/agX46Px6VaWq7cldERkJjARo1OjiPtL7hDbgVM0gfE3p5JNTsyYHgi4DILLW73j75JduDwopbq9TKxOv/EIAjP1v70Rw7eL2qNBdtl5nCafbvfIKqBNWPjFnB0dwIOgyfEwOEWHlE/OxoDocCLoM/1PHCAvbV649K7Au+4JbEJR9mJCwg+Xbg+qxP7gptYL2UTP8SLn2w0HRHApuRFjw7wTUyba/tjOJZV9wU47UbEB46C6iG+TZXvfpdhG2hySSFVqPqKhtNGi9CSNgvASDF8ZLWBvRhyOR9YhuuoGGnTdQ6OVDkZcPBd6+5HvV4OumwzkUFk2jk2tplb0ExAtEMAh4eZEZezc+AYEUHQ9iE76AYLy9wUsw4sX17RqAjy/epg/fN2+A8fEBL2+MtzfG19deWAwkYgjf7r0S4+0Fvr7g60thjQCe7BaHt5eQH/843xzLAj9f8PVD/PwoCgzgzSYt8PaC/Gv+w2IM4u+P1PDDy68G3j7efOEteIngO3ohp7wFHy8hxMuLcG8hxtuL67y98PXxws+7H77eoklbVTtiyiQthx9QJBV40hjTx/78bwDGmOfOtk1ycrJZsWKFkyJUSin3ICIrjTHJZZdbUaRtOdBCRJqIiB8wGJhlQRxKKeWRnD7UY4wpEJFRwFfYpnNONMZscHYcSinlqSwZ4zfGzAPmWXFspZTydFqPXymlPIwmfqWU8jCa+JVSysNo4ldKKQ+jiV8ppTyM0y/guhgicgAoX7vg/EQA5S9vtZ7GdWE0rgujcV2Y6hoXXFpsjY0xkWUXukTivxQisqKiK9espnFdGI3rwmhcF6a6xgWOiU2HepRSysNo4ldKKQ/jCYn/PasDOAuN68JoXBdG47ow1TUucEBsbj/Gr5RSqjRP6PErpZQqwaMSv4g8KCJGRCKsjgVARJ4SkXUiskZEFopIfatjAhCR50Vkkz22GSISanVMACJyk4hsEJEiEbF8BoaI9BWRzSLym4g8YnU8ACIyUUT2i8h6q2MpSUQaisi3IrLR/jMca3VMACLiLyI/i8hae1x/tzqmkkTEW0RWi8icqtyvxyR+EWkIXAXssjqWEp43xiQYYxKBOcDjFsdz2iIg3hiTAGwB/mZxPKetB64HFlsdiIh4A28C/YA44BYRibM2KgAmA32tDqICBcCDxpg4oBNwXzV5v3KBnsaYtkAi0FdEOlkbUiljgV+reqcek/iBl4G/UuruudYyxhwr8TSIahKbMWahMeb0jYqXAdFWxnOaMeZXY8xmq+Ow6wD8ZozZbozJAz4BrrU4Jowxi4HDVsdRljFmrzFmlf1xNrZk1sDaqMDYHLc/9bV/VYu/QxGJBvoDH1T1vj0i8YvItcAeY8zaSld2MhF5RkR2A7dRfXr8JQ0D5lsdRDXUANhd4nkm1SCRuQIRiQHaAT9ZHApQPJyyBtgPLDLGVIu4gFewdVaLqnrH1fZm6xdKRL4G6lbQNB54FNswj9OdKy5jzH+NMeOB8fZ7D48CnqgOcdnXGY/tI/pUZ8R0vnEp1yUiwcAXwP1lPvFaxhhTCCTaz2XNEJF4Y4yl50hEZACw3xizUkR6VPX+3SbxG2N6VbRcRNoATYC1IgK2YYtVItLBGPOHVXFVYCq2u5I5JfFXFpeIDAEGAFcaJ875vYD3y2p7gIYlnkfbl6mzEBFfbEl/qjHmS6vjKcsYc1REvsV2jsTqk+NdgDQRuRrwB2qJyL+NMbdXxc7dfqjHGPOLMaaOMSbGGBOD7SN5e2ck/cqISIsST68FNlkVS0ki0hfbR8w0Y8xJq+OpppYDLUSkiYj4AYOBWRbHVG2Jrdf1IfCrMeYlq+M5TUQiT89aE5EAoDfV4O/QGPM3Y0y0PWcNBr6pqqQPHpD4q7kJIrJeRNZhG4qqFlPcgDeAmsAi+1TTd6wOCEBErhORTCAVmCsiX1kVi/3k9yjgK2wnKqcbYzZYFc9pIjINWAq0FJFMEbnb6pjsugB3AD3tv1Nr7L1Zq9UDvrX/DS7HNsZfpVMnqyO9clcppTyM9viVUsrDaOJXSikPo4lfKaU8jCZ+pZTyMJr4lVLKw2jiVy5HRI5XvtZZtx1lr6ZZqkqr2Lxmb1snIu1LtNU7XR1RRHpUVaVEEUk/nyqjIpJRWUVZEflaRMKqIi7l/jTxK0/zI9AL2FlmeT+ghf1rJPB2ibYHgPedEt3F+xi41+oglGvQxK9clr2X/rz9IrhfRGSQfbmXiLxlv6fAIhGZJyI3AhhjVhtjMirY3bXAFHu1xmVAqIjUs7fdACyo4PgdRGSpvV76EhFpaV8+RERm2o+dYf+U8YB9vWUiUrvEbu6wX8y0XkQ62LcPF9v9GTaIyAeAlDjmTBFZaW8bWWI/s4BbLva9VJ5FE79yZddjq6HeFlsv/nl7sr4eiMFWJ/8ObFf6VqbCapsi0gQ4YozJrWCbTUBXY0w7bJVVny3RFm+PIwV4BjhpX28pcGeJ9QLt92O4F5hoX/YE8IMxpjUwA2hUYv1hxpgkIBkYIyLhAMaYI0CN08+VOhe3KdKmPNLlwDR7dcV9IvIdtkR7OfCZMaYI+MNeeOti1QMOnKUtBPjIXnPJYKvlftq39rrz2SKSBcy2L/8FSCix3jSw1dEXkVr2ujHdsP3TwBgzV0SOlFh/jIhcZ3/cENvQ1CH78/1A/RLPlaqQ9viVsjlbtc0cbNURK/IUtgQfD1xTZr2SnxCKSjwvonSHq2zNlLPWULGX5+0FpNrvGLW6zDH97fEqdU6a+JUr+x4YZL+RRiS2nvLP2E7g3mAf648CepzHvmYBd9rPG3QCsowxe7HdejLmLNuEcKYU85CLfA2nz0tcbj9mFrZbS95qX94POD1bJwTbsNNJEWmF7RaG2NcTbPcxyLjIOJQH0cSvXNkMYB2wFvgG+Ku93PYX2MboNwL/BlYBWQAiMsZe4TMaWGc/eQq2eyFsB37DNoPnXgBjzAlgm4g0r+D4/wKeE5HVXPyw6Sn79u8Apytp/h3oJiIbsA35nL5P9ALAR0R+BSZguy3maUnAshK3zFTqrLQ6p3JLIhJsjDluP9n5M9DlYu/BYB9TTzLG/F+VBlmFRORVYJYx5n9Wx6KqPz25q9zVHPuJUj/gqUu58Y4xZoYLzJZZr0lfnS/t8SullIfRMX6llPIwmviVUsrDaOJXSikPo4lfKaU8jCZ+pZTyMJr4lVLKw/w/y8f9sS3i/8EAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "import os\n", "import numpy as np\n", @@ -2039,9 +2040,7 @@ { "cell_type": "markdown", "id": "6bde8a5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## To think about, first part\n", "\n", @@ -2067,9 +2066,7 @@ { "cell_type": "markdown", "id": "18302ffc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More thinking\n", "\n", @@ -2102,9 +2099,7 @@ { "cell_type": "markdown", "id": "7ef0a4ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Still thinking\n", "\n", @@ -2116,10 +2111,7 @@ "cell_type": "code", "execution_count": 7, "id": "2b20819b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "#Model training, we compute the mean value of y and X\n", @@ -2141,9 +2133,7 @@ { "cell_type": "markdown", "id": "ab5cfc7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## What does centering (subtracting the mean values) mean mathematically?\n", "\n", @@ -2157,9 +2147,7 @@ { "cell_type": "markdown", "id": "78a93e96", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta_0, \\beta_1, ... , \\beta_{p-1}) = \\frac{1}{n}\\sum_{i=0}^{n} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij}\\beta_j\\right)^2,.\n", @@ -2169,9 +2157,7 @@ { "cell_type": "markdown", "id": "ef53e0f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Recall also that we use the squared value since this leads to an increase of the penalty for higher differences between predicted and output/target values.\n", "\n", @@ -2184,9 +2170,7 @@ { "cell_type": "markdown", "id": "ced64364", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_j} = 0,\n", @@ -2196,9 +2180,7 @@ { "cell_type": "markdown", "id": "4ae40674", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "for all $j$. For $\\beta_0$ we have" ] @@ -2206,9 +2188,7 @@ { "cell_type": "markdown", "id": "df1eae31", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial C}{\\partial \\beta_0} = -\\frac{2}{n}\\sum_{i=0}^{n-1} \\left(y_i - \\beta_0 - \\sum_{j=1}^{p-1} X_{ij} \\beta_j\\right).\n", @@ -2218,9 +2198,7 @@ { "cell_type": "markdown", "id": "5f52ce83", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying away the constant $2/n$, we obtain" ] @@ -2228,9 +2206,7 @@ { "cell_type": "markdown", "id": "91e8c0ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sum_{i=0}^{n-1} \\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} \\sum_{j=1}^{p-1} X_{ij} \\beta_j.\n", @@ -2240,9 +2216,7 @@ { "cell_type": "markdown", "id": "4bf5c771", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Further Manipulations\n", "\n", @@ -2253,9 +2227,7 @@ { "cell_type": "markdown", "id": "a41aa187", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "n\\beta_0 = \\sum_{i=0}^{n-1}y_i - \\sum_{i=0}^{n-1} X_{i1} \\beta_1.\n", @@ -2265,9 +2237,7 @@ { "cell_type": "markdown", "id": "c5841919", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We obtain then" ] @@ -2275,9 +2245,7 @@ { "cell_type": "markdown", "id": "59fc8eac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\beta_1\\frac{1}{n}\\sum_{i=0}^{n-1} X_{i1}.\n", @@ -2287,9 +2255,7 @@ { "cell_type": "markdown", "id": "eae08d74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we define" ] @@ -2297,9 +2263,7 @@ { "cell_type": "markdown", "id": "c0617587", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_1=\\frac{1}{n}\\sum_{i=0}^{n-1} (X_{i1},\n", @@ -2309,9 +2273,7 @@ { "cell_type": "markdown", "id": "05c2ee52", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and if we define the mean value of the outputs as" ] @@ -2319,9 +2281,7 @@ { "cell_type": "markdown", "id": "d7d01a2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mu_y=\\frac{1}{n}\\sum_{i=0}^{n-1}y_i,\n", @@ -2331,9 +2291,7 @@ { "cell_type": "markdown", "id": "0098e33a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have" ] @@ -2341,9 +2299,7 @@ { "cell_type": "markdown", "id": "117406d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\mu_y - \\beta_1\\mu_{1}.\n", @@ -2353,9 +2309,7 @@ { "cell_type": "markdown", "id": "6b663f7d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the general case, that is we have more parameters than $\\beta_0$ and $\\beta_1$, we have" ] @@ -2363,9 +2317,7 @@ { "cell_type": "markdown", "id": "d9a37db9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_0 = \\frac{1}{n}\\sum_{i=0}^{n-1}y_i - \\frac{1}{n}\\sum_{i=0}^{n-1}\\sum_{j=1}^{p-1} X_{ij}\\beta_j.\n", @@ -2375,9 +2327,7 @@ { "cell_type": "markdown", "id": "cf1a45ea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Replacing $y_i$ with $y_i - y_i - \\overline{\\boldsymbol{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)" ] @@ -2385,9 +2335,7 @@ { "cell_type": "markdown", "id": "282882c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}) = (\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta})^T(\\boldsymbol{\\tilde{y}} - \\tilde{X}\\boldsymbol{\\beta}).\n", @@ -2397,9 +2345,7 @@ { "cell_type": "markdown", "id": "41184eb0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wrapping it up\n", "\n", @@ -2409,9 +2355,7 @@ { "cell_type": "markdown", "id": "23644cc4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X})^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}},\n", @@ -2421,9 +2365,7 @@ { "cell_type": "markdown", "id": "ce98f4a7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\tilde{y}} = \\boldsymbol{y} - \\overline{\\boldsymbol{y}}$\n", "and $\\tilde{X}_{ij} = X_{ij} - \\frac{1}{n}\\sum_{k=0}^{n-1}X_{kj}$.\n", @@ -2434,9 +2376,7 @@ { "cell_type": "markdown", "id": "7b623b49", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}} = (\\tilde{X}^T\\tilde{X} + \\lambda I)^{-1}\\tilde{X}^T\\boldsymbol{\\tilde{y}}.\n", @@ -2446,9 +2386,7 @@ { "cell_type": "markdown", "id": "7d0ed96c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "What does this mean? And why do we insist on all this? Let us look at some examples." ] @@ -2456,9 +2394,7 @@ { "cell_type": "markdown", "id": "f6f8c0c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear Regression code, Intercept handling first\n", "\n", @@ -2468,13 +2404,44 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 13, "id": "e58d3f27", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "True beta: [2, 0.5, 3.7]\n", + "Fitted beta: [2.08376632 0.19569961 3.97898392]\n", + "Sklearn fitted beta: [2.08376632 0.19569961 3.97898392]\n", + "MSE with intercept column\n", + "0.004113634617443142\n", + "MSE with intercept column from SKL\n", + "0.004113634617443129\n", + "Manual intercept: 2.0837663229239056\n", + "Fitted beta (wiothout intercept): [0.19569961 3.97898392]\n", + "Sklearn intercept: 2.0837663229239043\n", + "Sklearn fitted beta (without intercept): [0.19569961 3.97898392]\n", + "MSE with Manual intercept\n", + "0.004113634617443132\n", + "MSE with Sklearn intercept\n", + "0.00411363461744314\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA7MElEQVR4nO3deVhV1frA8e8GEY+CoqKIQ+KQCsggaKKkgeY8hFNqZtqtzGuD10rTrFRuA4UN2vBrNLVraTmWDQ4BaQ4VCIihpCSKSIooCMjM+v2BnESmI9M5wPt5nvMIZ6999rs4+LLO2nu/S1NKIYQQwnSZGTsAIYQQ5ZNELYQQJk4StRBCmDhJ1EIIYeIkUQshhIlrVBMvamtrqxwcHCq1b0ZGBs2aNavegEyc9Ln+a2j9BenzrQoLC7uklGpT2rYaSdQODg6EhoZWat+QkBB8fHyqNyATJ32u/xpaf0H6fKs0TTtT1jaZ+hBCCBMniVoIIUycJGohhDBxNTJHXZrc3FzOnTtHVlZWue1atGjB8ePHaykq0yB9rhuaNGlCx44dsbCwMHYoooGptUR97tw5rK2tcXBwQNO0MtulpaVhbW1dW2GZBOmz6VNKkZyczLlz5+jSpYuxwxENTK1NfWRlZdG6detyk7QQpkrTNFq3bl3hJ0IhakKtzlFLkhZ1mfz+CmOptakPIYSor7aHJ/D9tvX4ND3LwIN5LBrliF+fDtX2+g3qqg9zc3Pc3d31j7i4OAYOHAhAXFwcX3zxhZEjFELUNdvDE1i15SfGHF1Ls1MhpFxNZcnWKLaHJ1TbMRpUotbpdEREROgfDg4OHDx4EJBELYSonNU/hPO+WSCDXK5y2nE612hCZm4+gbtiqu0YDSpRl8bKygqAxYsXs3//ftzd3XnrrbeMHJUQoi7IzcnmoailNM9JZH7+k5i3c9NvO5+SWW3HMcoc9Ypv/yD6/NVSt+Xn52Nubn7Lr+nUvjnLxjmX2yYzMxN3d3cAunTpwrZt2/TbAgICWLlyJTt37rzlYwshGqYtc3zwCM9hk86V/W1duYM8/bb2NrpqO06DOplYNPUhhBBV9dUL03A7nEJU7yZ8ZP8U5P+zTWdhzsIRPavtWEZJ1OWNfOvajRBCiIZn7/pX6Lk1krgOGqM//YnGf2Ven5NOo4ONjoUjelbrVR8NakRdHmtra9LS0owdhhDCxF04F0vu+s9JawY931mLVYtW+PUBvz4dCAkJ4YkZPtV+zAZ/MrGIq6sr5ubmuLm5yclEIUSpMjPSuPrZvbj1T8F82SK6Ot1RK8dtUCPq9PT0Mp+zsLAgKCiotkMSQtQR+Xl5fPPUUMa2jeXkkA8YMHRarR3boBG1pmk2mqZt1jTthKZpxzVNG1DTgQkhhCn5eq4vrj+nsTNzIO61mKTB8BH1KuBHpdRkTdMaA01rMCYhhDApW/wfwO2XS0Q7WTI54JtaP36FiVrTtBbAYGA2gFIqB8ip2bCEEMI0BG98k25f/c5Ze427P/kR80a1P2OsKaXKb6Bp7sBHQDTgBoQB85VSGTe1mwPMAbCzs/PcuHFjsddp0aIF3bt3rzCgyt7wUpdJn+uOU6dOkZqaesv7paen6++CbSjqQ58zUi6gW7kcy2y4MP8JWnV0Krd9Vfrs6+sbppTqW9o2QxJ1X+Aw4K2U+lXTtFXAVaXUC2Xt07dvX3XzKuTHjx/H0dGxwmAb4nXU0ue6w9Df45vJitx1T1ZmBmfeGEJB6jku9FuEz73zK9yniquQl5moDTmZeA44p5T69fr3mwGPSkUihBB1QH5eHt89P4rbc0+QdfcrBiXpmlRholZK/Q3Ea5pWdD/kUAqnQYQQol766vFhOH2XxObcu+gzYpaxwzH4hpcngA2aph0F3IFXaiyiGvTyyy/j7OyMq6sr7u7u/Ppr4YcEBwcHLl26VKK9MebXtm/fjr+/PwBJSUn079+fPn36sH//fkaPHk1KSgopKSm8//775b7O+fPnmTx5coXHe+UV472VlSkt6+DggIuLi76meFGZ2rLalva+VuSZZ56Ra+obsG0Bc3AN+ZsTPS2Y5L/V2OEUUkpV+8PT01PdLDo6usRzpbl69apB7W7VwYMHlZeXl8rKylJKKZWUlKQSEhKUUkp17txZJSUlldinWbNm1XLsvLy8crff2OcBAwboY/nyyy/VQw89VKL96dOnlbOzc7XEVpk+VtQfQ1y9elUFBwerMWPG3NJ+Zb1XVW17o7i4ODVs2LBStxn6e3yz4ODgSu1Xl9XFPu/b8q4Kc+mldg92VJcvnrvl/avSZyBUlZFTjXNn4g+L4e+oUjfp8vPAvBJhtXOBUQFlbk5MTMTW1hZLS0sAbG1tS7TJzMxk4sSJTJw4kUceeaTYtsDAQL766iuys7OZMGECK1asAMDPz4/4+HiysrKYP38+c+bMAQpH448++ih79+7lvffeY+TIkcyfP5+dO3ei0+nYsWMHdnZ2xY7x559/Ymlpia2tLRERESxatIjMzExCQ0M5dOgQjo6OhIaGsnjxYmJjY3F3d2fYsGEEBgaW6EtcXBxjx47l2LFjrF27lm+++YZr164RGxvLhAkTeP3111m8eLG+9KuzszMbNmzgf//7H6tXryYnJ4f+/fvz/vvvY25uXqI/f/31FytXrkTTNFxdXfn8889JSkpi7ty5nD17FoC3334bb29vli9fTmxsLKdOneLSpUssWrSIadOmsXjxYo4fP467uzuzZs1iwYIFt/CG/6Os96BIRkYG9957L+fOnSM/P58XXniBqVOnEhYWxlNPPUV6ejq2trasXbsWe3t7OnfuTHJyMn///Tft2rWrVEyi7vk7/iTq9XfJbgwdV66mZZvqK6pUVQ2m1sfw4cOJj4+nR48ezJs3j59//rnY9vT0dMaNG8f06dNLJOndu3dz8uRJfvvtNyIiIggLC2Pfvn0ArFmzhrCwMEJDQ1m9ejXJyclAYXLo378/kZGR3HnnnWRkZODl5UVkZCSDBw/m448/LhHjgQMH8PAoPE/r7u6Ov78/U6dOJSIiAp3un9q2AQEBdOvWjYiIiFKTdGkiIiLYtGkTUVFRbNq0ifj4eAICAvSlXzds2MDx48fZtGkTBw4cICIiAnNzczZs2FCiPy1btuSll14iKCiIyMhIVq1aBcD8+fNZsGABv//+O1u2bOHhhx/WH//o0aMEBQVx6NAh/P39SUxMJCAggEGDBhEREcGCBQuIiYkptlTajY+UlBT9a/n6+uLu7k7//v3LfQ+K/Pjjj7Rv357IyEiOHTvGyJEjyc3N5YknnmDz5s2EhYXxr3/9i6VLl+r38fDw4MCBAwb9bEXdl5OdxeX1szHvm0HG/Fk49rvb2CEVY5wRdTkj38waumzLysqKsLAw9u/fT3BwMFOnTiUgIIDZs2cDcM8997Bo0SJmzJhRYt/du3eze/du+vTpAxQm9ZMnTzJ48GBWr16tX4AgPj6ekydP0rp1a8zNzZk0aZL+NRo3bszYsWMB8PT0ZM+ePSWOk5iYSJs2baq76wAMHTqUFi1aAODk5MSZM2fo1KlTsTY//fQTYWFh9OvXDyj8hNG2bVuAYv0JCgpiypQp+k8lrVq1AmDv3r1ER/9znvnq1av6Wir33HMPOp0OnU6Hr68vYWFh2NvbFzt+z549DaoXHhwcXOwTUVnvQREXFxeefvppnn32WcaOHcugQYM4duwYx44dY9iwYUDhdd03xtO2bVvOnz9fYSyi7svPy2P3KxMYqx0jdNRKvMc8UvFOtaxBFWUyNzfHx8cHHx8fXFxcWLdunT5Re3t78+OPP3LfffehaVqx/ZRSLFmyhEcffbTY8yEhIezdu5dDhw7RtGlTfHx8yMrKAqBJkybFbuiwsLDQv665uTl5eXncTKfTVepmCkMUTfmUd3ylFLNmzeLVV18tse3m/pSmoKCAw4cP06RJkxLbbv6Z3vw9QExMDFOnTi31tUNCQrCxsSn1+bLegyI9evTgyJEjfP/99zz//PMMHTqUCRMm4OzszKFDh0o9XlZWVrFPMaL++nrBaFz2xrNt2jAmmGCShgY09RETE8PJkyf130dERNC5c2f99/7+/rRs2ZLHHnusxL4jRoxgzZo1+tFhQkICFy9eJDU1lZYtW9K0aVNOnDjB4cOHqxSjo6Mjp06dqrBdddbOtrCwIDc3FygcdW/evJmLFy8CcPnyZc6cOVNinyFDhvD111/rpxguX74MFE4vvfPOO/p2N46Od+zYQVZWFsnJyYSEhODh4VGiH0Uj6tIepSVpwKD34Pz58zRt2pT777+fhQsXcuTIEXr27ElSUpI+Uefm5vLHH3/o9/nzzz/p3bu3IT9CUYfteOMJXPbGc7JbI8Yu2WDscMrUYBJ1eno6s2bNwsnJCVdXV6Kjo1m+fHmxNqtWrSIzM5NFixYVe3748OHcd999DBgwABcXFyZPnkxaWhojR44kLy8PR0dHFi9ejJeXV5ViHDx4MOHh4agK7hZt3bo13t7e9O7dm4ULF1bpmHPmzMHV1ZUZM2bg5OTESy+9xPDhw3F1dWXYsGEkJiaW2MfZ2ZmlS5dy11134ebmxlNPPQUUTkGEhobi6uqKk5MTH3zwgX4fV1dXfH198fLy4oUXXsDe3r5aaoAb8h5ERUVxxx134O7uzooVK3j++edp3Lgxmzdv5tlnn8XNza3YpX65ubmcOnWKvn1LvUlM1BMHv/uMjuv28rctDPxwOxaNLSveyVjKuhykKg9TvDzPlN3Y5yeffFLt2bPHiNFUv2XLlqnAwMBiz5ny+7x161b1/PPPl7pNLs8znCn3OeGvaPVz/17qcJ9eKmL/jmp73Zq6PK/BjKjriueee45r164ZO4wGLS8vj6efftrYYYgakpuTTfKXc8jolU3KY1Nxu3O8sUOqUIM6mVgX2NnZMX684b84UVFRzJw5s9hzlpaW+rsuTcHNU0ymbsqUKcYOQdSgn99+gLtzjpLl9zL9/B43djgGkURdx7m4uBh0SZsQAjYuGM3tQaf5bupIxtSRJA0N6GSiEKJh2/nuM/T+8TQJ7c25e8Fnxg7nlkiiFkLUe7/v/pJ2n3xHUivo98FmLHV1azVBSdRCiHot4XQ0Gcv9KdCgub8/9p17GTukW9agErW5uXmx+hFxcXEMHDgQqLjkZmJiov4WcFMye/ZsNm/eXOL5F198kb1795a7b0hISLllQmva2rVri92mPW3atGI3JQlRVXm5OSRueJw0u3yS5vjhObRunihuUIm6qABR0cPBwUGfqCpK1G+++WaJYk2mzN/fn7vvLr+wTGUSdWm3nlfWzYn63//+N6+//nq1vb4QBz94lL55YdjNmMuYf5csjVBXNKhEXZqixQEWL17M/v37cXd3L/UuuS1btjBy5EigMMH4+fkxbNgwHBwcePfdd3nzzTfp06cPXl5e+luqP/74Y/r164ebmxuTJk3SXx89e/ZsnnzySQYOHEjXrl3Zvn07UJg4bxy1P/7446xduxYoTLz9+vWjd+/ezJkzp8K7F28caTs4OLBs2TI8PDxwcXHhxIkTxMXF8cEHH/DWW2/h7u7O/v37SUpKYtKkSfTr149+/frpq8ctX76cmTNn4u3tzcyZM7lw4QITJkzAzc0NNzc3fbL/3//+p78D8NFHHyU/P1//M16wYAHOzs4MHTqUpKQktm/fTmhoKDNmzMDd3Z3MzEwGDRrE3r17q/WPgWi4Ni3ygy8OsafpKO6Y/JSxw6kSo12e9+CPD5Z4boTDCMZ0GENmXibz9s4rsf2e7vfg192PK1lXeCqk+A/+s5EVn8Utqr0M0KVLF33FNSgsHbpy5Up27txZYr/Tp0/TsmXLYoWNjh07Rnh4OFlZWXTv3p3XXnuN8PBwFixYwPr16/nPf/5TrK71888/z6effsoTTzwBFE6l/PLLL5w4cYKxY8eWuBb6Zo8//jgvvvgiADNnzmTnzp2MGzeuwj4XsbW15ciRI7z//vusXLmSTz75hLlz52JlZcUzzzwDwH333ceCBQu48847OXv2LCNGjOD48eMAREdH88svv6DT6Zg6dSp33XUX27ZtIz8/n/T09GIlUi0sLJg3bx4bNmzggQceICMjg759+/LWW2/h7+/PihUrePXVV/n0009ZuXJlsVu1u3fvTmRkJJ6engb3TYgi28MTCNwVQ59zG3g0KIYzHc0Z+thHxg6ryhrUddRFUx+3qrTyo76+vlhbW2NtbU2LFi30SdPFxYWjR48Chcn8+eefJyUlhfT0dEaMGKHf38/PDzMzM5ycnEhKSqowhuDgYF5//XWuXbvG5cuXcXZ2vqVEPXHiRKCwxOrWraUvL1RemdLx48frq8kFBQWxfv16oHDev0WLFnz++edllkg1MzPTV8W7//779bGUpqi8qCRqcau2hyewZGsU3a/9zgMHDnK5BQR6PEnen2n49Wlu7PCqxGiJuqwRcFpaGrpGunJHyC2btDRoBF1ddDpdidKZN46uzczM9N+bmZnpP7rPnj2b7du34+bmxtq1awkJCSl1/6JpjEaNGlFQUKB/vuiYWVlZzJs3j9DQUDp16sTy5ctLxFORouOVVeIUyi9T2qxZs3JfX5VTIvVmpZU4LSLlRUVlBe6KoXnmWRYc/gKzAnjHaxJn6Ejgrhj8+pjOai2V0eDnqIuUVzq0R48exMXF3fJrpqWlYW9vT25urn6llPJ07tyZ6OhosrOzSUlJ4aeffgL+Sdi2trakp6eXepVHZdzc5/LKlN5o6NCh/N///R9QWHA/NTW13BKpBQUF+pi/+OIL7rzzzlKPD1JeVFReckoKARYfkddY8cWdnoRaDgDgfEqmkSOrOknU15VXcrNZs2Z069bNoFrRN/rvf/9L//798fb2pleviq/d7NSpE/feey+9e/fm3nvv1a8oY2NjwyOPPELv3r0ZMWKEfnqhqsaNG8e2bdv0JxPLK1N6o1WrVhEcHIyLiwuenp5ER0eXWyK1WbNm/Pbbb/Tu3ZugoCD9XPvs2bOZO3eu/mTihQsX0Ol0sk6huGUZaam8Y/42dzU+xXavSWy1nq7f1t6m7n9C0yq6eqAy+vbtq0JDQ4s9d/z4cRwdHSvcN62GluKqqm3bthEWFsZLL71U7a9tqn2uLlZWVvq57iKl9fmtt96iefPmPPTQQ7UZ3i0x9Pf4ZiEhIfj4+FR/QCastvqcm5PNt/fegS4zi4N9R/F5/jD9Np2FOa9OdKm1qY+q9FnTtDClVKlF0GVEbaAJEybg4OBg7DDqNRsbG2bNmmXsMEQdkp+Xx7ZZ3jieyCGlczs8Jy+kg40ODehgo6vVJF2TGtRVH1V146rawnA3j6bL8uCDJS/ZFKI8X8/1xS08g6OeVtz7/k+YN2pULxLzzWRELYSokzY9NQa3Xy4R7WTJxM9+wbxR/R13SqIWQtQ5oTs/wkULI9rZgtGf7zPt9Q6rgSRqIUSdEvzlStx+XwytuzPm84PomtXtm1kMIYlaCFFn7Pp0OS1f/pRvz7Wn47wdNGlqZeyQaoUkaiFEnbBvy7u0Xr2JVGtwefJDmtu0NnZItaZBJeqXX34ZZ2dnXF1dcXd31y8A6+DgwKVLl0q0L6qsV5u2b9+Ov78/AElJSfTv358+ffqwf/9+Ro8eTUpKCikpKbz//vvlvs758+eZPHlyhcd75ZVXqiXuyqiotOzN+vfvj7u7O7fddhtt2rQpVle8KqQOtukL++lrGr/yHjkW0CLgFW538zZ2SLVLKVXtD09PT3Wz6OjoEs+V5urVqwa1u1UHDx5UXl5eKisrSymlVFJSkkpISFBKKdW5c2eVlJRUYp9mzZpVy7Hz8vLK3X5jnwcMGKCP5csvv1QPPfRQifanT59Wzs7O1RJbZfpYUX8McfXqVRUcHKzGjBlzy/t+9tln6rHHHivxfG5ubqViCQkJUQ8//LBBbQ39Pb5ZcHBwpfary6qrz+dOH1chXr3Ub2691IEdH1XLa9aUqvQZCFVl5FSjjajPzHygxOPy9dFVQWZmqdtTthaWJc27cqXEtookJiZia2urL05ka2tL+/bti7XJzMxk1KhRfPzxxyX2DwwMpF+/fri6urJs2TL9835+fnh6euLs7MxHH/1TTtHKyoqnn34aNzc3Dh06hJWVFUuXLsXNzQ0vLy8uXLhQ4hh//vknlpaW2NraEhERwaJFi9ixY4f+Fuuikf/ixYuJjY3F3d2dhQsXltrfuLg4fc2MtWvXMnHiREaOHMntt9/OokWLgMIa3EWlX2fMmAGUX1P6xv6sX78eV1dX3Nzc9CVaK6pnPWDAAG6//Xb9z7eiGuCGuLlW9tq1a3n88X9Wlx47dqy+GNbu3bsZMGAAHh4eTJkyRX99t9TBNl2pVy6RvX4q+e6ZXPnPDAaOrzuLd1SnBjP1MXz4cOLj4+nRowfz5s3j559/LrY9PT2dcePGMX369BIruezevZuTJ0/y22+/ERERQVhYGPv27QNgzZo1hIWFERoayurVq0lOTgYgIyOD/v37ExkZyZ133klGRgZeXl5ERkYyePDgUv8YHDhwAA8PDwDc3d3x9/dn6tSpREREFKsoFxAQQLdu3YiIiCAwMNCg/kdERLBp0yaioqLYtGkT8fHxBAQE6Eu/btiwoVhN6YiICMzNzfXFpG7sT8uWLXnppZcICgoiMjKSVatWATB//nwWLFjA77//zpYtW4rdIHT06FGCgoI4dOgQ/v7+JCYmEhAQwKBBg4iIiGDBggXExMQUWyrtxkdKSkqZfYuOjmbv3r18+eWXZba5dOkSL730Env37uXIkSP07duXN998EyiseFhUB1uYjssX4jn00nA65sfTZvpbDJ/9vLFDMhqjXSHe+fP1pT6flpaGmU5X5naARi1blru9NFZWVoSFhbF//36Cg4OZOnUqAQEBzJ49G4B77rmHRYsW6UeWN9q9eze7d+/WF0lKT0/n5MmTDB48mNWrV+sXIIiPj+fkyZO0bt0ac3NzJk2apH+Nxo0b61dv8fT0ZM+ePSWOU1rd6+oydOhQWrRoAYCTkxNnzpyhU6dOxdr89NNPZdaUvrE/QUFBTJkyBVtbWwBatWoFlF/P+p577kGn06HT6fD19SUsLAx7e/tix+/Zs2el6oXfWCu7LIcPHyY6Ohpv78K5zZycHAYMGKDfLnWwTUtGWioHZo/itrP5/LLkGYYMnmDskIyq/t7KUwpzc3N8fHzw8fHBxcWFdevW6RO1t7c3P/74I/fdd1+JeslKKZYsWcKjjz5a7PmQkBD27t3LoUOHaNq0KT4+PvqSpE2aNMHc3Fzf1sLCQv+6ZdWE1ul0pKamVmeX9W6sf13W8VU5NaVv7k9pyqtnffPPtLSa1DExMfoFBm4WEhKCjY1NqdturJVdVk1vpRTDhg0rc9QtdbBNR25ONrtmDsbxdD6RY7sz7f7njB2S0TWYqY+YmJhiZ/YjIiLo3Lmz/nt/f39atmzJY489VmLfESNGsGbNGv3oMCEhgYsXL5KamkrLli1p2rQpJ06c4PDhw1WK0dHR0aBSquXVzr5VFhYW5ObmApRbU/pGQ4YM4euvv9ZP8xStEVlePesdO3aQlZVFcnIyISEheHh4lOhH0Yi6tEdZSfpmDg4OREREUFBQQHx8PL/99hsAXl5eHDhwQP/zzcjI4M8//9TvJ3WwTUN+Xh7bHigsshR5lx3TVn5r7JBMQoNJ1Onp6cyaNQsnJydcXV2Jjo5m+fLlxdqsWrWKzMxM/cm2IsOHD+e+++5jwIABuLi4MHnyZNLS0hg5ciR5eXk4OjqyePFivLy8qhTj4MGDCQ8Pr3Dh2tatW+Pt7U3v3r3LPJloqDlz5uDq6sqMGTPKrSl9I2dnZ5YuXcpdd92Fm5sbTz1VuH5lefWsXV1d8fX1xcvLixdeeAF7e/tya4BXlre3N126dMHJyYknn3xSP+ffpk0b1q5dy/Tp03F1dWXAgAGcOHECQOpgm5CtL07EJSKDo57WTHlvr7HDMR1lXQ5SlYcpXp5nym7s85NPPqn27NljxGiq37Jly1RgYGCx50zpfX7zzTfVJ598YlBbuTzPcLfa58NfvqLyX2yuvv7PIJVXyUstja3eXZ4nSvfcc89x7do1Y4fRoEgdbOPb7D+TtkfeIrLZQPxe31uvK+FVhkE/DU3T4oA0IB/IU2WsQiCqzs7OjvHjxxvcPioqSn8dcxFLS0v9XZem4OYpJlMjdbCNa/vKefTcGEpkt7aM/GozjSwaGzskk3Mrf7Z8lVIl77MWRuXi4lKpS9qEMAW7PlmGw9pg/raF/u9upomu/NXuGyqZ+hBCGMW+ze/S+p2vSLUGh3c+wb5zxQtAN1QGLW6radpp4AqggA+VUh+V0mYOMAfAzs7Oc+PGjcW2t2jRgu7du1d4rPz8/Aqv161vpM91x6lTpyp1rXt6erpRinwZU3l9zricQMGn/6X1RY2Euf+iTbd+tRxdzajK++zr61vm4raGJuoOSqkETdPaAnuAJ5RS+8pqXx9XIa9J0ue6Q1YhN1xZfb50/gzZHw+DvGskeL/KHSNnlty5jjLqKuRKqYTr/14EtgF3VCoSIzM3N8fd3R1nZ2fc3Nx44403it3FVppbLcUphCjb2ZOR7HtyFOY5qWRM+F+9StI1qcJErWlaM03TrIu+BoYDx2o6sJpQVIDojz/+YM+ePfzwww+sWLGi3H0kUQtRPS5fiCf60el0P64I6/Qvenj4GDukOsOQEbUd8IumaZHAb8B3SqkfazYs2B6egHdAEF0Wf4d3QBDbwxOq9fXbtm3LRx99xLvvvotSiri4OAYNGoSHhwceHh4cPHgQKFmKs6x2Qoiypade5sCskXRKVJycdgdj5pasJyPKVuHleUqpvwC3WohFb3t4Aku2RpGZW1gLOSElkyVbowDw69Oh2o7TtWtX8vPzuXjxIm3btmXPnj00adKEkydPMn36dEJDQwkICGDlypXs3LkTgGvXrpXaTghRutycbHY/4INjXAFHx/Vg6gvrjB1SnWOSt/8E7orRJ+kimbn5BO6KqdZEfaPc3Fwef/xxfR3mGwv2VKadEAJUQQHBb83EPiGXCJ92TA/cYeyQ6iSTTNTnUzJv6fnK+uuvvzA3N6dt27asWLECOzs7IiMjKSgoKLVUJ8Bbb71lUDshGqrt4QkE7ophWqc0olc8wjxtF0H3T2Dak2uMHVqdZZKJur2NjoRSknJ7m+qrF5yUlMTcuXN5/PHH0TSN1NRUOnbsiJmZGevWrdMvQXVzKc6y2gkhik9b6mLfpF3Keb7qOQiLQf9FM5P76yrLJH9yC0f0RGdR/GYInYU5C0f0rNLrFq0P6OzszN13383w4cP16x/OmzePdevW4ebmxokTJ/TF6G8uxVlWOyHEP9OWcy+txnv/RXIzLFmaM5uVe2SV96owyRF10Tx04K4Yzqdk0t5Gx8IRPas8P13e6Pf222/n6NGj+u9fe+01oLCwflBQULG2pbUTQhROTz5x4Q1GH0ok9jaNF3ovIddMV+3Tlg2NSSZqKEzWNXXiUAhRM5658BpDDiUR08WcuMnPkn7SBqjeacuGyCSnPoQQdYsqKODQZ8/Sp+lpjvVoxLMuyzBv0hyonmnLhq5WE7UhdUWEMFXy+1u6/Lw8vnl5KgPOfIBF94GYv7gT21atAehgo+PViS7y6biKam3qo0mTJiQnJ9O6detSV6AWwpQppUhOTpbLMW+Sm5PNtlneOEdk8M2UOxn75Bf0MzdnYt/OhISE8MQMH2OHWC/UWqLu2LEj586dIykpqdx2WVlZDe4/g/S5bmjSpAkdO3Y0dhgmIzvzGt/O9MblWBZH+1oz+cUdmNXB0rV1Qa0lagsLC7p06VJhu5CQEPr06VMLEZkO6bOoazLSUtk1czDOJ3KIHNiKKR/9LOsc1iA5mSiEuCXZWdf44bkROJ7IIcLHjmlrDkiSrmGSqIUQBsu6lk7M2+OZ2Oo4J2bdwfQPQowdUoMgiVoIYZCLCbHsurc/jZOOEeq6nAlLpApebZFELYSoUHzsMcLvH0e32AKirIZwx6QFxg6pQZFELYQo11/Rv3HiwSm0v6CIub8/U1bIike1TRK1EKJMf0bu58wjs2h7GWL/5cukpWuNHVKDJKdqhRCl+jv+FGZb53DNpoCM+8bj91igsUNqsCRRCyFKOBK8haZ7n6VDo3Tyln1OrzuGGTukBk0StRCimIPffQbLXufv1o0wf+1rerkPMnZIDZ4kaiGE3r7N72L5ynsAWD/6BLdLkjYJkqiFEADs/TwAmzfWkdMIzP67mEGjZhk7JHGdJGohBH8c/A6zj9aRaQlWAS/Rx3eSsUMSN5BELUQDF7VvB91+eoRM79Y08XuT3l6jjB2SuIkkaiEasO2BczE/HITO1Q6H/3yPbbtOxg5JlEJueBGigdri/wDdPvuZJpca0WTGF5KkTZiMqIVogL5aOgXnrceIt9dw+XQzHbo4GTskUQ5J1ELUU9vDEwjcFcP5lEza2+hYOKInfn06sHHheFy+PUlcJzP6r9+JrX3FC3oI45JELUQ9tD08gSVbo8jMzQcgISWTJVujUIfeo2t2GKe6t2TwZ7to2UYWna0LZI5aiHoocFeMPkkXeTDtAyZcfI9GnTwYtSVUknQdIolaiHrofErmP9+ofJ4/s4IxP8bw2WUXXP+zhcaWdWth4YZOErUQ9VB7G13hFyqfFX8txzs8jUinxnzSdimNLBobNzhxyyRRC1EPLRzRE2uzbF45+QJ3RGXyu4sOf8eXWTjG1dihiUqQk4lC1EO+nRuzIu1lHKNzONDHmk97v0TAaGf8+si8dF0kiVqIeib2j19pvHk2Y1td4Ic5U3n4qf/jYWMHJapEErUQ9cjOd5+h1brvsBicRfq9X3JP/xHGDklUA0nUQtQTm54ei/P3sSS1giZjAnGUJF1vSKIWoo7LzLjKtw8PwTU8g1NdzOn7wWbsO/cydliiGhmcqDVNMwdCgQSl1NiaC0kIYajUK5fYvWg4LuGZHPW0xu+TECx1TY0dlqhmt3J53nzgeE0FIoS4NWf/PErKO3cxul0cf9znwdQNv0mSrqcMStSapnUExgCf1Gw4QghDfPfes/z58FTystKIG/U/Jr+4wdghiRqkKaUqbqRpm4FXAWvgmdKmPjRNmwPMAbCzs/PcuHFjpQJKT0/HysqqUvvWVdLn+q86+xu3+RU8g+JJtoELsx/ArseAannd6tbQ3mOoWp99fX3DlFJ9S92olCr3AYwF3r/+tQ+ws6J9PD09VWUFBwdXet+6Svpc/1VHf7OuZahN9/VV0T17qW+HO6lzf/1R9cBqUEN7j5WqWp+BUFVGTjXkZKI3MF7TtNFAE6C5pmn/U0rdX6k/G0KIW3Y1JZldjw/FJSybKA8rxn38E7pmzY0dlqglFc5RK6WWKKU6KqUcgGlAkCRpIWpPwl/Hubz6LgbdlsBRP0fu/eJ3SdINjFxHLYQJ+/6D5yj4div9Pa5yYcwapnqPM3ZIwghuKVErpUKAkBqJRAhRzKZFfjjtjCHZRuPcsPfpI0m6wZIRtRAmJjvzGtsf8cE1NI3Y28xw+79NdOrW29hhCSOSRC2ECUlLvcyuh31wjcolyr0Z4z4NkvloIYlaCFNxPi6G7PVTcO58kaO39WXqGzuNHZIwEbLCixA1aHt4At4BQUQlpOIdEMT28IRS2/348Qv8vmQ8LfMvkTf+Q0nSohgZUQtRQ7aHJ7Bka1ThauCdICElkyVbowCKrbTy1ZJJOO6I5nILM84/tQ6XfsOMFbIwUTKiFqKGBO6KKUzSN8jMzSdwVwxQeNJw08z+uGyL5mxHM7qs34iTJGlRChlRC1FDzqdklvl8+tUr7J45GNeYPI656Rj9STDNrFvUcoSirpARtRA1pL2NrtTnXa1Tufi2D207pBA5sjNTNh2RJC3KJYlaiBqycERPdBbmxZ4bk/kNc+OWYVtwkRZTVzHt7R+NFJ2oS2TqQ4gaUnTCsHBOOo15V95j5C+nudyiERee2IaLq5dxAxR1hiRqIWqQX58OjHG25at/DcIjNI3THc1wemcdDo6llx0WojSSqIWoQefPxHBkzkQ8zhTwh0sTRq0JkflocctkjlqIGnLkx7U0+8wXzS6bX+/uxIQvf5ckLSpFRtRCVLOLCbHsmz+Z29pf5GLb9ri+8CmxCVcwbyT/3UTlyIhaiGq09/MATkwai+OxLP7K7obDogN0ut3N2GGJOk7+xAtRDbIzr7F1/ih6/3KRq80gbr4f0/79qrHDEvWEJGohqijxTAy/Bk7DfV8WJ3pa4Lnyc7xlFC2qkUx9CFEFe9f70+wzH4a0SuT47IGM33KE2yRJi2omI2ohKiHxzAkOLpiGw6lsjo3vQOd/fc7Ero7GDkvUU5KohbhFu9asoOmHG+l1FaIGtuaeJd/LKiyiRkmiFsJAOdlZbJ13Ny4Hk7liDeeemcq0h5YbOyzRAEiiFsIACX8dJ+2LB7G6nMyJXk0Z8NZG2jvIVIeoHZKohShHfl4eW5ZNp0fWYbpa55E66wX6+80zdliigZFELUQZ4mOPEfrUfbjE5HLcqQV2b22hf+eexg5LNECSqIUoxfcfPEeLNdvong4Rg9sycdUPWOqaGjss0UBJohbiBrk52ezwn4Lj5pMk28CFxTOZ/sBzxg5LNHCSqIW47vSJMHI3z2W89ifbBvTE96Uvaduhm7HDEkIStRD5eXlsfm4ydr/E0NX3KsfufJupox40dlhC6EmiFg3amRNhhD8zG9dTeZzuZEa631o8+g0zdlhCFCOJWjRYO999hlZrv6PbNYgcYs+kt3/AorGlscMSogRJ1KLBycnO4shnT2H+7U/kNDYj6emHmDb9GWOHJUSZJFGLBuXgd59hfnAVXhanCbp7BL3vfw1b+y7GDkuIckmiFg1CduY1ti6eiONPZzjXWRE+/z2GDL/f2GEJYRBJ1KLe+/6D57DcsA33JIjtbEZ3/3fo5TnE2GEJYTBJ1KLeunAuln2vzaT3nitcbg5bh/TiY+sH6bAHFpol4Neng7FDFMIgkqhFvZOSnMih9S9yV8oOfJrDDwNv5x27f5OkWgKQkJLJkq1RAJKsRZ0gS3GJeiM/L49trz1K1Ogh6Dbv53hTT3If3seHPf6rT9JFMnPzCdwVY6RIhbg1MqIW9cLhH9aRtOp1esUV8HcrSL93JD7z3wbgfEpsqfucT8msxQiFqLwKE7WmaU2AfYDl9fablVLLajowIQyRfvUKe958kO5fxWDRCCLv7si4VzfTzLqFvk17Gx0JpSTl9ja62gxViEozZOojGxiilHID3IGRmqZ51WhUQlQgNyebXR8/R9ab7owtCCa6rxVW6z9k2rt7iiVpgIUjeqKzMC/2nM7CnIUjpLa0qBsqHFErpRSQfv1bi+sPVZNBCVGeoC9WkvvxGmxSFfHjbLnst46pHj5lti86YRi4K4bzKZm0t9GxcERPOZEo6gyD5qg1TTMHwoDuwHtKqV9rNCohShEbdZBw/8dwjMoivSnEDe/FxKUbDarP4dengyRmUWdphQNmAxtrmg2wDXhCKXXspm1zgDkAdnZ2nhs3bqxUQOnp6VhZWVVq37pK+ly+gvw8Lv66ge6bDmOZC8dcm2I1eQHWth1rOMrqI+9xw1CVPvv6+oYppfqWulEpdUsP4EXgmfLaeHp6qsoKDg6u9L51lfS5bD9veUf9tcJFqWXN1Vf3u6oDOz6q2cBqiLzHDUNV+gyEqjJyqiFXfbQBcpVSKZqm6YBhwGuV+pMhhIGOHviW2NeW0jU2l8vj8rni+y6TX5yBZiaX/ouGx5A5antg3fV5ajPgK6XUzpoNSzRUKcmJ/PjsvfQ6fImuwPEBtox4eiMt28j8smi4KhyeKKWOKqX6KKVclVK9lVL+tRGYqJ+2hyfgHRBEVEIq3gFBbA9PAEAVFPDrzk+IHDsEt18uEdelEer/Apj2yX5J0qLBkzsTRa3ZHp7Akq1RZObmQ6d/am6kRO/BI349/bPC2NazI2l9R+H3+EpjhyuEyZBELWpN4K6YwiR9XZv8v3ns9Pt4brlG+vBrHO6/iHFLF9LIorERoxTC9EiiFrWmqLaGmcqh8YH/453D8Vhfg0jHxtzxr0/p7jrQyBEKYZokUYtaY9/ckh7pvzIxfAO3n1actdf45E5fjrefxH2SpIUokyRqUeMy0lL54Y15rFGR9Gocz7f2duzr2pXXrB7FsrGOV6XmhhDlkkQtasylxNPsfXUuHQ+fxfkqnB5mRpTbUt5p7MqUztnYx1tLzQ0hDCCJWlS7xDMn+dn/QboeScYtE8500EiaNIixC1Zj0diSKUBISAhPzPAxdqhC1AmSqEW1if3jVy4Ff4jThW+xj25Non0jrkycxMiHVxg7NCHqNEnUospCvn6Hyxs+of3ZHHqOvshx2+HYvTOHu/r6Gjs0IeoFSdSiUvLz8vjhwyXwzfd0O1NA88Zw0tUa26mfcoeLXMEhRHWSRC1uSX5eHpF7PiclaDXdvs0itSlE3mWH98J38OjuYuzwhKiXJFELg6QkJ7LrlTk0vniKCd3Oc87KnkNTXBn2n1V4tbY3dnhC1GuSqEW54mOPceD1x3H4/QKu1wqv4Aid+iZ9Rs5iSiP59RGiNsj/NFGqiwmnCXlrLt13n8UtB2I7m3Fp3GhGzX0Vc0nQQtQq+R8nijm081OuhG3l7oJDeGgWHO1iR6v7/sXYqf8xdmhCNFiSqAUAu9asIGvz1/T4K5/k7gWEj/Sj05hFTOzSy9ihCdHgSaJuwAry8/l21XwsdwbR+bwivQlEDmxNvwWv000usRPCZEiiboBORh7gwrFgOsZuotGJa1inNyZy+G3cveQD+tl3MXZ4QoibSKJuIM7GHOHQJyuwijyFQ3wB2pB08tu1xmbCY7iPfAhv6xbGDlEIUQZJ1PXY1ZRkwr//kKvrN+BwtgDXAkiygSivlnQZu5iuI2bSTVb1FsLkSaKuZy4mxBLywVK4eAq/dqcZRB678toT3bc5bUbdy11T5vNt1AWe2hXD+Z9/oL2NTkqNCmHiJFHXA5cvxBP04VIa/RZOl7g8XPLgfBsIc5tIqzumM3KZD9r1kXOxBWb5Z4FZQJK1ECZKEnUdlZKcyF+/70H9sY2EQ3/gfMyc1KYQ07sZzYeOYuis57FobFliv5sXmAXIzM0ncFeMJGohTJQk6jokPfUyez9aSv7BgzjE5tD47hQ6Njfnb9c+xPr2Yti/VuCla1ruaxQtMGvo80II45NEbeLycnP47YfPOL/uAxxis+iZBRmWcPp2S1q4PUav6QsZU8rIuSztbXQklJKU29voqjNsIUQ1kkRtgrIzr7H3sxVkxoUxpGkMnvlXUWfbc6ZLY8wHDODuR1+hb4tWlXrthSN6FpujBtBZmLNQFpgVwmRJojYRuTnZ/LTuJa7+9AO3ncygawYktFWcvscTrfck3J8dTzOrql/rXDQPHbgrhvMpmXLVhxB1gCRqI0pPvUz8iVByf1vH96vm0uO4Rk4jOO3QiHN93fB59L/Y1sCdgn59OkhiFqIOkURdS/Lz8jh64FtOBX2NOnWSFuczaHdR0XxMMnfp8tjV3ZFjzh0ZNOcl3G673djhCiFMiCTqGpJ45gRh33xMo4J02mWf4dKZU3T4yYLeQHYj+NtO4w83K6LMhmLvMJBNF9qxcERP2t0mI10hRHGSqKtBfl4eMWHBRH39Duanz9L672zaJkM34KxXFq266rjS0YnI0fm07TeE/mMfJu5UOv+9flLv6cZ5cuOJEKJMkqgr4fjve4ne/QV5MdE00mUxqt15OhfkUPCdPdeawIV25lxwbInOuQ8e4x6hU3cXbrvpNQJ3hcmNJ0IIg5hMot4enkDgrhimdUpjaUBQrV2JUHTcsq6ASEtN5tyJUFJPHuTc9u3Yn8mhVRo4AXlmcNLZjCiXsZjfdgfmQ9vieed4g5aqkhtPhBCGMolEXaz+RKfaqz9xc92LxCupfP2/90he/wfNz8fTMjETDcXQIYmF7fPbc7G9BfG3taOV5yDuGPcwLpVcgVtuPBFCGMokEnVt1Z9IvXyB038c5mJsFBl/nyXu1Gn+k5mGZef2dDW/SMqxJG4/Xli8KLMxJLYzI61DC8IHvkAn18H4tetUbbHIjSdCCEOZRKKuyjRARloqqZf/JuPKRc7+cYDko7+Sf+Uy5mkZWGTk0CQjD3svRVeLFA7G6nD43ZKidFu0GqDFbQlkay2J6diO460sCbfuw+ev++NxC7dm36obbzyBNDrIjSdCiDKYRKK+eRqgkcqhU94ZHBsl8M3bu7Fq2ZqmWi4X4/4g70gsltfyaJpRQLNrYJ0JuaNTcG1+jXPnbXDZV1iU6FpjSG8G15pqJFl0ILvNQHJ1uRy1v4plm3ZYt+/KxycaEZndgUxlBTlAs8JHBxtdqZXnqlvRjSchISE8McOnxo8nhKibTCJRLxzRk8VbInn72iJs3shnRyaYqX+2Jw9Jx6vtVQ5faUFuYjOuNdVIbdmISx0tyLdqRsseY8nv6kYLiyZkzTDHwekOWrapeGSaFp5A5NYokOkHIYQJM4lEXfRxP+XLdqQ6JJHepAlmNi1obm9Ps3a30bmPDxk9PfGytqmR40rdCyGEKaswUWua1glYD9gBCvhIKbWqugPx69MB+uwlJCSEqT4+1f3y5R5XErMQwpQZMqLOA55WSh3RNM0aCNM0bY9SKrqGYxNCCAFUuAS1UipRKXXk+tdpwHFAhqBCCFFLNKVUxa2KGmuaA7AP6K2UunrTtjnAHAA7OzvPjRs3Viqg9PR0rKysKrVvXSV9rv8aWn9B+nyrfH19w5RSfUvdqJQy6AFYAWHAxIraenp6qsoKDg6u9L51lfS5/mto/VVK+nyrgFBVRk6tcOoDQNM0C2ALsEEptbVSfy6EEEJUSoWJWtM0DfgUOK6UerPmQxJCCHEjQ0bU3sBMYIimaRHXH6NrOC4hhBDXVXh5nlLqF0CrhViEEEKU4pau+jD4RTUtCThTyd1tgUvVGE5dIH2u/xpaf0H6fKs6K6XalLahRhJ1VWiaFqrKukSlnpI+138Nrb8gfa5OBl31IYQQwngkUQshhIkzxUT9kbEDMALpc/3X0PoL0udqY3Jz1EIIIYozxRG1EEKIG0iiFkIIE2e0RK1p2khN02I0TTuladriUrZbapq26fr2X69X7quzDOjvU5qmRWuadlTTtJ80TetsjDirU0V9vqHdJE3TlKZpdf5SLkP6rGnavdff6z80TfuitmOsbgb8bt+maVqwpmnh13+/6/SdzZqmrdE07aKmacfK2K5pmrb6+s/jqKZpHlU+aFnVmmryAZgDsUBXoDEQCTjd1GYe8MH1r6cBm4wRay321xdoev3rf9fl/hra5+vtrCksnXsY6GvsuGvhfb4dCAdaXv++rbHjroU+fwT8+/rXTkCcseOuYp8HAx7AsTK2jwZ+oPCObi/g16oe01gj6juAU0qpv5RSOcBG4J6b2twDrLv+9WZg6PUCUXVRhf1VSgUrpa5d//Yw0LGWY6xuhrzHAP8FXgOyajO4GmJInx8B3lNKXQFQSl2s5RirmyF9VkDz61+3AM7XYnzVTim1D7hcTpN7gPWq0GHARtM0+6oc01iJugMQf8P35yi5aoy+jVIqD0gFWtdKdNXPkP7e6CEK/yLXZRX2+fpHwk5Kqe9qM7AaZMj73APooWnaAU3TDmuaNrLWoqsZhvR5OXC/pmnngO+BJ2onNKO51f/vFTKJVcjFPzRNux/oC9xl7FhqkqZpZsCbwGwjh1LbGlE4/eFD4aemfZqmuSilUowZVA2bDqxVSr2hadoA4HNN03orpQqMHVhdYawRdQLQ6YbvO15/rtQ2mqY1ovAjU3KtRFf9DOkvmqbdDSwFxiulsmsptppSUZ+tgd5AiKZpcRTO5X1Tx08oGvI+nwO+UUrlKqVOA39SmLjrKkP6/BDwFYBS6hDQhMLiRfWVQf/fb4WxEvXvwO2apnXRNK0xhScLv7mpzTfArOtfTwaC1PWZ+jqowv5qmtYH+JDCJF3X5y2hgj4rpVKVUrZKKQellAOF8/LjlVKhxgm3Whjye72dwtE0mqbZUjgV8lctxljdDOnzWWAogKZpjhQm6qRajbJ2fQM8cP3qDy8gVSmVWKVXNOKZ09EUjiZigaXXn/On8D8rFL6ZXwOngN+ArsY+21vD/d0LXAAirj++MXbMNd3nm9qGUMev+jDwfdYonPKJBqKAacaOuRb67AQcoPCKkAhguLFjrmJ/vwQSgVwKPyE9BMwF5t7wHr93/ecRVR2/13ILuRBCmDi5M1EIIUycJGohhDBxkqiFEMLESaIWQggTJ4laCCFMnCRqIYQwcZKohRDCxP0/5zUzMkNVbe8AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", @@ -2567,9 +2534,7 @@ { "cell_type": "markdown", "id": "294b9014", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The intercept is the value of our output/target variable\n", "when all our features are zero and our function crosses the $y$-axis (for a one-dimensional case). \n", @@ -2588,9 +2553,7 @@ { "cell_type": "markdown", "id": "9403b4b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=0}^{p-1}\\beta_j^2,\n", @@ -2600,9 +2563,7 @@ { "cell_type": "markdown", "id": "580e39e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "but when we take out the intercept, this equation becomes" ] @@ -2610,9 +2571,7 @@ { "cell_type": "markdown", "id": "3dcdcb5a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_2^2 = \\lambda \\sum_{j=1}^{p-1}\\beta_j^2.\n", @@ -2622,9 +2581,7 @@ { "cell_type": "markdown", "id": "255aa059", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For Lasso regression we have" ] @@ -2632,9 +2589,7 @@ { "cell_type": "markdown", "id": "bb150ba1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\lambda \\vert\\vert \\boldsymbol{\\beta} \\vert\\vert_1 = \\lambda \\sum_{j=1}^{p-1}\\vert\\beta_j\\vert.\n", @@ -2644,9 +2599,7 @@ { "cell_type": "markdown", "id": "4c0d85d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It means that, when scaling the design matrix and the outputs/targets, by subtracting the mean values, we have an optimization problem which is not penalized by the intercept. The MSE value can then be smaller since it focuses only on the remaining quantities. If we however bring back the intercept, we will get a MSE which then contains the intercept." ] @@ -2654,9 +2607,7 @@ { "cell_type": "markdown", "id": "8fbda7bf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code Examples\n", "\n", @@ -2665,13 +2616,117 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 14, "id": "90570eb7", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Beta values for own Ridge implementation\n", + "[ 1.03032441e+00 6.28336218e-02 -6.24175744e-01 5.21169159e-02\n", + " 2.80847477e-01 2.12552073e-01 8.13220609e-02 -1.69634578e-02\n", + " -6.50846112e-02 -7.38962192e-02 -5.94226022e-02 -3.50227564e-02\n", + " -9.80609615e-03 1.08299273e-02 2.41882037e-02 2.93492130e-02\n", + " 2.64742912e-02 1.63249532e-02 -5.01831224e-05 -2.15098090e-02]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.03032441e+00 6.28336218e-02 -6.24175744e-01 5.21169159e-02\n", + " 2.80847477e-01 2.12552073e-01 8.13220609e-02 -1.69634577e-02\n", + " -6.50846112e-02 -7.38962192e-02 -5.94226022e-02 -3.50227564e-02\n", + " -9.80609615e-03 1.08299273e-02 2.41882037e-02 2.93492130e-02\n", + " 2.64742912e-02 1.63249532e-02 -5.01831185e-05 -2.15098090e-02]\n", + "MSE values for own Ridge implementation\n", + "4.3632959128006053e-07\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "4.363295916429748e-07\n", + "Beta values for own Ridge implementation\n", + "[ 1.03630548 -0.01963611 -0.37900111 -0.07062318 0.12182967 0.16343471\n", + " 0.13003291 0.07490892 0.02365049 -0.01449782 -0.03814292 -0.04909093\n", + " -0.05009826 -0.04389027 -0.03279636 -0.01866537 -0.00289724 0.01348565\n", + " 0.02976145 0.04543942]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.03630548 -0.01963611 -0.37900111 -0.07062318 0.12182967 0.16343471\n", + " 0.13003291 0.07490892 0.02365049 -0.01449782 -0.03814292 -0.04909093\n", + " -0.05009826 -0.04389027 -0.03279636 -0.01866537 -0.00289724 0.01348565\n", + " 0.02976145 0.04543942]\n", + "MSE values for own Ridge implementation\n", + "5.1940428269262024e-06\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "5.194042826836048e-06\n", + "Beta values for own Ridge implementation\n", + "[ 1.04220758 -0.10931453 -0.17641709 -0.06020587 0.02208512 0.05789007\n", + " 0.06491736 0.05785343 0.04537385 0.03196357 0.01969145 0.00934499\n", + " 0.00107405 -0.00526348 -0.00992331 -0.01318643 -0.01531845 -0.01655318\n", + " -0.01708852 -0.01708781]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.04220758 -0.10931453 -0.17641709 -0.06020587 0.02208512 0.05789007\n", + " 0.06491736 0.05785343 0.04537385 0.03196357 0.01969145 0.00934499\n", + " 0.00107405 -0.00526348 -0.00992331 -0.01318643 -0.01531845 -0.01655318\n", + " -0.01708852 -0.01708781]\n", + "MSE values for own Ridge implementation\n", + "2.0940821989672366e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "2.0940821989628246e-05\n", + "Beta values for own Ridge implementation\n", + "[ 1.01219292 -0.06043581 -0.10391807 -0.05651951 -0.01898855 0.00312361\n", + " 0.01463049 0.01975848 0.02123176 0.02068067 0.01905883 0.01691985\n", + " 0.01458337 0.01223198 0.00996754 0.00784393 0.00588657 0.00410387\n", + " 0.00249435 0.00105081]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 1.01219292 -0.06043581 -0.10391807 -0.05651951 -0.01898855 0.00312361\n", + " 0.01463049 0.01975848 0.02123176 0.02068067 0.01905883 0.01691985\n", + " 0.01458337 0.01223198 0.00996754 0.00784393 0.00588657 0.00410387\n", + " 0.00249435 0.00105081]\n", + "MSE values for own Ridge implementation\n", + "0.0003153514830958068\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.00031535148309580843\n", + "Beta values for own Ridge implementation\n", + "[ 8.38916861e-01 1.31276579e-01 8.97497404e-03 -1.72271878e-02\n", + " -2.11744554e-02 -1.91492986e-02 -1.57201944e-02 -1.23002365e-02\n", + " -9.30466214e-03 -6.81048318e-03 -4.78184120e-03 -3.15130074e-03\n", + " -1.84923989e-03 -8.13661243e-04 7.46984697e-06 6.56636616e-04\n", + " 1.16805821e-03 1.56912044e-03 1.88168312e-03 2.12318726e-03]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 8.38916861e-01 1.31276579e-01 8.97497404e-03 -1.72271878e-02\n", + " -2.11744554e-02 -1.91492986e-02 -1.57201944e-02 -1.23002365e-02\n", + " -9.30466214e-03 -6.81048318e-03 -4.78184120e-03 -3.15130074e-03\n", + " -1.84923989e-03 -8.13661243e-04 7.46984697e-06 6.56636616e-04\n", + " 1.16805821e-03 1.56912044e-03 1.88168312e-03 2.12318726e-03]\n", + "MSE values for own Ridge implementation\n", + "0.015072388895177166\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.015072388895177053\n", + "Beta values for own Ridge implementation\n", + "[0.37396662 0.14174745 0.0764924 0.04892055 0.03447512 0.02586427\n", + " 0.02024962 0.01633913 0.01347916 0.0113104 0.0096208 0.00827728\n", + " 0.00719176 0.00630331 0.00556826 0.0049544 0.00443743 0.0039987\n", + " 0.0036237 0.003301 ]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[0.37396662 0.14174745 0.0764924 0.04892055 0.03447512 0.02586427\n", + " 0.02024962 0.01633913 0.01347916 0.0113104 0.0096208 0.00827728\n", + " 0.00719176 0.00630331 0.00556826 0.0049544 0.00443743 0.0039987\n", + " 0.0036237 0.003301 ]\n", + "MSE values for own Ridge implementation\n", + "0.26409315307910025\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.26409315307910025\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAr4ElEQVR4nO3deXyU5b338c8vIRAwCArUBRTQAsqSBAhQRFCrCMpO4gGV1qUWrVXrUan1eB61ejxi9Wld6nHpUXvqsS5NQEAQRYHiIwoJCsgigoga3NhlC2T5PX/MEIc4EAiZ3DOT77svXs69f+9A85vrvmauy9wdERGRqlKCDiAiIvFJBUJERKJSgRARkahUIEREJCoVCBERiapB0AFqS8uWLb1du3ZBxxARSSiLFi3a6O6tom1LmgLRrl07ioqKgo4hIpJQzOyzA23TIyYREYlKBUJERKJSgRARkaiSpg8imtLSUoqLiykpKQk6isSZ9PR02rRpQ1paWtBRROJWUheI4uJimjZtSrt27TCzoONInHB3Nm3aRHFxMe3btw86jkjcSupHTCUlJbRo0ULFQfZjZrRo0UItS5FqJHWBAFQcJCr9uxCpXtIXCBGRZPa3x8bzzCNXxOTcKhAxZmaMGzeucrmsrIxWrVoxdOhQAL755huGDh1KVlYWnTt35sILLwRg3bp1NG7cmOzs7Mo/f/vb3+o0e2SGzp078/Of/5zS0lIAioqKuOGGG6Ie165dOzZu3HhE13722Wcr77thw4Z069aN7Oxsfve73x3yOR566CF27dp1RDlE4plXVHDv2r/y97VTYnL+pO6kjgdHHXUUy5YtY/fu3TRu3JhZs2bRunXryu133HEHAwcO5De/+Q0AS5curdx26qmnsnjx4rqOvJ99GcrLyxk4cCAvv/wyl156KTk5OeTk5MTsuldccQVXXBF6V9SuXTvmzJlDy5YtD+scDz30EOPGjaNJkyaxiCgSuOXvTuHjo0v516aDY3J+tSDqwIUXXsj06dMBeOGFF7j44osrt3311Ve0adOmcjkzM/Owzv3WW2/RvXt3unXrxpVXXsmePXsoLCxk9OjRAEyZMoXGjRuzd+9eSkpKOOWUUwA4++yzufXWW+nduzcdO3bk7bffPuh1UlNT6d27N+vXrwdg7ty5la2gTZs2cf7559OlSxeuuuoqImcpvOeee+jUqRNnnnkmF198MQ8++CAAn3zyCYMHD6Znz57079+fjz766JDu94EHHqBXr15kZmZy5513ArBz506GDBlCVlYWXbt25aWXXuKRRx7hyy+/5JxzzuGcc845jJ+oSOIoePMRzGHkyENvWR+O+tOCuPFGqO1349nZ8NBD1e42duxY7r77boYOHcrSpUu58sorK38h//rXv2bMmDH8+c9/5rzzzuOKK67gxBNPBEK/RLOzsyvP8+ijj9K/f//K5ZKSEi6//HLeeustOnbsyM9//nMef/xxrrvuusqWx9tvv03Xrl0pLCykrKyMPn36VB5fVlbGwoULmTFjBr///e958803D3gPJSUlLFiwgIcffvgH237/+99z5plncscddzB9+nSefvppAAoLCykoKGDJkiWUlpbSo0cPevbsCcD48eN54okn6NChAwsWLODaa69l9uzZB/05vvHGG6xevZqFCxfi7gwfPpx58+axYcMGTjzxxMoivG3bNpo1a8Yf//jHGrU8RBJF/rZ36V/RjONPObw3loeq/hSIAGVmZrJu3TpeeOGFyj6GfQYNGsTatWuZOXMmr732Gt27d2fZsmVA9Y+YVq1aRfv27enYsSMAl112GY899hg33ngjp556KitXrmThwoXcdNNNzJs3j/Ly8v0KzL5WRs+ePVm3bl3Ua+wrUp9++ilDhgyJ2sKZN28ekyZNAmDIkCEcc8wxALzzzjuMGDGC9PR00tPTGTZsGAA7duxg/vz5XHTRRZXn2LNnz8F+hECoQLzxxht079698jyrV6+mf//+3Hzzzdx6660MHTp0v3sUSVarCl9jWbM9PNx4SMyuUX8KxCG804+l4cOHc8sttzB37lw2bdq037Zjjz2WSy65hEsuuYShQ4cyb968ynfaNTVgwABee+010tLSOO+887j88sspLy/ngQceqNynUaNGQOjxUVlZWdTz7CtSGzdupF+/fkydOpXhw4cfUbaKigqaN29+2P0r7s5tt93G1Vdf/YNt77//PjNmzODf//3fOffcc7njjjuOKKNIvCuY+ScARo+IzeMlUB9Enbnyyiu588476dat237rZ8+eXflJm+3bt/PJJ59w8sknH9I5O3XqxLp161izZg0Azz33HGeddRYA/fv356GHHqJv3760atWKTZs2sWrVKrp27Vqj/C1btmTixIncd999P9g2YMAA/v73vwPw2muvsWXLFgD69evHtGnTKCkpYceOHbz66qsAHH300bRv355//OMfQOgX/5IlS6rNMGjQIJ555hl27NgBwPr16/n222/58ssvadKkCePGjWPChAm8//77ADRt2pTt27fX6H5F4l3+5v9H3y0ZtOnUK2bXqD8tiIC1adMm6sdCFy1axHXXXUeDBg2oqKjgqquuolevXqxbt+4HfRBXXnnlfudIT0/n2Wef5aKLLqKsrIxevXpxzTXXANCnTx+++eYbBgwYAIQec3399ddH9AWxkSNHctddd/2gQ/vOO+/k4osvpkuXLpxxxhmVBa5Xr14MHz6czMxMjjvuOLp160azZs0AeP755/nVr37Ff/zHf1BaWsrYsWPJyso66PXPP/98Vq5cSd++fQHIyMjgf//3f1mzZg0TJkwgJSWFtLQ0Hn/8cSDUzzF48GBOPPFE5syZU+P7Fok3a5fM5YPmu3mw4dCYXsciP3FS6yc3Gww8DKQC/+3uE6tsvwm4CigDNgBXuvtn4W3lwIfhXT9394M+18jJyfGqEwatXLmS008/vTZuRWpox44dZGRksGvXLgYMGMBTTz1Fjx49go4F6N+HJK4H/nMIvy2dwae5b9Ou65lHdC4zW+TuUT+zHrMWhJmlAo8BA4FioNDMprr7iojdPgBy3H2Xmf0K+AMwJrxtt7tnxyqf1I3x48ezYsUKSkpKuOyyy+KmOIgksvwN/6QnTY64OFQnlo+YegNr3H0tgJm9CIwAKguEu0e2+98DxiFJZV/fhIjUjs9XvMvC5ju5L3VQzK8Vy07q1sAXEcvF4XUH8gvgtYjldDMrMrP3zGxktAPMbHx4n6INGzYccWARkXg3adofAMi94OaYXysuOqnNbByQA5wVsbqtu683s1OA2Wb2obt/Enmcuz8FPAWhPog6CywiEpD8r94i09Lp0HNgzK8VyxbEeuCkiOU24XX7MbPzgNuB4e5e+W0pd18f/u9aYC7QPYZZRUTi3per32d+8+3kHXNGnVwvlgWiEOhgZu3NrCEwFpgauYOZdQeeJFQcvo1Yf4yZNQq/bgn0I6LvQkSkPpo8ZSJukDfoX+vkejErEO5eBlwHvA6sBF529+VmdreZ7fvI6gNABvAPM1tsZvsKyOlAkZktAeYAE6t8+ilhxHq477PPPptOnTqRlZVFr1699vt28oUXXsjWrVt/cMxdd91VOWheTW3atKky1/HHH0/r1q0rl/fu3XtI55g7dy7z588/ohwi9UlB8SxO39aQ0/vE9vsP+8S0D8LdZwAzqqy7I+L1eQc4bj7QLdq2RFMXw30///zz5OTk8OyzzzJhwgRmzZoFwIwZM6o5suZatGhRme2uu+4iIyODW2655bDOMXfuXDIyMjjjjLppLosksm/XLeefzbZyO3U31piG2qgDsRzuO1Lfvn0rh+OG/Sfuuffee+nYsSNnnnkmq1atqtynsLCQzMxMsrOzmTBhQuVQHOXl5UyYMKFyaO0nn3zykDIsWrSIs846i549ezJo0CC++uorAB555BE6d+5MZmYmY8eOZd26dTzxxBP86U9/Ijs7u9rhxkXqu1deuY+KFMg997o6u2ZcfIqpLtw480YWf724Vs+ZfXw2Dw1+qNr9YjXcd1UzZ85k5MiRP1i/aNEiXnzxRRYvXkxZWdl+w25fccUV/OUvf6Fv3777zdb29NNP06xZMwoLC9mzZw/9+vXj/PPPp3379ge8fmlpKddffz1TpkyhVatWvPTSS9x+++0888wzTJw4kU8//ZRGjRqxdetWmjdvzjXXXFOjlodIfVSwbiY/Tk0j88y8OrtmvSkQQYrVcN/7XHrppezdu5cdO3ZE3f/tt99m1KhRlTOr7RuNdevWrWzfvr1ybKNLLrmkckC9N954g6VLl5Kfnw+E5lhYvXr1QQvEqlWrWLZsGQMHhj5+V15ezgknnFD5M7j00ksZOXJk1CImIge2+ctPmH30Jm7xn2Apdffgp94UiEN5px9LsRzu+/nnn6dnz55MmDCB66+/vnJuhiPh7jz66KMMGnTo39Z0d7p06cK77777g23Tp09n3rx5TJs2jXvvvZcPP/wwyhlEJJqpk+6jLBVy+/2qTq+rPog6EovhviOZGffccw/vvffeD6bvHDBgAK+88gq7d+9m+/btTJs2DYDmzZvTtGlTFixYAMCLL75YecygQYN4/PHHKS0tBeDjjz9m586dB83QqVMnNmzYUFkgSktLWb58ORUVFXzxxRecc8453H///Wzbto0dO3ZoOG6RQ5T/yTTabk+l50/rdjQiFYg6crDhvnNycsjMzKRv376Vw33D930Q+/488sgjB71G48aNufnmm/ebFAigR48ejBkzhqysLC644ILK80Oor+GXv/wl2dnZ7Ny5s3I47quuuorOnTvTo0cPunbtytVXX33ASYX2adiwIfn5+dx6661kZWWRnZ3N/PnzKS8vZ9y4cXTr1o3u3btzww030Lx5c4YNG8bkyZPVSS1yENu+/Zw3mn5LbsPsOn28BDEe7rsuabjvmtk3HDfAxIkT+eqrr6LOO52M9O9DEsHzT1zLuG8eZ37vJ+l7wfhaP38gw31LYpg+fTr33XcfZWVltG3blr/+9a9BRxKRCPmrXqF1gxT6nH9lnV9bBaKeGzNmDGPGjKl+RxGpczs2f83Mo77il2WZpKTW/a/rpC8Q7n5E02xKckqWR6uS3GZMup+SNMjrfVUg10/qTur09HQ2bdqkXwayH3dn06ZNpKenBx1F5KDyV+Tzo11GvwuuDuT6Sd2CaNOmDcXFxWgyIakqPT19vyFOROLNrm0bmdG4mJ+VdSY1rWEgGZK6QKSlpR30m78iIvHq9ckPsLMh5PW4PLAMSV0gREQSVcGHL9MizThr2PWBZUjqPggRkUS0Z+d3TG20jpHlHWjQMLi+MrUgRETizKzJD7K9EeR2uzTQHCoQIiJxpmDxCzRLg3NH3hRoDj1iEhGJI6Ulu5iS9gkjSk+hYeOMQLOoBSEiEkfmTH2YLelObuexQUdRgRARiSf5RX8jIw3OHzUh6Ch6xCQiEi/K9pYwOWUVQ0tOJj2jedBx1IIQEYkXb0//LzY2dvI6/kvQUQAVCBGRuJH/3rM0aQAX5N4adBRAj5hEROJCRXkZk3wFF+xuTZNmLYOOA6gFISISF+a/9hRfH1VB3gmjg45SSS0IEZE4kP/OX2hUBkNyfxd0lEpqQYiIBKyivIyCsg8ZVHI8TVucGHScSmpBiIgErPDN/6E4o5y8DiOCjrKfmBYIMxtsZqvMbI2Z/aDdZGY3mdkKM1tqZm+ZWduIbZeZ2erwn8timVNEJEgF854krRyG5d4WdJT9xKxAmFkq8BhwAdAZuNjMOlfZ7QMgx90zgXzgD+FjjwXuBPoAvYE7zeyYWGUVEQmKV1SQv+cDztveiubHta3+gDoUyxZEb2CNu691973Ai8B+7Sd3n+Puu8KL7wH75oAcBMxy983uvgWYBQyOYVYRkUB8MPcFPm1aRm77C4OO8gOxLBCtgS8ilovD6w7kF8Brh3OsmY03syIzK9K80yKSiArm/hepFTBiVHw9XoI46aQ2s3FADvDA4Rzn7k+5e46757Rq1So24UREYsQrKsjfWcg5246l5Umdgo7zA7EsEOuBkyKW24TX7cfMzgNuB4a7+57DOVZEJJEtf3cKHx9dSu7Jg4KOElUsC0Qh0MHM2ptZQ2AsMDVyBzPrDjxJqDh8G7HpdeB8Mzsm3Dl9fnidiEjSyJ/1MOYwatS/BR0lqph9Uc7dy8zsOkK/2FOBZ9x9uZndDRS5+1RCj5QygH+YGcDn7j7c3Teb2T2EigzA3e6+OVZZRUSCkP/de/SvaMZx7bsGHSWqmH6T2t1nADOqrLsj4vV5Bzn2GeCZ2KUTEQnORwtnsLzZHh5pMjToKAcUF53UIiL1TcHrfwJg9Ij4GXupKo3FJCISgILN79DXM2jdMSfoKAekFoSISB37ZPFsPmi+m7zjfxp0lINSgRARqWMF0x8EIHfYbwNOcnB6xCQiUscKNs4jhya07dIv6CgHpRaEiEgd+nzFuyxsvpO8lgOCjlItFQgRkTo0adofAMi98JaAk1RPj5hEROpQ/ldvkWXp/Lj7uUFHqZZaECIideTL1e/zzjHbyTsmvvse9lGBEBGpI5OnTAQgd9CNwQY5RHrEJCJSR/KLZ9E5pRGn94nf4TUiqQUhIlIHvl23nHnNtpLbtHfQUQ6ZWhAiInXglVfuoyIF8n56Q9BRDpkKhIhIHchf9xodUtPo1m900FEOmR4xiYjE2Kbi1cxutpncJj2xlMT5tasWhIhIjE2dfB/lKZA34NdBRzksKhAiIjGWv/ZV2qU2oMc5lwQd5bAkTltHRCQBbfv2c2Y13UBuo6yEerwEakGIiMTUtIL7KE2FvDOuCTrKYVOBEBGJoYKPp9C6QQq9B14edJTDlljtHRGRBLJ905e8lvEVuQ26kZKaeO/HEy+xiEiCmDHpfvY0gNxevwg6So2oQIiIxEjBykkcl5ZCvwuuDjpKjegRk4hIDOzatpHpjYsZzemkpjUMOk6NqAUhIhIDr09+gF0NIbfHZUFHqTEVCBGRGMj/8CVapBlnDbs+6Cg1pkdMIiK1bM/O75jW6DNGlnegQcP0oOPUmFoQIiK1bNbkB9neCPIyfxZ0lCMS0xaEmQ02s1VmtsbMfhdl+wAze9/Myswsr8q2cjNbHP4zNZY5RURqU/7iv9O8xPjpiBuDjnJEYtaCMLNU4DFgIFAMFJrZVHdfEbHb58DlwC1RTrHb3bNjlU9EJBb27t7BlLS1DC9tT8PGGUHHOSKxfMTUG1jj7msBzOxFYARQWSDcfV14W0UMc4iI1Jk5Ux9ma7qT1zmxRm6NJpaPmFoDX0QsF4fXHap0Mysys/fMbGStJhMRiZH8oufI2AsDR90cdJQjFs+d1G3dfb2ZnQLMNrMP3f2TyB3MbDwwHuDkk08OIqOISKWyvSW8kvoxw0rakp7RPOg4RyyWLYj1wEkRy23C6w6Ju68P/3ctMBfoHmWfp9w9x91zWrVqdWRpRUSO0LxXH2NjYyev678EHaVWxLJAFAIdzKy9mTUExgKH9GkkMzvGzBqFX7cE+hHRdyEiEo8KFvyVJnth8OjfBh2lVsSsQLh7GXAd8DqwEnjZ3Zeb2d1mNhzAzHqZWTFwEfCkmS0PH346UGRmS4A5wMQqn34SEYkr5aV7meQruHB3a5o0axl0nFoR0z4Id58BzKiy7o6I14WEHj1VPW4+0C2W2UREatP8mU/x9VEV5J4wOugoteagLQgzGxfxul+VbdfFKpSISKIpmP80jcpgSO4PvhOcsKp7xHRTxOtHq2y7spaziIgkpIryMgrKPmTwjuNp2uLEoOPUmuoKhB3gdbRlEZF6qfDN/6E4o5zcDiOCjlKrqisQfoDX0ZZFROql/HlPkFYOw3JvCzpKraquk/o0M1tKqLVwavg14eVTYppMRCQBeEUF+XsWM7C8Fc2Paxt0nFpVXYE4vU5SiIgkqA/mvsC6pmX8n2OGBB2l1h20QLj7Z5HLZtYCGAB87u6LYhlMRCQR5M95jNQUGDH634KOUuuq+5jrq2bWNfz6BGAZoU8vPWdmN8Y+nohI/PKKCvJ3FXHOtmNp0aZD0HFqXXWd1O3dfVn49RXALHcfBvRBH3MVkXpu2fzJrD66lLy2g4OOEhPVFYjSiNfnEv5WtLtvBzSHg4jUa/lvPkJKBYwcmVyfXtqnuk7qL8zsekJzOfQAZgKYWWMgLcbZRETiWsF3C+hf0Yzj2ncNOkpMVNeC+AXQhdC0oGPcfWt4/U+AZ2MXS0Qkvn20cAbLm+0hr/XAoKPETHWfYvoWuCbK+jmERlkVEamXCl7/EwCjRtwacJLYOWiBMLODzt/g7sNrN46ISGLI3/wOZ3hTWnfMCTpKzFTXB9GX0LzSLwAL0PhLIiJ8sng2i5vv5o+NkmvspaqqKxDHAwOBi4FLgOnAC+6+/KBHiYgksYLpDwIwetiEgJPE1kE7qd293N1nuvtlhDqm1wBzNReEiNRn+Rvn0WvrUbTt0q/6nRNYtTPKheeGHkKoFdEOeASYHNtYIiLx6bPl71DYfCcTGyTnl+MiVddJ/TegK6EvyP0+4lvVIiL10qRpDwCQe+EtASeJvepaEOOAncBvgBvMKvuoDXB3PzqG2URE4k7+17PJtsb8uPu5QUeJueq+B1HdF+lEROqN9R8XMf+Y7dxjyV8coPpvUouISNjkKfcDkDf4poCT1I1qO6lFRCSkYP2bdE5pxGm9Lww6Sp1QC0JE5BB88+ky5jXbSt7RfYKOUmfUghAROQSvvHIfFSmQd+4NQUepMyoQIiKHoOCzmXRITaPrGaOCjlJn9IhJRKQam4pXM7vZZvKa5GAp9efXploQIiLVmDr5PspTIHfAtUFHqVMqECIi1chf+yrtUhvQ45xLgo5Sp2LaVjKzwWa2yszWmNnvomwfYGbvm1mZmeVV2XaZma0O/7ksljlFRA5k6zefMavpBvIaZderx0sQwxaEmaUCjxEaLrwYKDSzqe6+ImK3zwlNZ3pLlWOPBe4EcgAHFoWP3RKrvCIi0bw6aSKlqZB7xtVBR6lzsSyHvYE17r7W3fcCLwL7za7h7uvcfSlQUeXYQcAsd98cLgqzgOQfOlFE4k7+x6/QZkcqvQdeHnSUOhfLAtGa0Gx0+xSH19XasWY23syKzKxow4YNNQ4qIhLN9k1fMjPja0Y36EpKav3rsk3oB2ru/pS757h7TqtWrYKOIyJJZsak+9nTAPLOuCroKIGIZYFYD5wUsdwmvC7Wx4qI1Ir8lQUcvzOFMwaPDzpKIGJZIAqBDmbW3swaAmOBqYd47OvA+WZ2jJkdA5wfXiciUid2bdvIjMbrGWWnk5rWMOg4gYhZgXD3MuA6Qr/YVwIvu/tyM7vbzIYDmFkvMysGLgKeNLPl4WM3A/cQKjKFwN3hdSIidWLmpD+wqyHk9bki6CiBMXcPOkOtyMnJ8aKioqBjiEiSuPTmdrye9jlf372LBg3Tg44TM2a2yN1zom1L6E5qEZFYKNmxlWmNPmNUecekLg7VqX+f2xIRqcasyf+X7Y0gL+tnQUcJlAqEiEgVBUteoHmacc7w3wQdJVB6xCQiEmHv7h1MSVvLiNJTaNg4I+g4gVILQkQkwpypD7M13cnrUr9Gbo1GBUJEJEJ+0XM0TYOBo26pfuckp0dMIiJhZXtLmJz6McP2tKXRUUcHHSdwakGIiITNe/UxNjV2cjv9S9BR4oIKhIhIWP6CZ2nSAAaP/m3QUeKCHjGJiADlpXuZxEqG7G5Dk2Ytg44TF9SCEBEB5s98im+aVJB74uigo8QNFQgRESB//n+TngoXjr416ChxQwVCROq9ivIyJpUtY1DJ8TRtcWLQceKG+iBEpN5bOOuvFGeUk9dxZNBR4ooKhIjUe/lvP0FaOQzLvS3oKHFFj5hEpF7zigoK9ixhYHkrmv3o5KDjxBW1IESkXnt/zt9Z17SMvFOGBh0l7qhAiEi9VjD3v0itgOGj9HipKj1iEpF6yysq+MeuIn5afiwt2nQIOk7cUQtCROqtD9+ZxJqjS8lrd0HQUeKSWhAiUm8VvPUoKQ4jR+rxUjQqECJSb+V/t4ABFc35UbsuQUeJS3rEJCL10soFr7Ki2R5yW58XdJS4pQIhIvVSwesPATBaj5cOSI+YRKReyt/yDv28KSd26BF0lLilFoSI1DtrPniLJc1LyD3+p0FHiWsqECJS7xTMeBCA3OEa2vtg9IhJROqdgo1v04ujOLlz36CjxLWYtiDMbLCZrTKzNWb2uyjbG5nZS+HtC8ysXXh9OzPbbWaLw3+eiGVOEak/Plv+DoXNd5LXckDQUeJezFoQZpYKPAYMBIqBQjOb6u4rInb7BbDF3X9sZmOB+4Ex4W2fuHt2rPKJSP1UMO0PAOQOuSXgJPEvli2I3sAad1/r7nuBF4ERVfYZAfxP+HU+cK6ZWQwziUg9V/D1HLK3NubUbHVQVyeWBaI18EXEcnF4XdR93L0M2Aa0CG9rb2YfmNk/zax/tAuY2XgzKzKzog0bNtRuehFJOus/LmL+MdvJO7Zf0FESQrx+iukr4GR37w7cBPzdzI6uupO7P+XuOe6e06pVqzoPKSKJZfKU+wHIHfSvASdJDLEsEOuBkyKW24TXRd3HzBoAzYBN7r7H3TcBuPsi4BOgYwyzikg9kL9+Fl22NeK03hcGHSUhxLJAFAIdzKy9mTUExgJTq+wzFbgs/DoPmO3ubmatwp3cmNkpQAdgbQyzikiS++bTZcxrvo28o38SdJSEEbNPMbl7mZldB7wOpALPuPtyM7sbKHL3qcDTwHNmtgbYTKiIAAwA7jazUqACuMbdN8cqq4gkv1deuQ83yD33+qCjJAxz96Az1IqcnBwvKioKOoaIxKmBN7bg89TtfPRACZYSr92vdc/MFrl7TrRt+imJSNLbVLyaOc02k9skR8XhMGioDRFJelMm/SflKZB31q+DjpJQVCBEJOnlfzqd9qkN6H72xUFHSShqa4lIUtv6zWe82XQDuY2y9XjpMKkFISJJbVrBfZSmQl6/a4KOknBUIEQkqRWsnsJJDVLpPfCKoKMkHLW3RCRpbd/0JTMzvmZ0g256vFQDakGISNKaXjCRPQ0gr88vg46SkFQgRCRpFXw0ieMbpHDGBeODjpKQ1OYSkaS0a9tGZjRez2jrTEqq3gvXhH5qIpKUZk76A7saQl6OOqdrSgVCRJJS/rKXaZlm9B9ybdBREpYeMYlI0inZsZVp6Z8xqqITDRqmBx0nYakFISJJZ9bk/8uOhpCbOS7oKAlNBUJEkk7+kr9zTJrx0xGaWvRI6BGTiCSVvbt3MDXtU0aUnkpaepOg4yQ0tSBEJKnMnvIQW9Od3C4aufVIqUCISFLJX/QcTdNg4Khbgo6S8PSISUSSRtneEl5JXc2wPW1pdNTRQcdJeGpBiEjS+Oe0R9nU2Mk7bUzQUZKCCoSIJByvqODLNe+zdNEMlq59lyWbV7LUv+ajpnvIKINBoyYEHTEpqECISFwr2bGVFQuns2T5Wyz9cjFLdn/K0vRtbGrslfu0tVQyK1oy0n/M6HN/RZNmLQNMnDxUIEQkLnhFBetXL/q+VbBlJUv9G1Y13UN5uLe0cSp0s6MYXdGJzCZdyTrtLLr1Hkbz49oGGz5JqUCISJ3b/d1mVhTOCLUKvlrMkt3rWJq+jc0RrYJ21oDMipaM5sdktelNZtb5nJp1DqlpDQNMXr+oQIhIzHhFBcWrCln6/mssWTufpVs+YinfsKrpXirCrYImKaFWQV5FJzKP6kbWaWfTrfdQmv3o5GDDiwqEiNSO3d9tZvnCV1myYvb3rYLG37ElPaJVkNKArPKW5NGBrJP6hFoF2edovoY4pb8VETksXlHBFx8tCLUKPn2XpVtWscS+YXXG962Co1Kgm2XwL+WnkZmRSdZpZ9G11xC1ChKMCoSIHNCubRtZtmAaS1fODbUKSj5jaePv2BrRKmif0oCs8laMsQ5kndyHzOxBnJJ5lloFSUB/gyKCV1Tw+cp3Wfr+zFCrYGu4VdC0FLfQPhmp0I0MxlacTmZGJpmnDaBb72Ec3apNsOElZmJaIMxsMPAwkAr8t7tPrLK9EfA3oCewCRjj7uvC224DfgGUAze4++uxzCpSX+zc8i3LFk5j6cp/suTrxSwNtwq2Rcyrc0q4VXAxHclqG2oVtO82QK2CeiZmf9tmlgo8BgwEioFCM5vq7isidvsFsMXdf2xmY4H7gTFm1hkYC3QBTgTeNLOO7l4eq7wi8cIrKnCvOPT/RrwGcPfK5W0b1/PhktdZ8ul74VbBt6yp0irIpCmXVHQhs2kmWaefTdfeQ2na4sQAfwISL2L5dqA3sMbd1wKY2YvACCCyQIwA7gq/zgf+bGYWXv+iu+8BPjWzNeHzvVvbITd/+QlnPti5tk8b97z6XQ7vfFbLJwS8llN65R/HLfoyB9rHwusqX1PltVf+DA64z8H+G4OfX1WnWgOy/Edcah3JavcTMrMH0a7rmWoVyAHF8l9Ga+CLiOVioM+B9nH3MjPbBrQIr3+vyrGtq17AzMYD4wFOPrlmn45okNaIrvyoRscmOqMWfys5tXm2SqH3C7V4vvBdm4demf1wObRflX2ciNcHX//9dQyziPMc7H++L4tV3vf3+ar5b+XrlO+vHV7fpGEGXTudSbc+w8k49vha/VlK8kvotw7u/hTwFEBOTk6N3m4e3aoNL//xi+p3FBGpZ2I5H8R64KSI5TbhdVH3MbMGQDNCndWHcqyIiMRQLAtEIdDBzNqbWUNCnc5Tq+wzFbgs/DoPmO3uHl4/1swamVl7oAOwMIZZRUSkipg9Ygr3KVwHvE7oY67PuPtyM7sbKHL3qcDTwHPhTujNhIoI4f1eJtShXQb8Wp9gEhGpWxZ6w574cnJyvKioKOgYIiIJxcwWuXtOtG2ak1pERKJSgRARkahUIEREJCoVCBERiSppOqnNbAPw2RGcoiWwsZbiBClZ7gN0L/EqWe4lWe4Djuxe2rp7q2gbkqZAHCkzKzpQT34iSZb7AN1LvEqWe0mW+4DY3YseMYmISFQqECIiEpUKxPeeCjpALUmW+wDdS7xKlntJlvuAGN2L+iBERCQqtSBERCQqFQgREYlKBaIKM7vZzNzMWgadpabM7B4zW2pmi83sDTNL2AmGzewBM/sofD+Tzax50JlqyswuMrPlZlZhZgn38UozG2xmq8xsjZn9Lug8NWVmz5jZt2a2LOgsR8rMTjKzOWa2Ivxv6ze1eX4ViAhmdhJwPvB50FmO0APununu2cCrwB0B5zkSs4Cu7p4JfAzcFnCeI7EMGA3MCzrI4TKzVOAx4AKgM3CxmSXqZO5/BQYHHaKWlAE3u3tn4CfAr2vz70UFYn9/An7L9/PXJyR3/y5i8SgS+H7c/Q13LwsvvkdodsGE5O4r3X1V0DlqqDewxt3Xuvte4EVgRMCZasTd5xGafybhuftX7v5++PV2YCXQurbOn9BzUtcmMxsBrHf3Jfsmrk9kZnYv8HNgG3BOwHFqy5XAS0GHqKdaA5GTtxcDfQLKIlGYWTugO7Cgts5ZrwqEmb0JHB9l0+3AvxF6vJQQDnYv7j7F3W8Hbjez24DrgDvrNOBhqO5ewvvcTqg5/XxdZjtch3IvIrXNzDKAAuDGKk8Qjki9KhDufl609WbWDWgP7Gs9tAHeN7Pe7v51HUY8ZAe6lyieB2YQxwWiunsxs8uBocC5Hudf3DmMv5dEsx44KWK5TXidBMzM0ggVh+fdfVJtnrteFYgDcfcPgR/tWzazdUCOuyfkSI9m1sHdV4cXRwAfBZnnSJjZYEL9Qme5+66g89RjhUAHM2tPqDCMBS4JNpJY6B3t08BKd/9jbZ9fndTJaaKZLTOzpYQem9XqR9/q2J+BpsCs8Md2nwg6UE2Z2SgzKwb6AtPN7PWgMx2q8AcFrgNeJ9QR+rK7Lw82Vc2Y2QvAu0AnMys2s18EnekI9AN+Bvw0/P+PxWZ2YW2dXENtiIhIVGpBiIhIVCoQIiISlQqEiIhEpQIhIiJRqUCIiEhUKhCSlMxsxxEce114xNL9RvW1kEfC25aaWY+IbSeY2avh12fve32kzGzuoYz8ambrqhuB2MzeNLNjaiOX1A8qECI/9A5wHvBZlfUXAB3Cf8YDj0dsuwn4S52kq7nngGuDDiGJQwVCklr4Xf8D4S8OfmhmY8LrU8zsv8JzTcwysxlmlgfg7h+4+7oopxsB/M1D3gOam9kJ4W25wMwo1+9tZu+a2QdmNt/MOoXXX25mr4SvvS7carkpvN97ZnZsxGl+Fv4C1DIz6x0+vkV4ro/lZvbfgEVc8xUzWxTeNj7iPFOBi2v6s5T6RwVCkt1oIBvIItQqeCD8S3000I7Q3AY/I/Tt5upEG9G0dXj4iS3uvifKMR8B/d29O6F5Of4zYlvXcI5ewL3ArvB+7xIaiXefJuG5Pa4FngmvuxP4f+7eBZgMnByx/5Xu3hPIAW4wsxYA7r4FaLRvWaQ6GotJkt2ZwAvuXg58Y2b/JPQL+UzgH+5eAXxtZnOO4BonABsOsK0Z8D9m1oHQvBxpEdvmhMfw325m24Bp4fUfApkR+70AoXkMzOxoC82qN4BQccHdp5vZloj9bzCzUeHXJxF6JLYpvPwtcGLEssgBqQUhcugONKLpbiD9AMfcQ6gQdAWGVdkvssVREbFcwf5v3qqOh3PA8XHM7GxCLaW+7p4FfFDlmunhvCLVUoGQZPc2MMbMUs2sFaF33gsJdUTnhvsijgPOPoRzTQV+Hu7X+Amwzd2/IjQVarsDHNOM74fFvryG97Cv3+TM8DW3EZq29JLw+guAfZ9OakbocdcuMzuN0DSUhPczQnNVrKthDqlnVCAk2U0GlgJLgNnAb8NzfBQQ6kNYAfwv8D6h2fcwsxvCo662AZaGO4EhNK/GWmANoU8sXQvg7juBT8zsx1Gu/wfgPjP7gJo/0i0JH/8EsG/k0d8DA8xsOaFHTfvmUZ8JNDCzlcBEQtO07tMTeC9iCleRg9JorlJvmVmGu+8Id9ouBPrVdIKo8DP/nu7+77UashaZ2cPAVHd/K+gskhjUSS312avhDt+GwD1HMnugu09OgE8HLVNxkMOhFoSIiESlPggREYlKBUJERKJSgRARkahUIEREJCoVCBERier/A5PsezvJwAMMAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -2743,9 +2798,7 @@ { "cell_type": "markdown", "id": "05e70aa8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The results here agree when we force **Scikit-Learn**'s Ridge function to include the first column in our design matrix.\n", "We see that the results agree very well. Here we have thus explicitely included the intercept column in the design matrix.\n", @@ -2756,22 +2809,146 @@ { "cell_type": "markdown", "id": "ea021c33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Taking out the mean" ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 15, "id": "be2dafd7", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Beta values for own Ridge implementation\n", + "[ 3.43579948e-02 -5.43330971e-01 -3.10141414e-03 2.47116868e-01\n", + " 2.18613217e-01 1.02054837e-01 -4.25617659e-04 -5.90475506e-02\n", + " -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02\n", + " 1.11482289e-02 2.88529063e-02 3.67047975e-02 3.38135733e-02\n", + " 2.02198702e-02 -3.46383925e-03 -3.63025821e-02]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[ 3.43579948e-02 -5.43330971e-01 -3.10141413e-03 2.47116868e-01\n", + " 2.18613217e-01 1.02054837e-01 -4.25617657e-04 -5.90475506e-02\n", + " -7.68534263e-02 -6.68929213e-02 -4.24906604e-02 -1.40927184e-02\n", + " 1.11482289e-02 2.88529063e-02 3.67047975e-02 3.38135733e-02\n", + " 2.02198702e-02 -3.46383925e-03 -3.63025821e-02]\n", + "Intercept from own implementation:\n", + "1.033030804518907\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.03303080451838\n", + "MSE values for own Ridge implementation\n", + "3.139255959141931e-06\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "3.1392559585690806e-06\n", + "Beta values for own Ridge implementation\n", + "[-0.05807125 -0.29822833 -0.08551306 0.08156108 0.13679863 0.12333649\n", + " 0.08251519 0.03815288 0.00111756 -0.02498832 -0.04010697 -0.04566964\n", + " -0.04355837 -0.03562355 -0.02348765 -0.00848904 0.00831018 0.0260906\n", + " 0.04423486]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.05807125 -0.29822833 -0.08551306 0.08156108 0.13679863 0.12333649\n", + " 0.08251519 0.03815288 0.00111756 -0.02498832 -0.04010697 -0.04566964\n", + " -0.04355837 -0.03562355 -0.02348765 -0.00848904 0.00831018 0.0260906\n", + " 0.04423486]\n", + "Intercept from own implementation:\n", + "1.0411487294305777\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0411487294305242\n", + "MSE values for own Ridge implementation\n", + "1.9601304850224855e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "1.9601304850078175e-05\n", + "Beta values for own Ridge implementation\n", + "[-0.1416398 -0.14021063 -0.05383795 0.01367553 0.04784395 0.05796251\n", + " 0.05447415 0.044613 0.03267527 0.02098261 0.01066519 0.00217499\n", + " -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081 -0.01416528\n", + " -0.01290947]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.1416398 -0.14021063 -0.05383795 0.01367553 0.04784395 0.05796251\n", + " 0.05447415 0.044613 0.03267527 0.02098261 0.01066519 0.00217499\n", + " -0.00440346 -0.00917248 -0.01231917 -0.01405935 -0.0146081 -0.01416528\n", + " -0.01290947]\n", + "Intercept from own implementation:\n", + "1.04955699662783\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0495569966278266\n", + "MSE values for own Ridge implementation\n", + "5.495916150937975e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "5.495916150936546e-05\n", + "Beta values for own Ridge implementation\n", + "[-0.13535942 -0.08593216 -0.03568439 -0.0036367 0.01397146 0.02229529\n", + " 0.02503753 0.0245528 0.02228115 0.01908936 0.01549377 0.01179792\n", + " 0.00817631 0.00472512 0.00149311 -0.00149956 -0.00424967 -0.00676387\n", + " -0.00905423]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.13535942 -0.08593216 -0.03568439 -0.0036367 0.01397146 0.02229529\n", + " 0.02503753 0.0245528 0.02228115 0.01908936 0.01549377 0.01179792\n", + " 0.00817631 0.00472512 0.00149311 -0.00149956 -0.00424967 -0.00676387\n", + " -0.00905423]\n", + "Intercept from own implementation:\n", + "1.0399676689527961\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "1.0399676689527975\n", + "MSE values for own Ridge implementation\n", + "7.571105947979378e-05\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "7.571105947979412e-05\n", + "Beta values for own Ridge implementation\n", + "[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706 -0.00517114\n", + " -0.00174276 0.00068734 0.00243186 0.00369758 0.00462287 0.0053018\n", + " 0.00579953 0.006162 0.00642221 0.00660427 0.00672607 0.0068011\n", + " 0.00683964]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.05100875 -0.04063602 -0.02723445 -0.01713366 -0.0100706 -0.00517114\n", + " -0.00174276 0.00068734 0.00243186 0.00369758 0.00462287 0.0053018\n", + " 0.00579953 0.006162 0.00642221 0.00660427 0.00672607 0.0068011\n", + " 0.00683964]\n", + "Intercept from own implementation:\n", + "0.999955585168597\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "0.999955585168597\n", + "MSE values for own Ridge implementation\n", + "0.0007698473260556331\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.0007698473260556325\n", + "Beta values for own Ridge implementation\n", + "[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335\n", + " -0.00323332 -0.00274989 -0.0023548 -0.00202756 -0.00175331 -0.00152117\n", + " -0.001323 -0.0011526 -0.00100519 -0.00087697 -0.00076495 -0.00066668\n", + " -0.00058016]\n", + "Beta values for Scikit-Learn Ridge implementation\n", + "[-0.00834567 -0.00803064 -0.00673407 -0.00554552 -0.00458878 -0.0038335\n", + " -0.00323332 -0.00274989 -0.0023548 -0.00202756 -0.00175331 -0.00152117\n", + " -0.001323 -0.0011526 -0.00100519 -0.00087697 -0.00076495 -0.00066668\n", + " -0.00058016]\n", + "Intercept from own implementation:\n", + "0.9637117593816477\n", + "Intercept from Scikit-Learn Ridge implementation\n", + "0.9637117593816477\n", + "MSE values for own Ridge implementation\n", + "0.0023813163025848865\n", + "MSE values for Scikit-Learn Ridge implementation\n", + "0.002381316302584886\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0tUlEQVR4nO3dd3xUVdrA8d+TDoQOIp2ICIYWpIkURUBRIthYiUoRfLHAsor6CjZEdC0oYAHWBjaWIliiUgTFF1dUitKLFFFBpISWEFLnef+YSzaEFAgzuZPk+X4+89m595x77nNGdp6ce+6cK6qKMcYYc66C3A7AGGNMyWAJxRhjjE9YQjHGGOMTllCMMcb4hCUUY4wxPmEJxRhjjE/4NaGISE8R2Soi20VkVC7l4SIy2yn/UUQaZCsb7ezfKiJXO/vqishSEdkkIhtF5B/Z6j8pIntEZI3zutaffTPGGHMq8dfvUEQkGPgF6AHsBlYCcaq6KVude4EWqnq3iPQDblDVW0QkGpgJtANqAUuAi4DzgJqq+pOIlAdWA9er6iYReRJIUtUX/dIhY4wx+fLnCKUdsF1Vd6pqGjAL6JOjTh/gXef9XKCbiIizf5aqpqrqr8B2oJ2q7lXVnwBUNRHYDNT2Yx+MMcacoRA/tl0b+CPb9m6gfV51VDVDRI4CVZ39P+Q49pTE4VweawX8mG33cBEZAKwCHlDVwzmDEpGhwFCAcuXKtW7SpMlZd8wYY0qz1atXH1TV6jn3+zOh+I2IRALzgPtU9ZizeyowDlDnf18CBuc8VlXfAN4AaNOmja5atapIYjbGmJJCRH7Lbb8/L3ntAepm267j7Mu1joiEABWBhPyOFZFQvMlkhqp+dLKCqu5T1UxV9QBv4r3kZowxpoj4M6GsBBqJSJSIhAH9gPgcdeKBgc77m4Gv1XuXQDzQz7kLLApoBKxw5lfeBjar6oTsDYlIzWybNwAbfN4jY4wxefLbJS9nTmQ4sAgIBqap6kYReQpYparxeJPD+yKyHTiEN+ng1JsDbAIygGGqmikinYD+wHoRWeOc6hFVnQ+8ICIxeC957QLu8lffjDHGnM5vtw0XB7nNoaSnp7N7925SUlJcisoEooiICOrUqUNoaKjboRjjOhFZraptcu4vlpPy/rR7927Kly9PgwYN8F5hM6WdqpKQkMDu3buJiopyOxxjApYtvZJDSkoKVatWtWRisogIVatWtVGrMQWwhJILSyYmJ/s3YUzBLKEYY0wpkpHp8VvbllACkIhw++23Z21nZGRQvXp1YmNjAdi3bx+xsbG0bNmS6Ohorr3Wuw7mrl27KFOmDDExMVmv9957r0hjzx5DdHQ0AwYMID09HYBVq1YxYsSIXI9r0KABBw8ePKdzT58+PavfYWFhNG/enJiYGEaNOm1d0jxNmjSJ5OTkc4rDmED2697D3DP1A3YfOFZw5bOlqqX21bp1a81p06ZNp+0rauXKldOWLVtqcnKyqqrOnz9fW7Zsqb169VJV1aFDh+qkSZOy6q9du1ZVVX/99Vdt2rRp0QecTfYYMjIytGvXrvrBBx8UeFz9+vX1wIEDPoujsO3ld1wg/NswJhDg/enHad+pNkIJUNdeey1ffPEFADNnziQuLi6rbO/evdSpUydru0WLFmfV9ldffUWrVq1o3rw5gwcPJjU1lZUrV3LjjTcC8Omnn1KmTBnS0tJISUnhggsuAOCKK67g4Ycfpl27dlx00UV8++23+Z4nODiYdu3asWePd4GEb775JmuUlZCQwFVXXUXTpk2588470Wy3r48bN47GjRvTqVMn4uLiePFF7wLSO3bsoGfPnrRu3ZrOnTuzZcuWM+rv+PHjadu2LS1atGDMmDEAHD9+nF69etGyZUuaNWvG7NmzeeWVV/jzzz/p2rUrXbt2PYtP1Jji4fX5y5n6xXd+a98SSgGuuOL015Qp3rLk5NzL33nHW37w4OllZ6pfv37MmjWLlJQU1q1bR/v2/11Xc9iwYQwZMoSuXbvyzDPP8Oeff2aV7dix45RLXjm/9FNSUhg0aBCzZ89m/fr1ZGRkMHXqVFq1asWaNWsA+Pbbb2nWrBkrV67kxx9/POXcGRkZrFixgkmTJjF27Nh8+5CSksKPP/5Iz549TysbO3YsnTp1YuPGjdxwww38/vvvAKxcuZJ58+axdu1aFixYQPbfCQ0dOpRXX32V1atX8+KLL3LvvfcW+Dl++eWXbNu2jRUrVrBmzRpWr17NsmXLWLhwIbVq1WLt2rVs2LCBnj17MmLECGrVqsXSpUtZunRpgW0bU9w8+OWDjFjan7T0TL+0b79DCVAtWrRg165dzJw5M2uO5KSrr76anTt3snDhQhYsWECrVq3YsMG70kzDhg2zEkNutm7dSlRUFBdddBEAAwcOZPLkydx33300bNiQzZs3s2LFCkaOHMmyZcvIzMykc+fOWcefHMW0bt2aXbt25XqOk0nt119/pVevXrmOoJYtW8ZHH3mXYuvVqxeVK1cG4LvvvqNPnz5EREQQERHBddddB0BSUhLLly+nb9++WW2kpqbm9xEC3oTy5Zdf0qpVq6x2tm3bRufOnXnggQd4+OGHiY2NPaWPxpREU7/4jqTK33Nz2VcJCw32yzksoRTgm2/yLitbNv/yatXyLy9I7969efDBB/nmm29ISEg4paxKlSrceuut3HrrrcTGxrJs2TJat25d+JMBXbp0YcGCBYSGhtK9e3cGDRpEZmYm48ePz6oTHh4OeC9nZWRk5NrOyaR28OBBOnbsSHx8PL179z6n2DweD5UqVco3WeZGVRk9ejR33XX6Sjw//fQT8+fP57HHHqNbt2488cQT5xSjMYFs3JIXkfAqTB5xh9/OYZe8AtjgwYMZM2YMzZs3P2X/119/nXUnUmJiIjt27KBevXpn1Gbjxo3ZtWsX27dvB+D999/n8ssvB6Bz585MmjSJDh06UL16dRISEti6dSvNmjUrVPzVqlXjueee49lnnz2trEuXLvz73/8GYMGCBRw+7H10TceOHfnss89ISUkhKSmJzz//HIAKFSoQFRXFhx9+CHgTxdq1awuM4eqrr2batGkkJSUBsGfPHvbv38+ff/5J2bJluf3223nooYf46aefAChfvjyJiYmF6q8xgWrRql/YW/FTOobfy3mVy/ntPDZCCWB16tTJ9Tbb1atXM3z4cEJCQvB4PNx55520bduWXbt2ZV1uOmnw4MGntBEREcH06dPp27cvGRkZtG3blrvvvhuA9u3bs2/fPrp06QJ4L7v99ddf5/Sjvuuvv54nn3zytLmcMWPGEBcXR9OmTbnsssuyEmLbtm3p3bs3LVq0oEaNGjRv3pyKFSsCMGPGDO655x6efvpp0tPT6devHy1btsz3/FdddRWbN2+mQ4cOAERGRvLBBx+wfft2HnroIYKCgggNDWXq1KmAd56mZ8+eWXMpxpQE3235haDjtZh673C/nscWh8yxOOTmzZu5+OKLXYrIgHeeIzIykuTkZLp06cIbb7zBJZdc4nZY9m/DFGtp6Zk+mzvJa3FIu+RlAs7QoUOJiYnhkksu4aabbgqIZGJMcTV/xRafJpP82CUvE3BOzq0YY87NwaPJxH7UiSZzb2bTC//y+/ksoRhjTAn197feRcskcO8ltxXJ+SyhGGNMCZSWnsm8PydQztOOe3t1KpJzWkIxxpgS6PEZ8aRX2M7f63xIUFDRPH7BEooxxpRAM9fNJiQoimcH3FBk57S7vAKQv5ev//zzz2nVqlXW8a+//joATz75ZNZCjHlp0KABzZs3p0WLFlx++eX89ttvWWWXXXZZrscMGjSIuXPnnt2HkMP69euz+lSlShWioqKIiYmhe/fuZ9zGJ598wqZNm84pDmOKi+3Pz2Bx/yVFcnfXSTZCCUDlypVjw4YNnDhxgjJlyrB48WJq166dVf7EE0/Qo0cP/vGPfwCwbt26rLKC1vJKT09n6NChrFixgjp16pCamprnmlx5Wbp0KdWqVWPMmDE8/fTTvPnmmwAsX778rNo5G82bN8/q16BBg4iNjeXmm28+qzY++eQTYmNjiY6O9kOExgSO5JR0ykaEckXLC4r0vDZCCVD+Wr4+MTGRjIwMqlatCnjX5mrcuHGhYuzQoUPW0vTg/RU6eJdFGT58OI0bN6Z79+7s378/q878+fNp0qQJrVu3ZsSIEVmjruPHjzN48GDatWtHq1at+PTTT88ohi+//JIOHTpwySWX0Ldv36wlVkaNGkV0dDQtWrTgwQcfZPny5cTHx/PQQw8RExPDjh07CtVnYwLdolW/UH5MXSZ8XPQrPdgIpQBXvHPFafv+1vRv3Nv2XpLTk7l2xrWnlQ+KGcSgmEEcTD7IzXNO/Sv6m0HfnNF5+/Xrx1NPPUVsbCzr1q1j8ODBWcuXDBs2jFtuuYXXXnuN7t27c8cdd1CrVi2A05ZeefXVV09ZSbdKlSr07t2b+vXr061bN2JjY4mLiyMo6Oz/tli4cCHXX3/9afs//vhjtm7dyqZNm9i3bx/R0dEMHjyYlJQU7rrrLpYtW0ZUVNQpSfKZZ57hyiuvZNq0aRw5coR27drRvXt3ypXLe92hgwcP8vTTT7NkyRLKlSvH888/z4QJExg2bBgff/wxW7ZsQUQ4cuQIlSpVonfv3oUa2RhTnIz8cCKesCNcFVP0I3FLKAHKX8vXA7z11lusX7+eJUuW8OKLL7J48WLeOfkQlzPQtWtXDh06RGRkJOPGjTutfNmyZcTFxREcHEytWrW48sorAdiyZQsXXHABUVFRAMTFxfHGG28A3pFGfHx81hxOSkoKv//+e75Lnfzwww9s2rSJjh07ApCWlkaHDh2oWLEiERERDBkyhNjY2KxRkDEl3cZd+9kU+g5NUgfQLKpGkZ/fEkoB8htRlA0tm295tbLVznhEkht/Ll/fvHlzmjdvTv/+/YmKijqrhLJ06VIqVarEbbfdxpgxY5gwYcIZH5sXVWXevHlndflNVenRowczZ848rWzFihV89dVXzJ07l9dee42vv/76nGM0JtANe2cKhKbwUu+Rrpzf5lACmD+Wr09KSuKbbA9pWbNmDfXr1z/r2EJCQpg0aRLvvfcehw4dOqWsS5cuzJ49m8zMTPbu3Zu1am/jxo3ZuXNn1k0As2fPzjrm6quv5tVXX816FPDPP/9cYAyXXnop3333XdZS/MePH+eXX34hKSmJo0ePcu211zJx4sSsZe5taXpTkh1JSmFZymRqHLmOa9s1cSUGSygBLL/l69u0aUOLFi3o0KFD1vL1cPojgF955ZVTjlVVXnjhBRo3bkxMTAxjxow5ZXTy9NNPU6dOnaxXfmrWrElcXByTJ08+Zf8NN9xAo0aNiI6OZsCAAVlLx5cpU4YpU6ZkPRe+fPnyWUvTP/7446Snp9OiRQuaNm3K448/XuDnU716dd555x3i4uKyPostW7aQmJhIbGwsLVq0oFOnTlkjqH79+jF+/HhatWplk/KmxKlQNpyXL5vHKzecfhm6qNjy9bZ8fZE6uTS9qjJs2DAaNWrE/fff73ZYZ8T+bRjjZcvXm4Dw5ptvEhMTQ9OmTTl69Giuj+Y1xpydx9//jBajhrP7wDFX47BJeVOk7r///mIzIjGmuJi0+llSQv7ivEovuxqHJZRcqOo5PfbWlDyl+dKwCWyvz19OUuXvubnsq0W6zEpu7JJXDhERESQkJNgXiMmiqiQkJBAREeF2KMacZuzi8ciJKky+8w63Q7ERSk516tRh9+7dHDhwwO1QTACJiIgo8K43Y4raolW/sLfip3TSRzmvct6rShQVvyYUEekJvAwEA2+p6nM5ysOB94DWQAJwi6rucspGA0OATGCEqi4SkbpO/RqAAm+o6stO/SrAbKABsAv4m6oePtuYQ0NDs37JbYwxgSyyTDiNjw9h6r3D3Q4F8OMlLxEJBiYD1wDRQJyI5FxcZghwWFUvBCYCzzvHRgP9gKZAT2CK014G8ICqRgOXAsOytTkK+EpVGwFfOdvGGFNidWxany3j33RlmZXc+HMOpR2wXVV3qmoaMAvok6NOH+Bd5/1coJt4Z8P7ALNUNVVVfwW2A+1Uda+q/gSgqonAZqB2Lm29C1zvn24ZY4z7Hnh7LtO/XOF2GKfwZ0KpDfyRbXs3//3yP62OqmYAR4GqZ3KsiDQAWgE/OrtqqOpe5/1feC+LnUZEhorIKhFZZfMkxpji6ODRZCZuu5vRC552O5RTFMu7vEQkEpgH3Keqp/2SR723aOV6m5aqvqGqbVS1TfXq1f0cqTHG+N7f33oXLZPAo10fdDuUU/gzoewB6mbbruPsy7WOiIQAFfFOzud5rIiE4k0mM1T1o2x19olITadOTWA/xhhTwqSlZzLvzwmUO9KOYbGdCz6gCPkzoawEGolIlIiE4Z1kj89RJx4Y6Ly/GfjaGV3EA/1EJFxEooBGwApnfuVtYLOq5lwzPXtbA4Eze+SfMcYUI4998CnpFbYztNmDBAUF1g+w/ZZQnDmR4cAivJPnc1R1o4g8JSK9nWpvA1VFZDswEufOLFXdCMwBNgELgWGqmgl0BPoDV4rIGud18ulTzwE9RGQb0N3ZNsaYEmXfsUOUPdKa5wbc6HYop7HVhnOsNmyMMYHO41FXRye22rAxxhRz7y1ZRUamJ+AudZ1kCcUYY4qBRat+YeB/2nHDCxPdDiVPllCMMaYYGPnhRMgM5dl+t7sdSp4soRhjTIDbuGs/m0LfoXHKgIBZZiU3ttqwMcYEuGHvTIHQFF68bqTboeTLEooxxgQwj0dZefRzanAdse0vdjucfFlCMcaYABYUJBx49nt+/eusn8ZR5CyhGGNMgEpLzyQlLYMK5cJp2uA8t8MpkE3KG2NMgHp8RjyVx0axYOVWt0M5I5ZQjDEmQE1ZM54gTwRdWzZ0O5QzYpe8jDEmAL0+fzlJlb/nprKvEBFWPL6qbYRijDEBaOziF5GUyky5c7DboZwxSyjGGBNglq37lb0VP+Gy0Hs5r3I5t8M5Y8VjHGWMMaVIp2YNeGHbYq65pJnboZwVSyjGGBNggoKEh27q5nYYZ80ueRljTADp9c/xtH5kJBmZHrdDOWuWUIwxJkAcPJrMgmMvsPvENkKCi9/Xs13yMsaYADHi7ffQMgd57LKH3A6lUCyhGGNMAEhLz2Tunpco52nLsNjObodTKJZQjDEmADw+I570CtsZVnt2wD7ityDF7yKdMcaUQJ0vbkKrlPt4dsCNbodSaDZCMcaYABDb/mJi2wfu8+LPhI1QjDHGZTe+8DJzlq11O4xzZgnFGGNctHj1Nj5Ovp9XlnzodijnzBKKMca46L45EyAzlCmDhrsdyjmzhGKMMS7Z/PsBNoW+w0Up/Wlxwfluh3PObFLeGGNccu/0KRCawkvXPeB2KD5hIxRjjHFJSFAI9Y/dSmz7i90OxSdshGKMMS5Z/PijbofgUzZCMcaYIpaWnsmEj5fi8ajbofiUJRRjjClij8+I54F1V/Lkv79wOxSfsktexhhTxKaueZGQkAY88reebofiU34doYhITxHZKiLbRWRULuXhIjLbKf9RRBpkKxvt7N8qIldn2z9NRPaLyIYcbT0pIntEZI3zutaffTPGmMJ4ff5yEisvp0+NkUSElay/6f2WUEQkGJgMXANEA3EiEp2j2hDgsKpeCEwEnneOjQb6AU2BnsAUpz2Ad5x9uZmoqjHOa74v+2OMMb4wdvGLSEplXhtyh9uh+Jw/RyjtgO2qulNV04BZQJ8cdfoA7zrv5wLdRESc/bNUNVVVfwW2O+2hqsuAQ36M2xhj/OKvQ0nsD1lJh9B7OL9KpNvh+Jw/x1u1gT+ybe8G2udVR1UzROQoUNXZ/0OOY2ufwTmHi8gAYBXwgKoezllBRIYCQwHq1at3Zj0xxhgfOL9KJEee2kHSiTS3Q/GLknSX11SgIRAD7AVeyq2Sqr6hqm1UtU316tWLMDxjTGl28GgyySnpRJYJK5GjE/BvQtkD1M22XcfZl2sdEQkBKgIJZ3jsKVR1n6pmqqoHeBPnEpkxxgSCvpPGU+HxC/kzIdHtUPzGnwllJdBIRKJEJAzvJHt8jjrxwEDn/c3A16qqzv5+zl1gUUAjYEV+JxORmtk2bwA25FXXGGOK0qFjJ/i/E69RNaMFtaqWdzscv/HbHIozJzIcWAQEA9NUdaOIPAWsUtV44G3gfRHZjneivZ9z7EYRmQNsAjKAYaqaCSAiM4ErgGoishsYo6pvAy+ISAygwC7gLn/1zRhjzsbwt95FyxzkkQ4Puh2KX4l3QFA6tWnTRletWuV2GMaYEiwtPZPIURcT5qnEsZd+JChI3A7pnInIalVtk3N/yfpVjTHGBJgX5i0mvcI2htWeXSKSSX4soRhjjB898rerCZn3Jff16ep2KH5nCcUYY/woKEgY1beH22EUiZL0OxRjjAkoFz00hKvG/dPtMIqMJRRjjPGDxau3sa3cdI6nH3c7lCJjCcUYY/zg/jkTITOUyQOHux1KkbGEYowxPrb59wNsDJ3ORSn9iWlYs+ADSgiblDfGGB+7d/oUCE3hpesecDuUImUJxRhjfGzo5b0pu7wsse0vdjuUImUJxRhjfCzuilbEXdHK7TCKnM2hGGOMj6SlZ9L+sYdZsHKr26G4whKKMcb4yJh/f8aK0Bf4cu1at0NxRb4JRURuz/a+Y46y0nMvnDHGnIHJP79ISGIDnh1wo9uhuKKgEcrIbO9fzVE22MexGGNMsfXGgu9JrPwdfWqMJCKsdE5PF5RQJI/3uW0bY0ypNXbxi0hKZV4bcofbobimoDSqebzPbdsYY0olj0epEV6P6LIPlNjnxZ+JghJKExFZh3c00tB5j7N9gV8jM8aYYiIoSPjp2Yluh+G6ghJK6fpVjjHGnKWtfxxkweoNjOh9eYl/gFZB8p1DUdXfsr+AJOASoJqzbYwxpdo906Zw/9quLF27w+1QXFfQbcOfi0gz531NYAPeu7veF5H7/B+eMcYErkPHTvBN8mucdySWbq0udDsc1xV0l1eUqm5w3t8BLFbV64D22G3DxphS7u9vv4eWPcAjVzzodigBoaCEkp7tfTdgPoCqJgIefwVljDGBLiPTw4e7X6LskTb8/boubocTEAqalP9DRP4O7MY7d7IQQETKAKF+js0YYwLW0jU7yAg5yrBGT5f6yfiTCkooQ4CngO7ALap6xNl/KTDdj3EZY0xA69G6EYca/1ZqfxWfm3w/CVXdD9ydy/6lwFJ/BWWMMYFs6x8HqV+jEpUiI9wOJaDkm1BEJD6/clXt7dtwjDEm8F058S4O6y6SXlpll7uyKWis1gH4A5gJ/Iit32WMKeW++nk7f1b4mMs8oy2Z5FBQQjkf6AHEAbcCXwAzVXWjvwMzxphAdN+siRAWytQ7/u52KAGnoF/KZ6rqQlUdiHcifjvwjT0LxRhTGm394yAbQqdzUUp/WlxwvtvhBJwCb08QkXCgF95RSgPgFeBj/4ZljDGBZ9S/P4DQE4yPHVlw5VKooEn594BmeH/QODbbr+aNMabU+fDBEby9qB29L412O5SAJKp5P9ZERDzAcWcze0UBVFUr+DE2v2vTpo2uWrXK7TCMMcWAx6M2Ce8QkdWq2ibn/oLmUIJUtbzzqpDtVb64JxNjjDlTGZkeKo7syMCX33I7lIBW0Fpe50REeorIVhHZLiKjcikPF5HZTvmPItIgW9loZ/9WEbk62/5pIrJfRDbkaKuKiCwWkW3O/1b2Z9+MMaXHEzM+I6ny91QpZ39H58dvCUVEgoHJwDVANBAnIjkvPA4BDqvqhcBE4Hnn2GigH9AU6AlMcdoDeMfZl9Mo4CtVbQR85WwbY8w5e+2n8YQkNuDZATe6HUpA8+cIpR2wXVV3qmoaMAvok6NOH+Bd5/1coJuIiLN/lqqmquqveG9XbgegqsuAQ7mcL3tb7wLX+7AvxphS6o0F35NY+Tt6n3e/rdtVAH8mlNp4f2V/0m5nX651VDUDOApUPcNjc6qhqnud938BNXKrJCJDRWSViKw6cODAmfTDGFOKPbX4JSSlEpPvtEdAFcSvcyhuUe+ta7nevqaqb6hqG1VtU7169SKOzBhT3DzR4wHuqT+Z86tEuh1KwPPn+G0PUDfbdh1nX251dotICFARSDjDY3PaJyI1VXWv87ji/ecSvDHGAAy9pgPeZQ1NQfw5QlkJNBKRKBEJwzvJnnP14nhgoPP+ZuBrZ3QRD/Rz7gKLAhoBKwo4X/a2BgKf+qAPxphSausfB2n28L18v+l3t0MpNvyWUJw5keHAImAzMEdVN4rIUyJyctn7t4GqIrIdGIlzZ5az+OQcYBPep0QOU9VMABGZCXwPNBaR3SIyxGnrOaCHiGzD+0Cw5/zVN2NMyXfPtClsLDuVhMTjBVc2QAG/lC/p7JfyxpjcHDp2gmrP1Kd6Wnv2TfzM7XACTl6/lLd74IwxJocR095Hyx7gkcsedDuUYsUSijHGZJOR6WHOHy9R1tOGv1/Xxe1wihVLKMYYk83+I8dpFHIlsRdfZYtBniVLKMYYk02tquXZ+PxUt8MolkrkDxuNMaYwPl2+kalffOd2GMWWJRRjjHEMmzuGYf+5joNHk90OpViyhGKMMcBXP29nT4WP6BB6D9UqlnU7nGLJEooxxgD3zZoImaFMHjjc7VCKLUsoxphSb+sfB9kQOp2LUvoT07Cm2+EUW5ZQjDGl3oLVG5CMcoy/aaTboRRrtvSKLb1ijAGSTqQRWSbM7TCKhbyWXrERijGmVPt+0++kpWdaMvEBSyjGmFLr2PFULn+7Bw0fjnM7lBLBEooxplTaf/g4UY9eR3qFXxjQ6na3wykRbOkVY0yp8/v+ozT/ZyzHKi7njirTeKZ/74IPMgWyhGKMKXUufXYQxyr8wP31ZjFhSF+3wykxLKEYY0qd6bc9z4/bhvJE3DVuh1Ki2ByKMaZU+M+GXXR7ahwej3J1m4ssmfiBJRRjTIk3f8UWLn+3E0tPTOD7zb+7HU6JZQnFGFOizVm2lth5XdCgdOZc+390bFrf7ZBKLEsoxpgS662FP9BvwRUEecL54uZl3Ny5hdshlWg2KW+MKbF2JyQQmlqTr4cssJFJEbC1vGwtL2NKnDU79matGpySlkFEmP3t7Eu2lpcxplS4/605tJp+AU/PWghgyaQI2SdtjCkx7nhlGu8k/A8VkjoyqNtlbodT6lhCMcaUCDe98AofnfgHVY9exZaxH9tjfF1gl7yMMcXelM//w0cn/kHNIzew85l4SyYusRGKMabYu/vajmzdO4vnR99kcyYushGKMaZYysj00P6xh5n3n/UEBQkv/88tlkxcZgnFGFPspKRl0PjhQawIfYF/LY13OxzjsHRujClWjh1P5eLHb+XPih/RTcax6NFH3A7JOCyhGGOKjYNHk2ny5I0kVFrE9RET+fjh+9wOyWTj10teItJTRLaKyHYRGZVLebiIzHbKfxSRBtnKRjv7t4rI1QW1KSLviMivIrLGecX4s2/GmKIXFhKMIAys/JYlkwDktxGKiAQDk4EewG5gpYjEq+qmbNWGAIdV9UIR6Qc8D9wiItFAP6ApUAtYIiIXOcfk1+ZDqjrXX30yxrhj2+4EgoKEhrWqsO+l+QQFidshmVz4c4TSDtiuqjtVNQ2YBfTJUacP8K7zfi7QTUTE2T9LVVNV9Vdgu9PembRpjClB1uzYS/MJl9P6hRvweNSSSQDzZ0KpDfyRbXu3sy/XOqqaARwFquZzbEFtPiMi60RkooiE+6ITxhj3fLfxN9pN6UJqmV083mWMJZMAV5JuGx4NNAHaAlWAh3OrJCJDRWSViKw6cOBAUcZnjDkLi1b9wuXvdCIj7CBvdl7CAzde6XZIpgD+TCh7gLrZtus4+3KtIyIhQEUgIZ9j82xTVfeqVyowHe/lsdOo6huq2kZV21SvXr2QXTPG+JPHo9z8wSA8QanM7LmUO3te6nZI5gz487bhlUAjEYnC+6XfD7g1R514YCDwPXAz8LWqqojEA/8WkQl4J+UbASsAyatNEampqnudOZjrgQ1+7Jsxxo+CgoQvBs8gKSWVa9s1cTscc4b8llBUNUNEhgOLgGBgmqpuFJGngFWqGg+8DbwvItuBQ3gTBE69OcAmIAMYpqqZALm16ZxyhohUx5t01gB3+6tvxhj/mPTJN0z7YR4/PfMyXVpEuR2OOUv2xEZ7YqMxAWHsv+fz5KabCEuOYtOD/6FhrSpuh2TyYE9sNMYErJFvf8iTW66nzPFo1t2/zJJJMWVLrxhjXDV08nu8uf8Oyh/rwIZHvqDeeRXdDskUko1QjDGuurBGLc47di3bxyyyZFLMWUIxxrhixtc/AfC/N3dn38TPOK9yOZcjMufKEooxpkh5PEqHx0dz+7eteTV+mdvhGB+yORRjTJHJyPRwyaMjWF9mMhcfv4t7enVyOyTjQzZCMcYUiZS0DJo8PJj1ZSbTOu0BNjw3lZBg+woqSey/pjGmSIybPZ8d5d/lSnmKFePG20KPJZBd8jLGFIln+vem/oLlDL2mg9uhGD+xEYoxxm92HzjG+ff35oOvVgNYMinhLKEYY/xix5+HaPLP7uwrv4Cfdu10OxxTBOySlzHG59bt/It2r/UgNXIbjzX6iHH9r3M7JFMELKEYY3xq3c6/aDO5C+ll9vBCzBc8dFM3t0MyRcQueRljfOrCWlWpQ3v+1XGxJZNSxkYoxhif+HT5Ri6sVZ2mDc5j50vvux2OcYGNUIwx5+zdxSu5Ib4LXV++w+1QjIssoRhjzsmr8csYtLQbwRkVmTPoVbfDMS6yhGKMKbSnZy1kxIqehKXU4fuh33JFywvcDsm4yOZQjDGFkpaeyTMr/pcy2oTV9y/i4nrV3Q7JuMwSijHmrHk8SlhoMN/evYDqFctRv0Ylt0MyAcAueRljzkq/l6ZwwUO3k5aeSZuLalsyMVksoRhjztg1zzzP7KRhpHqSSEnLcDscE2AsoRhjCuTxKB2feJSFGaOofyyOHc/OpUK5cLfDMgHGEooxpkCdxjzK8uB/0uT4//DLc+9TNiLU7ZBMALJJeWNMgW5tdw26QvnuuX/ag7FMnmyEYozJVdKJNJ744HMAhl/Xme/HPWvJxOTLEoox5jSHjp2g4SM3Mm7Hdcz7z3q3wzHFhF3yMsac4s+ERJqO68ORSt9wW8XXualTc7dDMsWEJRRjTJZf9x6m+QvXcLziKu45/32m3H2b2yGZYsQSijEBKC09k2PJqSQmp3I0OYUyYaE0rlsNgA++Ws2R5GSSU1NJTk3leGoKzerWY2CPtng8yk3jXyYlI5XUjFRSM72vHhd14tmB13PwaDKtxg4iQ1NJJ5UMTSFTUulT/w4+uG8oE+IXcjxyDaMbzuOfA/q4/CmY4sYSijE5JJ1IY8/BYxw4msSR4ydIPJFCkAi3XB4DwOvzl7N17x5OpKVyIj2VlPRUKpetwNR7bgeg7/jX2HFoB2meVNI8KaR5UqkX2ZBlY58C4IIH+rPf8wuZpOKRVDxBqdSjEztefA+AkAcbkFn+t1NiqnO0L39MmANA/yXdIOLoKeWNNt/BwB5tCQoSPkl+EIIynZIQIJyQbSHA9YQEB7Gf9QQRTrDzCtPylA2NAODVu+Lot/EyOjat7/sP1pR4llBMsZaSlsG+w0nsP5JE60a1CQoS5q/Ywndbf+HI8SSOpiRx9EQiJ9JPsOSJxwCIe2kq3+xeSKomkSZJZEgSoqGcmLgGgIse6cfeSh+fcp7gpHrccrn3S/7RL8eSUPnLU8rD90QzFW9CWbJnHkfKrkYknCAiEMLxJGZm1Q0NCifCU5lQwgklglDCaVYtJqv8mqp/53jaccJDwokICadMaAStmjXKKn/2kg8JEqFseDjlIyIoFxHOBedXyyrfefcBypcNp0LZcMJCg0+Js1JkBKkTNuf7mVoyMYVlCcUUub8OJbH+170cTEzi4LFEDiUlceh4Ig9dfw11qlfg9fnLmb48nuSMJI5nJJLiSSLFk8gPD86kUZ2q9HluAp8deRYNSYLQlKx29ww/Rq2q5Xni07dYHfbSaedNSx9NWGgwe47t5Qi/EUokEVQmgnqUD62SVe+OSwaxZe8VVIiIpHxEWcqFR1CtfIWs8tkDJ3MsOYXIiHAiy4RTvkw4lSLLZJUfnrQ03/5vHf9WvuWfjX4g3/JRfXvkWx5Vs3K+5cb4i18Tioj0BF4GgoG3VPW5HOXhwHtAayABuEVVdzllo4EhQCYwQlUX5demiEQBs4CqwGqgv6qm+bN/JUlauvcv6LDQYJJT0tn0+35S0tI5npJGcqr3dWmTBjSsVYU1O/YyeeGXHElOJDE1iaS0JI6nJzG2z1B6XxrNq/HLeOzrx0iXRDKCksgMScQTksSbnZdwZ89LeWTGh0w/NPi0GDpvWcvN1Vvw1aaf+VEmIpQniEhCiCREIkk8kQpAq7qN2XG4L2U1kvJSnvJhkVSIiKRsuPfX25PihrNrXxxVK0RSo1J5zqsUyXmVymX9te699PRUnp/FM/175/tZdWt1YWE+YmNKPL8lFBEJBiYDPYDdwEoRiVfVTdmqDQEOq+qFItIPeB64RUSigX5AU6AWsERELnKOyavN54GJqjpLRP7ltD3VX/0rSNKJNJJOpHEi1fulfDwljcgy4TRtcB4AM77+iaSUFE6kpXEiLY3U9HQurlObWy6PweNRBr82jbSMdFIz0kjNTCMtI40rm7RlVN8eHElKoeszo0j3pJHhSSdd08jQNG68+AYm3vk3tu1OoN2LfckkDY+k4SEdj6QR1+ABpo8YzFc/b6fH7EvRoDQIdl6i3FbxdT64byhz/7OWgd+1Pa1Pw7f9m1fviuOb9Vt56+Cg/xZIEFCedbt60vvSaEKCgwkihArUJYJIymh5ymkkF5zvfV7G7Z27EPHDB1QqG0nlcpFUiYykesXydGnmfTjTrJH3MidoWJ6f7ZO39eLJ23rlWd6pWQM6NWtw1v/NjDHnxp8jlHbAdlXdCSAis4A+QPaE0gd40nk/F3hNRMTZP0tVU4FfRWS70x65tSkim4ErgVudOu867fotoZx3/7UcDt6KShqeoDQISqNm6pXsmTAPgIpPXIAncs8px9Q9+jd+nzAbgNsXX3n6xOqawdxy+dsEBQnvJgyFIM8p5Qd++gej+vbA41HWyHREwhAJI0jDEEL57dClAIQEB+EhgxDCCaY8IYQRTBi1Knuvs9eqWpFm9COUMEI1jFANJSwojJ7NWwPQvnEDbtvwOhEhYYSHhBEeGkpEaBh92l7ijb1rO2KidlCjcnlqVIqkUmTEKb+gvqdXR+7p9XWen92VMQ25MqZhnuX2a2xjiid/JpTawB/ZtncD7fOqo6oZInIU7yWr2sAPOY6t7bzPrc2qwBFVzcil/ilEZCgwFKBevXpn16Ns6pdtSmRKVUIkjFDCCNFQmtVtmlV+U43RHE9L9n4hB4cRHhpGTMv/fok+HTObIBHCQ0MpExZG2bAwLqx1Xlb58lt+pVxEGOUiwigTHkpkRFjWgnxVKpRB/3lqMsouqmZljk5almf5xfWqs+651/Isb1y3Gh/cNzTP8moVy9qjXo0xpyl1k/Kq+gbwBkCbNm20sO2sfGZ8vuVzHsz7kg3Ao7dcnW95h+jCJztjjHGDP9fy2gPUzbZdx9mXax0RCQEq4p2cz+vYvPYnAJWcNvI6lzHGGD/yZ0JZCTQSkSgRCcM7yR6fo048MNB5fzPwtaqqs7+fiIQ7d281Albk1aZzzFKnDZw2P/Vj34wxxuTgt0tezpzIcGAR3lt8p6nqRhF5ClilqvHA28D7zqT7IbwJAqfeHLwT+BnAMFXNBMitTeeUDwOzRORp4GenbWOMMUVEvH/cl05t2rTRVatWuR2GMcYUKyKyWlXb5Nxvz0MxxhjjE5ZQjDHG+IQlFGOMMT5hCcUYY4xPlOpJeRE5APxWYMXcVQMO+jAcN1lfAk9J6QdYXwLVufSlvqpWz7mzVCeUcyEiq3K7y6E4sr4EnpLSD7C+BCp/9MUueRljjPEJSyjGGGN8whJK4b3hdgA+ZH0JPCWlH2B9CVQ+74vNoRhjjPEJG6EYY4zxCUsoxhhjfMISig+IyAMioiJSze1YCktExonIOhFZIyJfikgtt2MqDBEZLyJbnL58LCKV3I6psESkr4hsFBGPiBTLW1VFpKeIbBWR7SIyyu14CktEponIfhHZ4HYs50JE6orIUhHZ5Pzb+ocv27eEco5EpC5wFfC727Gco/Gq2kJVY4DPgSdcjqewFgPNVLUF8Asw2uV4zsUG4EYg7+c5BzARCQYmA9cA0UCciES7G1WhvQP0dDsIH8gAHlDVaOBSYJgv/5tYQjl3E4H/BYr13Q2qeizbZjmKaX9U9UtVzXA2f8D79M5iSVU3q+pWt+M4B+2A7aq6U1XTgFlAH5djKhRVXYb3mU3FmqruVdWfnPeJwGagtq/aL3XPlPclEekD7FHVtSLidjjnTESeAQYAR4GuLofjC4OB2W4HUYrVBv7Itr0baO9SLCYHEWkAtAJ+9FWbllAKICJLgPNzKXoUeATv5a5iIb++qOqnqvoo8KiIjAaGA2OKNMAzVFA/nDqP4h3ezyjK2M7WmfTFGF8TkUhgHnBfjqsT58QSSgFUtXtu+0WkORAFnByd1AF+EpF2qvpXEYZ4xvLqSy5mAPMJ0IRSUD9EZBAQC3TTAP+h1Vn8NymO9gB1s23XcfYZF4lIKN5kMkNVP/Jl25ZQCklV1wPnndwWkV1AG1UtliuRikgjVd3mbPYBtrgZT2GJSE+8c1qXq2qy2/GUciuBRiIShTeR9ANudTek0k28f/2+DWxW1Qm+bt8m5c1Jz4nIBhFZh/cynk9vJyxCrwHlgcXOLdD/cjugwhKRG0RkN9AB+EJEFrkd09lwbo4YDizCO/k7R1U3uhtV4YjITOB7oLGI7BaRIW7HVEgdgf7Alc7/P9aIyLW+atyWXjHGGOMTNkIxxhjjE5ZQjDHG+IQlFGOMMT5hCcUYY4xPWEIxxhjjE5ZQjHGISNI5HDvcWVH3lFWnxesVp2ydiFySraymiHzuvL/i5PtzJSLfnMnqxCKyq6AVskVkiYhU9kVcpuSzhGKMb3wHdAd+y7H/GqCR8xoKTM1WNhJ4s0iiK7z3gXvdDsIUD5ZQjMnBGVWMd37ouV5EbnH2B4nIFOd5K4tFZL6I3Aygqj+r6q5cmusDvKdePwCVRKSmU3YTsDCX87cTke9F5GcRWS4ijZ39g0TkE+fcu5xR0Uin3g8iUiVbM/2dH61tEJF2zvFVnWfdbBSRtwDJds5PRGS1UzY0WzvxQFxhP0tTulhCMeZ0NwIxQEu8o47xThK4EWiA99ke/fH+gr0gua24W9tZjuSwqqbmcswWoLOqtsL7XJp/Zitr5sTRFngGSHbqfY93peiTyjrPtrkXmObsGwP8R1WbAh8D9bLVH6yqrYE2wAgRqQqgqoeB8JPbxuTH1vIy5nSdgJmqmgnsE5H/w/sF3gn4UFU9wF8isvQczlETOJBHWUXgXRFphPe5NKHZypY6z7FIFJGjwGfO/vVAi2z1ZoL3OR4iUkG8T67sgjcZoapfiMjhbPVHiMgNzvu6eC/RJTjb+4Fa2baNyZWNUIzxr7xW3D0BRORxzDi8iaMZcF2OetlHNJ5s2x5O/QMx55pKea6xJCJX4B2JdVDVlsDPOc4Z4cRrTL4soRhzum+BW0QkWESq4/3LfgXeifebnLmUGsAVZ9BWPDDAmZe5FDiqqnvxPp64QR7HVOS/y7wPKmQfTs77dHLOeRTvo4RvdfZfA5y8e6si3stvySLSBO+jYXHqCd7ntewqZBymFLGEYszpPgbWAWuBr4H/dZ5xMw/vHMgm4APgJ7xPt0RERjgrA9cB1jmT3uB9rsxOYDveO7ruBVDV48AOEbkwl/O/ADwrIj9T+MvSKc7x/wJOrow7FugiIhvxXvr63dm/EAgRkc3Ac3gfnXxSa+CHbI9VNiZPttqwMWdBRCJVNcmZpF4BdCzsA9WcOYvWqvqYT4P0IRF5GYhX1a/cjsUEPpuUN+bsfO5McIcB487l6Zyq+nExuHtqgyUTc6ZshGKMMcYnbA7FGGOMT1hCMcYY4xOWUIwxxviEJRRjjDE+YQnFGGOMT/w/KpKrMndbP4YAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -2856,9 +3033,7 @@ { "cell_type": "markdown", "id": "2e58c926", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see here, when compared to the code which includes explicitely the\n", "intercept column, that our MSE value is actually smaller. This is\n", @@ -2873,9 +3048,7 @@ { "cell_type": "markdown", "id": "e03e422d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Friday September 9" ] @@ -2883,9 +3056,7 @@ { "cell_type": "markdown", "id": "67afcf2d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linking the regression analysis with a statistical interpretation\n", "\n", @@ -2912,9 +3083,7 @@ { "cell_type": "markdown", "id": "3d2ae4a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -2928,9 +3097,7 @@ { "cell_type": "markdown", "id": "d4a4325c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The randomness of $\\varepsilon_i$ implies that\n", "$\\mathbf{y}_i$ is also a random variable. In particular,\n", @@ -2947,9 +3114,7 @@ { "cell_type": "markdown", "id": "e7b6d38b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Assumptions made\n", "\n", @@ -2961,9 +3126,7 @@ { "cell_type": "markdown", "id": "c790eaa0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -2973,9 +3136,7 @@ { "cell_type": "markdown", "id": "39c60c78", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We approximate this function with our model from the solution of the linear regression equations, that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we want to minimize $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, our MSE, with" @@ -2984,9 +3145,7 @@ { "cell_type": "markdown", "id": "d8689c1e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -2996,9 +3155,7 @@ { "cell_type": "markdown", "id": "1120daa6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expectation value and variance\n", "\n", @@ -3008,9 +3165,7 @@ { "cell_type": "markdown", "id": "8ac202b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -3024,9 +3179,7 @@ { "cell_type": "markdown", "id": "13b366a7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "while\n", "its variance is" @@ -3035,9 +3188,7 @@ { "cell_type": "markdown", "id": "defeb04d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\mathbb{E} \\{ [y_i\n", @@ -3058,9 +3209,7 @@ { "cell_type": "markdown", "id": "e4f8de23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$ (not be confused with the singular values of the SVD)." @@ -3069,9 +3218,7 @@ { "cell_type": "markdown", "id": "464c41c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Expectation value and variance for $\\boldsymbol{\\beta}$\n", "\n", @@ -3081,9 +3228,7 @@ { "cell_type": "markdown", "id": "821074e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\beta}) = \\mathbb{E}[ (\\mathbf{X}^{\\top} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1}\\mathbf{X}^{T} \\mathbb{E}[ \\mathbf{Y}]=(\\mathbf{X}^{T} \\mathbf{X})^{-1} \\mathbf{X}^{T}\\mathbf{X}\\boldsymbol{\\beta}=\\boldsymbol{\\beta}.\n", @@ -3093,9 +3238,7 @@ { "cell_type": "markdown", "id": "f1602888", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This means that the estimator of the regression parameters is unbiased.\n", "\n", @@ -3107,9 +3250,7 @@ { "cell_type": "markdown", "id": "e879a87d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -3138,9 +3279,7 @@ { "cell_type": "markdown", "id": "ac6c75d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used that $\\mathbb{E} (\\mathbf{Y} \\mathbf{Y}^{T}) =\n", "\\mathbf{X} \\, \\boldsymbol{\\beta} \\, \\boldsymbol{\\beta}^{T} \\, \\mathbf{X}^{T} +\n", @@ -3160,9 +3299,7 @@ { "cell_type": "markdown", "id": "eb991bec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big]=(\\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I}_{pp})^{-1} (\\mathbf{X}^{\\top} \\mathbf{X})\\boldsymbol{\\beta}^{\\mathrm{OLS}}.\n", @@ -3172,9 +3309,7 @@ { "cell_type": "markdown", "id": "de8b7c6f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see clearly that \n", "$\\mathbb{E} \\big[ \\boldsymbol{\\beta}^{\\mathrm{Ridge}} \\big] \\not= \\boldsymbol{\\beta}^{\\mathrm{OLS}}$ for any $\\lambda > 0$. We say then that the ridge estimator is biased.\n", @@ -3185,9 +3320,7 @@ { "cell_type": "markdown", "id": "c95dcd8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{Ridge}}]=\\sigma^2[ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1} \\mathbf{X}^{T} \\mathbf{X} \\{ [ \\mathbf{X}^{\\top} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T},\n", @@ -3197,9 +3330,7 @@ { "cell_type": "markdown", "id": "9340ef2c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and it is easy to see that if the parameter $\\lambda$ goes to infinity then the variance of Ridge parameters $\\boldsymbol{\\beta}$ goes to zero. \n", "\n", @@ -3209,9 +3340,7 @@ { "cell_type": "markdown", "id": "f6fe0c77", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mbox{Var}[\\boldsymbol{\\beta}^{\\mathrm{OLS}}]-\\mbox{Var}(\\boldsymbol{\\beta}^{\\mathrm{Ridge}})=\\sigma^2 [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}[ 2\\lambda\\mathbf{I} + \\lambda^2 (\\mathbf{X}^{T} \\mathbf{X})^{-1} ] \\{ [ \\mathbf{X}^{T} \\mathbf{X} + \\lambda \\mathbf{I} ]^{-1}\\}^{T}.\n", @@ -3221,9 +3350,7 @@ { "cell_type": "markdown", "id": "0768c7e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The difference is non-negative definite since each component of the\n", "matrix product is non-negative definite. \n", @@ -3233,9 +3360,7 @@ { "cell_type": "markdown", "id": "0c7bd45f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving OLS from a probability distribution\n", "\n", @@ -3256,9 +3381,7 @@ { "cell_type": "markdown", "id": "c37124d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3268,9 +3391,7 @@ { "cell_type": "markdown", "id": "e03377d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Independent and Identically Distrubuted (iid)\n", "\n", @@ -3281,9 +3402,7 @@ { "cell_type": "markdown", "id": "f9e4cb19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", @@ -3293,9 +3412,7 @@ { "cell_type": "markdown", "id": "6bdc4a6f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", "\n", @@ -3305,9 +3422,7 @@ { "cell_type": "markdown", "id": "8712cac0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", @@ -3317,9 +3432,7 @@ { "cell_type": "markdown", "id": "80430601", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n", "in case we have a simple one-dimensional input and output case" @@ -3328,9 +3441,7 @@ { "cell_type": "markdown", "id": "793f35d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n", @@ -3340,9 +3451,7 @@ { "cell_type": "markdown", "id": "c18ad3e8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n", "We can now rewrite the above probability as" @@ -3351,9 +3460,7 @@ { "cell_type": "markdown", "id": "dca8f6bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3363,9 +3470,7 @@ { "cell_type": "markdown", "id": "eced810c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$." ] @@ -3373,9 +3478,7 @@ { "cell_type": "markdown", "id": "bd5abbdf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Maximum Likelihood Estimation (MLE)\n", "\n", @@ -3404,9 +3507,7 @@ { "cell_type": "markdown", "id": "11a1d3c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A new Cost Function\n", "\n", @@ -3416,9 +3517,7 @@ { "cell_type": "markdown", "id": "2ec1cabd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", @@ -3428,9 +3527,7 @@ { "cell_type": "markdown", "id": "1554445a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which becomes" ] @@ -3438,9 +3535,7 @@ { "cell_type": "markdown", "id": "d7242de0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", @@ -3450,9 +3545,7 @@ { "cell_type": "markdown", "id": "1edfabbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" ] @@ -3460,9 +3553,7 @@ { "cell_type": "markdown", "id": "32b3e8c3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", @@ -3472,9 +3563,7 @@ { "cell_type": "markdown", "id": "9dcf6043", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] @@ -3482,9 +3571,7 @@ { "cell_type": "markdown", "id": "b6b40930", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", @@ -3494,9 +3581,7 @@ { "cell_type": "markdown", "id": "b95d46a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Before we make a similar analysis for Ridge and Lasso regression, we need a short reminder on statistics." ] @@ -3504,9 +3589,7 @@ { "cell_type": "markdown", "id": "7dee92e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More basic Statistics and Bayes' theorem\n", "\n", @@ -3524,9 +3607,7 @@ { "cell_type": "markdown", "id": "b22f76fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X \\cup Y)= p(X)+p(Y)-p(X \\cap Y).\n", @@ -3536,9 +3617,7 @@ { "cell_type": "markdown", "id": "12b5e02d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**The product rule (aka joint probability) is given by.**" ] @@ -3546,9 +3625,7 @@ { "cell_type": "markdown", "id": "aae18731", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X \\cup Y)= p(X,Y)= p(X\\vert Y)p(Y)=p(Y\\vert X)p(X),\n", @@ -3558,9 +3635,7 @@ { "cell_type": "markdown", "id": "beda3345", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we read $p(X\\vert Y)$ as the likelihood of obtaining $X$ given $Y$.\n", "\n", @@ -3570,9 +3645,7 @@ { "cell_type": "markdown", "id": "00c88335", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Marginal Probability\n", "\n", @@ -3582,9 +3655,7 @@ { "cell_type": "markdown", "id": "5f3eaf66", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X)=\\sum_{i=0}^{n-1}p(X,Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert Y=y_i)p(Y=y_i)=\\sum_{i=0}^{n-1}p(X\\vert y_i)p(y_i).\n", @@ -3594,9 +3665,7 @@ { "cell_type": "markdown", "id": "5139188c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditional Probability\n", "\n", @@ -3606,9 +3675,7 @@ { "cell_type": "markdown", "id": "1500564b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)}=\\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}.\n", @@ -3618,9 +3685,7 @@ { "cell_type": "markdown", "id": "1ec3367c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem\n", "\n", @@ -3630,9 +3695,7 @@ { "cell_type": "markdown", "id": "d84cddc7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n", @@ -3642,9 +3705,7 @@ { "cell_type": "markdown", "id": "ad4c41f7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we can rewrite as" ] @@ -3652,9 +3713,7 @@ { "cell_type": "markdown", "id": "4c51899e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n", @@ -3664,9 +3723,7 @@ { "cell_type": "markdown", "id": "156aa916", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$." ] @@ -3674,9 +3731,7 @@ { "cell_type": "markdown", "id": "68006bf2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpretations of Bayes' Theorem\n", "\n", @@ -3693,9 +3748,7 @@ { "cell_type": "markdown", "id": "60a10f26", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example of Usage of Bayes' theorem\n", "\n", @@ -3714,9 +3767,7 @@ { "cell_type": "markdown", "id": "82dfdaae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X=1\\vert Y=1) =0.8.\n", @@ -3726,9 +3777,7 @@ { "cell_type": "markdown", "id": "af7f86b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This obviously sounds scary since many would conclude that if the test is positive, there is a likelihood of $80\\%$ for having cancer.\n", "It is however not correct, as the following Bayesian analysis shows." @@ -3737,9 +3786,7 @@ { "cell_type": "markdown", "id": "c8a39201", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Doing it correctly\n", "\n", @@ -3750,9 +3797,7 @@ { "cell_type": "markdown", "id": "b084cf75", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(Y=1) =0.004.\n", @@ -3762,9 +3807,7 @@ { "cell_type": "markdown", "id": "d6e9eabf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need also to account for the fact that the test may produce a false positive result (false alarm). Let us here assume that we have" ] @@ -3772,9 +3815,7 @@ { "cell_type": "markdown", "id": "0ed96660", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X=1\\vert Y=0) =0.1.\n", @@ -3784,9 +3825,7 @@ { "cell_type": "markdown", "id": "afd67cd4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using Bayes' theorem we can then find the posterior probability that the person has breast cancer in case of a positive test, that is we can compute" ] @@ -3794,9 +3833,7 @@ { "cell_type": "markdown", "id": "34a584ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(Y=1\\vert X=1)=\\frac{p(X=1\\vert Y=1)p(Y=1)}{p(X=1\\vert Y=1)p(Y=1)+p(X=1\\vert Y=0)p(Y=0)}=\\frac{0.8\\times 0.004}{0.8\\times 0.004+0.1\\times 0.996}=0.031.\n", @@ -3806,9 +3843,7 @@ { "cell_type": "markdown", "id": "f4094b37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "That is, in case of a positive test, there is only a $3\\%$ chance of having breast cancer!" ] @@ -3816,9 +3851,7 @@ { "cell_type": "markdown", "id": "7aeaf0f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem and Ridge and Lasso Regression\n", "\n", @@ -3833,9 +3866,7 @@ { "cell_type": "markdown", "id": "da4c3fcd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Test Function for what happens with OLS, Ridge and Lasso\n", "\n", @@ -3852,10 +3883,7 @@ "cell_type": "code", "execution_count": 11, "id": "4ec2b5fd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -3926,9 +3954,7 @@ { "cell_type": "markdown", "id": "caff09a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "How can we understand this?" ] @@ -3936,9 +3962,7 @@ { "cell_type": "markdown", "id": "bd3a9cca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Invoking Bayes' theorem\n", "\n", @@ -3950,9 +3974,7 @@ { "cell_type": "markdown", "id": "243ef030", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n", @@ -3962,9 +3984,7 @@ { "cell_type": "markdown", "id": "7e41e345", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is given by" ] @@ -3972,9 +3992,7 @@ { "cell_type": "markdown", "id": "7200c900", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -3984,9 +4002,7 @@ { "cell_type": "markdown", "id": "d847fb1f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] @@ -3994,9 +4010,7 @@ { "cell_type": "markdown", "id": "792df834", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", @@ -4006,9 +4020,7 @@ { "cell_type": "markdown", "id": "6c269fe7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Bayes' theorem comes to our rescue here since (omitting the normalization constant)" ] @@ -4016,9 +4028,7 @@ { "cell_type": "markdown", "id": "f445c73f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", @@ -4028,9 +4038,7 @@ { "cell_type": "markdown", "id": "9df306df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!" ] @@ -4038,9 +4046,7 @@ { "cell_type": "markdown", "id": "2012e122", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and Bayes\n", "\n", @@ -4054,9 +4060,7 @@ { "cell_type": "markdown", "id": "8d363e72", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -4066,9 +4070,7 @@ { "cell_type": "markdown", "id": "2c97b8dd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -4076,9 +4078,7 @@ { "cell_type": "markdown", "id": "7f50f4b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -4088,9 +4088,7 @@ { "cell_type": "markdown", "id": "7c32f26a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", @@ -4101,9 +4099,7 @@ { "cell_type": "markdown", "id": "7041fe5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -4113,9 +4109,7 @@ { "cell_type": "markdown", "id": "a00209ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/2\\tau^2$ with $\\lambda$ we have" ] @@ -4123,9 +4117,7 @@ { "cell_type": "markdown", "id": "622d7e54", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -4135,9 +4127,7 @@ { "cell_type": "markdown", "id": "d577c8e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Ridge cost function! Nice, isn't it?" ] @@ -4145,9 +4135,7 @@ { "cell_type": "markdown", "id": "637345d5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso and Bayes\n", "\n", @@ -4157,9 +4145,7 @@ { "cell_type": "markdown", "id": "9f6301a9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -4169,9 +4155,7 @@ { "cell_type": "markdown", "id": "106b9ad1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -4179,9 +4163,7 @@ { "cell_type": "markdown", "id": "8e0aeaaa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -4191,9 +4173,7 @@ { "cell_type": "markdown", "id": "8a0604bf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", @@ -4203,9 +4183,7 @@ { "cell_type": "markdown", "id": "9f2904a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4215,9 +4193,7 @@ { "cell_type": "markdown", "id": "18dee61b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/\\tau$ with $\\lambda$ we have" ] @@ -4225,9 +4201,7 @@ { "cell_type": "markdown", "id": "63672572", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -4237,9 +4211,7 @@ { "cell_type": "markdown", "id": "e3c3010e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Lasso cost function!" ] @@ -4247,9 +4219,7 @@ { "cell_type": "markdown", "id": "46ff5c79", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 1: mean values and variances in linear regression\n", "\n", @@ -4263,9 +4233,7 @@ { "cell_type": "markdown", "id": "987ebe65", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y} = f(\\boldsymbol{x})+\\boldsymbol{\\varepsilon}\n", @@ -4275,9 +4243,7 @@ { "cell_type": "markdown", "id": "3db260a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We then approximate this function with our model from the solution of the linear regression equations (ordinary least squares OLS), that is our\n", "function $f$ is approximated by $\\boldsymbol{\\tilde{y}}$ where we minimized $(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2$, with" @@ -4286,9 +4252,7 @@ { "cell_type": "markdown", "id": "fb640678", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta}.\n", @@ -4298,9 +4262,7 @@ { "cell_type": "markdown", "id": "79301962", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The matrix $\\boldsymbol{X}$ is the so-called design matrix." ] @@ -4308,9 +4270,7 @@ { "cell_type": "markdown", "id": "4c47b7f5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**a)**\n", "Show that the expectation value of $\\boldsymbol{y}$ for a given element $i$" @@ -4319,9 +4279,7 @@ { "cell_type": "markdown", "id": "7918dc5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \n", @@ -4333,9 +4291,7 @@ { "cell_type": "markdown", "id": "d7f554ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and that\n", "its variance is" @@ -4344,9 +4300,7 @@ { "cell_type": "markdown", "id": "cf7f457d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*} \\mbox{Var}(y_i) & = \\sigma^2. \n", @@ -4357,9 +4311,7 @@ { "cell_type": "markdown", "id": "ac435746", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Hence, $y_i \\sim \\mathcal{N}( \\mathbf{X}_{i, \\ast} \\, \\boldsymbol{\\beta}, \\sigma^2)$, that is $\\boldsymbol{y}$ follows a normal distribution with \n", "mean value $\\boldsymbol{X}\\boldsymbol{\\beta}$ and variance $\\sigma^2$." @@ -4368,9 +4320,7 @@ { "cell_type": "markdown", "id": "9fd2a8ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**b)**\n", "With the OLS expressions for the parameters $\\boldsymbol{\\beta}$ show that" @@ -4379,9 +4329,7 @@ { "cell_type": "markdown", "id": "3b44a157", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}(\\boldsymbol{\\beta}) = \\boldsymbol{\\beta}.\n", @@ -4391,9 +4339,7 @@ { "cell_type": "markdown", "id": "e5ffb127", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**c)**\n", "Show finally that the variance of $\\boldsymbol{\\beta}$ is" @@ -4402,9 +4348,7 @@ { "cell_type": "markdown", "id": "8af2115c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{eqnarray*}\n", @@ -4416,9 +4360,7 @@ { "cell_type": "markdown", "id": "3ad6fb92", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Exercise 2: Adding Ridge and Lasso Regression\n", "\n", @@ -4442,10 +4384,7 @@ "cell_type": "code", "execution_count": 12, "id": "e50e3da5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = np.random.rand(100)\n", @@ -4455,9 +4394,7 @@ { "cell_type": "markdown", "id": "ec466f66", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**a)**\n", "Write your own code for the Ridge method (see chapter 3.4 of Hastie *et al.*, equations (3.43) and (3.44)) and compute the parametrization for different values of $\\lambda$. Study the dependence on $\\lambda$ while also varying the strength of the noise in your expression for $y(x)$." @@ -4466,16 +4403,32 @@ { "cell_type": "markdown", "id": "f9de2dd8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**b)**\n", "Our next step is to study the variance of the parameters $\\beta_1$ and $\\beta_2$ (assuming that we are parameterizing our function with a second-order polynomial). We will use standard linear regression and the Ridge regression. You can now opt for either writing your own function or using **Scikit-Learn** to find the parameters $\\beta$. From your results calculate the variance of these parameters (recall that this is equal to the diagonal elements of the matrix $(\\hat{X}^T\\hat{X})+\\lambda\\hat{I})^{-1}$). Discuss the results of these variances as functions of $\\lambda$. In particular, try to link your discussion with the discussion in Hastie *et al.* and their figures 3.10 and 3.11. **Scikit-Learn** may not provide the variance of the parameters $\\beta$. This needs to be checked. With your own code you can however do so." ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index 76af05aad..4372552a4 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -1721,15 +1721,19 @@ and using our SVD decomposition of $\bm{X}$ we have \tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{\Sigma}\bm{V}^T\left(\bm{V}\tilde{\bm{\Sigma}}^{2}(\bm{V}^T\right)^{-1}\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{y}, \] !et -which gives us, using the orthogonality of the matrices $\bm{U}$ and $\bm{V}$,, +which gives us, using the orthogonality of the matrices $\bm{U}$ and $\bm{V}$, !bt \[ -\tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{U}^T\bm{y}=\sum_{i=0}^{n-1}\bm{u}_i\bm{u}^T_i\bm{y}, +\tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{U}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_i\bm{u}^T_i\bm{y}, \] !et -It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$. +It means that the ordinary least square model (with the optimal +parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal +transformation of the output (or target) vector $\bm{y}$ by the +vectors of the matrix $\bm{U}$. Note that the summation ends at $p-1$, +that is $\bm{\tilde{y}}\ne \bm{y}$. !split ===== Further properties (important for our analyses later) =====